Slow webserver response

It is true that eclips.is is US GOVT funded, so if you consider anything
funded by the US Government or Open Tech Fund (OTF) to be suspect, then
you’ll need to also mistrust:

  • SELinux
  • the NIST/MITRE CVE vulnerability reporting system
  • Signal (OTF-funded)
  • Tor Project (OTF, State DRL, DARPA, NSF, etc)
  • F-Droid (OTF, ISC)
  • and many more

It is important to remember that the US Government is vast (~5 million
people). And we’re fully aware of the risks of people with money asking
for backdoors. That’s why we only develop open source software, and we
are public about the funding we accept, and what the goals of that
funding are. See the https:/f-droid.org and
https://guardianproject.info blogs for more details. F-Droid is now
starting to gather this info in https://opencollective.com/f-droid

1 Like

I should add, the DARPA/OTF/USAGM/DRL/etc funding definitely comes out
of interventionist US foreign policy. It’s clear to see that its
related. But I still think that organizations can accept that money and
still be widely trusted. For example, I’ve also received grant money
from the City of Vienna, which is very neutral, being the capitol of a
neutral country.

1 Like

letsencrypt was also OTF-funded:
https://www.opentech.fund/results/supported-projects/certbot-improvements/

It is important to be skeptical about funding sources, they are not
neutral. They have a point of view. Keeping things open and public
means people can review the work, but then of course someone actually
has to review the work and point out problems. So I welcome your
skepticism :slight_smile:

1 Like

Nice, I was wondering about Xi’an Jiaotong-Liverpool University venture located in China, maybe they could be interested in hosting F-Droid ?

They are running free(?) http://sanddroid.xjtu.edu.cn/#home

Although I have no idea how technological us ban could impact on open source licenses.

1 Like

Do you have any monitoring set up for CPU (core utilization, interrupts, softirqs, etc), disk I/O (read/write MB/s, read/write IOPS, backlog, utilization time, etc), networking (used sockets, sent/received packets, etc) and your web server (number of workers, number of idle workers, requests per seconds, KiB/s, etc)? That would be really helpful.

Like I asked on IRC: what’s net.core.somaxconn set to (check with sudo sysctl net.core.somaxconn)? If it’s some low value like the previous default 128, could you try increasing it to 4096 (which is also the default since Linux kernel 5.4; you could do that with sudo sysctl -w net.core.somaxconn=4096 and when you want to keep it after reboots, by adding net.core.somaxconn = 4096 to /etc/sysctl.conf)? Of course you then also have to increase ListenBackLog in your Apache config to 4096 (from the defaullt 511) as well.

1 Like

As far as I know, there is not performance monitoring in place on those servers. They are maintained by the founder, who maintains a lot of things and does not have so much time these days. So we’re trying to move things off of his plate. Troubleshooting the existing setup means using external monitoring. I’m happy to set up 3 VMs and an Anycast IP for someone to set up an ansible-based caching frontend setup for f-droid.org. Once we have that, then we can migrate to that for production. And we’ll be able to get detailed troubleshooting information from those boxes.

2 Likes

@uniqx and I were just discussing fdroid-frontend-deployment, very cool to see @Bubu got a prototype up and running already (https://fdroid-frontend1.bubu1.eu/). I think we should take this approach to production. The open issues we could think of here:

  • change the security headers to be passed through from the webserver so we have a single place to maintain them
  • maintain the webserver hostname in an ansible encrypted vault variable using the GPG-based setup we have in fdroid-website-server, fdroid-deployserver, etc.
  • intrusion detection monitoring and fail2ban
  • disabling sshd password auth, allowing only ssh key auth

In a related effort, I recently setup staging.f-droid.org to be built with https://gitlab.com/fdroid/fdroid-website-server, so that is something we can also discuss migrating to. I think that will be a bit more complicated so something to do later.

Oops, and also performance monitoring like @wb9688 suggests. I think that could go to monitor.f-droid.org

I notice Fosshost list F-Droid on their project page. They just launched an anycast CDN last month.

yup, we’re in discussions with them about the privacy policy of their
CDN. Ideally, we can move to a CDN without leaking more personal data.

Their new privacy policy is already looking pretty good, more review is
appreciated:

Hey there. I know it’s been a topic one or two years ago, but as somebody hosting stuff myself I’ve got to ask again: Is there any way we can improve the page speed ?

It’s kind of annoying when you’re checking the current build queue or for people that are using the “get it on f-droid” link and simply don’t get any website to display for seconds (or simply a timeout depending on the time).

Examples:
https://monitor.f-droid.org/builds/build
8,24seconds waiting for the response
image
And it’s not a fluke, here again with 13,47s waiting for response.
image

Ok that may be just the build system, but the packages aren’t better:
https://f-droid.org/packages/vocabletrainer.heinecke.aron.vocabletrainer/
It varies a lot, here instant with 95ms and disabled cache.
image
And again with 21 seconds.
image

I’m connecting over AS3320 - Deutsche Telekom AG, no WiFi. And there isn’t anything else that is as slow.
And I’ve seen even worse times using the network of my workplace.

Cloudflare also won’t help with that, obviously as AFAIK they also only play reverse-proxy with some caching:
https://cloudflare.f-droid.org/packages/vocabletrainer.heinecke.aron.vocabletrainer/
image
Even worse, now all content requests like images are slow:
https://cloudflare.f-droid.org/assets/ic_repo_app_default_KNN008Z2K7VNPZOFLMTry3JkfFYPxVGDopS1iwWe5wo=.png
image
Even with reloading (so it should be cached?)
https://cloudflare.f-droid.org/assets/apple-touch-icon_ypJwtCrcixeH_qV6LdcMYk1anFIR9o-_ufR__1wNdJY=.png
image

Traceroute to hetzner is a little broken but it looks ok.

traceroute to f-droid.org (148.251.140.42), 30 hops max, 60 byte packets
 1  XXXX  1.129 ms
 2  XXXX.dip0.t-ipconnect.de (62.155.246.4)  14.975 ms  14.964 ms  14.953 ms
 3  * * *
 4  * * *
 5  * core24.fsn1.hetzner.com (213.239.252.250)  24.674 ms core23.fsn1.hetzner.com (213.239.252.246)  25.285 ms
 6  ex9k1.dc12.fsn1.hetzner.com (213.239.203.174)  25.289 ms  18.615 ms ex9k1.dc12.fsn1.hetzner.com (213.239.203.186)  18.896 ms
 7  * * *
 [...]
3 Likes

That’s not the build system actually, just a site with stats

The solution to slow web sites is using Tor. You get used to everything being slow. Patience is a virtue, so they say.
/1/2 sarcasm

Makes it even worse, I’ve got a 50% chance for a gateway timeout on weekends.

We know, there’s some work going on for front runners to proxy and help.

1 Like

@Ox0p54 thanks for digging into this! @uniqx has created https://gitlab.com/fdroid/fdroid-http-fronters which is running as https://fdroid.h4x.at/. We’re working on putting that into production now. But we’d love to have some testing on https://fdroid.h4x.at/ now, if you’re willing @Ox0p54.

1 Like

Would be happy to do that, do you have some specific in mind ?
The gitlab link you send seems to be broken.

ah sorry, that repo is still private. You could just profile the domain name,
the setup isn’t particularly special, its basically nginx as a reverse proxy.

That was a misconfiguration, and now it should be fixed.

Cache performance overview (by requests) of our Cloudflare configuration in the past 14 days

Checkly monitoring in the last 24 hours (grouped by locations)

1 Like

Ok so in the hopes of actually helping with that, another round:
15.05’22 18:00 CEST, some interesting stuff to see this time.

“Dauer” column stands for “Duration”, so full request duration.

First the new solution h4x.at
https://fdroid.h4x.at/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/
Disabled cache, first hit


Notably some of the fastlane images are served via http and not https and biggest part of them are served over f-droid, not h4x.at. Everything coming from h4x.at does get delivered very fast.

F-droid itself, for a baseline
https://f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/
Disabled cache, first hit


As expected initial request takes 9 seconds, everything else varies a lot.

Now for cloud flare, where it’s getting interesting:
https://cloudflare.f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/
Disabled cache, first hit


1,6minutes for the first hit itself! (Certainly cold cache waiting for f-droid)
Nearly all fastlane images are 404, though not if you directly browse the URLs.
Example URL 404: https://cloudflare.f-droid.org/repo/vocabletrainer.heinecke.aron.vocabletrainer/en-US/phoneScreenshots/Editor.png

Let’s reload cloud flare to see the cache in effect
Disabled cache, second hit
https://cloudflare.f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/


Stuff that doesn’t 404 is mostly fast. Some stuff seems to not get cached, marked by firefox with a turtle (and also the “Dauer” column).

And for completeness another cloudflare request, with caches.
https://cloudflare.f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/