Slow webserver response

hans · February 19, 2020, 7:49pm

It is true that eclips.is is US GOVT funded, so if you consider anything
funded by the US Government or Open Tech Fund (OTF) to be suspect, then
you’ll need to also mistrust:

SELinux
the NIST/MITRE CVE vulnerability reporting system
Signal (OTF-funded)
Tor Project (OTF, State DRL, DARPA, NSF, etc)
F-Droid (OTF, ISC)
and many more

It is important to remember that the US Government is vast (~5 million
people). And we’re fully aware of the risks of people with money asking
for backdoors. That’s why we only develop open source software, and we
are public about the funding we accept, and what the goals of that
funding are. See the https:/f-droid.org and
https://guardianproject.info blogs for more details. F-Droid is now
starting to gather this info in https://opencollective.com/f-droid

hans · February 20, 2020, 3:54pm

I should add, the DARPA/OTF/USAGM/DRL/etc funding definitely comes out
of interventionist US foreign policy. It’s clear to see that its
related. But I still think that organizations can accept that money and
still be widely trusted. For example, I’ve also received grant money
from the City of Vienna, which is very neutral, being the capitol of a
neutral country.

hans · February 21, 2020, 10:38am

letsencrypt was also OTF-funded:

It is important to be skeptical about funding sources, they are not
neutral. They have a point of view. Keeping things open and public
means people can review the work, but then of course someone actually
has to review the work and point out problems. So I welcome your
skepticism

oF2pks · February 21, 2020, 1:20pm

Nice, I was wondering about Xi’an Jiaotong-Liverpool University venture located in China, maybe they could be interested in hosting F-Droid ?

They are running free(?) http://sanddroid.xjtu.edu.cn/#home

Although I have no idea how technological us ban could impact on open source licenses.

wb9688 · June 16, 2020, 2:24pm

Do you have any monitoring set up for CPU (core utilization, interrupts, softirqs, etc), disk I/O (read/write MB/s, read/write IOPS, backlog, utilization time, etc), networking (used sockets, sent/received packets, etc) and your web server (number of workers, number of idle workers, requests per seconds, KiB/s, etc)? That would be really helpful.

Like I asked on IRC: what’s net.core.somaxconn set to (check with sudo sysctl net.core.somaxconn)? If it’s some low value like the previous default 128, could you try increasing it to 4096 (which is also the default since Linux kernel 5.4; you could do that with sudo sysctl -w net.core.somaxconn=4096 and when you want to keep it after reboots, by adding net.core.somaxconn = 4096 to /etc/sysctl.conf)? Of course you then also have to increase ListenBackLog in your Apache config to 4096 (from the defaullt 511) as well.

hans · June 18, 2020, 7:06am

As far as I know, there is not performance monitoring in place on those servers. They are maintained by the founder, who maintains a lot of things and does not have so much time these days. So we’re trying to move things off of his plate. Troubleshooting the existing setup means using external monitoring. I’m happy to set up 3 VMs and an Anycast IP for someone to set up an ansible-based caching frontend setup for f-droid.org. Once we have that, then we can migrate to that for production. And we’ll be able to get detailed troubleshooting information from those boxes.

hans · November 4, 2020, 11:45am

@uniqx and I were just discussing fdroid-frontend-deployment, very cool to see @Bubu got a prototype up and running already (https://fdroid-frontend1.bubu1.eu/). I think we should take this approach to production. The open issues we could think of here:

change the security headers to be passed through from the webserver so we have a single place to maintain them
maintain the webserver hostname in an ansible encrypted vault variable using the GPG-based setup we have in fdroid-website-server, fdroid-deployserver, etc.
intrusion detection monitoring and fail2ban
disabling sshd password auth, allowing only ssh key auth

In a related effort, I recently setup staging.f-droid.org to be built with F-Droid / fdroid-website-server · GitLab, so that is something we can also discuss migrating to. I think that will be a bit more complicated so something to do later.

hans · November 4, 2020, 11:45am

Oops, and also performance monitoring like @wb9688 suggests. I think that could go to monitor.f-droid.org

curben · November 9, 2020, 7:05am

I notice Fosshost list F-Droid on their project page. They just launched an anycast CDN last month.

hans · November 9, 2020, 9:28am

yup, we’re in discussions with them about the privacy policy of their
CDN. Ideally, we can move to a CDN without leaking more personal data.

Their new privacy policy is already looking pretty good, more review is
appreciated:
https://delivr.dev/docs/privacy-policy

Ox0p54 · May 6, 2022, 5:36pm

Hey there. I know it’s been a topic one or two years ago, but as somebody hosting stuff myself I’ve got to ask again: Is there any way we can improve the page speed ?

It’s kind of annoying when you’re checking the current build queue or for people that are using the “get it on f-droid” link and simply don’t get any website to display for seconds (or simply a timeout depending on the time).

Examples:
https://monitor.f-droid.org/builds/build
8,24seconds waiting for the response

And it’s not a fluke, here again with 13,47s waiting for response.

Ok that may be just the build system, but the packages aren’t better:
https://f-droid.org/packages/vocabletrainer.heinecke.aron.vocabletrainer/
It varies a lot, here instant with 95ms and disabled cache.

And again with 21 seconds.

I’m connecting over AS3320 - Deutsche Telekom AG, no WiFi. And there isn’t anything else that is as slow.
And I’ve seen even worse times using the network of my workplace.

Cloudflare also won’t help with that, obviously as AFAIK they also only play reverse-proxy with some caching:
https://cloudflare.f-droid.org/packages/vocabletrainer.heinecke.aron.vocabletrainer/

Even worse, now all content requests like images are slow:
https://cloudflare.f-droid.org/assets/ic_repo_app_default_KNN008Z2K7VNPZOFLMTry3JkfFYPxVGDopS1iwWe5wo=.png

Even with reloading (so it should be cached?)
https://cloudflare.f-droid.org/assets/apple-touch-icon_ypJwtCrcixeH_qV6LdcMYk1anFIR9o-_ufR__1wNdJY=.png

Traceroute to hetzner is a little broken but it looks ok.

traceroute to f-droid.org (148.251.140.42), 30 hops max, 60 byte packets
 1  XXXX  1.129 ms
 2  XXXX.dip0.t-ipconnect.de (62.155.246.4)  14.975 ms  14.964 ms  14.953 ms
 3  * * *
 4  * * *
 5  * core24.fsn1.hetzner.com (213.239.252.250)  24.674 ms core23.fsn1.hetzner.com (213.239.252.246)  25.285 ms
 6  ex9k1.dc12.fsn1.hetzner.com (213.239.203.174)  25.289 ms  18.615 ms ex9k1.dc12.fsn1.hetzner.com (213.239.203.186)  18.896 ms
 7  * * *
 [...]

Licaon_Kter · May 8, 2022, 5:00pm

That’s not the build system actually, just a site with stats

anon46495926 · May 8, 2022, 5:52pm

The solution to slow web sites is using Tor. You get used to everything being slow. Patience is a virtue, so they say.
/1/2 sarcasm

Ox0p54 · May 8, 2022, 6:56pm

Makes it even worse, I’ve got a 50% chance for a gateway timeout on weekends.

Licaon_Kter · May 8, 2022, 7:00pm

We know, there’s some work going on for front runners to proxy and help.

hans · May 12, 2022, 2:28pm

@Ox0p54 thanks for digging into this! @uniqx has created https://gitlab.com/fdroid/fdroid-http-fronters which is running as https://fdroid.h4x.at/. We’re working on putting that into production now. But we’d love to have some testing on https://fdroid.h4x.at/ now, if you’re willing @Ox0p54.

Ox0p54 · May 12, 2022, 4:51pm

Would be happy to do that, do you have some specific in mind ?
The gitlab link you send seems to be broken.

hans · May 12, 2022, 7:56pm

ah sorry, that repo is still private. You could just profile the domain name,
the setup isn’t particularly special, its basically nginx as a reverse proxy.

proletarius101 · May 15, 2022, 7:09am

That was a misconfiguration, and now it should be fixed.

Cache performance overview (by requests) of our Cloudflare configuration in the past 14 days

Checkly monitoring in the last 24 hours (grouped by locations)

Ox0p54 · May 15, 2022, 4:28pm

Ok so in the hopes of actually helping with that, another round:
15.05’22 18:00 CEST, some interesting stuff to see this time.

“Dauer” column stands for “Duration”, so full request duration.

First the new solution h4x.at
https://fdroid.h4x.at/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/
Disabled cache, first hit

Notably some of the fastlane images are served via http and not https and biggest part of them are served over f-droid, not h4x.at. Everything coming from h4x.at does get delivered very fast.

F-droid itself, for a baseline
https://f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/
Disabled cache, first hit

As expected initial request takes 9 seconds, everything else varies a lot.

Now for cloud flare, where it’s getting interesting:
https://cloudflare.f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/
Disabled cache, first hit

1,6minutes for the first hit itself! (Certainly cold cache waiting for f-droid)
Nearly all fastlane images are 404, though not if you directly browse the URLs.
Example URL 404: https://cloudflare.f-droid.org/repo/vocabletrainer.heinecke.aron.vocabletrainer/en-US/phoneScreenshots/Editor.png

Let’s reload cloud flare to see the cache in effect
Disabled cache, second hit
https://cloudflare.f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/

Stuff that doesn’t 404 is mostly fast. Some stuff seems to not get cached, marked by firefox with a turtle (and also the “Dauer” column).

And for completeness another cloudflare request, with caches.
https://cloudflare.f-droid.org/en/packages/vocabletrainer.heinecke.aron.vocabletrainer/