Troubleshooting f-droid.org slowness periods

website
#22

My guess is that one key part of the problem is that the vast majority of downloads from fdroidclient are from f-droid.org. By moving all of the APK downloads to mirrors, that should lighten the load on f-droid.org. That’s how I’m thinking, but we’ll see.

1 Like
#23

The actual site and the content are hosted in the same place? Then yeah…

#24

Yes, the client fetches from f-droid.org by default

#25

@hans ,

It’s an important point to distribute the F-Droid Servers, for independence and Community-Driven content.
Coudn’t we make a Service to distribute F-Droid-Mirrors to clients? We could make decisions based on the mirrors power, internet-speed and location of the client-ip, so the load is equally and smart distributed between several community-driven-mirrors.

Also the clients could save the assigned points with location, so in case of errors they could decide themselves wich mirror to try and use.

sincerly, Dennis

#26

sounds like that would be nice to have. There is currently only the
very basic https://fdroid.gitlab.io/mirror-monitor

#27

We could make a central Server wich the clients can get their Mirrors off of.
If this server Crashes, we could try to re-route the traffic, by changing the DNS-IP of the subdomain via the Domain-Providers API - if possible.

The Central-Server could also work as a cluster (connected and positioned troughout the world) where different Central-Nodes distribute different global-positions - but i think this is a bit overkill for now, and we can add it later quite easily because i try to use Apache’s Kafka, ZooKeeper and maybe Cassandra.

The Goal is:

  • A Client can trust our API completely
    Well … we use certificates and TLS for our server (it’s the Web-of-trust, what can we do…)
  • We don’t have to trust our Clients
    Where should we know it’s not a gouverment trying to get someones personal data trough an App
  • Clients can access a change-of-data (Update, Change) ASAP
    They should be able to know when to download updates

The Method:

  • Clients Ask the DNS-Provider for the Mirror-Service
  • Clients Access the mirror Service
  • The Service gives them an IP or Domain to access the Data
  • Clients can ask for Publicly Available Resources (Images, App-Files(Signed), etc )

Big question:
Whht Data should be made distributed (Images, App-Files(Signed) … Meta-Data, App-Data, What apps are recently updated?)
Because, people could just as easily not update their DB and Clients woudn’t know of updates or new Apps, if we made the App-Table distributed as well.

I mean, if we decentralise the index, how do we make sure it’s up-to-date and unchanged by the distributor?

#28

Btw. Can you explain me a little how the current Mirroring-system works.
I don’t really see how mirrors get distributed right now.

#29

There are either “official” mirrors that are included in the index file.
And then “user” mirrors which are added by the user clicking the URL on
the device, or copying and pasting it in.

1 Like
#30

Ok, i read the Apps are signed by the developers key, and we don’t have to worry about App-Data corruption.
I think we can at first ignore the index_up-to-date problem for now, so we can firstly have a working distribution.

As much as i understood, every developer posts the app in a public repository, and then gets copied into your official Repo, am i right?
And someone can register a Mirror, wich copies from your Main-Repository.

Currently the App has a Timeout deployed for wich mirror to try to use, and changed it after a while with no response.

I’d make a random-number generator decide a mirror by random, so the load gets more evenly distributed. Also i’d try to make a Service where clients can ‘ask’ for a mirror, and the Service manages the loads. Maybe even with interacting with Mirrors-apis in the future?

#31

@Denis.Mootz The devs just program the apps, in their publicly available code repos (eg. Gitlab Github etc), fdroidserver pulls the code, builds it, signs it then publishes the resulting APK. The APK (repo) gets mirrored…

#32

Thanks a lot, Licanon!
I guess i was a little overenthusiastic with my problem-solving. I’ve read a little into the app, but the documentation is a little rudimentary - i have to look at the Code to know how it’s working.

I think for now we just need a Mirror-Distributer-Service.
Basicly: the Client can ask for a free mirror, and get the address of a free one, wich it keeps for like one minute or so.
So the Service can redistribute the load within the mirrors and later even based on their metrics (Internet-ussage, System-Ussage, etc).

If you’d like, i can create a http-based Server to manage the Mirrors, check them for up-time and respond one within a JSON-String to the client.
So then at least a redistribution-infrastructure is set - The App-Changes could be made later, to firstly try the Suggested-Mirror, and then - like it’s now - try randomly.

Is this a deal? Do you have other suggestions or wishes?
independently from this, the Website-Server and mirror (where the community-mirrors copies from) does also need some more distribution i think - but w can manage that later.

#33

The client has its own mirror selection logic. If you want to set up a
server that somehow selects different mirrors, you’ll have to set that
up without coordination with the client. This is part of the deliberate
design of F-Droid. Also, FYI, though F-Droid uses HTTP libraries and
servers, it is not a browser, and only uses the parts of the HTTP spec
that it needs. For example, redirects are not supported.

1 Like
#34

Hmm … ok,
You have more experience and knowlege of the basic system - so tell me what you think is needed to speed up performence.
Fact: the images and Apps downlaod slow (for me and other people) - so what can we do about it?

We could maybe try having a CDN for the images and icons to distribute the load … maybe even the whole Website and search engine (Cassandra, GitHub-Pages and cloning the Repository).
I actually can’t really find a wiki-page where it says, how the data is distributed and where the client get’s it’s data from … can you explain it please to me a little more?
Or tell me what you think is slowing down the traffic from the project.

And thank you very much for your patience with an inexperienced user like me ^^

#35

Ok, as i gained a little more insigt in this Project, i found out:

  • The App get’s it’s App-Index File from the MainServer (Signed by F-Droids Key)
  • The App selects a Random Mirror, wich hosts the Images, APKs and Meta-Data of the Apps
  • Everyone can become a Mirror by installing the ‘FDroid-Mirror-Server’ and registering by the GitHub-Index

Am i right? @hans

Because then it just means, the Mirrors are overloaded - either by one being more used than others, or by being all equally frequent used.
Do we have a method to finding that out? Like Statistics from the Community-Mirror (Or your experience)?

Also i heard the Website and Search-Function is quite slow as well…
Would also be quite useful to distribute this Service within the Community or at least have a distributed CDN (Do we have one already - and how does the server perform)

I think we should monitor the Services a little better to keep up with upcomming demands and Traffic-Spikes. Like, just Logging the ussages should be enough - but to have a grasp of how it is right now.

What do you think?

#36

Step 1 is the issue right now, since everyone gets the index from the same place, afaics.

#37

We could offload the Index and image Server for speed-improvement for now (a second server).
And later Implement a cdn-subdomain to automatically select a mirror for a user.
In the worst case, the Client app can just make a random select for a session.

#38

@Licaon_Kter @hans @Bubu
I’ve made a Slide where i wrote down the current situation of the Server & Mirrors, their speed and the User-Experience with the App, Search and Website.

https://codi.mootz.eu/s/ryjK0wn9E

I’ve made a conclusion of the current situation - wich i’d like to know if you’d agree with.
And served some possible Solutions, Pro & Con and why i prefer the last one.


I know that currently other Issues rise in the GitLab, yet this CDN could reduce vastly the MaintenanceCost of the Servers, enable more Community Mirrors to Help and a Modular approach would enable faster develoopment with less friction in the long run.

If you have geniune interest, i could make a RoadMap for the Deployment of the CDN-Project. I’d document excactly what would need to be done, how and that would be to implement. Then we could talk about improvement and what can be done, till we’re hapyp with the results.

After a Solid Tecnical Dokumentation, i would start the Logical one, to describe Logic and Code - so changes could be made more easily.

Currently i’m very interested in Distributed Databases, Real-Time Application and Secure-Access. I’ve read trough some DatabaseSystems and contribute in this field.
Also i try to become more aware for Solid documentaions and ProjectManagement.


I’d like to ask you, since you know exactly what this Project currently has for Issues, how ProjectManagement is done and what FDroid is trying to become in the long run.
So what do you guys think?

If you have wishes or requirements, feel free to tell me - so we can brainstorm a solution.

sincerly, Dennis

#39

You are not allowed to access this page. Maybe try logging in?

Umm?

#40

I’m so sorry <:’-D

The UAC is quite intransparent.
It should work now @Licaon_Kter

Quite emberessing.

#41

Hey @Dennis.Mootz,

thanks a lot for the extensive write-up! That seems like a lot of work you’ve put into it. Let me comment on it and excuse me writing it in bullet points for time reasons.

  • Using a commercial CDN likely won’t happen because of protecting users against tracking
  • Offloading parts of F-Droid’s repo (e.g. images) is difficult because it breaks backward compatibility
  • Even moving the repo to repo.f-droid.org is not easily possible because F-Droid client doesn’t follow redirects for security reasons (see latest large apt bug which was possible because of following redirects)
  • f-droid.org is a blackbox; only @ciaran has access to it and even @hans who probably knows the most about it beside him don’t know that much
  • the way to proceed is really to work on the client to use f-droid.org less
  • Updating mirrors for all users doesn’t require the app to be rebuilt; instead a list of “official mirrors” is included in the index which will automatically update mirrors for every user
  • Indexes are already cached inside the client by using the HTTP ETag header
  • F-Droid currently download the index with all languages because of tracking reasons
  • One of the major goal of F-Droid is to make it hard to track people; therefore, we tried to make the fingerprint of a person as small as possible, i.e. making it hard to distinguish between F-Droid users

You see, changes to all this infrastructure aren’t that easy because a lot of things have to be kept in mind. There was a lot of work in the past making tracking F-Droid users hard which at some places results in being inefficient, meaning clients download more data than they need. We want to make it as easy as possible for independent, voluntary persons and organizations to run mirrors. Currently, hosting an F-Droid repo only consists of syncing a directory and statically serving it with any HTTP server. Shortly after supporting mirrors in the client, we received a lot of mirror contributions by a lot people around the world which you nicely displayed in your graphic. Therefore, the computational power to make F-Droid faster is there and we currently do not need a CDN which we would have to pay (resulting in less money for other parts) and which would probably not fit into our high anti-tracking standards.

Instead, it’s now the task to optimize the F-Droid client. @hans has put and is still putting a ton of work into that, making the client use mirrors as often as possible. But as you can easily note by using the newest versions, this still does’t work that good. What F-Droid currently really needs are persons that are willing to work on the client, as there’s the biggest lack of contribution.

Thanks a lot again for your write-up!

1 Like
Mastodon