Is the F-Droid build process currently broken?

The same happened to My Expenses, where tag r301 has not been taken up yet: My Expenses | F-Droid - Free and Open Source Android App Repository

1 Like

I can confirm this glitch. Iā€™ve released SkyTube 2.70 around a week ago, yet itā€™s still not availableā€¦

1 Like

It looks like there was a batch released on November 7 and another on November 13.

https://f-droid.org/wiki/index.php?title=Special:RecentChanges&hidebots=0

Perhaps F-Droid only pulls from Git after finishing a batch, and for some reason batches are taking a while to process.

It looks like another batch went through on November 14.

See What is the index update timeframe? (#17) Ā· Issues Ā· F-Droid / admin Ā· GitLab too

1 Like

I updated Privacy Browser to version 2.7.2 early on November 22. A couple of build cycles have run since then, but F-Droid has not yet noticed the updated tag and attempted to build 2.7.2. On November 23, F-Droid attempted to build 2.7.1 again. This leads me to believe that the metadata is not updated (or at least not updated successfully) with each build cycle.

A week without any repo updates is just an inconvenience. A week with no updates, no explanation, no acknowledgement, and no ETA is simply unacceptable. Is the repo dead? Is the project finished? Should users move back to Google Play for updates? Not everybody understands that this is a volunteer effort, and a full week without any outward sign of life looks really, really bad.

A concise status page/bar would go a long way to avert frustration - even a one-liner explanation. Anything is better than silence.

2 Likes

Hello there,

It seems that every updates of my app failed for the same reason:

Failed to install the following SDK components:
[Android SDK Platform 27]
Please install the missing components using the SDK manager in Android Studio.

Few weeks ago, I needed to move on SDK 27 due to weird bugs. But no release has been published since. Some users start to go back on Google :frowning:

Any help would be greatly appreciated. Thanks.

The file
https://gitlab.com/fdroid/fdroiddata/commits/master/stats/known_apks.txt
has been updated. Just noticing - I have no further insight.

1 Like

Syncthing is also not getting updated. Which sucks even more because for some reason F-Droid released a beta version as stable.

I agree that this situation sucks. Weā€™re aware of missing platform-27 and gradle-4.3.x; this has been fixed weeks ago. Weā€™re waiting for @ciaran to redeploy the buildserver image.

2 Likes

The simple answer is that building thousands of apps from sources is a
complicated and error prone process. A few of us have recently been
working to work out the worst delays. You can follow the tests here:

https://jenkins.debian.net/job/reproducible_fdroid_build_apps/

Weā€™ve gotten up to 350 APKs built in a single session before hanging.
We recently fixed a number of issues that cause the whole buildserver to
hang until manual intervention:

We have also addressed somethings that slow the process down:

6 Likes

Thanks hans. It is nice to have some insights into what is going on. I appreciate the difficulty of getting something with so many moving parts to work well. For those who have never worked on automating a project like this, it can be much more difficult than initially appears.

1 Like

@ciaran updated the buildserver. This version includes fixes that made fdroidserver faster and more robust. Currently itā€™s working hard to process a huge backlog. Things are going fine, most of build failures are resolved.

4 Likes

@relan Thatnks for the update. In general, how often does F-Droid have a queue that takes longer than 1 day to process? From what I have read there have been issues in the past where the build system halts waiting for manual intervention before continuing, which caused things to back up. Assuming all those issues are now resolved, is it possible for the F-Droid server to process 100 or even 1,000 updates per day?

1 Like

Sometimes, when fdroidserver builds huge apps like Fennec or VLC (multiplied by the number of architectures). This takes many hours.

It was one Git-related issue thatā€™s fixed now.

We never had 100 pending builds. Also remember thereā€™s singing step that still requires manual intervention.

Would it make sense to get a second build server to help with that? I would certainly donate toward that if you deem it useful.

Help with new builds availability delays? There are a lot of factors here, and hardware isnā€™t that important IMHO. Anyway, Iā€™m not the right person to discuss this with.

Unfortunately we are still seeing hanging buildservers.
We definitely need to implement some sort of timeout for every build. But this is not trivial to fit into the current architecture.

Right now this would unfortunately not solve anything. The whole process is architectured around one server running all pending builds in a loop in a VM instance that is reset after very app. After that is complete it proceeds to sign and then build and publish the index update. Signing the apps and then again signing the index is apparently done on a separate offline machine.

I think this whole architecture needs to be rethought and having a build master/slave system would be the way to go. But this would basically require rewriting most/all of the current build related scripts in fdroidserver. So I donā€™t think weā€™ll be there anytime soon.

Until this we are trying to fix those hanging issues and introduce timeouts for all builds and maybe have a certain 24 time limit after that an index getting published no matter if there are still apps in the backlog. But this is all very much future work right now.

3 Likes

Thanks @Bubu. It sounds like what the build process needs the most is some additional developers to volunteer their time. Perhaps at some point in the future I will be able to do so.