@ciaran updated the buildserver. This version includes fixes that made fdroidserver faster and more robust. Currently it’s working hard to process a huge backlog. Things are going fine, most of build failures are resolved.
4 Likes
@relan Thatnks for the update. In general, how often does F-Droid have a queue that takes longer than 1 day to process? From what I have read there have been issues in the past where the build system halts waiting for manual intervention before continuing, which caused things to back up. Assuming all those issues are now resolved, is it possible for the F-Droid server to process 100 or even 1,000 updates per day?
1 Like
Sometimes, when fdroidserver builds huge apps like Fennec or VLC (multiplied by the number of architectures). This takes many hours.
It was one Git-related issue that’s fixed now.
We never had 100 pending builds. Also remember there’s singing step that still requires manual intervention.
Would it make sense to get a second build server to help with that? I would certainly donate toward that if you deem it useful.
Help with new builds availability delays? There are a lot of factors here, and hardware isn’t that important IMHO. Anyway, I’m not the right person to discuss this with.
Unfortunately we are still seeing hanging buildservers.
We definitely need to implement some sort of timeout for every build. But this is not trivial to fit into the current architecture.
Right now this would unfortunately not solve anything. The whole process is architectured around one server running all pending builds in a loop in a VM instance that is reset after very app. After that is complete it proceeds to sign and then build and publish the index update. Signing the apps and then again signing the index is apparently done on a separate offline machine.
I think this whole architecture needs to be rethought and having a build master/slave system would be the way to go. But this would basically require rewriting most/all of the current build related scripts in fdroidserver. So I don’t think we’ll be there anytime soon.
Until this we are trying to fix those hanging issues and introduce timeouts for all builds and maybe have a certain 24 time limit after that an index getting published no matter if there are still apps in the backlog. But this is all very much future work right now.
3 Likes
Thanks @Bubu. It sounds like what the build process needs the most is some additional developers to volunteer their time. Perhaps at some point in the future I will be able to do so.
Yes please! Anyone with buildbot experience would be very helpful right now. buildbot would let us run multiple buildservers easily, and handle other aspects as well.
1 Like
I don’t know anything about buildbot, but I’m sure I could pick it up. I don’t have time right now to take on a new project, but in the future I might be interested in becoming an F-Droid developer. I think that what F-Droid provides for Android is indispensable.
2 Likes
With our jenkins.debian.net setup, we’re able to build ~500 APKs a day.
That’s not a huge machine. One big bottleneck recently has been that
the build process has been hanging a lot. I think we’ve found almost
all the causes of these hangs, which means a build/release cycle should
now finish without manual intervention.
Beyond that, there are many things we could do to speed things up. One
nice would be to make the buildserver build for ~20 hours, then stop,
and publish, no matter what. Right now, the cycle waits for the full
build process to finish before running a publish. That means most
updates would always happen within a day, but some large, troublesome
builds like VLC might be even more delayed.
As for adding more hardware, the whole build process needs to be
rearchitected to allow for multiple build VMs. Right now, it only uses
one. To do that, we should use buildbot. That’s a big project. If
anyone knows buildbot, or just wants to help out, they could put
together a prototype of running fdroid builds using buildbot.
3 Likes
Hi all 
This Tuesday, I updated my app to version 2.9 (1ST automatic update ^^).
Ref: Bible Multi "The Light" (#351) · Issues · F-Droid / Requests For Packaging · GitLab
The source is tagged…
If you want to check the server build…
Thanks a lot,
I reopened the RFP in FDroid client.
6 days and not built!
Is there a way to automatically trigger a re-build on the official f-droid server for a unsuccessful build?
One of my apps has encountered a build failure because of an error either in the build server or the connection (as described in this post).
The search function in the wiki couldn’t find any other apps with the same error, so I suppose this is most likely a temporary issue (though the search function sometimes malfunctions – not returning all matches).
The problem may be related to SDK 27 on the f-droid build machine, since
- the build log yields problems related to SDK 27 (two logs with different problems)
- it happens on the first release (and the second) after switching to 27 (p.s. there isn’t a newer release yet, since I’m hoping to see the problem solved before relasing it)
Is there a way to automatically trigger a re-build on the official f-droid server for a unsuccessful build?
No. Failed builds are retried automatically on the next buildserver run until they succeed or someone disables them in the metadata.
I think we need to add this into FAQ.
1 Like
Hi @relan, how can I check the auto-build of my app (above)?
I don’t have info.
Rem: +1 to stop some app building that will slow the whole build process.
I also see an app in Fdroid with maybe10 separated apps, maybe they need a dedicated server lol 
(or special conditions to build them as last)
Edit: ticket solved about Gitlab shared runners
@hotlittlewhitedog, there is a wiki that is auto-updated by the build process, although sometimes it is a trailing indicator (it can take a few days to get updated after the actual build has processed).
There is also build log (also a trailing indicator. but usually updated more quickly than the general wiki):
https://f-droid.org/wiki/index.php?title=org.hlwd.bible/lastbuild&redirect=no
This shows that there hasn’t been an attempt to build your APK since October 10.
Other apps are building fine:
https://f-droid.org/wiki/index.php?title=Special:RecentChanges&hidebots=0
The metadata for you app shows that it isn’t trying to build any newer version. Because Auto Update Mode is set to none, you or someone else will need to manually edit the metadata file to initialize the build process.
I would recommend changing the metadata file to use some form of auto updating:
For my app, I use Tags, which automatically adds a new build section based on release tags in my repo.
1 Like
I was checking the build status just now, and find many apps are failed with this error:
INFO: Cleaning Gradle project...
ERROR: Could not build app APP_ID: OSError while trying to execute gradle clean: [Errno 8] Exec format error
This seems to indicate some (or one) gradle-related files lack their shebang?
1 Like
Hmm, buildserver/gradle does have #!/bin/bash at the top…
1 Like
Hi all,
Will be the FAILED applications rebuilt?
Or do we have to make an update to force the auto-update?
1 Like