I had to install a lot of apps yesterday and some of my “must have” apps are now gone from F-Droid. That includes VLC and ZXing Barcode Scanner. I followed the discussion in Gitlab issues and there seems to me a common cause for this problem: F-Droid build server uses old Debian release which doesn’t have the necessary versions of tools like Maven.
Question is: is it just an issue of nobody having the time to move the build infrastructure to Unstable, or are there deeper technical issues with using a later version of Debian?
I ran an F-Droid server on Ubuntu Server 16.04 and it ran without any problems. I’ve also built ZXing on my 16.04 workstation and it proceeded without any issues. I imagine it would be similar with Debian Unstable.
Debian Unstable is not meant to be used in any live system. You can use it at home on a test machine to search for bugs but that’s all. If there’s a problem with software which is too old you have to wait until a migration to Debian Stretch is done (if that’s even planned).
The large majority of Android apps build on any supported version of Debian, so the base platform works fine there. VLC and Barcode Scanner both have uncommon, customized build setups.
Moving the build infrastructure to a newer version of Debian is a large project, switching to Ubuntu is an even larger project, and then updating to each Ubuntu version would be a pretty large project in itself. Updating the base platform is not the only way to solve these kinds of issues. If you look at CI (Continuous Integration) build platforms like gitlab-ci, travis-ci, circle-ci, etc, they are generally quite conservative in which base platform they use. The base platform needs to be compatible with lots and lots of setups, and updating it means lots of potential breakage.
Perhaps that could be solved in another way, by using a tool like Ansible or Salt, where all the tools needed to install all packages are installed automatically and reproducibly. If a packager approaches a package like VLC they could edit the Ansible template and ask for the build server to be rebuilt. This would also help with migrating between distro versions, when that becomes necessary.
I know I’ve built Java code with latest Maven and Java on ancient CentOS 6 because I’ve had Ansible playbooks to deploy these. That’s, of course, a matter of taste, some people oppose to any non-distro-packaged software and that’s understandable.
What you describe is certainly possible, it would require quite a bit of work. If someone wants to take on that project, they should first sketch out the implementation and how it would integrate with our existing infrastructure.
I’ve found that most custom build systems have instructions that include lines like pip install compare-locales rather than Ansible/etc. profiles. So having something like sudo=pip install compare-locales in the build metadata file makes it easy to understand and manage.
IMHO Docker container is the best alternative among existing. Industry
supported, developing etc.
It is automatically deployed so no Ansible and friends required.
Network-deployable, portable, scalable etc. Strongly recommended.
Docker is okay but it’s in the same domain as my Ansible solution (dockerfile vs cookbook - Linux packages separated from fdroiddata).
What Hans is proposing makes sense to me - it’s proven to work for Travis and others, it may not be too convenient but it works and developers are used to it. It will not necessarily be perfect for people outside of F-Droid organization who may not reset their build server to last snapshot after every build but it’s their problem really.
We’d happily accept merge requests to add Docker support to the fdroid tools:
Docker is very useful for CI builds, but I don’t think its the right tool for our release builds. VirtualBox/KVM provides much stronger isolation than Docker, so we can run random code on our build machine without worrying about malicious code so much. Also, to run Docker at a decent level of security requires a full time sysadmin. That’s easier to do with full VMs.
@gdr if people use the fdroid buildserver, then it is automatically reset before each build.
At least now that Debian 9 has been released you should be able to switch to Stretch, should not you? If this would fix the problems with VLC and the barcode app, then you should do this ASAP. And it’s a supported release, so certainly a good short-term “solution” for this problem…
We do plan on updating to stretch. Updating running infrastructure is a lot of work, so it will not happen soon. Plus updating to stretch will probably break some builds as well, so its not really the solution to this problem. Apps with specific needs should have those needs as part of the build metadata file.