Call for Help: making free software builds of the Android SDK

free-software
debian

#1

There is an odd quirk with the Android SDK: while the source code is Apache-2.0 licensed, free software, the binaries that Google ships are not free software and even put substantial restrictions on the apps that are built with the Google binaries. @beuc has been leading the effort to make actual free software binaries of the Android SDK. The Debian Android Tools Team has been working to package the Android SDK Tools as part of Debian.

There has been a great discussion on the FSFE Android mailing list about doing free software rebuilds of all the Android SDK components where we have outlined what is left to be done to actually ship this binaries to app developers who care about free software.

I’m proposing to use the F-Droid resources to both run a builder, which is easy now that we have the gcc cfarm machines. Then I also propose hosting the results in an sdkmanager repo on mirror.f-droid.org as the official source of this repo so that it can be easily mirrored elsewhere.

If we can get those builds to be reproducible, that will make it much easier to manage building the official binaries since we won’t need to have some super locked down builder machine.

This effort definitely needs help, here are some specific things where people can jump in:

  • quite hard to even find the git tags that Google uses to make their builds
  • write the scripts to make the sdkmanager repos from our binaries
  • get sdkmanager packaged in Debian
  • creating an Ansible automation to create the buildserver for these machines
  • building all versions of the NDK based on the existing build scripts
  • create build script for the Android Support libraries
  • setup maven repo for all of the free software libraries
  • scripted setup to automatically publish diffoscope output comparing our builds to Google’s official builds

Reverse engineer library used to receive push notifications through Google servers (GCM/Firebase)
#2

JFYI, ReplicantOS team did SDK builds but gave up for some reason (there’s a relevant bug in their redmine). Yet I’m sure you still can find people with that experience.


split this topic #3

5 posts were split to a new topic: Reverse engineer library used to receive push notifications through Google servers (GCM/Firebase)


#4

Replicant shifted focus off SDK/NDK because they thought Debian would do it - but providing several clean Debian package sets of complete SDK platforms turned out quite hard and time consuming, so possibly they could revise their position and help with this project :slight_smile:

On other news while perusing git history I stumbled on the sdk_repo goal modifier in the SDK build system to create separate packages as well as XML package lists.
And also found out how to build system images for alternate architectures e.g. make PRODUCT-sdk_arm64-sdk.
More doc at https://android-rebuilds.beuc.net/SDK/
Two steps closer to drop-in rebuilds :slight_smile:

I put some of the generated files for SDK 8 at:
https://android-rebuilds.beuc.net/testrepo/
Maybe it’s time to start playing around with a free sdkmanager repo? :slight_smile:


#5

Great news! An sdkmanager repo would be great to have. I won’t have time to work on that until January. Feel free to ping me about it.


#6

OK!

By the way you mentioned getting some disk space and I begin to be short of backed-up space on my end.

How do you see things moving forward on that front? :slight_smile:


#7

We will have space on the mirror server for these rebuilds. Do you think they’ll be larger than 100gigs?


#8

If we rebuild all host archs and tools and system images for all Android releases, we may reach that, but:

  • when we reach that goal I’ll offer a bottle of fine champagne :wink:
  • if space becomes a problem we can restrict to a subset (e.g. the packages/archs that F-Droid is most interested in)

Currently there is around 15-20GB of files for everything that is linked at https://android-rebuilds.beuc.net/ (which I just clarified some more).


#9

Ok, so we can put this repo on mirror.f-droid.org without a problem. Before copying your existing binaries, I’d like to try a quick test of whether they are reproducible. It would be nice to launch this repo with reproducible builds.


#10

If you mean reproducible as in https://reproducible-builds.org/ I highly doubt so, but independent binaries would still be nice.

If you intend to use the CI rules you added, beware that these differ from the normal build system (typically: not the same working directory, not the same UID, etc.). I was thinking of pushing android-rebuilds:* base images somewhere at dockerhub so they could be grabbed by the CI and so we could properly (re)build everything using the same recipes.


#11

Something like this https://hub.docker.com/u/androidrebuilds possibly.
I didn’t expect to have to use a non-dash prefix (androidrebuilds/android-rebuilds:sdk-XXX) OTL
I’m open to better naming schemes.


#13

I think we could probably cheat our way into reproducible builds to start with, using Docker images and disorderfs to provide a sorted filesystem.

Let me know what you need from me and I’ll do it! I want to see this happen!


#14

I think we’re getting side-tracked here :wink:
We’ve been discussing this additional disk space for nearly 2 month, went from “create a sdkmanager” repo to “rebuild locally” to “rebuild reproducibly”, but I still need disk space :stuck_out_tongue:

There’s a chance in 1 000 000 that the SDK buids reproductibly even with disorderfs, and from experience debugging a monolithic build for reproducibly issues takes an eternity and then some. You can hack the android-rebuilds Makefile locally to try and install disorderfs just for checking (I don’t know how precisely) but let’s do that aside from this particular sub-project.

Right now I think we need to focus on:

  • setting up the storage area
  • check how to provide a sdkmanager repo, and how to combine the bits of generated XML from the rebuilds
  • unify the CI / non-CI builds because right now there’s duplication and discrepancies between Dockerfile+Makefile and the .gitlab-ci.yml.

#15

The disk space is there already! I didn’t know you were ready to
transfer binaries. How would you like to do that? I could set up a
dedicated account, then could SSH/rsync the binaries to it. You’ll have
to send me an SSH public key for that.

The sdkmanager XML format is easy. We can just use the Google ones as
an example. When I did it before, it was quicker to hand edit it rather
than even script it.


#16

Ok, so now there is a android-rebuilds VM on one of our GCC boxes to provide somewhere to work. Now the question is: how should the repository look on the mirror box? It doesn’t yet have a webserver, but that’ll take me 15 minutes to setup.

Should we follow the Google layout somewhat? e.g.

I think we might actually want to find a better than android or android-rebuilds for the root name there. Perhaps android-sdk? But maybe just android works.


#17

I was considering that “android-rebuilds” wasn’t that good of a name.

The main focus points are:

  • free/libre license
  • DEVelopment tools (for Android)

Non-goals:

  • Android OS (cf. Replicant, LineageOS, etc.)
  • 3rd-party software / libraries (cf… F-Droid repo? ;))

“SDK” may be too limited since we also target the NDK, the Gradle plugin and Studio. NDK and Studio may be downloaded separately but the Gradle plugin is distributed through a repo as well AFAIK.

What do you people think?
“android-libredev”?


#18

android-libredev seem OK, but a little opaque. Other related ideas:

  • android-liberated
  • android-free-dev
  • android-dfsg-free

#19

android-free-dev (or android-freedev to avoid dashitis? ;)) then.

liberated -> as opaque?
dfsg-free -> somewhat redundant, debian-centric

Google uses plain “android” pretty ambiguously (“Android is open source”…) so we probably want to be a little more precise (“dev”), unless there’s a reason to stay compatible with their naming scheme (I wouldn’t know, I don’t use their repos ;))


#20

Hi Hans,
I’m not sure if you’re waiting for something from me (or somebody)? :slight_smile:


#21

these seems like a good idea