Call for Help: making free software builds of the Android SDK

Yeah that what I called “user-defined sites” - but that doesn’t do the trick, since we want to have priority. E.g. if the user’s build environment requests “platform-28”, we want sdkmanager to download our build, rather than attempt to download a newer revision with a proprietary license from google.

Also the URL scheme doesn’t seem to work:

$ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 tools/bin/sdkmanager --update
Warning: Failed to fetch URL https://mirror.f-droid.org/android-free/repository/repository-10.xml/sys-img.xml, reason: File not found
Warning: Failed to fetch URL https://mirror.f-droid.org/android-free/repository/repository-10.xml/addon.xml, reason: File not found

About the F-Droid system images: how are they built - i.e. what is the license?

I guess I haven’t tried doing the repository replacement before, it seems to have different rules than the system images. I guess we have to dig into the sdkmanager source code to figure what the requirements are for setting up a repository to replace Google’s. I see an error in the XML:

https://clbin.com/RkrPN

Warning: Errors during XML parse:                                                                                
Warning: cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.                                                                                
Warning: org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.                                                                                
Warning: javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.]                                                                                
Warning: Additionally, the fallback loader failed to parse the XML.

I think the Google repositories are hard coded in. There used to be a way to be able to disable them in repositories.cfg, something like adding an entry for those URLs in the same style as I outlined above, but then adding something like enabled02=false.

@cdesai build the F-Droid system images, I think they are probably Apache-2.0 licensed.

I had tested with a modified and an empty “repository-10.xml” and there’s still the same XML error, I figure it’s related to the 404 error above.

By the way the current XMLs are manual copy/pastes of bits produced by the rebuilds, with modified filenames. This is for initial testing. Next we’ll want to automate some of this - if there are existing tools, that would help.

I put together a quick script to scrape all the XML from the Google repository:

#!/bin/sh -ex

wget="wget --continue --quiet"

#baseurl=https://dl.google.com/android/repository
baseurl=https://dl.google.com/android/repository/sys-img/x86
#baseurl=https://dl.google.com/android/repository/sys-img/android

for name in addon repository sys-img; do
    $wget $baseurl/${name}.xml &
    for i in x86 x86_64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do
        $wget $baseurl/${name}-${i}.xml &
    done
done

It seems that your repository-10.xml is totally different from the one that I downloaded. How about starting with the one downloaded from Google, then use symlinks to match the filenames, and change the sha256 sums. Then we can see what we need to change, I think it won’t be much.

As for auto-generating the XML, that should be relatively easy to do once we figure out what matters in that file.

This looks like the script they use to generate the XML:
https://android.googlesource.com/platform/tools/buildSrc.git/+/refs/heads/master/servers/mk_sdk_repo_xml.sh

And these seem to be the .xsd files used by that script:
https://android.googlesource.com/platform/tools/base.git/+/refs/heads/master/sdklib/src/main/java/com/android/sdklib/repository/

I built the F-Droid system images using the AOSP source code without any modifications.

To include the privileged extension it simply needs to be placed in the AOSP source tree, and then built via PRODUCT_PACKAGES += F-DroidPrivilegedExtension

You should definitely generate the XMLs via the script, that’s what I did for the system images.

https://gitlab.com/fdroid/emulator-system-images contains the scripts and files used.

Thanks for the pointers.
@cdesai There doesn’t seem to be any occurrence of PRODUCT in your scripts - did you use some equivalent?

(also the XML bits generated by the rebuilds already use that script:)

/bin/bash -c "development/build/tools/mk_sdk_repo_xml.sh	            out/host/linux-x86/sdk/sdk_x86_64/repository.xml prebuilts/devtools/repository/sdk-repository-10.xsd  build-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip platform-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip docs linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-docs-userdebug.9.0.0_r33.zip:sdk-repo-linux-docs-userdebug.9.0.0_r33.zip platforms linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip:sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip samples linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-samples-userdebug.9.0.0_r33.zip:sdk-repo-linux-samples-userdebug.9.0.0_r33.zip sources linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-sources-userdebug.9.0.0_r33.zip:sdk-repo-linux-sources-userdebug.9.0.0_r33.zip"

Starting from a file from Google with a huge proprietary license on top doesn’t sound good.
Before rebuilding the XML this or that way, we need to be able to actually test it though :slight_smile:
I had made a quick skim in sdklibs, it’s a bit hard to figure things out, some of it is depracated, and some of the sdkmanager logic is in another subproject afaics.

@Beuc For the privext, I did this hack temporarily.

Unsure as to what proprietary license you’re referring too, all of the code linked above / used by me is Apache 2.

I was answering to @hans wrt using google’s repository.xml as a starting point, sorry I wasn’t clear.

Congratulations and thank you to the people who worked for those builds to happen. If there are some guides/posts on how to use them that 'd be really helpful (if anyone can post them here or point to where they are).

Thanks for your supporting message!
Right now the easiest is downloading and extracting a bundle from https://android-rebuilds.beuc.net/ but we’d like to setup the new repository so people could use grab individual components (and pick versions) directly from sdkmanager.
This requires overriding google’s repos in that tool, and setup the new repo accordingly. All help is welcome :slight_smile:

Btw I added a few static index.html so people can browse the files and get a better idea of what we got already.
@hans is the NoIndex web configuration a security measure or just a default that we could change?

I read this and I am not sure: Is a google maven repository clone planned too (with reproducible builds of the libraries if possible)? (this is the repository with the support libraries AND some closed source components which should not be cloned) This would have the benefit that Google can’t break builds by doing something stupid (it once was required to move the google maven repo to the top of the repo list because otherwise projects won’t build anymore). Moreover, it would allow developing Android Apps without interactions with Google servers (at every hit of the run button in android studio, a request is sent to check for library updates, this is not really something I like).

@l-jonas speaking for myself my main focus is the base NDK and SDK, but the project goal is providing Android dev tools without the non-free binary license.
A secondary goal is to provide dev tools to countries blocked by google.
Is the maven repository impacted by these?

Also for the privacy issue, maybe you can mitigate with a local caching proxy :slight_smile:

I set up a proxy (with HTTPS interception). The request at very run seems to be wrong. I was even able to use Android Studio without internet connection (without enabling the offline mode in the settings in Android Studio). Bad brain or this has changed during the last months.

FYI I added API-27 packages.
https://mirror.f-droid.org/android-free/repository/
Listing:
https://android-rebuilds.beuc.net/

2 Likes

And @cdesai 's F-Droid-flavored sys-image for emulators :slight_smile:

1 Like

What do you mean by this?

@JosephJones check a few posts above starting at :

Looks like it is quite easy to build the “sources” zipballs, that would be nice to include in this repo: https://gist.github.com/cketti/71475220a08c9ee43fd48ecaaf7046a5

Mastodon