Call for Help: making free software builds of the Android SDK

debian
free-software
#22

While we’re waiting for public access to the files, some info for the repository.

I just contacted by a Qt Creator user who couldn’t get it to use our SDK.
One issue is that it used both sdkmanager and did some manual checks for component versions; in particular it checked if build-tools/28.0.3 exists.
We ship build-tools under build-tools/android-9 - because that’s how the official build system generates it.
To avoid this, and also allow parallel installation with e.g. build-tools/28.0.1, I think we’ll have to repackage the generated repo .zips, unless somebody find the appropriate flag :slight_smile:

Also Qt Creator seemed to insist on using 28.0.3 and not 28.0.2 (the one we provide, last time I checked 28.0.3 wasn’t tagged anywhere at AOSP). It would help to know if Qt Creator just enforces using the latest version, or if this version is hard-coded somewhere.
Renaming the directory was enough to make it stop complaining though :wink:

0 Likes

#23

maybe edit source.properties too, eg. 25.2.5 has this:

Pkg.Revision=25.2.5
Platform.MinPlatformToolsRev=20

:wink:

1 Like

#24

That wasn’t required by the Qt Creator configuration dialog, but that can help, yes.
Make sure you do that before running sdkmanager or update the locally-generated package.xml as well.

0 Likes

#25

Doing my weekly reminder…
@hans any problem?

0 Likes

#26

Sorry, I’m the bottleneck on too many things. Thanks for keeping on nagging. I was waiting for the details like the name and proper packaging and setup before setting up the official repo.

About the naming, I think sdkmanager renames the directory based on data in sources.properties. The official packages also have a base dir of android-9 for build-tools-28.0.3.

Have you tried doing a diffoscope run on your builds vs the original builds? That could be quite useful to see what the diffs are.

0 Likes

#27

About the name, I think I like /android-free/ the best. It is close to Google’s name, which is just /android/, with the essential difference that these builds are actually free, while theirs are not.

1 Like

#28

Let’s go with android-free then. It would be good to start the repo so we can experiment with it.

1 Like

#29

works for me, I’ll set it up this week

0 Likes

This forum's notifications
#30

Ok, it is setup:

I think we should follow the conventions of the Google repositories as much as possible. For example repository.xml should have the version number in it for the format that we are using, e.g. repository-12.xml And the dir structure should be the same. Otherwise, I think we’ll see weird issues, since a lot of the Android SDK was not designed to be decentralized and is quite brittle.

Here are some examples:

That also means naming the ZIP files the same, e.g. platform-27_r02.zip or build-tools_r28.0.3-linux.zip

1 Like

This forum's notifications
#31

FYI, that rsync command is strict, as enforced on the server/remote side. That means if you run it with different options, it will fail. You can add these though for debugging: --verbose --progress The local path can be freely changed, but the remote path cannot.

0 Likes

This forum's notifications
#32

Thanks a lot - I didn’t get notified so I only noticed today while about to complain :wink:
I’m merging a fresh rebuild (that includes the aforementioned build-tools-28.0.3 at last).
https://mirror.f-droid.org/android-free/repository/repository-10.xml
https://mirror.f-droid.org/android-free/repository/sys-img/android/sys-img-3.xml
(give it some time to upload)

There seem to be another xml location for ‘samples’ and ‘sources’ packages, which I didn’t find.
Also I’m not sure if we should symlink e.g. repository.xml to repository-10.xml.

Is there a way to pass this repository URL to sdkmanager without recompiling it?

0 Likes

#33

Added a new system image compiled during the night!

https://dl.google.com/android/repository/addons_list-3.xml seems important to locate the sys-img additional repository. Lots a naming conventions and lack of provider-oriented documentation makes it difficult to figure things out.

There’s a thing called “user defined sites” but I don’t think this’d work as we want to have priority over the default google repositories.

I don’t use this kind of tools so any help is appreciated :slight_smile:

0 Likes

#34

It is possible to add new repositories both via the GUI and via a config file. In Android Studio, go to Tools --> SDK Manager --> SDK Update Sites. For a config file, this should work for these repos if put in ~/.android/repositories.cfg:

count=2
disp00=Android Free SDK
src00=https\://mirror.f-droid.org/android-free/repository/repository-10.xml
disp01=Android Free System Images
src01=https\://mirror.f-droid.org/android-free/repository/sys-img/android/sys-img-3.xml

No need for symlinks or aliases for the XML files, there isn’t a standard path that sdkmanager users for discovery. It is based on providing full URLs.

@Beuc Could you include the existing F-Droid system image in this setup? It is currently available here: https://staging.f-droid.org/emulator/sys-img.xml

It would be amazing to have F-Droid flavors of emulators in this new repo. @cdesai is your build setup for that F-Droid system image somewhere accessible?

Also, FYI, I forgot to change the access logging on the webserver, so the default was enabled up until now. I’m switching it to this proven config which strips out user-identifying info: https://guardianproject.info/2017/06/08/tracking-usage-without-tracking-people/

0 Likes

#35

Yeah that what I called “user-defined sites” - but that doesn’t do the trick, since we want to have priority. E.g. if the user’s build environment requests “platform-28”, we want sdkmanager to download our build, rather than attempt to download a newer revision with a proprietary license from google.

Also the URL scheme doesn’t seem to work:

$ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 tools/bin/sdkmanager --update
Warning: Failed to fetch URL https://mirror.f-droid.org/android-free/repository/repository-10.xml/sys-img.xml, reason: File not found
Warning: Failed to fetch URL https://mirror.f-droid.org/android-free/repository/repository-10.xml/addon.xml, reason: File not found

About the F-Droid system images: how are they built - i.e. what is the license?

0 Likes

#36

I guess I haven’t tried doing the repository replacement before, it seems to have different rules than the system images. I guess we have to dig into the sdkmanager source code to figure what the requirements are for setting up a repository to replace Google’s. I see an error in the XML:

https://clbin.com/RkrPN

Warning: Errors during XML parse:                                                                                
Warning: cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.                                                                                
Warning: org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.                                                                                
Warning: javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.]                                                                                
Warning: Additionally, the fallback loader failed to parse the XML.

I think the Google repositories are hard coded in. There used to be a way to be able to disable them in repositories.cfg, something like adding an entry for those URLs in the same style as I outlined above, but then adding something like enabled02=false.

@cdesai build the F-Droid system images, I think they are probably Apache-2.0 licensed.

0 Likes

#37

I had tested with a modified and an empty “repository-10.xml” and there’s still the same XML error, I figure it’s related to the 404 error above.

By the way the current XMLs are manual copy/pastes of bits produced by the rebuilds, with modified filenames. This is for initial testing. Next we’ll want to automate some of this - if there are existing tools, that would help.

0 Likes

#38

I put together a quick script to scrape all the XML from the Google repository:

#!/bin/sh -ex

wget="wget --continue --quiet"

#baseurl=https://dl.google.com/android/repository
baseurl=https://dl.google.com/android/repository/sys-img/x86
#baseurl=https://dl.google.com/android/repository/sys-img/android

for name in addon repository sys-img; do
    $wget $baseurl/${name}.xml &
    for i in x86 x86_64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do
        $wget $baseurl/${name}-${i}.xml &
    done
done

It seems that your repository-10.xml is totally different from the one that I downloaded. How about starting with the one downloaded from Google, then use symlinks to match the filenames, and change the sha256 sums. Then we can see what we need to change, I think it won’t be much.

As for auto-generating the XML, that should be relatively easy to do once we figure out what matters in that file.

0 Likes

#39

This looks like the script they use to generate the XML:
https://android.googlesource.com/platform/tools/buildSrc.git/+/refs/heads/master/servers/mk_sdk_repo_xml.sh

And these seem to be the .xsd files used by that script:
https://android.googlesource.com/platform/tools/base.git/+/refs/heads/master/sdklib/src/main/java/com/android/sdklib/repository/

0 Likes

#40

I built the F-Droid system images using the AOSP source code without any modifications.

To include the privileged extension it simply needs to be placed in the AOSP source tree, and then built via PRODUCT_PACKAGES += F-DroidPrivilegedExtension

You should definitely generate the XMLs via the script, that’s what I did for the system images.

https://gitlab.com/fdroid/emulator-system-images contains the scripts and files used.

0 Likes

#41

Thanks for the pointers.
@cdesai There doesn’t seem to be any occurrence of PRODUCT in your scripts - did you use some equivalent?

(also the XML bits generated by the rebuilds already use that script:)

/bin/bash -c "development/build/tools/mk_sdk_repo_xml.sh	            out/host/linux-x86/sdk/sdk_x86_64/repository.xml prebuilts/devtools/repository/sdk-repository-10.xsd  build-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip platform-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip docs linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-docs-userdebug.9.0.0_r33.zip:sdk-repo-linux-docs-userdebug.9.0.0_r33.zip platforms linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip:sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip samples linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-samples-userdebug.9.0.0_r33.zip:sdk-repo-linux-samples-userdebug.9.0.0_r33.zip sources linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-sources-userdebug.9.0.0_r33.zip:sdk-repo-linux-sources-userdebug.9.0.0_r33.zip"

Starting from a file from Google with a huge proprietary license on top doesn’t sound good.
Before rebuilding the XML this or that way, we need to be able to actually test it though :slight_smile:
I had made a quick skim in sdklibs, it’s a bit hard to figure things out, some of it is depracated, and some of the sdkmanager logic is in another subproject afaics.

0 Likes

Mastodon