While we’re waiting for public access to the files, some info for the repository.
I just contacted by a Qt Creator user who couldn’t get it to use our SDK.
One issue is that it used both sdkmanager and did some manual checks for component versions; in particular it checked if build-tools/28.0.3 exists.
We ship build-tools under build-tools/android-9 - because that’s how the official build system generates it.
To avoid this, and also allow parallel installation with e.g. build-tools/28.0.1, I think we’ll have to repackage the generated repo .zips, unless somebody find the appropriate flag
Also Qt Creator seemed to insist on using 28.0.3 and not 28.0.2 (the one we provide, last time I checked 28.0.3 wasn’t tagged anywhere at AOSP). It would help to know if Qt Creator just enforces using the latest version, or if this version is hard-coded somewhere.
Renaming the directory was enough to make it stop complaining though
That wasn’t required by the Qt Creator configuration dialog, but that can help, yes.
Make sure you do that before running sdkmanager or update the locally-generated package.xml as well.
Sorry, I’m the bottleneck on too many things. Thanks for keeping on nagging. I was waiting for the details like the name and proper packaging and setup before setting up the official repo.
About the naming, I think sdkmanager renames the directory based on data in sources.properties. The official packages also have a base dir of android-9 for build-tools-28.0.3.
Have you tried doing a diffoscope run on your builds vs the original builds? That could be quite useful to see what the diffs are.
About the name, I think I like /android-free/ the best. It is close to Google’s name, which is just /android/, with the essential difference that these builds are actually free, while theirs are not.
from the VM: rsync -ax --delete /home/android-rebuilds/testrepo/ mirror.f-droid.org:/var/www/html/android-free/repository/
I think we should follow the conventions of the Google repositories as much as possible. For example repository.xml should have the version number in it for the format that we are using, e.g. repository-12.xml And the dir structure should be the same. Otherwise, I think we’ll see weird issues, since a lot of the Android SDK was not designed to be decentralized and is quite brittle.
FYI, that rsync command is strict, as enforced on the server/remote side. That means if you run it with different options, it will fail. You can add these though for debugging: --verbose --progress The local path can be freely changed, but the remote path cannot.
There seem to be another xml location for ‘samples’ and ‘sources’ packages, which I didn’t find.
Also I’m not sure if we should symlink e.g. repository.xml to repository-10.xml.
Is there a way to pass this repository URL to sdkmanager without recompiling it?
It is possible to add new repositories both via the GUI and via a config file. In Android Studio, go to Tools → SDK Manager → SDK Update Sites. For a config file, this should work for these repos if put in ~/.android/repositories.cfg:
count=2
disp00=Android Free SDK
src00=https\://mirror.f-droid.org/android-free/repository/repository-10.xml
disp01=Android Free System Images
src01=https\://mirror.f-droid.org/android-free/repository/sys-img/android/sys-img-3.xml
No need for symlinks or aliases for the XML files, there isn’t a standard path that sdkmanager users for discovery. It is based on providing full URLs.
It would be amazing to have F-Droid flavors of emulators in this new repo. @cdesai is your build setup for that F-Droid system image somewhere accessible?
Also, FYI, I forgot to change the access logging on the webserver, so the default was enabled up until now. I’m switching it to this proven config which strips out user-identifying info: Tracking usage without tracking people - Guardian Project
Yeah that what I called “user-defined sites” - but that doesn’t do the trick, since we want to have priority. E.g. if the user’s build environment requests “platform-28”, we want sdkmanager to download our build, rather than attempt to download a newer revision with a proprietary license from google.
Also the URL scheme doesn’t seem to work:
$ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 tools/bin/sdkmanager --update
Warning: Failed to fetch URL https://mirror.f-droid.org/android-free/repository/repository-10.xml/sys-img.xml, reason: File not found
Warning: Failed to fetch URL https://mirror.f-droid.org/android-free/repository/repository-10.xml/addon.xml, reason: File not found
About the F-Droid system images: how are they built - i.e. what is the license?
I guess I haven’t tried doing the repository replacement before, it seems to have different rules than the system images. I guess we have to dig into the sdkmanager source code to figure what the requirements are for setting up a repository to replace Google’s. I see an error in the XML:
Warning: Errors during XML parse:
Warning: cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.
Warning: org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.
Warning: javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.]
Warning: Additionally, the fallback loader failed to parse the XML.
I think the Google repositories are hard coded in. There used to be a way to be able to disable them in repositories.cfg, something like adding an entry for those URLs in the same style as I outlined above, but then adding something like enabled02=false.
@cdesai build the F-Droid system images, I think they are probably Apache-2.0 licensed.
I had tested with a modified and an empty “repository-10.xml” and there’s still the same XML error, I figure it’s related to the 404 error above.
By the way the current XMLs are manual copy/pastes of bits produced by the rebuilds, with modified filenames. This is for initial testing. Next we’ll want to automate some of this - if there are existing tools, that would help.
I put together a quick script to scrape all the XML from the Google repository:
#!/bin/sh -ex
wget="wget --continue --quiet"
#baseurl=https://dl.google.com/android/repository
baseurl=https://dl.google.com/android/repository/sys-img/x86
#baseurl=https://dl.google.com/android/repository/sys-img/android
for name in addon repository sys-img; do
$wget $baseurl/${name}.xml &
for i in x86 x86_64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do
$wget $baseurl/${name}-${i}.xml &
done
done
It seems that your repository-10.xml is totally different from the one that I downloaded. How about starting with the one downloaded from Google, then use symlinks to match the filenames, and change the sha256 sums. Then we can see what we need to change, I think it won’t be much.
As for auto-generating the XML, that should be relatively easy to do once we figure out what matters in that file.
I built the F-Droid system images using the AOSP source code without any modifications.
To include the privileged extension it simply needs to be placed in the AOSP source tree, and then built via PRODUCT_PACKAGES += F-DroidPrivilegedExtension
You should definitely generate the XMLs via the script, that’s what I did for the system images.
Thanks for the pointers. @cdesai There doesn’t seem to be any occurrence of PRODUCT in your scripts - did you use some equivalent?
(also the XML bits generated by the rebuilds already use that script:)
/bin/bash -c "development/build/tools/mk_sdk_repo_xml.sh out/host/linux-x86/sdk/sdk_x86_64/repository.xml prebuilts/devtools/repository/sdk-repository-10.xsd build-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip platform-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip docs linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-docs-userdebug.9.0.0_r33.zip:sdk-repo-linux-docs-userdebug.9.0.0_r33.zip platforms linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip:sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip samples linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-samples-userdebug.9.0.0_r33.zip:sdk-repo-linux-samples-userdebug.9.0.0_r33.zip sources linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-sources-userdebug.9.0.0_r33.zip:sdk-repo-linux-sources-userdebug.9.0.0_r33.zip"
Starting from a file from Google with a huge proprietary license on top doesn’t sound good.
Before rebuilding the XML this or that way, we need to be able to actually test it though
I had made a quick skim in sdklibs, it’s a bit hard to figure things out, some of it is depracated, and some of the sdkmanager logic is in another subproject afaics.
