I guess I haven’t tried doing the repository replacement before, it seems to have different rules than the system images. I guess we have to dig into the sdkmanager source code to figure what the requirements are for setting up a repository to replace Google’s. I see an error in the XML:
Warning: Errors during XML parse:
Warning: cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.
Warning: org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.
Warning: javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 70; cvc-elt.1: Cannot find the declaration of element 'sdk:sdk-repository'.]
Warning: Additionally, the fallback loader failed to parse the XML.
I think the Google repositories are hard coded in. There used to be a way to be able to disable them in repositories.cfg, something like adding an entry for those URLs in the same style as I outlined above, but then adding something like enabled02=false.
@cdesai build the F-Droid system images, I think they are probably Apache-2.0 licensed.
I had tested with a modified and an empty “repository-10.xml” and there’s still the same XML error, I figure it’s related to the 404 error above.
By the way the current XMLs are manual copy/pastes of bits produced by the rebuilds, with modified filenames. This is for initial testing. Next we’ll want to automate some of this - if there are existing tools, that would help.
I put together a quick script to scrape all the XML from the Google repository:
#!/bin/sh -ex
wget="wget --continue --quiet"
#baseurl=https://dl.google.com/android/repository
baseurl=https://dl.google.com/android/repository/sys-img/x86
#baseurl=https://dl.google.com/android/repository/sys-img/android
for name in addon repository sys-img; do
$wget $baseurl/${name}.xml &
for i in x86 x86_64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do
$wget $baseurl/${name}-${i}.xml &
done
done
It seems that your repository-10.xml is totally different from the one that I downloaded. How about starting with the one downloaded from Google, then use symlinks to match the filenames, and change the sha256 sums. Then we can see what we need to change, I think it won’t be much.
As for auto-generating the XML, that should be relatively easy to do once we figure out what matters in that file.
I built the F-Droid system images using the AOSP source code without any modifications.
To include the privileged extension it simply needs to be placed in the AOSP source tree, and then built via PRODUCT_PACKAGES += F-DroidPrivilegedExtension
You should definitely generate the XMLs via the script, that’s what I did for the system images.
Thanks for the pointers. @cdesai There doesn’t seem to be any occurrence of PRODUCT in your scripts - did you use some equivalent?
(also the XML bits generated by the rebuilds already use that script:)
/bin/bash -c "development/build/tools/mk_sdk_repo_xml.sh out/host/linux-x86/sdk/sdk_x86_64/repository.xml prebuilts/devtools/repository/sdk-repository-10.xsd build-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-build-tools-userdebug.9.0.0_r33.zip platform-tools linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip:sdk-repo-linux-platform-tools-userdebug.9.0.0_r33.zip docs linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-docs-userdebug.9.0.0_r33.zip:sdk-repo-linux-docs-userdebug.9.0.0_r33.zip platforms linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip:sdk-repo-linux-platforms-userdebug.9.0.0_r33.zip samples linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-samples-userdebug.9.0.0_r33.zip:sdk-repo-linux-samples-userdebug.9.0.0_r33.zip sources linux out/host/linux-x86/sdk/sdk_x86_64//sdk-repo-linux-sources-userdebug.9.0.0_r33.zip:sdk-repo-linux-sources-userdebug.9.0.0_r33.zip"
Starting from a file from Google with a huge proprietary license on top doesn’t sound good.
Before rebuilding the XML this or that way, we need to be able to actually test it though
I had made a quick skim in sdklibs, it’s a bit hard to figure things out, some of it is depracated, and some of the sdkmanager logic is in another subproject afaics.
Congratulations and thank you to the people who worked for those builds to happen. If there are some guides/posts on how to use them that 'd be really helpful (if anyone can post them here or point to where they are).
Thanks for your supporting message!
Right now the easiest is downloading and extracting a bundle from https://android-rebuilds.beuc.net/ but we’d like to setup the new repository so people could use grab individual components (and pick versions) directly from sdkmanager.
This requires overriding google’s repos in that tool, and setup the new repo accordingly. All help is welcome
Btw I added a few static index.html so people can browse the files and get a better idea of what we got already. @hans is the NoIndex web configuration a security measure or just a default that we could change?
I read this and I am not sure: Is a google maven repository clone planned too (with reproducible builds of the libraries if possible)? (this is the repository with the support libraries AND some closed source components which should not be cloned) This would have the benefit that Google can’t break builds by doing something stupid (it once was required to move the google maven repo to the top of the repo list because otherwise projects won’t build anymore). Moreover, it would allow developing Android Apps without interactions with Google servers (at every hit of the run button in android studio, a request is sent to check for library updates, this is not really something I like).
@l-jonas speaking for myself my main focus is the base NDK and SDK, but the project goal is providing Android dev tools without the non-free binary license.
A secondary goal is to provide dev tools to countries blocked by google.
Is the maven repository impacted by these?
Also for the privacy issue, maybe you can mitigate with a local caching proxy
I set up a proxy (with HTTPS interception). The request at very run seems to be wrong. I was even able to use Android Studio without internet connection (without enabling the offline mode in the settings in Android Studio). Bad brain or this has changed during the last months.
