Open Source Android Development Tools

I had some free time this morning and took a long hard look at all four of these ‘EULA’ problems. I actually think we are good to go, and here’s why:

  1. As I thought previously, Google/AOSP is not trying to change clang or these build tools, but that they have to pick a specific version in the space/time continuum to make sure the builds always work the same, which is why they fork their own version. They do cherry pick features, both up and down stream for their clang/build tools. But now we have definitive mention of that which you can read.

https://android.googlesource.com/toolchain/llvm_android/+/refs/heads/main/README.md

  1. Google’s lld license can be found here, and is Apache 2.0.

https://android.googlesource.com/toolchain/lld/+/refs/heads/main/LICENSE.TXT

  1. Google’s LibLLVM_android latest build source code can be found here, and it includes “MODULE_LICENSE” files, all of which are Apache 2.0, MIT, and BSD.

https://android.googlesource.com/toolchain/llvm_android/+/refs/heads/llvm-r510928

  1. Google’s renderscript can be viewed here, and each file has a license statement at the top of Apache2.0.

https://android.googlesource.com/platform/frameworks/base/+/master/rs/java/android/renderscript

  1. Google’s libclang_android building all takes part in the android clang/libllvm building, which we already discussed above, but you can read the build instructions here:

https://android.googlesource.com/toolchain/llvm_android/+/refs/heads/llvm-r510928

Interesting to note that in step 4, the prebuilts are downloaded for testing.

  1. In all cases, (lld, renderscript, libllvm, libclang) their source code is found inside the Android Open Source Project code, whereas when something is not open source, Google puts that in a separate repository which is pulled in the manifest.xml file when syncing.

  2. @hans is right about renderscript, it is depreciated in Android 12 and onward. However, if our users plan to build for say Android 11, they may want/need it. Especially if they are rebuilding someone else’s previous open source work which may include code that renderscript reads/uses.

I recommend that we press forward, simply removing the EULA statement by replacing it with a blank file prior to build (which can be scripted), since it doesn’t seem to actually apply. E.g., it shouldn’t be there, so we get rid of it. But I’m not a lawyer… I’m also the dumbest person I know, so keep that in mind. :smiley:

P.S.
All Google’s Clang/LLVM prebuilts are here:
https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86/+/refs/heads/main
and have an Android.bp license of:

license {
    name: "prebuilts_clang_host_linux-x86_license",
    visibility: [":__subpackages__"],
    license_kinds: [
        "SPDX-license-identifier-Apache-2.0",
        "SPDX-license-identifier-BSD",
        "SPDX-license-identifier-MIT",
        "SPDX-license-identifier-NCSA",
        "SPDX-license-identifier-PSF-2.0",
        "SPDX-license-identifier-Zlib",
        "legacy_unencumbered",
    ],
1 Like

I +100 that notion.

Alright, thanks for looking at them all. I wanted a second opinion. I had a similar conclusion regarding these modules. Actually I had just assumed the libLLVM_* and libclang_* were derivative works of the upstream clang/llvm, and thus would also have to be apache, but had not found the sources to verify. Good work! And I see we’re both looking at the same apache licensed sources of renderscript. :wink:

Really, it’s awesome to have this clearly laid out. Yes, I think we can proceed removing the EULA with a clean consciousness. When I get the time I can try digging deeper in the things I detailed previously, and see if I can find if we can simply patch away some specific lines. Or we could just quick and dirty ‘echo “” > prebuilts/sdk/NOTICE’, although that feels a bit… asking for trouble? Maybe one day (or in some older version) there’s something else correctly pulling in the EULA, and we sort of just scrub it away? :rofl: Finding a way of patching out the specific root cause for these 4 modules might be safest and cleanest?

That said, I’m still a bit busy, so it will not be just yet. I don’t suppose this could be a clue? tools/Android.bp - platform/prebuilts/sdk - Git at Google it does seem to pull in something called “prebuilts_sdk_license”, which might (directly/indirectly) turn into pulling in the EULA file?

And @hans I also do like you idea of simply removing the parts that could be a problem. If it turns out there were something legally we missed here, I’m all in favor of simply removing any offending parts. Until then, it would be good if we don’t, and try provide something that will be 1:1 compatible with the official sdk packages. Just to avoid scaring away people if something breaks when they try migrate.

Also (and trust me, I know I’m the oddball here) I’m actually targeting android as old as kitkat (4.4) or even older with my little project. And even then I have no need for any of these 4 modules. But that might just be because I use my own mini build system.

edit: okay, what in the world is this link supposed to be?! http://go/android-license-faq (taken from that Android.bp, seems broken, or is it internal?)

1 Like

Looking forward to trying these packages!

I also agree with @alaskalinuxuser’s assessment. Plus there is a clear record
here of in depth due diligence, and that matters in copyright cases, and with
contract law (licenses are contracts). It is clear you are trying to do the
right thing.

The http://go/ link is an internal Google thing. On their internal networks,
they have set up the domain name go for their employees. So
http://go/android-license-faq is an internal FAQ.

1 Like

Alright, that makes us three people agreeing this looks OK. :smiley:

Soon, when things calm down a bit, I will resume digging into the build system. But if anyone else figures it out before then, I don’t mind. I’m trying to help, doing this because I believe in and care about the project.

Also a little trick, for anyone who cares: I found using chmod to remove read/write from files/directories was the easiest way of figuring out the parts of the build system - just observe what breaks when trying to read or write. That’s pretty much how all previous insights were obtained, because it certainly isn’t a clean and well documented build system…

1 Like

Okay, some further digging…

First, just to recap: I had found that something was running the build_license_metadata script to generate the meta_lic files (storing arguments in the meta_lic.rsp files), but what?

Turns out this is done in a ninja-file. For me the ldd-module is specifically in out/soong/build.sdk.4.ninja, where among many others, the following can be found the lld module:

[...]

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# Module:  build-tools-lld-linux
# Variant:
# Type:    genrule
# Factory: android/soong/android.moduleType.register.ModuleFactoryAdaptor.func1
# Defined: prebuilts/sdk/tools/Android.bp:102:1

m.build-tools-lld-linux_.moduleDesc = //prebuilts/sdk/tools:build-tools-lld-linux
m.build-tools-lld-linux_.moduleDescSuffix =

rule m.build-tools-lld-linux_.generator
    command = out/host/linux-x86/bin/sbox --sandbox-path out/soong/.temp --output-dir out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/gen --manifest out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/genrule.sbox.textproto

build $
        out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/genrule.sbox.textproto: $
        g.android.rawFileCopy $
        out/soong/raw-sdk/ec/ec4d5d0ffdbec7c56da8cc32d014b9428b6adb43
    description = ${m.build-tools-lld-linux_.moduleDesc}raw genrule.sbox.textproto${m.build-tools-lld-linux_.moduleDescSuffix}
    tags = module_name=build-tools-lld-linux;module_type=genrule;rule_name=rawFileCopy

build $
        out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/gen/lld.zip: $
        m.build-tools-lld-linux_.generator | $
        prebuilts/sdk/tools/linux/lib64/libc++.so.1 $
        prebuilts/sdk/tools/linux/lld-bin/lld prebuilts/sdk/tools/lld $
        prebuilts/sdk/tools/lld-dummy $
        out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/genrule.sbox.textproto $
        out/host/linux-x86/bin/go/soong_zip/obj/soong_zip $
        out/host/linux-x86/bin/sbox
    description = ${m.build-tools-lld-linux_.moduleDesc}generate lld.zip${m.build-tools-lld-linux_.moduleDescSuffix}
    tags = module_name=build-tools-lld-linux;module_type=genrule;rule_name=generator

build $
        out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/gen/lld.zip: $
        m.build-tools-lld-linux_.generator | $
        prebuilts/sdk/tools/linux/lib64/libc++.so.1 $
        prebuilts/sdk/tools/linux/lld-bin/lld prebuilts/sdk/tools/lld $
        prebuilts/sdk/tools/lld-dummy $
        out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/genrule.sbox.textproto $
        out/host/linux-x86/bin/go/soong_zip/obj/soong_zip $
        out/host/linux-x86/bin/sbox
    description = ${m.build-tools-lld-linux_.moduleDesc}generate lld.zip${m.build-tools-lld-linux_.moduleDescSuffix}
    tags = module_name=build-tools-lld-linux;module_type=genrule;rule_name=generator

build $
        out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/meta_lic: $
        g.android.licenseMetadataRule | ${g.android.licenseMetadataCmd} || $
        dedup-8d89789fba2a7c89
    description = ${m.build-tools-lld-linux_.moduleDesc}license metadata${m.build-tools-lld-linux_.moduleDescSuffix}
    tags = module_name=build-tools-lld-linux;module_type=genrule;rule_name=licenseMetadataRule
    args = -mn build-tools-lld-linux -mt genrule -r prebuilts/sdk/tools -mc UNKNOWN -k SPDX-license-identifier-Apache-2.0 -k SPDX-license-identifier-BSD -k SPDX-license-identifier-CC0-1.0 -k SPDX-license-identifier-CPL-1.0 -k SPDX-license-identifier-MIT -k SPDX-license-identifier-NCSA -k SPDX-license-identifier-OFL -k SPDX-license-identifier-Unicode-DFS -k SPDX-license-identifier-W3C -k legacy_unencumbered -c notice -c reciprocal -c unencumbered -n prebuilts/sdk/NOTICE -d 'out/soong/.intermediates/build/soong/zip/cmd/soong_zip/linux_glibc_x86_64/meta_lic:toolchain' -s out/host/linux-x86/bin/soong_zip

[...]

and there are many more such files build.sdk.*.ninja there. They are all autogenerated and gigantic. As the comment suggests, this part of the file is in some way generated by prebuilts/sdk/tools/Android.bp, as previously linked to: tools/Android.bp - platform/prebuilts/sdk - Git at Google in one of the “genrule{}” blocks.

Nowhere is the EULA or the NOTICE files mentioned. The only thing there that specifies the licenses seems to be right at the top: default_applicable_licenses: ["prebuilts_sdk_license"]. So what turns this into the long list? I guess somewhere there’s a prebuilts_sdk_license defined, which I assume is then applied to all these “genrules”.

Perhaps this go-written thing is involved… android/licenses.go - platform/build/soong - Git at Google

The genrules in this file are specifically for the build-tools-lld-*. Similarly for renderscript: out/soong/.intermediates/prebuilts/sdk/renderscript/renderscript_sdk_prebuilts/meta_lic.rsp comes from out/soong/build.sdk.9.ninja which in turn is generated by prebuilts/sdk/renderscript/Android.bp, containing a single “genrule” for “renderscript_sdk_prebuilts”, and the only license is default_applicable_licenses: ["prebuilts_sdk_license"] in the package{} block.

As an experiment I tried commenting out this line from two Android.bp-files: prebuilts/sdk/tools/Android.bp and prebuilts/sdk/renderscript/Android.bp. And sure enough, the files .intermediates/prebuilts/sdk/tools/build-tools-lld-linux/meta_lic.rsp and out/soong/.intermediates/prebuilts/sdk/renderscript/renderscript_sdk_prebuilts/meta_lic.rsp do not contain the EULA file any more (the -n prebuilts/sdk/NOTICE argument previously)!

But it also removes all the other licenses, like Apache. It’s possible all important licenses will still be included in the final zip, because there will be more modules pulling in similar licenses, but I have not tried this yet.

This gives me a few ideas:

  1. Find this “prebuilts_sdk_license” definition and remove the reference to the EULA file. But it could possibly affect other modules, unintentionally removing EULA from something where it’s applicable?
  2. just remove the entire EULA text from the NOTICE file. Same result as above, just much simpler, and same possible concerns as above.
  3. Figure out if the specific Android.bp files can be modified to explicitly list licenses instead of referring to the common “prebuilts_sdk_license” list. Or, if not possible, define a custom list instead of “prebuilts_sdk_license” and change to using it for these 4 modules.
  4. Simply remove the line with “prebuilts_sdk_license” from the 4 Android.bp files, and try build everything and double-check that the other modules still pulls in all related licenses. Just diff the resulting NOTICE files to make sure the only difference is the removal of the EULA.

Obviously (3) above would be the most clean (less risk of accidentally removing the EULA from another module, or removing some license that applies). Since we all 3 agree they are all under Apache, so it would just be a matter of adding that as an explicit license instead of “prebuilts_sdk_license” to the 4 Android.bp files…

edit: short but important update! It seems alternative 3 will work. Here are all licenses to choose from: licenses/Android.bp - platform//build/soong - Git at Google so maybe something like default_applicable_licenses: ["SPDX-license-identifier-Apache-2.0"] could work. And if that doesn’t work, here is an example of defining a custom license{} block to selectively pull only the licenses we want Android.bp - platform/system/ca-certificates - Git at Google (and use it instead of “prebuilts_sdk_license”). I haven’t tried this yet, but I believe this is close to the perfect solution.

1 Like

Good news everybody! I now have the build-tools package for “linux” and “windows” building without the EULA being included! At least for SDK 15, but it probably will work for all of them - the affected files seem mostly unchanged since 2013…

Turns out only two files needed modification: prebuilts/sdk/tools/Android.bp and prebuilts/sdk/renderscript/Android.bp. The first is pretty much just the different clang/llvm-derived works, and the second is for renderscript. They both use default_applicable_licenses: ["prebuilts_sdk_license"] (which can be found defined here: Android.bp - platform/prebuilts/sdk - Git at Google the “it may not be entirely correct” line makes me chuckle).

I changed it to the already defined “Android-Apache-2.0”, (as defined here: licenses/Android.bp - platform//build/soong - Git at Google) which only pulls in “SPDX-license-identifier-Apache-2.0”, and no EULA (no license_text block). But it does claim “Copyright (C) The Android Open Source Project”, which I’m not happy with. It’s probably valid for the renderscript, but the rest is not their copyright. But it seemed like the smallest and cleanest solution, since it doesn’t seem to have any impact on the resulting packages outside of the EULA now not being included. All the copyrights and licenses in the NOTICE file remains unchanged, so it seems perfect.

I also tried changing it to “prebuilts_clang_host_linux-x86_license” as @alaskalinuxuser mentioned above, but it is not defined as visible globally. So it would have to be edited (“You may need to add “//prebuilts/sdk/tools” to its visibility”).

So, I propose simply adding the following line anywhere in the docker build.sh, before the make (and just to be safe, probably the lunch as well):

sed -i 's/prebuilts_sdk_license/Android-Apache-2.0/' prebuilts/sdk/tools/Android.bp prebuilts/sdk/renderscript/Android.bp

Also, I do apologize for the triple-post. I could add this to the previous post, but I’m worried it will all get too big and cluttered…

2 Likes

Great work Slinger!

I just got back from my vacation, so after I settle in, I can rebuild with this change.

Do you want me to fork the build scripts, and then add this edit, or do you want to fork and edit so you get the git credit? Or do you want me to fork, add you to colab, then you edit for credit? Or do you care, and just want me to fork, edit and in the description add you for credit?

I have to edit some of the build scripts as well, especially for 12 if I recall.

Also, this works well from Android 12 up to the present, but does not work for Nougat and down, as they do not have an Android.bp file, and does not work for Android 8 through 11, as the Android.bp file does not contain this line that we are using sed to fix.

I don’t see this as a big issue. A question I did have was how old of each version of the SDK and NDK do we plan to build? My thoughts are that we should start by building the latest on down as far as we can go. But that’s just my opinion. In some of the older variations, we may need to make the EULA license a blank file if we can’t figure out how it is baked in.

Thoughts?

Yes, the older versions used a (somewhat) cleaner approach with Makefiles. If we want to support them, it should be possible to figure out. Especially to verify that there is no other components pulling in the EULA (again, me being paranoid about the legal stuff). But also, as you suggest, deleting the text is a fallback.

For versions, I think the idea of starting with the latest makes sense, so IMO the much older versions can wait. My own personal needs are simple: I want to support devices as old as kitkat (android 4.4, api 19), or maybe older. And even the latest SDK still supports this, so I am fine with whatever here.

The NDK is different: I would need r25 (it’s the last to support kitkat). r23 might also be interesting (supports down to api 16), but I’m not sure yet if I will even use it. But since the NDKs “builds” are just repackaged prebuilts, building the older versions should be pretty smooth sailing.

As for credits, I’m honestly not sure. Where are you planing on putting it? gitlab? codeberg? But I am truly thankful that you are concerned about it. I have sometimes contributed code and ideas to projects without getting much credit (even on some things mentioned on the f-droid forum and blog). So I really appreciate that you care about it. :slight_smile:

1 Like

Gitlab is my go to place for such things. I’ll probably fork the original build scripts tomorrow or Friday. :+1:

I have forked the codeberg source to gitlab (to retain all of Startfish’s work and credit) and the repo can be found here:

If you want to grab some credit for your edit, you can either fork and pull request or ask to be a member and I’ll add you. Next week I plan to start the rebuilds, but I can wait while you add your work.

Or, if you’d rather not have your name on it, let me know too. I can add your edits for you as well.

I’m really amiable. Just want to make sure you have the opportunity to get some credit.

1 Like

Awesome work. :slight_smile:

1 Like

I’m so sorry for the late response! I hope you have not been held back waiting for me.

Again, I really appreciate your concern regarding crediting. It warms my heart. I’ve been thinking about this, is the git log the best approach for this? I mean in two ways:

  1. I believe much of the work by Starfish is probably based on the previous work by Beuc, who’s commits are not included.
  2. Do people really look at the commit history to see contributors? If it’s further forked, it might disappear. Especially if people just copy the files directly.

What I’m thinking is: we should probably try to list everyone involved (and what they did) in the files themselves? It could be in the README, but also in the individual files? I also noticed none of the files themselves contain any copyright information, which we might want to fix as well.

I honestly have no experience with crediting+copyright info when several people are involved (mostly just developing solo). But typically, I would expect each file to begin with something like this (perhaps real names, emails, similar, depending on what people want):

#!/bin/sh
#Copyright (C) ...years... Beuc
#Copyright (C) ...years... Starfish
#Copyright (C) ...years... AlaskaLinuxUser
#Copyright (C) ...years... Slinger
#...

And somewhere detailing what each person did (either in the file or the README I guess). Again, I have no experience, so I shouldn’t be the one to guide this. But I feel we should make sure to credit Beuc, and look at what Starfish changed compared to his work.

(Again: the nicknames could also be real names and similar, depending on what people want, I think I would prefer that for myself)

Also, I don’t have an account on gitlab (well, not a “personal” one - I got one I plan to use as a mirror of my project which is on savannah), and I don’t know if I have anything to gain from creating one. If we figure out a way of crediting people directly in the files, I think I’ll be fine with that. I could either send you what edits needs to be made.

Or this could be an excuse (for me) to try the whole git patch stuff. Originally git collaboration wasn’t meant to be through this “fork” workflow. Instead git has its own built-in system for generating and applying patches, which includes metadata like who made it. Typically used to send patches over email, like on the linux kernel mailing list. Might be worth trying, just for the experience. But again: anything is fine for me, I don’t want to cause extra complexity for you.

By the way, since we discussed older versions, I had a look at the older build-tools package for android 11, 12, 13, 14 (“build-tools_r30-linux.zip” and similar 31, 32, 33… also 31 and 32 are both for android 12): the ones for 11 and 12 do not contain the EULA in the NOTICE.txt, only the one for 13 and later!

I haven’t looked at the other zips for these versions (and this definitely NEEDS to be done! Also to verify the EULA doesn’t exists somewhere else), but fingers crossed, perhaps only the recent SDK versions even need this tweak? This was with the official zips, I think. Downloaded from the tencent mirror.

1 Like

Yes. That is the right way to go as a mention and credit. But it would be wise to add unless they are active for us, then mention such, else as a historical approach and us carrying it forward.

Not necessarily to do that. Good to have, but not necessary.

This too is not necessary if @alaskalinuxuser is going to do everything there, or unless you are going to contribute there directly. Any extra accounts just are burdens tbvh.

For the rest, let me know when ready for things, and I will open my site for others with our files.
I have kept them, only excluded from daily backups for my server. :innocent:

1 Like

You bet, no need to make yet another account. We could add the work to a file. We can add it to the readme, or we could make a contributing.md file, too, which explains how to contribute and can list past contributors as well. At least, that’s how I’ve seen those used in the past, but I’m open to suggestions.

Absolutely, and thanks again for hosting!

A little delay here on my end, but I should have everything sorted out soon. My current real life job was cut, so I got a new one. Just moving some things around, but I still plan to start on this sometime this week. I can make the edits Slinger suggested and rebuild 15, 14, 13, 12.1, and 12.0 for sure, then we can look at the older ones.

1 Like

Honestly, good that you landed another one. I would hate to see anyone suffering tbvh.

1 Like

Okay, in the interest of moving forwards, I made a contribution file:

https://gitlab.com/alaskalinuxuser/sdk-rebuilds/-/blob/main/CONTRIBUTIONS.md

Where I list the prior contributors and state how to contribute going forward. Not very flashy, but it works. Anyone with time to spare can clean it up or change the format, I will not be offended at all. :smiley:

This way we will press forward and those who contributed will be recognized. :+1:

Like @vdbhb59 said, it’s good to hear things have worked out. Please feel no hurry with any of this! Just take your time, and that way I don’t need to feel bad for replying slowly. :wink:

In the interest of figuring out the copyright and crediting, I looked at some other projects. I think the CONTRIBUTIONS.md is a good idea, but probably not where people should be listed. Nano seems to have a good structure:

An AUTHORS file, listing proper authors:
https://cgit.git.savannah.gnu.org/cgit/nano.git/tree/AUTHORS

A THANKS file, listing contributions like translations:
https://cgit.git.savannah.gnu.org/cgit/nano.git/tree/THANKS

And for contrast, a simpler example from Indent (maybe a bit too bare-bones):
https://cgit.git.savannah.gnu.org/cgit/indent.git/tree/AUTHORS

In case the cgit service is overloaded (I remember rwp mentioned the savannah servers getting too much traffic from bots/scrappers, I suspect AI thrawlers), just use “apt source” or similar.

Since the amount of code is small, I guess a single AUTHORS (or AUTHORS.md if you want to be modern and fancy) can list everyone and what they did? And then the script files can just have a short copyright info (and apache license) at the top, listing those that made edits.

Do you think that makes sense? When we have decided, I think I will give my name and email. And the code with comments to insert.

I suspect the only edits needed (for removing the EULA) will be for versions 15 and 14. Assuming they are the only one with the EULA included in packages. But we need need to check if the EULA exists somewhere else for older versions. I didn’t get around to checking it myself, since the tencent mirror was throttled. And also, double check that our own packages don’t contain anything. I only built 15 to verify, due to limited time and disk space.

ps: I think we can remove some of the stuff in the READMEs, like the whole “Microsoft Windows is a trademark of the Microsoft group of companies. Android is a trademark of Google LLC.” :joy: (yes, I know it’s from forking the Starfish repo)

1 Like

All of these look like great options! :smiley:

Here’s my suggestion:

While I build the SDKs, could you put together an AUTHORS file, formatted any of the above ways? If you post it here, or message it to me, I will remove the CONTRIB file and put up the AUTHORS file.

By the way, just built SDK-15.0.0_r9 using the aforementioned edit, works great, and no unneeded EULA found. It is uploading now, but my internet is slow, so it will take about 2 hours from now to finish.

@vdbhb59 Since you have the files from before, would you please delete your copies of SDK15, and SDK14? Also, would you be able to extract SDKs 13, 12.1, and 12 that were previously built and check them for the EULA statement?

If it exists in the older SDKs, I will edit and rebuild them, if it doesn’t exist, then those versions are good to go.

I will keep you posted for when SDK15 is uploaded and ready for you to move to your server for hosting, if you are still willing to do so.

Thanks all! Glad to see progress being made. :grinning: