Open Source Android Development Tools

Nice to see activity has continued here. Unfortunately I have gotten quite sick since I last wrote and have not been able to work on this as much as I want.

@hans What you wrote makes me hopeful:

I hope this means that we are already done for the generated zips that do not contain the EULA (which is all of them, except for one). I don’t feel quite relieved here, but I do hope it’s that simple. One thing of note: one of the EULA’d prebuilts is a repo called “sdk”, which contains a bunch of jar-files, like conscrypt.jar. My worry is that they are used as-is (perhaps packaged inside the android.jar?), and then might “infect” the end results with the EULA terms. In fact, this “sdk” even contains several android.jar files, and I do hope the build process doesn’t just copy one of them directly…

But for the one zip with an EULA, “build tools”, I agree we can’t just pass it along to users and say “just ignore that EULA part”. I also think we need to be careful with removing the EULA, in case it does apply. The good part is that this specific zip can mostly be built independent from the SDK, as the work by debian (and not the least, by you yourself) has already demonstrated. If I recall, what remains outside of the already debian-provided programs is:

  • r8/d8: this tool is more modern than the other build tools (like aapt, zipalign, …) and has its own repo, and a clean license. However, it also uses gradle to build, so I don’t know if it pulls in some badly licensed/EULA’d dependency.
  • there’s a thing called “renderscript”. I don’t know anything about it, maybe it’s affected by the EULA. I read it’s deprecated, so in worst case (it’s affected by the EULA) maybe we can just throw it away.

@Legimet Neat. I had heard of zram, but never tried it. I just enabled another swap partition during my build. Not as pretty, and probably slower than zram, but it worked. I also saw very long build times, that’s just normal for the mess that is android. :wink:

@alaskalinuxuser I have tried building a simple app using your SDK 14 (it’s the only version I had downloaded before vdbhb59 removed them). The good news is: the app worked perfectly fine!

This was only tested against an old 32 bit arm (kitkat) and a more modern 64 bit arm phone. I also only used the android.jar and d8, the rest (zipalign, aapt, apksigner and friends) was those provided by debian.

I know It’s not much. But it’s what I had energy to try, and at least this shows the result are truly usable and useful. When I get better I should try using all tools from your build. I also plan to install android x86 (32 bit and/or 64 bit) on an old eeepc to have a complete test of all targets.

Also this did not use your NDKs, since they don’t support kitkat (I didn’t have the energy to go through and change everything to a later api, and remember to change them back to my beloved kitkat afterwards). But I am very confident it will work, since (I think we agreed) it doesn’t actually recompile the binaries so there should be no real change.

Still, finding some way of building the NDK (at least the clang and utility binaries) from source is something I really hope we figure out.

@alaskalinuxuser update: today I had the energy to change all remaining build tools (aapt, apksigner and zipalign) I use for the app build. The resulting apk installed and ran without any problems, just like before. So to summarize: for me personally your rebuilt sdk zips are sufficient and will work for my project. And hopefully for other apps. Still remains to solve the legal issue, and hopefully rebuild the NDKs from source if possible. But at least, hopefully, make sure there’s older versions of the NDK with kitkat support.

1 Like

Gradle itself is free but is built using a proprietary plugin that enables integration with their services. Debian is working on packaging a new version of Gradle, and they have a patch to drop the plugin.

This is good news!

But the EULA question still remains.

Should I continue to build the Android SDK, and complete the less than 12.0 versions? Or should I wait until we receive some resolution on the EULA?

Should I continue to build the prepackaged NDK for all lower versions (less than 27) or should I wait?

You mentioned Replicant had a version 14 SDK that was completely free of a EULA, perhaps we should look at their build process. And perhaps the end product as well. I don’t know how they could build it without getting the same issue we had.

My point was that d8 itself is built using gradle. And builds using gradle will typically pull in further dependencies. So even if d8 looks cleanly licensed in the source code, the resulting program might still contain software with other licenses and, possibly, the EULA from the SDK could apply somehow. But probably not.

In contrast: all the other build tools (like aapt and apksigner) can be built like proper unix programs, without pulling in external dependencies.

(rant mode engaged: IMO gradle is part of a big issue we’re seeing right now with build systems just pulling down and bundling dependencies. It runs completely counter to the unix tradition and has repeatedly a prime attack target for malicious actors.)

It was more about it being free of the prebuilts. Unfortunately, as I mentioned somewhere above, I think that was just a mistake on their part. Their PKGBUILD attempts to use sed to remove all prebuilt repos, but it’s not applied to the correct file. Still, you can have a look: SDK - Replicant - notice that $srcdir is undefined, so sed is applied to //.repo/manifests/default.xml which fails for obvious reasons, but the pkgbuild continues blindly even though sed fails. When I fixed this, so sed modified the correct file, the build failed for me.

As for what you should do… I don’t think I’m the one to decide. Do what you want! :wink:

Personally, I only need a reasonably modern SDK (as the only java code I use is a standard SDL2 template that runs a native binary containing the real code). If you do make some older NDK (r25) I will definitely test it against my simple/proof-of-concept app. But I think we can already be pretty sure it will work, because of the prebuilt binaries. I actually “built” NDK r25 a while back and can confirm it does work.

Again, I can only speak about me. I can also say that my main interests right now are: figuring out the legal questions and try building the NDK a bit more from source. It might be something you would like to try? I just tried moving the “clang” prebuilt directory and checkbuild.py complains about:

ndk.builds.ModuleValidateError: clang: notice file /buildmnt/25/prebuilts/clang/host/darwin-x86/clang-r450784d/NOTICE does not exist

So… that’s a bit silly. But it might still be possible to just explicitly delete the binaries…?

There seems to be a list of dependencies at prebuilts/r8/LICENSE, and they all seem to be under free licenses.

I had assumed the d8 being included in the build-tools zip was built from source. We just need to be really careful when verifying that the EULA doesn’t affect anything in that zip. If we can be sure d8 is not affected, and if that “renderscript” thing also isn’t, then the EULA can probably be worked around. Perhaps by packaging our own build-tools zip.

d8+r8 also has its own git repo (r8 - Git at Google), and looking through the files I see several build.gradle.kts with further dependencies. I’m too tired to dig into this properly. It also seems it uses a thing called depot_tools (taken from chromium) for downloading further dependencies. But I also find a LIBRARY-LICENSE file that seems to list all dependencies and their licenses and so far they seem pretty cleanly licensed. Two things of note, though:

[...]
- artifact: com.android.tools:sdk-common
  name: com.android.tools.sdk-common
  copyrightHolder: The Android Open Source Project
  license: The Apache License, Version 2.0
  licenseUrl: http://www.apache.org/licenses/LICENSE-2.0.txt
  url: http://tools.android.com/
[...]
- artifact: com.android.tools:sdklib
  name: com.android.tools.sdklib
  copyrightHolder: The Android Open Source Project
  license: The Apache License, Version 2.0
  licenseUrl: http://www.apache.org/licenses/LICENSE-2.0.txt
  url: http://tools.android.com/
[...]

Although it says Apache, could this still be the reason the EULA is being included? To recap: the EULA text being copied into the build-tools zip does come from prebuilts/sdk

1 Like

hello.
Is there a summary of where things stand with regards to open source alternatives to Studio? I have found Studio difficult to use, but i understand it is the only way to build signed apps.

I have been trying to cross compile my own open source SDL2 apps for fdroid/Android, so I can use my own prefered IDE to write the app, but I get beaten by the studio tools, even in command line mode. See:

https://forum.f-droid.org/t/sconnex-an-open-source-sdl-game-for-f-droid

Thanks.

Most folks who want an IDE and do not want AStudio use Eclips. Others just use gradle on the command line.

Though I did build an Apache2.0 version of AStudio 3, but it typically downloads proprietary parts/pieces when you ask it to do things.

Is that what you meant?

1 Like

If you are determined and don’t mind some extra manual work, you don’t really need to use android studio. You don’t even need to use gradle.

In case it helps: I am also using SDL2 and instead of using gradle and similar, I’m building the app directly using the underlying tools (aapt, apksigner and all those). I’ve automated this process with a set of makefiles. My project and all libraries it depends on are using autotools, and the android version is just a port. It’s not released yet, but I can give some guidance if you want.

To get an understanding of how to put together apps from sources, I recommmend this blog-post: Building an Android App from the Command Line Perhaps you want to use aapt2 instead of aapt, also the latest version of that post uses d8 instead of dx. d8 is not available directly in debian right now, but it’s part of the build-tools package we’re working on here. You’ll also need the android.jar file, which we’re also dealing with here.

For NDK (like C code and the SDL2 library), that post also gives some basic overview. If you are going to use different C libraries, it’s the normal cross compilation. Have a look at this: Use the NDK with other build systems  |  Android NDK  |  Android Developers I’ve only done the autoconf stuff, because that is all my software and libraries need. I HIGHLY recommend passing varaibles as arguments to configure instead of setting them as environment variables (so ./configure CC="$TOOLCHAIN/bin/clang --target=$TARGET$API" <etc...>). Use this with the --prefix argument to install to your own little “sysroot” any additional libraries you will need.

Finally, if you decide to follow this, you might run into an issue with SDL2 not building properly for android when using autoconf. It seems the developer didn’t pay much attention and forgot a detail there. It’s easy to fix, but I’ll share it if you need it, because I’ve probably written a lot already.

1 Like

I’d add that Emacs’s port to Android is likewise configured with autotools, and compiled with GNU make, OpenJDK, and the SDK build-tools, and you may be interested in these files:

https://git.savannah.gnu.org/cgit/emacs.git/tree/configure.ac#n824
https://git.savannah.gnu.org/cgit/emacs.git/tree/m4/ndk-build.m4

It also ships a bespoke implementation of Android’s ndk-build build system for dependencies:

https://git.savannah.gnu.org/cgit/emacs.git/tree/cross/ndk-build

2 Likes

By the way, I do appreciate that you looked into this. I have just managed to build r8/d8 from source* and can also confirm that it doesn’t seem to pull down any nasty licenses or EULAs.

So at this point, I think we can safely assume that the following programs inside build-tools are free from EULA contamination:

  • The traditional tools already packaged by debian: aapt, aapt2, aidl, apksigner (and related lib/apksigner.jar), dexdump, split-select, zipalign
  • The d8/r8 stuff: d8 and lib/d8.jar
  • Most likely the clang stuff: lld

This leaves the following to check:

*) But man, building it was a true mess. It pulled down 29GB(!!!) of dependencies and the build instructions were wrong: there is no “d8” target for gradle any more (it seems all is directed through the r8.jar these days…).

Oh, that is lovely. Just lovely! And as a lisper/schemer wannabe, I find it even more lovely! I see it somehow uses autoconf in the actual app build process.

Personally I have left all kinds of cross compiling and packaging outside of my autotools process (things like android apk, windows installer, etc… are created as a separate step). My set of makefiles works as a layer on top, and simply downloads+./configure+make+install on its own, and then copies files around and runs the various “alien” tools that creates various packages/installers.


edit: okay, what did I just stumble across? https://android.googlesource.com/platform/prebuilts/clang/host/linux-x86/+/20a8d53/3.8/README.md quote: " Prebuilts are then generated using these projects and placed into the proper prebuilts/clang/host subprojects." and " Prebuilts for llvm-rs-cc and bcc_compat also need to be generated for prebuilts/sdk. This is done by running frameworks/rs/update_rs_prebuilts.sh on both Mac and Linux."… okay, seems google has some kind of separate workflow for creating those prebuilts…

1 Like

wow, thanks slinger&alu, tons for me to look at there! I have been coding for many decades using C89/99 (TurboC, MSVC, gcc…), often just with batch files for building, or various make/project systems. But always at the base was something like cc that could take hello.c, compile and link obj to a binary of just a few k without too much fuss.

As john carmack noted in your links, Android Studio seems to generate 50 mbytes of content in thousands of files for that. Maybe Android needs a “64k app/game programming competition” like the ones you see for the PC…

It would be nice to have something that sweeps up all the LTS versions of the build tools targeting a recent version of android, that would last for a couple of years before needing updating. That would be a service to the programming community. Something that doesnt use gradle might be a bit easier to work with too.

In puppy linux they do this with the gcc compiler tools, it is just one devx.sfs file that you attach with a single command, with all the baseline libraries there. But I did have to use “apt install default-jdk” to get java, so something might be missing from puppy, i might have to setup debian/ubuntu.

The linux version of the sdkmanager command line tool seems to be not very informative when there is an problem setting up, I may try the windows version under wine.

Reading around the forums it seems others are having similar issues with Studio.
Anyway I will read through that material slinger, thanks for taking the time to answer.

1 Like

I hope it’s OK that I bump this thread, so it doesn’t get closed. This still feels very important and I care about it, but have been too busy to work more on this.

So I don’t have much new to add. I think I stumbled upon some documentation regarding how the prebuilts are created. At least the implication that they exist and that google have some internal build server they copy the files out of.

And given that they have shipped GCC and other GPL’d binaries through prebuilt git repos, I assume they have the instructions somewhere (otherwise they would violate the part of the GPL requiring both source and build instructions/process to replicate binaries).

2 Likes

So my question now is: I have around 70+GB of the files which @alaskalinuxuser generated. Should I keep those, or remove them from my server? Are we looking at any further in this or stuck in infinite loop?

1 Like

Oh, I certainly hope we’re not stuck! I think we’re just in a time of the year when people are very busy. Or at least I am.

Honestly, I feel like I’m the cause for this stop. If I had just not mentioned the EULA, this wouldn’t even have been considered a problem. I do think @hans suggested that as an initial step we just patch the build (process?) so the EULA doesn’t get copied. This would be the easiest way forward and shouldn’t be too complicated, as long as we can identify what script is doing that. Or even simpler: just programmatically remove the EULA from the file we know it gets copied from: prebuilts/sdk/NOTICE

After all it’s only included in this one small zip (the one with basic tools), and I couldn’t find any indication that any of those tools had special restrictive licenses or similar in their sources. I could be wrong, of course.

So what does everyone else say? Should we just do that so things can get back in action, and worry about the rest later? Or should we make more effort into figuring out what part of the build process is doing this? Perhaps the responsible script has some comment that might shed light on it? Or we could try to find the upstream source of the “prebuilt sdk”, see why there’s an EULA there in the first place, if it’s an oversight? Or start reaching out to seek legal advice?

Also, @vdbhb59, in case you are just worried about publishing the files or not: in my opinion I don’t think you should feel worried at all. I don’t see any reason to be afraid of legal issues here. A lot of people (and companies) are already sharing the official zips, and I’ve not heard of google taking any action against them. And even then, in the worst case, they would likely just ask you to take them down.

1 Like

Alright, everyone, I think I cracked the nut. After the post by @vdbhb59 I felt motivated to start digging, and I’ve found the culprit (what causes the EULA to be included). I assume double posting is not appreciated here? But it will get so long to append all this to the previous post… :worried: I just want to document this while it’s clear in my head:

First of all, as we know, the EULA gets copied from prebuilts/sdk/NOTICE, but why is it included? This was a loooooong detective work trying to dig through the android build process (not recommended if you want to keep you sanity):

In short, soong generates ninja rules, which in turn runs a program out/host/linux-x86/bin/build_license_metadata which generates meta_lic files for different modules (how this is done I’ve not looked at yet). These files are then run through out/host/linux-x86/bin/textnotice which collects all license files and combines them. These meta_lic files in turn include each other in different ways… so: out/soong/.intermediates/development/build/build-tools/linux_glibc_common/meta_lic is used to create the final NOTICE file with the EULA in the build-tools zip.

But it includes other files, such as out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/meta_lic which contains the following:

module_name:  "build-tools-lld-linux"
module_types:  "genrule"
module_classes:  "UNKNOWN"
projects:  "prebuilts/sdk"
license_kinds:  "SPDX-license-identifier-Apache-2.0"
license_kinds:  "SPDX-license-identifier-BSD"
license_kinds:  "SPDX-license-identifier-CC0-1.0"
license_kinds:  "SPDX-license-identifier-CPL-1.0"
license_kinds:  "SPDX-license-identifier-MIT"
license_kinds:  "SPDX-license-identifier-NCSA"
license_kinds:  "SPDX-license-identifier-OFL"
license_kinds:  "SPDX-license-identifier-Unicode-DFS"
license_kinds:  "SPDX-license-identifier-W3C"
license_kinds:  "legacy_unencumbered"
license_conditions:  "notice"
license_conditions:  "reciprocal"
license_conditions:  "unencumbered"
license_texts:  "prebuilts/sdk/NOTICE"
is_container:  false
sources:  "out/host/linux-x86/bin/soong_zip"
deps:  {
  file:  "out/soong/.intermediates/build/soong/zip/cmd/soong_zip/linux_glibc_x86_64/meta_lic"
  annotations:  "toolchain"
}

Notice the “prebuilts/sdk/NOTICE”? :wink:

It also includes the following files with similar contents (and many, many more, but only these 4 refers to the EULA file):

  • out/soong/.intermediates/prebuilts/sdk/renderscript/renderscript_sdk_prebuilts/meta_lic
  • out/soong/.intermediates/prebuilts/sdk/tools/prebuilt_libLLVM_android/linux_glibc_x86_64_shared/meta_lic
  • out/soong/.intermediates/prebuilts/sdk/tools/prebuilt_libclang_android/linux_glibc_x86_64_shared/meta_lic

So, do we think the EULA applies to this lld, renderscript, libLLVM_android and libclang_android? Probably not. From what I can tell, the lld binary being shipped in build-tools is the usual from the LLVM project which is APACHE. And the libLLVM libclang should be fine I think? They might be android-specific libraries, so I’m not sure yet. And renderscript I believe I checked before and the source was apache as well?

Actually, we really need to double-check this to be sure first. But I’m feeling pretty confident.

So how best do we patch out this? Assuming we can’t get upstream to adjust this. Yeah, no big chance of the latter, I think.

Well… these files are in turn generated by out/host/linux-x86/bin/build_license_metadata from the files like out/soong/.intermediates/prebuilts/sdk/tools/build-tools-lld-linux/meta_lic.rsp. This in turn is simply an automatically created long list of arguments by ninja (or maybe instructed by soong)… I haven’t found yet what build rule is creating it. But, well it contains the following arguments:

-mn build-tools-lld-linux -mt genrule -r prebuilts/sdk/tools -mc UNKNOWN -k SPDX-license-identifier-Apache-2.0 -k SPDX-license-identifier-BSD -k SPDX-license-identifier-CC0-1.0 -k SPDX-license-identifier-CPL-1.0 -k SPDX-license-identifier-MIT -k SPDX-license-identifier-NCSA -k SPDX-license-identifier-OFL -k SPDX-license-identifier-Unicode-DFS -k SPDX-license-identifier-W3C -k legacy_unencumbered -c notice -c reciprocal -c unencumbered -n prebuilts/sdk/NOTICE -d 'out/soong/.intermediates/build/soong/zip/cmd/soong_zip/linux_glibc_x86_64/meta_lic:toolchain' -s out/host/linux-x86/bin/soong_zip

So… if we can remove the -n prebuilts/sdk/NOTICE part of the arguments to these, it also removes the line containing license_texts: "prebuilts/sdk/NOTICE" in the generated file (while keeping the other license information), which finally makes the content of this file not getting copied!

And to be clear: I strong believe the resulting NOTICE file in the zip will still contain all the valid licenses (like apache). Just not the EULA from prebuilts/sdk/NOTICE.

Oh, I’m a bit tired after this detective work. :rofl:

What remains is: figure out exactly what build rules are calling out/host/linux-x86/bin/build_license_metadata (knowing google, it’s probably another layer being generated and parsed again…), and modify them according to above. Or keep it simple and just replace the contents of prebuilts/sdk/NOTICE with an empty file. :stuck_out_tongue_closed_eyes:

Also, again: it’s obviously very important that we double-check that lld, renderscript, libLLVM_android and libclang_android are in no actually under the EULA before doing this.

4 Likes

I’ve been on vacation but will be back in two weeks. If this is a good way to go, I can rebuild what I’ve done before with these edits, also posting an updated repository with the build scripts, having the changes, so people can verify/prove integrity.

What do you think?

2 Likes

Yes. I think that should be good to go. We are okay, and there should be no DMCA afaiu from reading everything here.

1 Like

Welcome back! :slight_smile:

Keep in mind, these are not yet actual edits you can apply. I have yet to dig deeper and see what underlying build rules needs patching, but it feels like a start. So no hurry to start rebuilding.

Also, I would still love it if we could look closer at the 4 parts that are pulling in the EULA: lld, renderscript, libLLVM_android and libclang_android. Just to make sure they truly don’t have some nasty legalese in upstream. There are matching files inside the build-tools zip (renderscript IIRC has a couple of files in a directory).

So far I am almost 100% sure the lld is just the standard binary from llvm/clang. renderscript I recall seeing in android sources (so apache?), but not sure. The other two lib* I can’t recall what they even are and do. Maybe someone else got some insight here? :sweat_smile:

2 Likes

renderscript can probably be put off til last, or maybe just left out entirely. It is deprecated and probably rarely used. It would also be good to check the status of lld, libLLVM_android and libclang_android. Maybe they are in a similar state? I know that the linker in use has changed over the years, so it could be that lld is only there to support old apps. Its git repo hasn’t been touched in 6 years:
https://android.googlesource.com/toolchain/lld/

How about just deleting those four from the final package for now? Then we can see if anyone complains that they are missing.

1 Like