Open Source Android Development Tools

42

Personally I’m just happy to do whatever small things I can to help. At this point making these tools free is a matter of principle. That being said, after loosing the files hosted by Beuc, ensuring long term hosting of the resulting packages feels important. Do you think f-droid would be able to host/mirror downloads of them?

Also, even though the NDK is standalone, the sources are still downloaded from many separate git repos (using the “repo” tool), and this includes a lot of prebuilt binaries. This includes the clang binaries (and a version of python, and a bunch more). I just hope it’s possible to filter out the prebuilts and get it all to compile from source.

1 Like
43

Slinger, and the group,

Just to make sure it is clear, are you saying no prebuilt tool can be used to build the SDK/NDK/Android Studio? Or just no prebuilt tool with an EULA from Google?

I ask in earnest, again, to be really, really clear about the goal of the project.

I can understand we don’t want a prebuilt tool that has a EULA from Google, even if it is an open source tool.

But some of the prebuilts are literally just the actual prebuilt tool in a repository, such as Go, and Rust, both of which I checked, are still under their open source license with no EULA from Google in their repository.

The reason I ask, and again, typing doesn’t convey emotion very well, so I am just asking analytically, not in defense or anything, is where do we draw the line? I used Ubuntu to build it, which uses many prebuilt packages, such as when I run the command - required for building all of the above:

sudo apt-get install git-core gnupg flex bison build-essential zip curl zlib1g-dev libc6-dev-i386 x11proto-core-dev libx11-dev lib32z1-dev libgl1-mesa-dev libxml2-utils xsltproc unzip fontconfig

I am using a lot of prebuilt tools.

At present, to build the Android-SDK from source is a 100GB download already, with 62 prebuilt repositories, though it is really about a third, when considering it is the same tool, prebuilt for Linux, Windows, and Mac.

So, we need to clearly define where the line is drawn on prebuilts, and identify which (or all) of the prebuilts that need to built from source, and then we can start determining how we do that. Keep in mind, this will dramatically slow the project down, as each prebuilt will need the appropriate version for each version of the SDK/NDK/Android Studio we build. Achievable, but will take some work.

Do you have the skills to look through this, or should we find other helpers? :smiley:

1 Like
44

Thanks Hans!

I don’t mind doing my portion of the work for free. For the most part, I’m just following the scripts of what to do to build from source, the only issue I’ve had so far is with Android-SDK 14.0.0_r18, to which I just rolled back and built _r3.

Once we identify if prebuilts or which prebuilts are acceptable or not, we can continue, but as of now I will stop building these tools until we make that determination. I can’t see downloading, building, and then determining we can’t use it.

Do you have some thoughts on the prebuilts? You do a lot more on F-Droid and open source work than me.

Is it that you used an open source screwdriver to screw that in that open source screw, but you didn’t build the screwdriver from scratch so it doesn’t count.

Or is it that you used a screw driver with a EULA to screw in that open source screwdriver, so it doesn’t count.

I think it is best to clearly define the goal of the project before we start. Because within a month I can build all of these Android development tools from source and they end up with an Apache2.0 license file when I’m done. But if no one will use them because they had a prebuilt, then I’d rather we didn’t build them and work out how to manage the prebuilt tools.

And just to reiterate, I know I keep saying this, but I want to be very clear, I am just asking in earnest, not defending, attacking, or anything.

1 Like
45

You express yourself very clearly and kind! I just meant NDK: When I do a “repo sync” of the NDK “sources”, it pulls down toolchain/prebuilts/ndk/r25 - Git at Google as one of the “prebuilts”, which contains a fully compiled NDK. I have looked and used the “building” scripts for the NDK and it will just copy these files out and repackage them. I hope it possible to rebuild them instead by just removing the “prebuilts” repos from the manifest.

As for your question, in general, I don’t think I’m the one to decide on this. However, the following is my understanding:

  1. Using anything provided directly by the distribution (ubuntu/debian in this case) is 100% fine, no doubt at all!
  2. Ideally, the prebuilt binaries provided upstream from google would be avoided. This is part of why projects like debian don’t package the SDK and similar - they want everything built from source and trusted.
  3. But I think it will be hard/impossible to avoid all prebuilt binaries that are part of the android build process. My understanding is that all modern versions of android SDK (and more) have some nasty circular dependencies, so it might be unavoidable.
  4. To me, PKGBUILD in the replicant wiki (SDK - Replicant) shows that by just removing the lines containing “prebuilt” from “.repo/manifests/default.xml” before doing the “repo sync” the SDK will still automatically be built (by “lunch”). So outside taking more compiling time (and actually MUCH LESS to download) I think these prebuilts can be avoided without much more work. Again, just removing the lines like this:
sed -i '/prebuilt/d' "$builddir"/.repo/manifests/default.xml

My own, personal, concern simply boils down to two things:

  1. I just hope we can avoid the EULA. So, if nothing more, the “prebuilts/sdk” and “prebuilts/tools/common/api-versions” should be avoided as they legally contaminate the result. But hopefully all other things in the “prebuilts/” directory can be avoided anyway. :slightly_smiling_face:
  2. I hope the NDK can similarly be rebuilt by just removing the all lines containing “prebuilt” from the manifest.

edit: I forgot to mention, when I write “prebuilts”, I simply mean the contents of the directory called “prebuilts” at the top of the SDK/NDK source tree. The one which is populated from various git repositories containin the name “prebuilt” which are in “.repo/manifests/default.xml”. I am willing to overlook any other prebuilt binaries that might be hidden elsewhere, and anything directly from a distro is 100% fine.

1 Like
46

So, when we follow that link, the folder itself, has the notice file, claiming Apache2.0 license. And it doesn’t just repackage the same NDK prebuilt, or the build would only take a few seconds. :smiley:

To build the NDK from scratch, following these instructions:

https://android.googlesource.com/platform/ndk.git/+/HEAD/docs/Building.md

I decided to walk through all the prebuilt objects by starting with NDK-r29. Had I known you would reference r25, I would have picked that, I just picked r29, because it was modern. We can pick another if you like. In the manifest file for r29 of the NDK, there are 17 prebuilt repositories, so I went to each repository, and went through all the files.

In each one, there was no EULA, and along with the binar(y/ies), there was a license file provided, they were as follows:

Apache2.0
  <project path="prebuilts/ninja/linux-x86" name="platform/prebuilts/ninja/linux-x86" groups="presubmit" revision="8a10824f74fe0e22af9bf314a837f5b70e2bb67f" upstream="ndk-r29-release" />
  <project path="prebuilts/ninja/darwin-x86" name="platform/prebuilts/ninja/darwin-x86" revision="f321e197944c19d273cec788b9a3e8ca94331248" upstream="ndk-r29-release" />
  <project path="prebuilts/ninja/windows-x86" name="platform/prebuilts/ninja/windows-x86" revision="4d526002f0158a5d103bb6e54d43943c0e1c2778" upstream="ndk-r29-release" />
  <project path="prebuilts/clang/host/darwin-x86" name="platform/prebuilts/clang/host/darwin-x86" revision="7d20032ca1970a8390be09b73f6a894905ee8542" upstream="ndk-r29-release" />
  <project path="prebuilts/clang/host/linux-x86" name="platform/prebuilts/clang/host/linux-x86" groups="presubmit" revision="a92c8229e4d69f3552cdcd956e3f2b0380a41931" upstream="ndk-r29-release" />
  <project path="prebuilts/clang/host/windows-x86" name="platform/prebuilts/clang/host/windows-x86" revision="c8a25b3935d01457157f8094bff6b1ad5e6f7ca5" upstream="ndk-r29-release" />
  <project path="prebuilts/ndk" name="platform/prebuilts/ndk" groups="presubmit" revision="c0815fea3a8081be6215440de330c63246e6551f" upstream="ndk-r29-release" />
  <project path="prebuilts/simpleperf" name="platform/prebuilts/simpleperf" groups="presubmit" revision="a63e5b546388f4b947d1b310ab3d9bada63bb242" upstream="ndk-r29-release" />

BSD-3
  <project path="prebuilts/cmake/darwin-x86" name="platform/prebuilts/cmake/darwin-x86" revision="7aea7e9880110799088cd1de509886871078306f" upstream="ndk-r29-release" />
  <project path="prebuilts/cmake/linux-x86" name="platform/prebuilts/cmake/linux-x86" groups="presubmit" revision="e57c88d59d7d8408a32b16425158fd2aa64e2b3e" upstream="ndk-r29-release" />
  <project path="prebuilts/cmake/windows-x86" name="platform/prebuilts/cmake/windows-x86" revision="f6283abb0a655968016437be07d04370fd815e6c" upstream="ndk-r29-release" />

GPL 2
  <project path="prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.17-4.8" name="platform/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.17-4.8" groups="presubmit" revision="49917e030f76c01f35d5a46b66a2b85b2976647c" upstream="ndk-r29-release" />
  <project path="prebuilts/gcc/linux-x86/host/x86_64-w64-mingw32-4.8" name="platform/prebuilts/gcc/linux-x86/host/x86_64-w64-mingw32-4.8" groups="presubmit" revision="d6b93efcc5c2d123cb44eee4f74d6fd35d5efd9f" upstream="ndk-r29-release" />

GPL 3
<project path="prebuilts/build-tools" name="platform/prebuilts/build-tools" revision="434d914996ecc2abcda46a1841c82946241b1c35" upstream="ndk-r29-release" />

GPL Compatible
  <project path="prebuilts/python/darwin-x86" name="platform/prebuilts/python/darwin-x86" groups="darwin" revision="3400487f6a77cc05ee5d4799575372713ce63232" upstream="ndk-r29-release" />
  <project path="prebuilts/python/linux-x86" name="platform/prebuilts/python/linux-x86" groups="linux,presubmit" revision="fce2346610379fdcce9dc7423c0e9a04e1a43cbf" upstream="ndk-r29-release" />
  <project path="prebuilts/python/windows-x86" name="platform/prebuilts/python/windows-x86" revision="6d619afe3f821388a57b7b2b473bee85debfdd7c" upstream="ndk-r29-release" />

So is it not acceptable to build revision 29 of the NDK as is, with these prebuilds, all including their original license, based on this?

I don’t think this is a slight of hand by Google to trick you into using a contaminated binary, but rather that the build process gets complicated, so you would need this exact revision of say python3.11 to guarantee the build is successful. This gets hard when using one server to build all 29 NDK versions, because each one needs a different revision of python 2,3,3.11, etc. to build.

Also, Replicant is a great project, but unfortunately, they are still stuck on Android 6.0, where the build system is simpler. If I remove these downloaded binaries, I have to have that version/revision number of that binary installed on my server to build with, or I have to replace it with a binary that I compile from source myself.

The only issue I see with compiling them myself is that to compile NDK-r29, I have to compile 17 other projects as well, and not the current version of that project, but the specific version and revision number of the project that corresponds with this version needed for the NDK.

As you mentioned, in recent versions, it also becomes circular. To build the modern NDK, you need the modern Android build tools, which comes from the Android-SDK, which will require you to download the prebuilt NDK to build.

But, if the end product was built using repositories that had no EULA, and just had the precompiled binary, with the original license, is that okay to use?

I apologize if I am missing the point of your question. I literally am the dumbest person I know, so please help me understand better if I am not understanding correctly. I will not take offense. :slight_smile:

2 Likes
47

You did a very thorough job. Sorry if I caused a lot of extra work. But actually I agree. I have also looked through the NDK repos, trying to find the EULA and (as I said before) have also not found anything like that. r25 was just the version I had at hand, it doesn’t matter. Still, good job.

I have no problem with any of those licenses. And you really don’t need to worry so much about my opinion here, I just personally believe that if the NDK can be built from source without too much trouble, then it should. But that’s just my opinion, and I’m not anyone important.

But I suspect it’s quite possible that just removing some of the “prebuilts” subdirectories is enough to make checkbuild.py rebuild them automatically. But as you noticed, some are probably needed to get things working. Like removing the “prebuilts/ndk” and maybe the “clang/host/*” ones from the manifest might mean they get recompiled from source (and for me I think they would compile faster than downloading anyway). But I haven’t tried that.

However, I am very confident that the clang binaries (and most/all other binaries) are not being recompiled and that checkbuild.py pretty much just repackage the existing binaries (if it detects they exist). It is just very slow and applies heavy compression on the generated archives. It was a while since I last looked at that script (it was when I wrote the other topic), but I remember even adding debug printouts in the checkbuild.py script to verify this.

In fact, I can even build some proof-of-concept C+SDL2 based apps using only than that single linked prebuilts repo (and no other NDK-related repos) plus the SDK.

edit: Just wanted to mention, the PKGBUILD in the replicant wiki (which builds the SDK without downloading the “prebuilts”) was for SDK 14, which is not very old at all. Also I just noticed the NDK used to have a “build/tools/rebuild-all-prebuilt.sh” script. Not any more, but it was interesting to see.

1 Like
48

Everyone is somebody important.

2 Likes
49

Thanks. :slightly_smiling_face: My point is still that you don’t need to listen to me for permission, or what to do/don’t. I think it’s amazing you are doing this, reviving the Android Rebuilds. I tried building the SDK a few months back but got stuck, I think because I only had 16GiB of ram.

I do think avoiding the EULA for the SDK can be important. Redistributing those prebuilt files will probably have some legal implication. The NDK can wait until the SDK looks good. For the SDK, I know we can’t avoid some prebuilt binaries, but again, it seems the important ones (inside the “prebuilts” directory, especially the two containing the EULA) can be removed and the automated build process will just continue to work. Just take a bit more time.

I assume you are using the scripts originally written by Beuc, so I would love to see if it is enough to just add the following sed line to download.sh:

VERSION=......
repo init -u https://android.googlesource.com/platform/manifest -b android-$VERSION --depth=1
sed -i '/prebuilt/d' .repo/manifests/default.xml
repo sync --current-branch -j4

(I did this kind of checkout a while back, and can confirm it didn’t download the EULA-encumbered parts.)

Also: when/if both the SDK and NDK are ready, I would be happy to do some testing. I’m using a custom build system with makefiles, and mostly tools directly packaged by debian, so no need for gradle and similar. Pretty much just the android.jar and d8 from SDK, and the NDK.

1 Like
50

Okay, I have the builds that @alaskalinuxuser builds. I am confused as to how to share them. I do not wanna use Nextcloud or Owncloud to share as they make a mess of my server and uses hell lot of resources.

@Slinger suggest me a good and simple maybe PHP alternative to provide the files and secure. I have 4 hours free this weekend to upload and share the same.
As of now I have 9.2GB worth of files that he has built.

Ohh: & it will be available to all for ever (at least till I do not go broke or my server provider does not break up).

51

Slinger,

I will give this a try later with the Android-SDK 14.0.0_r3 just built, and see if that works. If it does, that’s great! Seems like some things might be missing, though, so if it doesn’t, we will proceed with the current build process for now.

As for listening, you bring up good points, similar to my brother’s objections, and we want people to use this stuff, which means it has to pass the bar, so to speak. Granted, we can’t make everyone happy all the time, but we can work on it. :smiley:

1 Like
52

Since you are only serving static files, you don’t need PHP. If you already got a web server, like nginx or apache, you can just use them directly. I think autoindex for nginx might be a good starting point: Module ngx_http_autoindex_module and Best Practice of Nginx File Indexer ｜ fernvenue's Blog

Or maybe share them over ftp using something like ftpd-ssl?

1 Like
53

Yeah I just recalled. I do have another static site where I share my auto builds.
Will create a page similar to that over the weekend.
@alaskalinuxuser let me know when you have any other file to be added. I will share those files by this weekend and revert.
@Licaon_Kter @hans can we create a news topic for it once we are ready to share those on fd/news section?

54

will see when we see how it works in practice too :person_shrugging:

1 Like
55

Slinger, I just tried this with the 14.0.0_r3 Android-SDK build, and it failed immediately once starting the build process for no such file or directory for the prebuilts.

@vdbhb59 @Licaon_Kter We really need some testers to prove these builds work properly. Perhaps that would be a newsworthy item? Asking for testers?

Next up I will start building the NDK from source. When I finish those, I hope to build the remaining older SDK versions, and then reviewing the Android Studio builds for versions above 3.0, since the newer versions of AS will not build due to proprietary vendor files.

1 Like
56

I have uploaded these to my server now. Waiting for @alaskalinuxuser and @Slinger to let me know.

2 Likes
57

Sadly, I can confirm. I really appreciate that you tried this and I hope I didn’t cause you too much work. I had high hopes and also tried building manually with the same bad news.

I tried with both SDK 11 and 14. Removing all prebuilts breaks the entire build, yes. I also tried selectively removing just the two containing the EULA (“prebuilts/sdk” and “prebuilts/tools”) and the build did work for a bit, but eventually failed by missing files. Also I tried running a script that was meant to rebuild some prebuilts, but that also failed because it needed some things from “prebuilts/tools”… for now I think we will have to accept this situation.*

I’m not a lawyer, I don’t know what legal implications there might be in potentially redistributing these files that seem be under the EULA, while claiming it’s not under the EULA. But if no one else objects, I’m happy to pretend I never saw this, and let everyone move on with this awesome work. :fast_forward:

Now, good luck with the NDK! As I said, I am certain checkbuild.py is just zipping up prebuilts (using very strong and slow compression). It would be great if I’m wrong, I know that script also contains code to rebuild the binaries (but in my testing I found it’s not done by default, and I’ve literally managed to use the clang prebuilts as-is to make working apps). Hopefully, rebuilding the NDK from sources might be more achievable, but at least there’s no EULAs involved there. Ironically the NDK sources used to have a script called " rebuild-all-prebuilt.sh"…

When the SDKs and NDKs are ready, I’ll be happy to try them out on a couple of simple apps. I am currently preparing a range of different devices with different architectures and android versions to be guinea pigs. I hope you plan to build the NDK version 25 (or earlier). It is the last with Kitkat support, which is what my beloved old phone runs. It’s now my dedicated “podcast player”. :laughing:

*Footnote: for those it may concern, I think there’s a bug in that replicant wiki PKGBUILD which causes all prebuilts to be downloaded anyway: it attempts to modify the file at “$builddir”/.repo/manifests/default.xml, where builddir=“$srcdir/” and srcdir was never defined. This means it tries (and fails) to modify //.repo/manifests/default.xml. Since the PKGBUILD doesn’t check for errors it just continues the build process anyway. This is why it succeeded for those people, and no one noticed it. And it’s why they still saw binaries inside the prebuilts directory.

58

NDK-r29 is complete, and NDK-r28 is under construction. NDK-r27 is downloading the portions of the repository. I’m working from 29 backwards, since I downloaded 29 to check all the prebuilts, which had no EULA, as we discussed before.

As for the SDK, we built it and the end result is Apache2.0, per the license that comes with it when you build it. So, it will be up to the end user if that is acceptable to them or not.

1 Like
59

On the plus side, the NDK builds very quickly…

It takes me longer to download the source than to build it with my slow internet. :sweat_smile:

IMG_20250307_081733
IMG_20250307_0817331920×2560 640 KB

1 Like
60

That sounds perfect, but I’m not sure I understand/agree. I’ve been l looking at the zips built for version 14 (downloaded from @vdbhb59’s server) and I see many different licenses, and google EULA in some of them…

Please let me know what you think about this. Maybe I’m just misunderstanding. Looking at the file sdk-repo-linux-build-tools-eng.14.0.0_r3.zip, which contains the android-14 directory (with things like d8 and android.jar), it has a “NOTICE.txt” with many licenses and the google EULA.

At the same time, I don’t see any EULA like this in android-sdk_eng.14.0.0_r3_linux-x86.zip. It doesn’t even have a top-level license file. It has many different licenses in subdirectories, but luckily no EULA. Interestingly, both this zip and the previous mentioned do have the same android.jar, but in this archive there is no clear license/eula associated with it…

Yup, it builds very quickly for me as well. As said, I’m pretty sure it’s just copying prebuilt binaries around, stripping and zipping them up. For instance, when it says “Building modules: clang” it’s not actually compiling it. At least not normally. I need to dig deeper into this because it was several months since I last figured it out and can’t remember the details, but you can look inside the source ndk/ndk directory to see the the python code for all this. The file ndk/ndk/checkbuild.py (not to be confused with ndk/checkbuild.py which is the one we actually run) it defines most of these “modules”. Just search for name = "clang" in the code to find where the “clang” module starts. There’s a bunch of shutil.copytree(). But again, I’d love to be proven wrong here.

edit: I just remembered, if I remember correctly I think shader-tools is the only module that results in any real compilation being done when building the NDK on my system.

1 Like
61

@alaskalinuxuser can help. I am just providing the host, and honestly I do not understand these things a lot. But I trust he has checked everything.
:sunglasses:

1 Like