Interestedin Linux Alpine i am familiar with the penguin..
Oof. Reminds me of what I stumbled upon recently: GNU GUIX: The Full-Source Bootstrap (mirror).
@Licaon_Kter Thanks a lot for reopening this thread! I saw it close and thought we’d have to make a new one.
Also, to clarify for anyone interested in this topic: I still care a lot about this project and think it would be very important to see a finished solution, ideally with the packages hosted officially by f-droid. And if needed, I would be prepared to take on a bigger role to get the ball rolling.
I just can’t promise to have the time to support this long-term, at least not on my own. But I also don’t want to step on anyone’s toes and “hijack” this and take over the project. @alaskalinuxuser, how are things for you? But please, I don’t want you to feel any pressure regarding this project.
Haha, great minds think alike! I actually posted the same link earlier in this thread.
1 Like
i spam, i spam, i spam spam spam spam spam~
Well, to be honest, I got a little confused over what we were doing. E.g., the end goal, and the steps in between.
As @hans said:
The DFSG level means rebuilding
everything from source, which would be a lot more work. Google uses lots of
“prebuilt” binaries of tools in their builds.
So as I built these SDKs/NDKs, it seemed like the group as a whole wants to have everything built from source, no prebuilts at all, which is really cool, but also really, really time consuming and in many cases beyond my limited skills.
In the meantime, I have switch over to Ubuntu Touch and am using all of my free time to help develop with my meager skills there. Not having a clear direction, I repurposed my build server for other projects.
1 Like
Oh, no worries! It’s relevant and you’re just trying to bring attention to it.
Sadly, that so called “SDK” are just some of the build tools, not the actual SDK. There are some similar projects out there, like GitHub - nmeum/android-tools: Unoffical CMake-based build system for android command line utilities (but the one you link claims to build some binaries use musl?)
The NDK does seem to have everything! But the repo does not contain much actual information on the build process. It all seems to be defined within big github workflows files (.github/workflows/make_ndk_musl.yml)? Unless I’m missing something? Oh… the commit history makes me a bit concerned… (“umm ok ig”, “funny rename”, “oops”,…). Anyway, it might be interesting to look into to see if it really solves full NDK rebuilds.
Regarding both repos, I don’t understand why they’ve put so much work into formatting the readme and adding emojis, but not said anything about the actual build process.
Oh, hi @alaskalinuxuser! You posted while I was typing the reply to @namark. Nice to hear from you again!
I can’t speak for anyone else, but my standpoint is this: my goal is to have some kind of “officially unofficial” sdk+ndk (hopefully blessed by being hosted by f-droid). If that means using prebuilt binaries, I can live with that, as long as there are no hidden EULA issues. And I think my EULA concerns are properly solved.
So (again, my perspective), all that’s left is to just build the packages and host them? For my use case, I will then just point my custom build system at the new download URL. BUT: I’m actually quite a newbie when it comes to android development, so I’m not sure how people in general will be able to use these packages. Can these packages just be dropped into place somewhere?
From what I’ve seen/experienced: building using gradle will results in a bunch of downloads (from google and eula enforcements), so maybe we also need to find some solution to that…
Edit: to be clear about the DFSG, these are requirements for official inclusion in debian. I don’t think there’s any mortal soul capable to doing a clean from-source build of this mess. And we don’t need to. The goal is not to make official debian packages, it is simply to offer an alternative to downloading directly from google, without agreeing to licenses/eulas.
yeah everything is in github actions atm, since that’s what the author is comfortable with I suppose. If it works though it’s worth looking through and converting to something more sensible. In general there is no way there is going to be one forever good instruction on how to build this mess I think, it would need a maintainer who would constantly maintaining the build scripts, and homu homu there seems to be up to the challange.
Building the Android SDK packages from source even using the prebuilts would
still be quite valuable. If those where available, I think app devs would
switch to it. It might be enough to label the binaries as FOSS. Did you manage
to find the license of any prebuilt binaries?
alaskalinuxuser via F-Droid Forum:
I may have forgotten something, but the only license concern I can remember was regarding the EULA that sort of “leaked” into the final license files. If that is what you are referring to, then I believe I solved it a while back (in the chain of license text inclusions some overly-aggressive rules incorrectly pulled in the EULA text when really only Apache was the only applicable).
So that part is fixed. A small tweak to the docker files is all it takes to build zips without the EULA sneaking in. At least for the recent versions. I don’t remember if we checked older builds.
By the way, can I ask a question: how easy would it be to use these zips as “drop in” replacements of the official ones? Doesn’t tools like gradle always download from google? Or could these be just put/unzipped in the right directory and get used instead? Or can we just configure gradle to use an alternative URL to download these files? And can we configure gradle to not require users to accept the eula (when using our own packages)? Essentially I’m wondering if we’ll need to look into making a patched version of gradle(w) to make it possible for devs to actually switch to these packages?
I’m a noob when it comes to android app building. For my own project I’m using a simple solution with makefiles, which makes it really easy to swap parts like this, but I have no idea how flexible gradle would be for this scenario.
What gradle does is a good question. I’m not sure how that works, but would
really like to know.
F-Droid already has gradlew-fdroid, so we can adjust things there.
Mats Wahlberg via F-Droid Forum:
Ah, that sounds perfect.
Right now I’m considering if I should take a more active role on this. I really don’t want to “hijack” any of this, or step on anyone’s toes, but I would really like to see some packages online that could be used, even if I have to do it myself.
I was able to successfully build several SDK and NDK versions manually on a normal debian install without using docker. I haven’t actually used docker before, but would it be best to still use it? I feel a bit lazy for not trying it yet, perhaps it is important for older versions and to ensures easier reproducible builds?
IMHO, it would be great to have some binaries posted somewhere, regardless of
who does it so people can start trying. Also that would make it easy for
someone else to run a build and check whether the builds are reproducible.
Mats Wahlberg via F-Droid Forum:
Agreed. I think I will open a project on codeberg and use that to upload some packages. Unless something changes, or @alaskalinuxuser wants to upload somewhere instead. Again, I don’t want to step on toes or make it feel like I’m hijacking someone’s work.
It will still take a little while, I want to make sure I provide a sane system for reproducing the builds (either clean up my own scripts, extend those by Starfish, or do a mix of both).
Oh, embarrassingly, I noticed that the build solution by Starfish (for more recent SDK versions) actually didn’t use docker. It was plain scripts for building directly on the host OS, so I shouldn’t have worried about that. I had mistaken it with the work by @Beuc, which is what I mostly looked at in the past.
Edit/Update: okay, my little build solution can now build some NDK and SDK versions (without non-free licenses or EULAs). However, I’ve hit a snag: I didn’t remember the SDK generating so many zip-files, and them together being so big. Codeberg seems to have a (default) size limit of 1.5GB which I think will only be enough for one or two version of each. Technically, the NDK also creates some xz-compressed archives which are about half the size of the zip files, and I guess I could skip uploading the windows versions (only the gnu/linux versions). But I think I will need to find a more long-term solution.
@hans: If I recall, didn’t f-droid used to host @Beuc’s rebuild files? Would something similar be possible here, or at least is there some respectable site that would support bigger uploads? I don’t want to start putting things on Mega or similar. Or, @vdbhb59, would you still be up for hosting these files?
Also, I mentioned Starfish before, turns out I was thinking of the work by wchen342 which extends on the work by @Beuc and uses docker. My current solution does not use docker, but I could look into moving to docker if it would be preferred (I just don’t have prior experience with it).
IntelliJ IDEA Open/Community Edition
Yeah, I think there’s a good availability of options in the editor space, but initial question was more about the SDK and build tools. I guess there isn’t really a FOSS solution (yet).
Agreed. The input is absolutely appreciated, but editors are not the big problem.
But, some good news: right now my little build solution has built the following (with patches as necessary to avoid EULA/licensing issues):
For the NDK:
- 29 - the current stable version
- 27d - the current LTS release
- 25c - the last release to support KitKat (this one is important to me personally)
For each, I got packages for both gnu/linux and windows (I skipped Mac for now). Each one being both a compressed zip-file, and some other format (tar.xz for the older one, and tar.br for the two recent ones, it seems upstream is trying out ways of reducing the archive size?).
For the SDK:
- 15 r9
Again, here it applies a small patch to prevent the EULA from being included in the packages. Turns out there has been many new revisions since r9 (right now it’s at 30-something?). And there is a new SDK 16 to look at. Unlike the NDK, the SDK does does not seem to have an LTS version? So I need to do some more building here.
There are some small things to clean up (like renaming the files to conform with the official zips - I am already doing this for the NDK releases), but that’s not a big problem.
Right now my main concern is deciding on where to upload all of these! To @hans and @vdbhb59 : how should I proceed with uploading these? I’m going to put the sources of my build solution on codeberg, but I think it will be too much to also upload the actual results there (at least if I make more versions, it will be too big). Is this something any of you or f-droid could host? Obviously you might want to build these yourself, but the build system should make that easy when it’s done (just needs a debian install with some standard packages and the “repo” script).
Upload it on my server and my git/forgejo instance.
For full files, I can upload to my server and give urls as earlier.
Awesome!
I just need to find some time to figure out how to rename the files, and build the most recent sdk.
From the current build of SDK 15 (15.0.0_r9), there’s a lot of zip-files created:
aapt2-protos.zip
android-sdk_linux-x86-slinger.zip
public-sdk-28-stub.zip
public-sdk-29-stub.zip
public-sdk-30-stub.zip
public-sdk-31-stub.zip
public-sdk-32-stub.zip
public-sdk-33-stub.zip
public-sdk-34-stub.zip
public-sdk-35-stub.zip
sdk-repo-linux-build-tools-slinger.zip
sdk-repo-linux-docs-slinger.zip
sdk-repo-linux-platforms-slinger.zip
sdk-repo-linux-platform-tools-slinger.zip
sdk-repo-linux-samples-slinger.zip
sdk-repo-linux-sources-slinger.zip
sdk-repo-windows-build-tools-slinger.zip
sdk-repo-windows-platform-tools-slinger.zip
Of these, I only think the following will be useful/interesting:
android-sdk_linux-x86-slinger.zip
sdk-repo-linux-build-tools-slinger.zip
sdk-repo-linux-docs-slinger.zip
sdk-repo-linux-platforms-slinger.zip
sdk-repo-linux-platform-tools-slinger.zip
sdk-repo-linux-samples-slinger.zip
sdk-repo-linux-sources-slinger.zip
sdk-repo-windows-build-tools-slinger.zip
sdk-repo-windows-platform-tools-slinger.zip
Obviously the names need to be adjusted a bit. The official naming of these files is very varied, both the the versions are random like “build-tools_r35.0.1_linux.zip”, "samples-23_r02.zip”, and I’m a bit confused on some of these, like both android-sdk_linux-x86-slinger.zip and sdk-repo-linux-platforms-slinger.zip seem to contain the important android.jar, which in upstream I found in a zip called platform-35_r02.zip.
I’m not sure, but I could try renaming them like:
android-sdk-r15.0.0_r9.zip
android-sdk-r15.0.0_r9-linux-build-tools.zip
android-sdk-r15.0.0_r9-docs.zip
...
This would match the sdk naming (android-ndk-r29-linux-x86_64.zip and similar), but not quite with upstream. Each zip seems to contain a source.properties with the actual version number, but it seems tedious to try to extract it for renaming each individual file, and I’m not sure if it would be of any use.
Also, now I noticed the zip called “android-sdk_linux-x86-slinger.zip” is much bigger than the others, and seem to contain everything from the other archives packaged together… curious.
edit/update: okay, it seems android-sdk_linux-x86-slinger.zip only contains the same files from other packages, and not all of them (it’s the docs, platform, etc, but not all of them), and nothing more I think? I think it can be dropped (the replicant sdk page seems to suggest this is the one to use, but it is missing a bunch of tools, and I didn’t see a similar zip on the tencent mirror). Also I found a notice in the docs zip saying it has been deprecated since 2016. So this narrows it down to 7 zips. Still thinking about how to name them.
Okay, first of all, I don’t know how people feel about double posting here. I didn’t want to add a big edit to the last, but I’m worried I’m making a faux pas here. I hope this is okay…
Some updates: the latest SDK 15 (r36) has been built without containing the EULA. But I’m having trouble with SDK 16 (r4) I’m getting Too many open files (os error 24) despite increasing the limit by both ulimit and by editing my /etc/security/limits.conf. I will dig deeper into this when I have some more time. Maybe r4 is just broken or I’m having too many cores causing too many parallel jobs?
Apart from this, I saw the file repository.xml in the build output. It seems (and according to the work of Beuc) to be an old solution for package hosting metadata. The point is, it had a nice naming for the files: no versions at all, so just remove the “-username” from the end of the file. So to answer my previous pondering, I’m now going with “android-sdk_linux-x86.zip”, “sdk-repo-linux-build-tools.zip” and similar. I’ve also cleaned up most of the code and added an EULA scanner script, so every time a build finishes the results are checked for traces of it (it can also be used directly to check files).
Some more exciting news! It seems Starfish on codeberg has continued to work on their SDK rebuild solution, and have just posted a guide on how to actually use the custom built files for normal (unmodified gradle) builds: https://codeberg.org/Starfish/SDK-Rebuilds/src/branch/main/how_to_use.md. This seems important, I was wondering how to do this (although my own project doesn’t need it). Maybe I should reach out and see if our effort could be combined?
4 Likes