Automated way of downloading source code of all apps from repo

Hello all,

I’m trying to do develop a machine translation model and train it using the source code of all the apps in the official F-Droid repo. I did come across this topic:How can I download all the apps in F-Droid repository but the stated solution using fdroidcl downloads apps as APKs and I want the source code of each of the apps. Does anyone know of an automated way or a package using which I can download the source code of all the apps in the official f-droid repo (preferably through Linux command line)?

Sorry for bad English

Easy…

Download this file first: https://f-droid.org/repo/index.xml

(this is one line!)

sed -i -e 's/apkname>/apkname>\n/g' index.xml|grep \/apkname index.xml|cut -d "<" -f 1|sort|sed -e 's!^!https://f-droid.org/repo/!' -e 's/.apk/_src.tar.gz/'

Buuut… you’ll also need to prune and leave only the latest version AND pretty please DON’T DDOS our servers by downloading them all, please.

1 Like

I downloaded the file and ran the command and it does indeed show a url to all of the apps but nothing was downloaded apparently. In which location can I find these files?image

These are just the URLs which you need to download.
But I would use a simpler command:
rsync -a --info=progress2 --human-readable 'rsync://ftp.fau.de/fdroid/repo/*' --include='*_src.tar.gz' --exclude='*' destination_dir
These are around 50GiB, so this needs some time.
You can also use a different rsync mirror.
Here is a list: https://marzzzello.gitlab.io/mirror-monitor/repo/f-droid.org:repo.html

2 Likes

Of course, it’s just a list, you can feed it to curl or wget :wink:

1 Like

This works great! Is it possible to only get the latest version of the app instead of multiple versions of the same app through this? It won’t make much of a difference having all versions of the app in my case and it would take less time too.

Awesome! Can this command be modified to include only the latest version or can you suggest a way for me to accomplish the same? Thanks

I’m trying to do develop a machine translation model and train it using the source code of all the apps in the official F-Droid repo.

Machine translation from Java to Kotlin? :slight_smile:

1 Like

Actually I’m looking into making a deobfuscator for program elements like class names, method names etc.

Mass downloads should happen via rsync from the mirrors. Many of the mirrors support rsync. And if you are downloading this to a machine outside of the EU, consider offering a local mirror:

https://f-droid.org/docs/Running_a_Mirror/

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.