more like a human factor.
Generating the index takes a few hours (fdroid update has to gather all kind of metadata information from all the apps source repos). You can probably imagine why it’s done in batches 
.
Most of the builds get automatically added by checkupdates. There is no point in this process where a build entry can be marked as more critical as any other.
Also it would not help at all because the server would still build all pending builds and everything is gated by the index signing in the end… :-/
вт, 11 сент. 2018 г., 19:36 Marcus Hoffmann:
Generating the index takes a few hours (fdroid update has to gather all kind of metadata information from all the apps source repos). You can probably imagine why it’s done in batches
This is a very interesting and useful information. I’m sure there’s a bug hanging somewhere for it. Please someone share if not difficult: besides obvious “caching” improvement I got another idea to purpose.
I’m just a lowly user who enjoys F-Droid immensely… I appreciate the time all of you put into making all this work. I always read in awe. A big THANKS.
6 Likes
Haha, everybody loses their minds when their YouTube scraper app breaks, brb, gotta get some popcorn
*IS SCREAMING INTERNALLY TOO*
Guys, I understand your frustration about how slow F-Droid delivers updates. Let me explan why this happens. F-Droid cycle looks like this:
- Check for updates. F-Droid downloads recent changes of 2365 apps and parses them to find new releases.
- Build. For each new release F-Droid starts a VM and builds it from scratch. Usually there are dozens. Even the smallest utility needs 7 minutes to build, while each Fennec APK will take up to 5 hours.
- Generate index. F-Droid gathers metadata from fdroiddata (2773 entries), apps source code (98 file trees) and APKs (19229 of them, each needs to be unzipped and parsed).
- Sign APKs and index. Ciaran manually transfers APKs and the new index to an air-gapped system (it has no Internet connection to avoid signing keys leaks), signs them and brings signed files back.
Hopefully you now see why each stage takes many hours and the whole cycle takes days.
5 Likes
No, it does not, NO USE IN THAT if you want to actually watch the video, not just read about it¹.
And it never did, as far as I can remember. Google never wanted you to do without running their nonfree programs on your machine, whether installed, updated and removed at your will, or pushed into your browser without your consent unless you are careful enough to prevent them from being executed.
This is the whole point of existence of software like NewPipe.
So this was essentially a command to give up even a mere freedom to decide what software to run on your little machine! And that while we could just install an upstream build [0]. This is a really strange thing to hear from a maintainer of free software distribution.
[0] https://github.com/TeamNewPipe/NewPipe/releases/download/v0.14.1/NewPipe-v0.14.1.apk
¹ And even for reading video titles, descriptions and seeing thumbnails with just a browser you have to instruct it to introduce itself as something that is incapable of running JS (Googlebot is a good choice), otherwise youtube.com would return you nothing besides a nonfree program to run.
1 Like
Really? You went there? 
Check for updates. F-Droid downloads recent changes of 2365 apps and parses them to find new releases.
Does it start building as soon as first app with changes fetched? Does it query multiple apps simultaneous?
Build. For each new release F-Droid starts a VM and builds it from scratch. Usually there are dozens. Even the smallest utility needs 7 minutes to build
It should be possible to optimize the VM start - maybe starting VM before other build finishes but not starting build - “empty” VM should not consume much resources. Or have such VMs saved state, restoring it also should be faster than complete boot.
while each Fennec APK will take up to 5 hours.
It’s 4 APKs per update so 20 hours + some other heavy apps = 24+ hours, so this can be done separately if signing is not a bottleneck.
APKs (19229 of them
not all of them are freshly build, right? So old ones can be cached.
each needs to be unzipped and parsed
Is it necessary to perform a full unzip? Or only few files such as manifest needed? AFAIK it’s possible to perform a partial unzip.
Refresh repos now!
Enjoy…
Hope we see you all here back lending a hand to avoid such issues in the future.
Cheers 
2 Likes
No, stages don’t overlap.
No.
VM in restored from a saved state.
Most are old, yes.
It’s not a full unzip, it’s an aapt call.
1 Like
Disclaimer: I want to learn how F-Droid works and not offend anybody !
I wonder what is the missing step between when an app is built successfully (in the case of NewPipe it is 0.14.1, successfully built 2 days ago: https://f-droid.org/wiki/page/org.schabi.newpipe/lastbuild) and when it becomes visible in the repository (still the non-functional 0.14.0 in that case: NewPipe | F-Droid - Free and Open Source Android App Repository)? Is it the manual signing process that was mentioned before, or is there something else?
Is the current status of app updates documented somewhere?
Please read above, it was explained.
Also, if you want updates use F-Droid the client app (get the APK: https://f-droid.org , install it, set an update check, slide Wi-Fi and mobile data sliders to the right), where NewPipe has been available for 4-5 hours already, and you would have seen the update notification. The website will update at some point in the next day.
Thanks for the answer, but I still do not understand entirely:
Hm, that’s what I thought I did. But I did not understand everything, especially what happens after the build succeeded. Therefore I asked if the manual signing process is what is happening in between, and how to check which apps are waiting for that step.
Just before I posted, I checked in F-Droid and 1.4.1 was not there, which caught my interest and triggered my research. Now it is, but I am still curious . I’ll read one more time, maybe I understand better then.
To make it more clear: What happens after
and where are files waiting for this step listed?
Before signing they’re build, you can see current and past build apps here: F-Droid Monitor (almost real time, date in UTC), not sure what you mean, they wait for the whole queue to finish and be transfered in bulk to the singing machine. (no they can’t be signed right after each one is build, this process needs a human!)
Really?
Unless I am deeply mistaken, yes.
I do not think, I am, though, since there is nothing to be surprised about: a good half of the today’s Web is useless without nonfree programs. It actually went so broken, that there are pages, which are supposed primarily to be read (i. e. they comprise noninteractive text), and yet they heavily relies on ad-hoc programs they try to run in your browser.
If you mean JavaScript please say that, else saying non free programs means something else.
a good half of the today’s Web is useless without nonfree programs
If you mean JavaScript please say that
What does the language in which the program is written matter, when you are not allowed to modify it?
Indeed, programs, that are run in a web-browser, are almost exclusively written in Javascript now (though this might change in very near future), but I could not raise any objection to that.
else saying non free programs means something else.
Sure, it means something else! Namely a program that you are not allowed to either: use for any purpose, modify to suite your needs and wishes, and distribute whether modified or not. And you are very much aware of that, I believe. 
But that is not all! Compared to nonfree programs that are not tied to certain webpages (which, by the way, are written in Javascript sometimes too), most programs that are found on webpages strips you of a mere freedom to decide what to run and whether to run it at all. Those, who designed the browser client for Youtube and similar software, did that with an implication in their mind, that they, not you, are in charge of what programs to run on your machine, when to update and delete them. That is what Web became — a thing, that is more hostile to user, than orthodox nonfree software.
So I cannot quite understand, why, given all possible options, did you, to put it mildly, firmly recommend to stick with the least friendly one. Indeed, some users, that are not fully aware even of the very fact, that web-browser runs any programs at all, might not recognize it as such. But is not the task of a free software developer to educate them, rather than to take advantage of their lack of knowledge to wave them off?
1 Like
I was also worried about the crash and one of the seriously looking guys for the update;
Came to know that the latest version is readily available to download for all Android devices from the official source; https://github.com/TeamNewPipe/NewPipe/releases/download/v0.14.1/NewPipe-v0.14.1.apk
Kindly don’t blame f-droid; it’s one of the finest app catalogues ever I have seen and the safest as well.
Regards,
Rahul.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.