Making it easier for F-Droid to package Mozilla Firefox

firefox
mozilla

#9

Yeah, the build script calls a script within the same repo that holds the locales. These files are removed because the fdroid-server checks for the presence of any executable/binary data.

If i remove just the files, but keep the .ini s, i get something like

And if i remove all the directory with the conflicting data including the .ini s, the error is:

In this example happens with a xpcshell.ini file, but the same happens with something like MOCHITEST_MANIFESTS += ['test/mochitest.ini']. I opted to remove all the test directories with conflicting files, and then grep the moz.build to remove the .ini s

There was some problems with telemetry code being included even when disabling it with .mozconfig. I see that many of them were fixed in the last versions.

Definitely i have to test the last versions, but with a packet analyzer we discovered some time ago that some information was being sent even when opting out from the user preferences. As i said, the technology used by firefox for android has changed a lot since then.

I will try a gradle build with more recent versions (last time i tried was Fx54), but if you use a custom maven repo it could be problems as there are just a couple of them whitelisted. @hans should know more.

Many thanks to you! :slight_smile:

Matías


#10

@Manizuca,

Manizuca https://forum.f-droid.org/u/manizuca
December 1

nalexander:

Can you explain what this means? Do binaries mean things like proprietary
codec blobs? I didn’t see anything like this when I looked in the F-Droid
build scripts at https://gitlab.com/fdroid/fdroiddata/blob/master/
metadata/org.mozilla.fennec_fdroid.txt#L3959-3989.

Yeah, the build script calls a script
https://gitlab.com/Manizuca/fenneclocales/raw/master/prebuild.sh within
the same repo that holds the locales. These files are removed because the
fdroid-server checks for the presence of any executable/binary data.

Oh my. In general, this approach is always going to be broken, and Mozilla
would much prefer to hear about and potentially accommodate your use cases
than have you require such blunt instruments. But this approach does have
the advantage of not requiring upstream changes, which I understand!

nalexander:

More details would be helpful here. I would expect that --disable-tests
in your mozconfig wouldn’t fail the builds, but I don’t know what’s at
play.

If i remove just the files, but keep the .ini s, i get something like

0:25.26 ==============================
0:25.26 ERROR PROCESSING MOZBUILD FILE
0:25.26 ==============================
0:25.26 The error occurred while processing the following file or one of
the files it includes:
0:25.26 /Sources/F-droid/build/org.fdroid.fennec/modules/libjar/
zipwriter/moz.build
0:25.26
0:25.26 The error occurred when validating the result of the execution.
The reported error is:
0:25.26 Error processing test manifest file /Sources/F-droid/build/org.
fdroid.fennec/modules/libjar/zipwriter/test/unit/xpcshell.ini: Traceback
(most recent call last):
0:25.26
0:25.26 File “/Sources/F-droid/build/org.fdroid.fennec/python/mozbuild/mozbuild/frontend/emitter.py”,
line 1294, in _process_test_manifest
0:25.26 path, ', '.join(missing)), context)
0:25.26
0:25.26 SandboxValidationError:
0:25.26 ==============================
0:25.26 ERROR PROCESSING MOZBUILD FILE
0:25.26 ==============================
0:25.26 The error occurred while processing the following file or one of
the files it includes:
0:25.26 /Sources/F-droid/build/org.fdroid.fennec/modules/libjar/
zipwriter/moz.build
0:25.26
0:25.26 The error occurred when validating the result of the execution.
The reported error is:
0:25.26 Test manifest (/Sources/F-droid/build/org.
fdroid.fennec/modules/libjar/zipwriter/test/unit/xpcshell.ini) lists test
that does not exist: test_asyncadd.js, test_asyncbadadd.js,
test_asyncbadremove.js, test_asyncremove.js, test_bug399727.js,
test_bug419769_1.js, test_bug419769_2.js, test_bug425768.js,
test_bug433248.js, test_bug446708.js, test_bug467740.js,
test_createempty.js, test_deflatedata.js, test_directory.js,
test_editexisting.js, test_storedata.js, test_sync.js, test_undochange.js,
test_zipcomment.js, test_zippermissions.js, test_bug717061.js,
test_alignment.js

And if i remove all the directory with the conflicting data including the
.ini s, the error is:

0:23.25 ==============================
0:23.25 ERROR PROCESSING MOZBUILD FILE
0:23.25 ==============================
0:23.25 The error occurred while processing the following file:
0:23.25 /Sources/F-droid/build/org.fdroid.fennec/modules/libjar/
zipwriter/moz.build
0:23.25
0:23.25 The error was triggered on line 7 of this file:
0:23.25 XPCSHELL_TESTS_MANIFESTS += [‘test/unit/xpcshell.ini’]
0:23.25
0:23.25 An error was encountered as part of executing the file itself. The
error appears to be the fault of the script.
0:23.25
0:23.25 The error as reported by Python is:
0:23.25 [‘IOError: Missing files: /Sources/F-droid/build/org.
fdroid.fennec/modules/libjar/zipwriter/test/unit/xpcshell.ini\n’]

In this example happens with a xpcshell.ini file, but the same happens
with something like MOCHITEST_MANIFESTS += [‘test/mochitest.ini’]. I
opted to remove all the test directories with conflicting files, and then
grep the moz.build to remove the .ini s

Why is it that you need to do anything with these test files? They don’t
feature in the packaged artifacts at all, and if you use --disable-tests
in your mozconfig you should not really see any interaction with them.

nalexander:

These should all be supported as mozconfig options. If they’re not doing
the right things for you, I consider that a bug, and would appreciate being
CCed on a ticket in the Core :: Build Config component at
https://bugzilla.mozilla.org. In particular, Telemetry is growing in
scope and intrusiveness, so I wouldn’t be surprised if we need a little
focused attention to ensure we’re not accidentally pushing bits to Mozilla
that F-Droid users don’t want exposed.

There was some problems with telemetry code being included even when
disabling it with .mozconfig. I see that many of them were fixed in the
last versions.

nalexander:

Firefox for Android has been pretty focused on using newer technologies to
A/B test and really understand our user base, so we’ve used Adjust (a
German tracker with a good privacy policy), Switchboard (an A/B module that
we run an in-house backend for), and Leanplum (a marketing application
about which I know little). It’s my belief that we’ll be able to make
building with Google Play Services imply no Adjust and Leanplum. I don’t
know what the right outcome for Switchboard is for F-Droid – you don’t want
Mozilla’s instance, but you probably don’t want to run your own (which has
a non-trivial cost). I’m pretty sure that everything will work if you “opt
out” of Switchboard, but I don’t know if that’s easy to arrange at
build-time. We can try to make it so.

Definitely i have to test the last versions, but with a packet analyzer we
discovered some time ago that some information was being sent even when
opting out from the user preferences. As i said, the technology used by
firefox for android has changed a lot since then.

I would appreciate you revisiting, if and when you can, but I don’t think
this should be high priority for you.

nalexander:

Surely the majority of F-Droid’s packages are built with Gradle, so this
problem must be addressed in some way. Firefox for Android’s Gradle
configuration is pretty complex, but it should “just work” to run mach
build after the dependencies are in place. I expect that F-Droid could
work through these dependency issues fairly easily. I can provide links to
what Firefox for Android does: the tl;dr is that we collect Gradle
dependencies using a proxy Maven repository running Nexus Sonatype. It’s
not that complicated; start reading at https://searchfox.org/mozilla-
central/source/taskcluster/scripts/misc/android-gradle-dependencies.sh
and the linked scripts.

I will try a gradle build with more recent versions (last time i tried was
Fx54), but if you use a custom maven repo it could be problems as there are
just a couple of them whitelisted. @hans
https://forum.f-droid.org/u/hans should know more.

Mozilla does not run its own Maven repository, and is unlikely to depend on
one anytime soon: it’s a non-trivial cost when you run continuous
integration at the scale that we do. The set of Maven dependencies evolves
slowly, so when it changes we capture the entire dependency set, archive
it, and offer that to subsequent builds. Right now, those dependencies are
fetched from jcenter (run by Bintray) and maven.google.com (run by
Google). It might be possible to avoid Google’s repository right now, but
the evolution is definitely towards everything Android related being from
there so it’s probably not possible to avoid for long.

In any case, I don’t think you would have big problems doing what we do;
and, if you so chose, you could also use Mozilla’s dependency archives –
they’re just .tar.xz files of Maven file repositories.

Firefox 58 will probably be the last Firefox for Android that will not
require Gradle to build – watch
https://bugzilla.mozilla.org/show_bug.cgi?id=1414415

to see when the non-Gradle build is removed. In order to have a smooth
transition as Firefox 58 becomes release, F-Droid will need to do some work
to support this.

nalexander:

Thanks for your feedback!
Nick

Many thanks to you! [image: :slight_smile:]

Is it easy to test changes to F-Droid’s build script? I think we might be
able to do a little focused work and get a much simpler script that
achieves the same outcomes.

Best,
Nick


#11

May it be cleaner to empty those .ini files instead of removing?

F-Droid server scans the whole source tree for executable binaries because binaries under source control often lack source code (i.e. non-free software). F-Droid server has no idea which files will be packaged, so it triggers an error even if they are unused.

Those are allowed, no problem here.

Yes. F-Droid server (despite its name) is just a Python program that runs build recipes. See https://f-droid.org/docs/Building_Applications/


#12

@nalexander about Google Play Services, what is the push stuff used for? I never understand why Firefox needed its own push services. But I do understand why push requires GCM on Android.

About reproducible builds, check out fdroid verify and our verification server:
https://f-droid.org/docs/Verification_Server

The process is not well documented, but there are currently two ways to get APKs into f-droid.org via reproducible builds:

Happy to help you try that stuff out. Perhaps that would be better in a new thread.


#13

This I can answer, 'cuz I built most of this integration for Fennec :slight_smile: The Firefox product family wants push services to support the Web Push HTML5 spec. This is a big part of the Progressive Web Apps initiative that aims to make developing rich App experiences possible using the Web Platform. Firefox for Desktop runs its own custom push service called autopush that keeps long-lived Web Sockets open. As you surely know, that’s not viable on a phone – the OS is best positioned to do that work, potentially using side-channels from existing (carrier) channels. Therefore, Firefox for Android uses an autopush bridge to Google Cloud Messaging (and eventually will have to migrate to Firebase Cloud Messaging), and Firefox for iOS uses an autopush bridge to Apple Push Notification Service.

I concur, a new thread would be best for this. Let me try to get this --without-google-play-services business across the line before I get carried away and try to push on reproducible builds :slight_smile:

Many thanks for this information, @hans.

Yours,
Nick


#14

Hi folks,

Circling back to this: I finally landed Bug 1419581, which added a --without-google-play-services configure flag and a continuous integration job (labelled BnoGPS, like this job to prevent us regressing the build parts at least. (There’s no test that verifies that the produced artifacts don’t reference Google Play Services in some way, mostly 'cuz that’s a hard problem.)

These jobs without Google Play Services are not shipped to end users, nor are they signed for release, so they’re not directly usable by F-Droid consumers. But hopefully this will let F-Droid simplify its build script (starting with Firefox for Android 60), and help keep F-Droid’s own builds green in the future!

Yours,
Nick Alexander


#15

@nalexander, thank you very much for saving Fennec’s freedom!


#16

This is great, thanks! It would be nice to have an official Firefox in F-Droid again. Do you think that would be possible using --without-google-play-services?


#17

I’m confident that Mozilla Corporation doesn’t want to support an officially branded Firefox for Android that doesn’t have the full complement of Web Platform features. (In this case, GCM is required for Web Push, which is a big part of the Progressive Web Apps strategy on mobile.)

So while it’s technically possible, it’s not something that Mozilla is likely to commit resources to. I am interested in doing small things to ensure that downstream projects (including, but not limited to, F-Droid) can build a free-as-in-freedom Firefox for Android without fragile build system hacks for the foreseeable future.

That means that F-Droid will need to continue branding Firefox for Android, just like it does now.


#18

makes sense. I’ve been lobbying Google through all the channels I have
to ask them to open source the GMS/Play Services libraries. That’s the
only thing that’s not free software for many important apps, like
Firefox. MicroG has done some work towards making free replacements,
hopefully that can also be used to pressure Google. Its not like the
business logic is in those small GMS jars.

They do seem to be moving more on open sourcing the Firebase libraries.
Is Firebase Push a possibility for Firefox?


#19

Yes, we (Mozilla Corporation) did too. I can’t link to the ticket since it’s MoCo-confidential, but there was nothing really interesting – our council reached out and I don’t think we got anywhere.

I concur, although the division of responsibility between the client code and the privileged on-device service code is not clear to me. I think the sticking point is the Chromecast code, which – as far as I know – does the interesting bits of the protocol on the device, and potentially even in the consuming App’s process space. This is hearsay/speculation, though.

Technically, yes. If we (I!) had built our Web Push integration perhaps 12-18 months later, we would have undoubtedly used FCM instead of GCM. As it stands, I think there’s a race between Google deprecating GCM entirely and Firefox for Android modernizing its Web Push implementation. It’s probably not a lot of work but there’s nobody working on it right now (and nothing scheduled that I know of). (Tongue only somewhat in cheek: patches wanted!)


#20

Here’s some good news: VLC has implemented fully open source Chromecast support:


#21

#22

#23

Fennec 60.0 is available on F-Droid now! I’d like to thank everyone who put so much efforts into this release!


#24

Is it me or is version 60.0 available for certain devices? I’ve updated the repos on my F-Droid app and I couldn’t see any update for Fennec.

I’ve also tried installing it using the APK from the website: https://f-droid.org/packages/org.mozilla.fennec_fdroid/ and I received this message from adb install:

adb: failed to install org.mozilla.fennec_fdroid_600010.apk: Failure [INSTALL_FAILED_NO_MATCHING_ABIS: Failed to extract native libraries, res=-113]

#25

This could be two things. It’s most likely a legit ABI issue – you’re installing an APK with ARM libraries on an x86 device (or, less likely, vice versa). It could also be an issue that Firefox for Android is seeing with recent devices (particularly the Samsung Galaxy S8) – see https://bugzilla.mozilla.org/show_bug.cgi?id=1450793 and related tickets.


#26

So is https://bugzilla.mozilla.org/show_bug.cgi?id=1450793 the reason why F-droid shows no updates available for Fennec on my device? I have a Samsung S5 device (codenamed klte, model SM-G900F), it’s processor is based on the ARM instruction set as far as I’ve investigated so far.


#27

No. Given that you’re using an S5 (a very common ARM device) I think the issue is probably the F-Droid APKs are giving you x86 native libraries, which isn’t what you want. This could be a version number issue – if F-Droid isn’t behaving like the Play Store and is offering the linearly highest version number, rather than the highest version number appropriate for your device, you will get this issue. But I don’t know enough about F-Droid to say what’s really happening.


#28

No, the reason is that, while available, the “recommended version” didn’t get bumped. This should happen with the next index update. In the meantime, you can install it manually from the “versions” fold of the app info screen.