Build infrastructure for Javascript-based apps


#1

Hi. I’m working on the inclusion of a Javascript-based app (Expo) and it leaves me wondering how the fdroid build process could make this easier. There are currently mechanisms to make the Java-aspect of building apps quite smooth in many cases, and there are also useful checks (on the presence of binaries and non-free dependencies and a Maven repository whitelist, for example).

For Javascript-based apps, in many cases there is a second build-infrastructure (next to the Java-part), which gathers source files and dependencies and generates one or more Javascript bundles (see also here). This may benefit from a similar infrastructure, with various node versions (compare Android SDK versions), validation of dependencies (like a Maven whitelist). Instead of putting this in a build recipe (like in this prebuild), I wonder how it could be improved.

Perhaps it’s too early to ask this question, and we need a couple of (larger) Javascript-based apps to get more experience, but I don’t think it hurts gathering some ideas already. Feel free to add them here.

(Just in case, I would like to mention that I hope this question will not block new Javascript-based apps from becoming part of F-Droid, pending a full answer here.)


#2

Idea: multiple nodejs versions

Some apps depend on a specific node.js version for building. It would be useful to be able to set the node version in the metadata. nvm could be used for selection of the appropriate version (system packages don’t usually cut it, as apps often require much newer versions). In the first version, we may trust the official binaries, in an improved we could also pre-build them regularly from source and use those.

This would be controlled by a build parameter like jsversion, a version number (major only “4”, or full “5.12.0”), and that nodejs version will be put on the PATH.

Idea: license scanner

For each node_modules directory, run a license checker (like node-license-validator, licensee or package-license-type-based) to check that all packages conform to an accepted open-source license (no clue how often an npm-package has a closed license).

Still, even if the Javascript code has a certain license, the package may contain binaries without source code. I’m not sure how often this happens, but ngrok would be an example (binary code is closed). It may be enough to put those on a blacklist (or just remove binaries with the scanner - often they aren’t needed for building the app anyway).

Idea: npm blacklist/whitelist

Instead of scanning licenses, a blacklist may also work. There would be some responsibility to keep it up-to-date though, and I don’t really know how to easily find this out. A whitelist would be safer, but there are so many modules with varying usage, that it’s not really feasible, I would think.

In any case, I think a blacklist for certain modules (or module regexps) would be useful, just like the scanner does for gradle dependencies. Things like (proprietary) analytics services could be detected when implemented in Javascript as well.

Idea: Javascript build step

Some projects require an explicit Javascript build step before the normal Android build process can begin. This happens both for Cordova and React Native based apps.

Initial suggestions for extra build parameters in the metadata:

  • jsdeps - npm modules installed globally (or at least on PATH). Sometimes gulp or react-native-cli is required for building, but not specified in package.json.
  • buildjs - paths to build Javascript in (similar to buildjni); in these directories, npm install will be run. These can be multiple, since some projects have multiple Javascript components (Expo has it; perhaps it’s a corner-case).
  • jsbuild - Javascript build command, which usually needs to be specified

The naming could use some clearing up, perhaps.

Idea: allow (some) relative gradle repositories

Some React Native modules have Java code with gradle build files, which reference a local path to another npm module (see, for example, lottie’s top-level build.gradle: $rootDir/node_modules/react-native/android). These should be allowed by the scanner. (I think we could whitelist some of these expansion parameters that are really local.)

Question: include npm modules in source archive?

Do Javascript npm dependencies need to be part of the source code or not?

  • for comparison: I think Maven modules are not stored in the source archive (but they are compiled JARs).
  • Javascript modules from npm usually consist of the source code (there may be a bundled version, but even then the source is usually included).
  • npm versions might sometimes disappear (not often, if I’m correct), in that case it would be useful to include them in the sources. Nevertheless, npm modules are publically available, so there may be no need.

This is also relevant for the build order: if npm packages are part of the source package, it needs to happen somewhere between patching and packaging the sources. This also means items built need to be cleaned up, and they need to pass the scanner. If they are downloaded and installed at build-time (as part of it or as a kind of prebuild step), it would be a little easier, but we need to be sure the packages are really in-line with the policies.

Question: binaries in modules

Several npm modules have either binaries included, or download binaries as part of the install step (e.g. sqlite3). This is so that developers can easily install dependencies without having to bother about compilation and setup (on common platforms). Most often (always?) source code is provided as a fallback (some packages respect the --build-from-source option - though not all, like ngrok with its binary for which source is not available).


#3

Thanks for the detailed post. Although I only really work on the client, I’d think that the server folks would be very open to merge requests which improved support for JS based apps, especially seeing as they only seem to be getting more common.

There is recently worked on support for a sudo= field in the metadata, which allows people to install their own dependencies (e.g. specific node versions), but it would be much nicer if indeed some of these suggestions you have (e.g. various node-js versions) were made more of a first class citizen over time.

I especially like that you’ve broken it down into some pretty concrete and well defined tasks. Perhaps these could be moved into the fdroidserver issue tracker once you get a little bit of feedback.