[Help wanted] How to create a reproducible build (FairEmail)

Nope, disorderfs doesn’t improve things, same error.

Please explain what/how to verify…

Would you like me to provide an APK so you can compare?

I would start by checking the diff of the manifest file because this is a readable XML file.

You can just download the latest APK file from GitHub and unzip it.

You can also send the APK file you’ve built to me for inspection, but please make sure first that the source code is the same because it is odd that the manifest and classes are different.

@hans can you maybe tell what to look for considering this?

ERROR: SHA-256 digest of AndroidManifest.xml does not match the digest specified in META-INF/MANIFEST.MF. Expected: <oTOZfOh8OUkJZrrW3ByzKfy/zl1Rb2Ywg4l3PYpHrwQ=>, actual: <kSPZ+GnUbxRRZw0EvDerKqqEx9jebp9GJ8MVQ/jBjDA=>
ERROR: SHA-256 digest of classes.dex does not match the digest specified in META-INF/MANIFEST.MF. Expected: <IypiPzw/bBi5loZvI01Po2n++AYrLWfo5FSMz7knjWM=>, actual: <asXmVEQg4NzX6H/YLniWSTUgARHS5FRjGP3OrwUiX64=>
ERROR: SHA-256 digest of classes2.dex does not match the digest specified in META-INF/MANIFEST.MF. Expected: <VP6C33A0DM40xWvoMP/5VuN5l2xJuS9gnu//uhzzpQA=>, actual: <B/3V7v91IF5/q0+a+ighqdP8PIbx/U9EdwLgCJKN7Ts=>

Attached here: https://gitlab.com/snippets/1945581

This is the APK with disorderfs.

Metadata used:

  - versionName: '1.992'
    versionCode: 992
    commit: '1.992'
    subdir: app
    submodules: true
    init:
      - cd ../..
      - mv eu.faircode.email eu.faircode.email_underlying
      - mkdir eu.faircode.email
      - disorderfs --sort-dirents=yes --reverse-dirents=no eu.faircode.email_underlying
        eu.faircode.email
      - cd eu.faircode.email
    gradle:
      - full
    prebuild: sed -i -e '/keystoreProperties/d' build.gradle
    ndk: r21

We do a clone then a hard cleaning dffx, same for modules (umm maybe https://gitlab.com/fdroid/fdroidserver/issues/662). Your workflow does the same?

I did build with:

./gradlew clean assembleGitHub

Unfortunately the manifest files are binary files and there are many differences. I don’t know why :frowning:

https://email.faircode.eu/AndroidManifest-fdroid.xml.bin

https://email.faircode.eu/AndroidManifest-github.xml.bin

Yes, I never saw them as readable XML file unless one uses a special unpacker.

So I’m extracting both from APKs with https://ibotpeaches.github.io/Apktool/ as readable XML

The only difference is the Bugsnag BUILD_UUID

/LE: the extracted smali is plenty different but maybe you can judge why

Which file is the Bugsnag BUILD_UUID in?

I have no idea why the classes (smali) are different.
If the source set is the same, the classes should be the same.

GitHub / F-Droid:

classes.dex 9.3 MB / 9.5 MB
classes2.dex 1.9 / 4.8 MB

So, there is a huge size difference for some reason.

Is F-Droid using proguard? To reduce size, not to obfuscated, these rules are used:

AndroidManifest.xml

As far as I know yes.

Looking at apktool output there’s apktool.yml for your APK

!!brut.androlib.meta.MetaInfo
apkFileName: FairEmail-v1.992-full-github.apk
compressionType: false
doNotCompress:
- resources.arsc
- png
isFrameworkApk: false
packageInfo:
  forcedPackageId: '127'
  renameManifestPackage: null
sdkInfo:
  minSdkVersion: '21'
  targetSdkVersion: '29'
sharedLibrary: false
sparseResources: false
unknownFiles:
  LICENSES: '8'
  billing.properties: '8'
  biweekly/biweekly.license: '8'
  biweekly/biweekly.properties: '8'
  biweekly/commons-codec.license: '8'
  biweekly/google-rfc-2445.license: '8'
  biweekly/messages.properties: '8'
  org/bouncycastle/x509/CertPathReviewerMessages.properties: '8'
  org/bouncycastle/x509/CertPathReviewerMessages_de.properties: '8'
  org/commonmark/internal/util/entities.properties: '8'
usesFramework:
  ids:
  - 1
  tag: null
version: 2.4.1
versionInfo:
  versionCode: '992'
  versionName: '1.992'

vs for F-Droids APK

!!brut.androlib.meta.MetaInfo
apkFileName: FairEmail-v1.992-full-release-unsigned.apk
compressionType: false
doNotCompress:
- resources.arsc
- META-INF/androidx.activity_activity.version
- META-INF/androidx.annotation_annotation-experimental.version
- META-INF/androidx.appcompat_appcompat-resources.version
- META-INF/androidx.appcompat_appcompat.version
- META-INF/androidx.arch.core_core-runtime.version
- META-INF/androidx.biometric_biometric.version
- META-INF/androidx.browser_browser.version
- META-INF/androidx.cardview_cardview.version
- META-INF/androidx.coordinatorlayout_coordinatorlayout.version
- META-INF/androidx.core_core.version
- META-INF/androidx.cursoradapter_cursoradapter.version
- META-INF/androidx.customview_customview.version
- META-INF/androidx.documentfile_documentfile.version
- META-INF/androidx.drawerlayout_drawerlayout.version
- META-INF/androidx.exifinterface_exifinterface.version
- META-INF/androidx.fragment_fragment.version
- META-INF/androidx.interpolator_interpolator.version
- META-INF/androidx.lifecycle_lifecycle-extensions.version
- META-INF/androidx.lifecycle_lifecycle-process.version
- META-INF/androidx.lifecycle_lifecycle-runtime.version
- META-INF/androidx.lifecycle_lifecycle-service.version
- META-INF/androidx.lifecycle_lifecycle-viewmodel-savedstate.version
- META-INF/androidx.lifecycle_lifecycle-viewmodel.version
- META-INF/androidx.loader_loader.version
- META-INF/androidx.localbroadcastmanager_localbroadcastmanager.version
- META-INF/androidx.paging_paging-runtime.version
- META-INF/androidx.preference_preference.version
- META-INF/androidx.recyclerview_recyclerview.version
- META-INF/androidx.savedstate_savedstate.version
- META-INF/androidx.sqlite_sqlite-framework.version
- META-INF/androidx.sqlite_sqlite.version
- META-INF/androidx.swiperefreshlayout_swiperefreshlayout.version
- META-INF/androidx.transition_transition.version
- META-INF/androidx.vectordrawable_vectordrawable-animated.version
- META-INF/androidx.vectordrawable_vectordrawable.version
- META-INF/androidx.versionedparcelable_versionedparcelable.version
- META-INF/androidx.viewpager2_viewpager2.version
- META-INF/androidx.viewpager_viewpager.version
- META-INF/androidx.work_work-runtime.version
- META-INF/com.google.android.material_material.version
- META-INF/javamail.default.address.map
- META-INF/services/java.nio.charset.spi.CharsetProvider
- png
isFrameworkApk: false
packageInfo:
  forcedPackageId: '127'
  renameManifestPackage: null
sdkInfo:
  minSdkVersion: '21'
  targetSdkVersion: '29'
sharedLibrary: false
sparseResources: false
unknownFiles:
  LICENSES: '8'
  billing.properties: '8'
  biweekly/biweekly.license: '8'
  biweekly/biweekly.properties: '8'
  biweekly/commons-codec.license: '8'
  biweekly/google-rfc-2445.license: '8'
  biweekly/messages.properties: '8'
  org/bouncycastle/x509/CertPathReviewerMessages.properties: '8'
  org/bouncycastle/x509/CertPathReviewerMessages_de.properties: '8'
  org/commonmark/internal/util/entities.properties: '8'
usesFramework:
  ids:
  - 1
  tag: null
version: 2.4.1
versionInfo:
  versionCode: '992'
  versionName: '1.992'

Helpful??

So, there is meta info missing in the F-Droid build, but why?

Another possible problem is that for the GitHub build R8 is explicitly disabled:

I see this during build:

INFO: Scanning source for common problems...
INFO: Removing gradle-wrapper.jar at gradle/wrapper/gradle-wrapper.jar
INFO: Creating source tarball...
INFO: Building Gradle project...
DEBUG: Directory: build/eu.faircode.email/app
DEBUG: > /home/strech/fdroidserver-1106/gradlew-fdroid assembleFullRelease
Found 5.4.1 via distributionUrl
Running /home/strech/.cache/fdroidserver/gradle/5.4.1/bin/gradle assembleFullRelease

> Configure project :app
WARNING: DSL element 'useProguard' is obsolete and will be removed soon. Use 'android.enableR8' in gradle.properties to switch between R8 and Proguard..
WARNING: DSL element 'useProguard' is obsolete and will be removed soon. Use 'android.enableR8' in gradle.properties to switch between R8 and Proguard..
WARNING: DSL element 'useProguard' is obsolete and will be removed soon. Use 'android.enableR8' in gradle.properties to switch between R8 and Proguard..

…strange?

And

> Transform artifact classes.jar (project :openpgp-api) with DexingWithClasspathTransform
Injecting the input artifact of a transform as a File has been deprecated. This is scheduled to be removed in Gradle 6.0. Declare the input artifact as Provider<FileSystemLocation> instead.

Would these differences be if I’m building the wrong flavour?

eg. the APK above is with gradle: full as the rest of the builds.

But you said ./gradlew clean assembleGitHub which means that I need to setup gradle: github I guess, but for me fails with Task 'assembleGithubRelease' not found in project ':app'.

Mastodon