pure luck
we have some contrib machines so we can build, besides the CI. The VMs allow us to enter and look an the state directly, maybe try some tricks there. As opposed to the Ci or the main build server where you get a log and an APK (if so).
now, not all machines are the same, we have several with 24cores/32GbRAM but also a smaller one with 8 cores/16Gb.
In past apps, we saw the same error, baseline.prof file non-deterministic, so when testing I used the smaller machine and its 6cores/12Gb RAM VM and was confused to see that… the error was not present here.
Then retried on the bigger ones and voila… error again
difference being cores? let’s try limiting
badum-tss… discovered that even on 24cores machine, IF WE LIMIT to 6 (or 7 for some reason), suddenly apps that errored out on main server either error out less (say 3 times out of ten vs ten) or not error out at all
