Hi, In August, I had two weeks off with my family, so I made fewer patches, but had the machines build all of ring0 reproducibly for the first time and did start builds of ring1 (500GB binaries each). In ring0, I also managed to build packages with OBS, osc and pbuild and all methods agreed on the build-result. That is with debuginfo enabled everywhere to keep compatibility with Tumbleweed. Only pesign-obs-integration is still causing diffs in OBS. With my reproducibleopensuse tools (devel branch), the lines to reproduce are: osc co home:bmwiedemann:reproducible:distribution:ring0/zstd && cd $_ nachbau # or for a double-build with more variations: debuginfo="--debuginfo --baselibs" repo=standard \ project=home:bmwiedemann:reproducible:distribution:ring0 \ rbk and with pbuild (patched with cd /usr/lib/build && curl https://github.com/openSUSE/obs-build/commit/f47374bc2027b3a21c2232e3d88b2b2... | patch -p1 ), it is osc co home:bmwiedemann:reproducible:distribution:ring0 && cd $_ ln -sf 000pbuildconf/_* . pbuild --kvm --vm-memory=9000 --vm-disk-size=30000 --jobs 4 \ --buildjobs 4 --no-checks --release 1.1 --baselibs sha256sums To reach a 100% reproducible ring0, some trade-offs had to be accepted. One is that we use %do_profiling 0 which costs some ~8% performance in gcc, bash, python and others that used profile-guided-optimization (PGO). Another is that LTO triggered a reproducibility issue in llvm's libomp, so all versions of llvm build without LTO here, probably losing ~10% of performance when compiling with these as well. There are also some unmerged patches integrated: * SR 1192491 rpm-config-SUSE * https://github.com/rpm-software-management/rpm/pull/2762 * https://github.com/apache/xmlgraphics-fop/pull/65 for pdf-rendering with a PoC/WIP portion for deterministic UUIDs All this brings me close to the first goal of the project: to have all the 328 ingredient rpms for a minimalVM image build 100% bit-reproducible. https://build.opensuse.org/package/show/home:bmwiedemann:reproducible:distri... needs some parts from ring1 and those still need checking, but hopefully not much is left to fix. Image-creation itself is not yet reproducible with random IDs in filesystems and maybe even mtimes. In past years, %post scripts also caused some non-determinism. However, this part is out-of-scope for this project. Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/D... Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2024-07/ I uploaded https://rb.zq1.de/compare.factory-20240904/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15708 (+42) build-tried: 15641 (+39) build-failed: 7 (-6) build-n-a: 180 (-6) build-succeeded: 15454 (+51) build-official-failed+na: 158 (+133) build-compare-failed: 152 (-2) build-compare-succeeded: 15302 (+53) verify-failed: 251 (+2) verified-semi-reproducible: 15011 (+67) verified-bit-identical: 13646 (+55) bit-by-bit-identical: 14825 (+46) not-bit-by-bit-identical: 624 (-1) not-bit-by-bit-identicalcheck: 629 (+5) https://rb.zq1.de/compare.factory-20240904/graph.png shows the change over time https://rb.zq1.de/compare.factory-20240904/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 3 were in ring0 28 were in ring1 That makes it 31/4043 => 0.77 % which is below the overall average of 152/15454 => 0.98 % 624/15454 => 4.04 % of packages are not perfectly reproducible package notes: ==> ./agama-integration-tests/.rb.notes <== => SR 1194375 random port in .lock file ==> ./apache-arrow/.rb.notes <== OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --noservice --vm-type=kvm --clean standard ==> ./binutils/.rb.notes <== => SR 1171128 PGO from --enable-pgo-build=lto no problem? ==> ./blaspp/.rb.notes <== => SR 1194838 => https://github.com/icl-utk-edu/blaspp/pull/87 ==> ./ca-certificates-mozilla/.rb.notes <== => SR 1192626 = https://bugzilla.opensuse.org/show_bug.cgi?id=1229003 date /usr/share/factory/var/lib/ca-certificates/java-cacerts differs at offset '30' (Java KeyStore) ca-certificates-mozilla:ca-certificates-mozilla-prebuilt unknown - likely timestamp in milliseconds ==> ./clamav/.rb.notes <== => SR 1190176 = https://github.com/Cisco-Talos/clamav/issues/1300 FTBFS-2024-07-28 found range good=1722124782 bad=1722126706 ==> ./contrast/.rb.notes <== unknown rust/llvm ==> ./cosmic-edit/.rb.notes <== = https://github.com/pop-os/cosmic-edit/issues/221 hash order issue, rust -4CXlibnamebaseblue of +inf-inf+NaN-NaNopendataRectBlurDx12kindlineDarkFontB4x4B5x4B5x5B6x5B6x6B8x5B8x6B8x8filetextnodeItemmimeGridListNameThinBoldFifocodesizePollInitKeyAKeyBKeyCKeyDKeyEKeyFKeyGKeyHKeyIKeyJKeyKKeyLKeyMKeyNKeyOKeyPKeyQKeyRKeySKeyTKeyUKeyVKeyWKeyXKeyYKeyZHelpHomeMetaCopyFindOpenUndoCellNV12AstcaxisLeftBackRedoAttnPlayEisuSaveCallExitInfoLinkLockWinkDragViewDropPathUserCodeNoneSomeSizelistgridviewpathexeciconDeadIterRgba.gitgapsrectOhos]*/ AtomFullLostMisctrueiterhwndBindWiretimehighdiffundoredocopysavequiteditfindwarnhashdate -> dark ==> ./cosmic-term/.rb.notes <== unknown asm diff - from lto? ==> ./elixir/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1205134 FTBFS-j1 stuck osc build --vm-type=kvm --clean --noservice -j1 standard ==> ./elixir115/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1205134 FTBFS-j1 stuck ==> ./emacs/.rb.notes <== .pdmp from ["./temacs" "--__aslr-disabled" "-batch" "--no-build-details" "-l" "loadup" "--temacs=pdump" "--bin-dest" "/usr/bin/" "--eln-dest" "/usr/lib64/emacs/29.4/"] ==> ./fonttosfnt/.rb.notes <== => SR 1190940 1190278 => https://gitlab.freedesktop.org/xorg/app/fonttosfnt/-/merge_requests/22 toolchain, date ==> ./fractal/.rb.notes <== rust/llvm symbol order variation ==> ./gawk/.rb.notes <== PGO ==> ./git-annex/.rb.notes <== parallelism ==> ./git-credential-keepassxc/.rb.notes <== rust codegen-units, low-entropy +_ZN4clap5error5Error7for_app17h3a0e3b5c590fcdc5E.llvm.17478145784603506657 ==> ./gromacs/.rb.notes <== FTBFS-j1 failed - needs 4+ cores succeeds with parallelism=6 parallelism2=5 multibuildrbkall ==> ./gstreamer-plugins-rs/.rb.notes <== date ; rust/llvm asm diff variations in /usr/lib64/pkgconfig/gstwebrtchttp.pc ==> ./helm/.rb.notes <== FTBFS-2028-04-05 2032-08-21 SSL ==> ./kanidm/.rb.notes <== low-entropy unknown rust/llvm ==> ./lapackpp/.rb.notes <== => SR 1194839 => https://github.com/icl-utk-edu/lapackpp/pull/68 hostname in /usr/include/lapack/defines.h from cmake + defines.h.in ==> ./libdb-4_8/.rb.notes <== => SR 1190247 1187675 /usr/share/java/db-4.8.30.jar mtimes from by pid=171773 dir=/home/abuild/rpmbuild/BUILD/db-4.8.30/build_nptl/classes exec="/usr/bin/jar", ["jar", "cfm", "../db.jar", "../../dist/../java/jarManifestEntries", "./com/sleepycat"] - started ==> ./libguestfs/.rb.notes <== hostname in /usr/lib64/guestfs/supermin.d/base.tar.gz in etc/hosts ==> ./lua-lmod/.rb.notes <== => SR 1195841 => https://github.com/TACC/Lmod/pull/702 date+time from `date` /usr/share/lmod/8.7.37/libexec/Version.lua differs (ASCII text) ==> ./mayavi/.rb.notes <== tvtk_classes.zip varies (mtimes?) from python build_wheel ==> ./openblas/.rb.notes <== => SR 1190320 = https://bugzilla.opensuse.org/show_bug.cgi?id=1228177 CPU = https://bugzilla.opensuse.org/show_bug.cgi?id=1181083 FTBFS-j1 ==> ./pop-launcher/.rb.notes <== => SR 1190933 = https://github.com/pop-os/launcher/issues/230 parallelism ==> ./python-EditorConfig/.rb.notes <== #=> 1191536 .pyc mtime from %check ==> ./python-cfn-lint/.rb.notes <== FTBFS-2024-10-13 OSC_BUILD_ROOT=/var/tmp/build-root.$slot time osc build --noservice --vm-type=kvm --build-opt=--vm-custom-opt="-rtc base=2024-10-14T00:00:00" standard ==> ./python-libcst/.rb.notes <== unknown rust/llvm /usr/lib64/python3.11/site-packages/libcst/native.cpython-311-x86_64-linux-gnu.so ==> ./python-paramiko/.rb.notes <== FTBFS-CPU+j1 ==> ./python-pysnmp/.rb.notes <== => https://github.com/lextudio/pysnmp/pull/35 FTBFS-2038 ==> ./python-pyvo/.rb.notes <== FTBFS-2024-12-29 found range good=1735428808 bad=1735430733 ==> ./python-spyder-notebook/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228441 report FTBFS OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --noservice --vm-type=kvm --clean -j4 standard ==> ./python-torch/.rb.notes <== = python-torch https://bugzilla.opensuse.org/show_bug.cgi?id=1229033 CPU ==> ./python-yt/.rb.notes <== = https://github.com/yt-project/yt/issues/4611 parallelism # old: ==> ./python311/.rb.notes <== PGO only as of 2024-08-01 ==> ./python312/.rb.notes <== see python311 ==> ./python313/.rb.notes <== = https://github.com/python/cpython/issues/122433 FTBFS-j1 OSC_BUILD_ROOT=/var/tmp/build-root.$slot time osc build --noservice --vm-type=kvm --clean standard ==> ./qemu/.rb.notes <== #= https://bugzilla.opensuse.org/show_bug.cgi?id=1221340 FTBFS /usr/share/qemu/slof.bin differs at offset '108' (data) ==> ./rabbitmq-server/.rb.notes <== FTBFS https://build.opensuse.org/package/show/network:messaging:amqp/rabbitmq-serv... ==> ./resources/.rb.notes <== unknown rust/llvm ==> ./rmw/.rb.notes <== #=> https://github.com/theimpossibleastronaut/rmw/pull/442 = https://github.com/theimpossibleastronaut/rmw/issues/439 FTBFS-2038 found range 2147482293 -> 2147484218 ==> ./rpm-config-SUSE/.rb.notes <== => SR 1192491 date+time /usr/lib/rpm/macros.d/macros.opensuse differs (ASCII text) ==> ./seahorse/.rb.notes <== parallelism Binary files /var/tmp/build-root.12/.mount/home/abuild/rpmbuild/BUILD/seahorse-43.0+22/x86_64-suse-linux/src/seahorse.p/meson-generated_application.c.o and /var/tmp/build-root.12b/.mount/home/abuild/rpmbuild/BUILD/seahorse-43.0+22/x86_64-suse-linux/src/seahorse.p/meson-generated_application.c.o differ ==> ./swipl/.rb.notes <== date;FTBFS-2029 swipl found range good=1867791402 bad=1867793326 (2029-03-09T23:28:46) ==> ./tigervnc/.rb.notes <== minor jar mtimes ; from jar call #= https://bugzilla.opensuse.org/show_bug.cgi?id=1208478 RSA key ==> ./weblate/.rb.notes <== => SR 1194546 = https://github.com/WeblateOrg/weblate/issues/8556 FTBFS-2038 toolchain mercurial ERROR: test_upstream_changes_rename (weblate.vcs.tests.test_vcs.VCSHgTest) ==> ./xdg-desktop-portal-cosmic/.rb.notes <== low-entropy order issue --- old /usr/libexec/xdg-desktop-portal-cosmic (objdump) ==> ./zig/.rb.notes <== debuginfo ==> ./zls/.rb.notes <== random path in debuginfo=--debuginfo multibuildrbkall -/usr/src/debug/zls-0.13.0/zig-cache/o/dd3075db6d19db75877b8d14c5f6fc11/version_data_0.13.0.zig 0 (none) 100644 root root 0 4294967295