Hi, This month, I declared https://en.opensuse.org/openSUSE:Reproducible_openSUSE/Part1 - the Minimal part of the R-B-OS (sponsored by the NLNet foundation) completed (minus the VM image) and am pretty close to complete https://en.opensuse.org/openSUSE:Reproducible_openSUSE/Part2 (not fully satisfied with the VM image, but that was out of scope here anyway) I started to use https://build.opensuse.org/package/show/home:rb-checker/notes for online note-taking about found issues and my rb-checker bot now checks all submissions that go to Factory through "adi" staging projects. This already found the first two regressions. And I sent many nice patches, including 3 toolchain patches that fix multiple packages: * https://build.opensuse.org/request/show/1230042 scons * https://build.opensuse.org/request/show/1233216 cargo-packaging * https://github.com/openSUSE/hpc/pull/12 suse-hpc I also did two more test runs over the ring1 packages and found two new surprising classes of issues that broke reproducibility. 1. .changes file entries with certain timezones (WEST, IST, UCT) were mis-parsed by osc build on one machine. Replacing these with UTC was easy enough. 2. packages such as atftp, ppp, texlive and sendmail use the new system-user-foo packages, but their user-id differed between OBS and local (no-preinstallimage) builds and was embedded into cpio headers. Maybe it was already fixed in sysuser-tools after my fork? While investigating an issue with r-b, I also stumbled across a silent data corruption: https://bugzilla.opensuse.org/show_bug.cgi?id=1234897 This is far from the first. And it shows once again, that r-b is not only good for security, but also for quality. Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/K... Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2024-11/ I uploaded https://rb.zq1.de/compare.factory-20241231/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15907 (+109) build-tried: 15854 (+125) build-failed: 25 (+0) build-n-a: 175 (+0) build-succeeded: 15654 (+125) build-official-failed+na: 151 (-182) build-compare-failed: 161 (+12) build-compare-succeeded: 15493 (+113) verify-failed: 250 (-57) verified-semi-reproducible: 15100 (+56) verified-bit-identical: 13394 (+123) bit-by-bit-identical: 15415 (+131) not-bit-by-bit-identical: 240 (-3) not-bit-by-bit-identicalcheck: 239 (-6) https://rb.zq1.de/compare.factory-20241231/graph.png shows the change over time https://rb.zq1.de/compare.factory-20241231/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 1 were in ring0 25 were in ring1 That makes it 26/4043 => 0.64 % which is below the overall average of 161/15654 => 1.03 % 240/15654 => 1.53 % of packages are not perfectly reproducible package notes: ==> ./LiE/.rb.notes <== => SR 1225975 ASLR+toolchain ghostscript ==> ./Rivet/.rb.notes <== #=> https://gitlab.com/hepcedar/rivet/-/issues/255 FTBFS-j1 #osc build --vm-type=kvm --noservice -j1 --clean standard ==> ./alex/.rb.notes <== debuginfo="--debuginfo" rbk nothing helps ==> ./bpftool/.rb.notes <== /usr/src/debug/bpftool-libbpf-v7.3.0-sources/src/pid_iter.skel.h varies from kernel ==> ./cargo-packaging/.rb.notes <== => SR 1233216 toolchain for rusty_v8 ==> ./ccache/.rb.notes <== => https://github.com/ccache/ccache/pull/1525 = https://github.com/ccache/ccache/issues/1524 FTBFS-2038 found range good=2145915653 bad=2145917578 ==> ./cockpit/.rb.notes <== => https://github.com/cockpit-project/cockpit/pull/21460 readdir FTBFS-2030-07-14 SSL ==> ./collectd/.rb.notes <== => SR 1231851 minor jar ==> ./colord/.rb.notes <== = https://github.com/hughsie/colord/issues/174 https://bugzilla.opensuse.org/show_bug.cgi?id=1217747 .icc only-CPU AVX2 ; toolchain from lcms2? ==> ./conky/.rb.notes <== => https://github.com/brndnmtthws/conky/pull/2096 date ==> ./cosmic-ext-tasks/.rb.notes <== see cosmic issue ==> ./cosmic-ext-tweaks/.rb.notes <== see cosmic issue ==> ./cosmic-files/.rb.notes <== see cosmic issue ==> ./cosmic-store/.rb.notes <== see cosmic issue ==> ./dealii/.rb.notes <== doxygen parallelism/race ==> ./deepin-daemon/.rb.notes <== => SR 1230049 tar mtime / meta /usr/share/dde-daemon/polkit.tar.gz ==> ./deepin-file-manager/.rb.notes <== => SR 1230061 tar ==> ./esbuild/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1234374 .note.go.buildid + .note.gnu.build-id from random go tmp path debuginfo=--debuginfo rbk ==> ./examine/.rb.notes <== cosmic issue ==> ./ghc-zlib/.rb.notes <== .note.gnu.build-id debuginfo=--debuginfo rbk ==> ./grpc/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1234751 uname -r vmtype=--vm-type=podman rbk ==> ./hawk2/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230275 date, other .cache files vary for unknown reason ==> ./ibmtss/.rb.notes <== => SR 1228038 1227925 => https://github.com/kgoldman/ibmtss/commit/3a17ac01bea73d3568272d61b895a16a0b... = https://sourceforge.net/p/ibmtpm20tss/tickets/49/ FTBFS-2026-02-20 found range good=1771613260 bad=1771615185 (2026-02-20T19:19:45) ==> ./icedtea-web/.rb.notes <== => SR 1227576 jar mtime ==> ./installation-images/.rb.notes <== bug or ordering in usr/share/debuginfodeps/root ==> ./ispc/.rb.notes <== FTBFS-CPU --build-opt=--vm-custom-opt=-cpu qemu64 ==> ./java-23-openjdk/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1221224 ==> ./java-atk-wrapper/.rb.notes <== minor jar mtime /usr/lib64/java-atk-wrapper/java-atk-wrapper.jar ==> ./julia/.rb.notes <== stuck once in -j1 build ==> ./kicad/.rb.notes <== => https://gitlab.com/kicad/code/kicad/-/merge_requests/2087 tar from ==> ./less/.rb.notes <== => SR 1231194 #=> less https://github.com/gwsw/less/pull/567 date ==> ./libmfx-gen/.rb.notes <== vmtype=--vm-type=podman rbk uname -r? ==> ./libmfx/.rb.notes <== uname -r in /usr/lib64/mfx/libmfx_hevc_fei_hw64.so vmtype=--vm-type=podman rbk ==> ./libnice/.rb.notes <== FTBFS-j1 ==> ./liboqs/.rb.notes <== uname -r vmtype=--vm-type=podman rbk ==> ./libraw/.rb.notes <== => SR 1227792 fdupes-induced non-determinism ==> ./librespeed-cli/.rb.notes <== => SR 1233732 date ==> ./libsoup/.rb.notes <== FTBFS-2040 found range good=2239732379 bad=2239734304 ==> ./mraa/.rb.notes <== => SR 1229658 minor .jar mtime from jar cvmf ==> ./nst/.rb.notes <== minor date in /usr/share/man/man5/nst.conf.5.gz from ["a2x", "-f", "manpage", "-D", "./doc", "doc/nst-msg.1.adoc"] ==> ./nunit3/.rb.notes <== minor mono dll+exe ==> ./ollama/.rb.notes <== => SR 1230608 gz ==> ./opa-fmgui/.rb.notes <== => SR 1230004 /usr/share/java/opa-fmgui/opa-fmgui.jar differs - timestamps? ==> ./openCryptoki/.rb.notes <== => https://github.com/opencryptoki/opencryptoki/pull/829 https://github.com/opencryptoki/opencryptoki/pull/828 copyright year + date+time in /usr/src/debug/opencryptoki-3.24.0/usr/lib/api/mechtable.c debuginfo=--debuginfo rbk ==> ./openmpi4/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1234014 rpm toolchain, CPU: missing provides oscbuildparams=' -M=gnu-hpc ' rbk ==> ./openttd/.rb.notes <== ASLR into build-id ==> ./openwsman/.rb.notes <== => SR 1228990 minor jar mtimes from /usr/lib64/jvm/java-21-openjdk-21/bin/jar", ["/usr/lib64/jvm/java-21-openjdk-21/bin/jar", "cvf", "openwsman-x86_64-2.8.0.jar", "libjwsman.so", "org/openwsman" ==> ./papi/.rb.notes <== minor gzip mtime from %{hpc_compress_man 3} ==> ./presage/.rb.notes <== => SR 1233892 = https://bugzilla.opensuse.org/show_bug.cgi?id=1234897 ==> ./procps/.rb.notes <== = https://gitlab.com/procps-ng/procps/-/issues/362 FTBFS-2038 uptime --raw test osc build --noservice --clean --vm-type=kvm "--build-opt=--vm-custom-opt=-rtc base=2038-01-20T00:00:00" standard ==> ./python-HyperKitty/.rb.notes <== => https://gitlab.com/mailman/hyperkitty/-/merge_requests/656 = https://gitlab.com/mailman/hyperkitty/-/issues/527 toolchain issue?, random from ["python3.11", "build_static_files/manage.py", "compress", "--force"] ==> ./python-aiosmtpd/.rb.notes <== FTBFS-j1 ==> ./python-exiv2/.rb.notes <== = https://github.com/jim-easterbrook/python-exiv2/issues/44 FTBFS-2038 ==> ./python-pexpect/.rb.notes <== FTBFS-j1 ; osc build --noservice --clean --vm-type=kvm -j1 standard = flaky test https://github.com/pexpect/pexpect/issues/749 ==> ./python-sherpa/.rb.notes <== FTBFS-j1 FTBFS-date?random? rbkt 801 works ==> ./python-yt/.rb.notes <== = https://github.com/yt-project/yt/issues/4611 parallelism # old: ==> ./qt6-webengine/.rb.notes <== copyright year: /usr/src/debug/qtwebengine-everywhere-src-6.7.2/build/src/core/RelWithDebInfo/x86_64/gen/qtwebengine/common/extensions/api/webrtc_desktop_capture_private.h Copyright 2041 The Chromium Authors Qt toolchain issue ; hard to debug because of slow compile time ==> ./resources/.rb.notes <== unknown rust/llvm LTO becomes reproducible with lto="off" ==> ./sad/.rb.notes <== = https://github.com/ms-jpq/sad/issues/359 rust+python random uuid https://github.com/ms-jpq/sad/blob/6c65d52211e79298c0e1b1496a7287e98cb8e813/... ==> ./scap-security-guide/.rb.notes <== => SR 1229844 = https://bugzilla.opensuse.org/show_bug.cgi?id=1230361 date ==> ./scons/.rb.notes <== => SR 1230042 from https://github.com/SCons/scons/pull/4261 / https://github.com/SCons/scons/pull/4239 => https://github.com/SCons/scons/pull/3312 hostname ==> ./scummvm/.rb.notes <== ASLR in debuginfo /usr/bin/scummvm ==> ./sendmail/.rb.notes <== => SR 1231690 1231685 = https://bugzilla.opensuse.org/show_bug.cgi?id=1234629 uname -r debuginfo=--debuginfo vmtype=--vm-type=podman rbk ==> ./singularity-ce/.rb.notes <== random go.buildid / debuginfo ; from /dev/(u)random debuginfo="--debuginfo" rbk ==> ./static-initrd/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1234709 date+host+CPU => SR 1232164 ==> ./swipl/.rb.notes <== date; => https://github.com/SWI-Prolog/swipl/pull/32 FTBFS-2029 swipl found range good=1867791402 bad=1867793326 (2029-03-09T23:28:46) ==> ./swtpm/.rb.notes <== => SR 1229015 nocheck ==> ./tiny/.rb.notes <== https://github.com/osa1/tiny/issues/437 rust HashMap order from HashMap https://github.com/osa1/tiny/blob/ee8615a55256b242b02e6f8a2350ca7f39aca517/c... ==> ./toipe/.rb.notes <== rust/llvm ? ==> ./trippy/.rb.notes <== rust/llvm? ==> ./vtk/.rb.notes <== => SR 1231633 minor jar mtime /usr/lib64/java/vtk.jar ==> ./xdg-desktop-portal/.rb.notes <== => SR 1234111 xdg-desktop-portal Sphinx doctrees