Hi, I did plenty of patches this month. I spent a week in Hamburg on the r-b-world-summit, having good discussions and finding nice patches from Debian (thanks Mattia). With ring0 being reproducible, I concentrated on ring1 packages. I dropped efl and mono-core that are not needed there. I started to collect various details about my R-B-OS in https://en.opensuse.org/openSUSE:Reproducible_openSUSE/Part1 and https://en.opensuse.org/openSUSE:Reproducible_openSUSE/Part2 I worked with Fridrich Strba to greatly improve the reproducibility of java-related packages. This cut the number of remaining packages with issues in half. He made nice toolchain-patches for ant, xmvn and several maven-plugins, mostly for normalizing jar mtimes. https://github.com/mesonbuild/meson-python/issues/671 also stands out, because this affects at least 6 python packages. I have a workaround now. The kernel became reproducible after switching generation of Sphinx docs and BTF to -j1 to workaround race-conditions (partial fixes for the races exist). Two major items on my todo-list are: * write an rb-checker bot that tests submissions to Factory * fix non-determinism in emacs .pdmp files ( https://mail.gnu.org/archive/html/emacs-devel/2024-10/msg00004.html ) ; the alternative would be to drop it from my R-B-OS and lose tools that depend on it: a2ps anthy autoconf avfs caml-mode ddskk global gnugo gnuplot idutils librep maildir-utils notmuch psgml quilt rtags scheme48 supercollider tamago translate-shell uim vagrant xdp-tools xemacs-packages xslide Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/2... Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2024-08/ I uploaded https://rb.zq1.de/compare.factory-20241002/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15806 (+98) build-tried: 15741 (+100) build-failed: 12 (+5) build-n-a: 180 (+0) build-succeeded: 15549 (+95) build-official-failed+na: 52 (-106) build-compare-failed: 153 (+1) build-compare-succeeded: 15396 (+94) verify-failed: 255 (+4) verified-semi-reproducible: 15106 (+95) verified-bit-identical: 13634 (-12) bit-by-bit-identical: 15292 (+467) not-bit-by-bit-identical: 258 (-366) not-bit-by-bit-identicalcheck: 257 (-372) https://rb.zq1.de/compare.factory-20241002/graph.png shows the change over time https://rb.zq1.de/compare.factory-20241002/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 3 were in ring0 27 were in ring1 That makes it 30/4043 => 0.74 % which is below the overall average of 153/15549 => 0.98 % 258/15549 => 1.66 % of packages are not perfectly reproducible package notes: ==> ./agama-integration-tests/.rb.notes <== => https://github.com/openSUSE/agama/pull/1576 (old: SR 1194375) random port in .lock file ==> ./binutils/.rb.notes <== #=> SR 1171128 PGO from --enable-pgo-build=lto no problem? ==> ./biome/.rb.notes <== rust/llvm some order issue visible in strings diff ==> ./blaspp/.rb.notes <== #=> SR 1194838 => https://github.com/icl-utk-edu/blaspp/pull/87 ==> ./ca-certificates-mozilla/.rb.notes <== #=> SR 1192626 = https://bugzilla.opensuse.org/show_bug.cgi?id=1229003 date /usr/share/factory/var/lib/ca-certificates/java-cacerts differs at offset '30' (Java KeyStore) ca-certificates-mozilla:ca-certificates-mozilla-prebuilt unknown - likely timestamp in milliseconds ==> ./clutter/.rb.notes <== = https://gitlab.gnome.org/GNOME/libxslt/-/issues/123 = https://bugzilla.gnome.org/show_bug.cgi?id=751621 needs toolchain xsltproc patch for stable IDs ==> ./colord/.rb.notes <== = https://github.com/omgovich/colord/issues/122 https://bugzilla.opensuse.org/show_bug.cgi?id=1217747 .icc only-CPU ==> ./contrast/.rb.notes <== unknown rust/llvm => SR 1203242 contrast FTBFS-nocheck ==> ./crash/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230281 parallelism, race ==> ./descent3/.rb.notes <== => descent3 https://github.com/DescentDevelopers/Descent3/pull/606 mtime/date = archive with embedded mtimes from ["../tools/HogMaker", "/home/abuild/rpmbuild/BUILD/Descent3-616f921e97b0fb4745a2d36de149c737bf720214/build/Descent3/d3-linux.hog", "/home/abuild/rpmbuild/BUILD/Descent3-616f921e97b0fb4745a2d36de149c737bf720214/scripts/data/fullhog/d3-linux-fullhog.txt", "/home/abuild/rpmbuild/BUILD/Descent3-616f921e97b0fb4745a2d36de149c737bf720214/build/scripts"] ==> ./difftastic/.rb.notes <== rust/llvm ==> ./elixir/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1205134 FTBFS-j1 stuck osc build --vm-type=kvm --clean --noservice -j1 standard ==> ./emacs/.rb.notes <== = https://mail.gnu.org/archive/html/emacs-devel/2024-10/msg00004.html .pdmp cd ~/rpmbuild/BUILD/emacs-29.4/src && ./temacs "--__aslr-disabled" "-batch" "--no-build-details" "-l" "loadup" "--temacs=pdump" "--bin-dest" "/usr/bin/" "--eln-dest" "/usr/lib64/emacs/29.4/" && md5sum emacs.pdmp ==> ./fonttosfnt/.rb.notes <== #=> SR 1190940 1190278 => https://gitlab.freedesktop.org/xorg/app/fonttosfnt/-/merge_requests/22 toolchain, date ==> ./ghostscript/.rb.notes <== date in .h + https://bugs.ghostscript.com/show_bug.cgi?id=696765 ==> ./git-annex/.rb.notes <== parallelism ==> ./git-credential-keepassxc/.rb.notes <== rust codegen-units, low-entropy +_ZN4clap5error5Error7for_app17h3a0e3b5c590fcdc5E.llvm.17478145784603506657 ==> ./glycin-loaders/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230879 FTBFS-j1 osc build --vm-type=kvm --noservice -j1 standard ==> ./groff/.rb.notes <== groff:full postscript toolchain date+time in /usr/share/doc/packages/groff/pdf/pdfmark.pdf ==> ./grpc/.rb.notes <== #=> SR 1102203 grpc drop log #=>https://github.com/grpc/grpc/pull/35687 ==> ./gstreamer-plugins-rs/.rb.notes <== = https://gitlab.freedesktop.org/gstreamer/gst-plugins-rs/-/issues/599 date ; rust/llvm asm diff variations in /usr/lib64/pkgconfig/gstwebrtchttp.pc ==> ./helm/.rb.notes <== FTBFS-2028-04-05 2032-08-21 SSL ==> ./joker/.rb.notes <== #=> https://github.com/candid82/joker/commit/d2817bb18cd9cc0274460113a97dc87f261... #= https://github.com/candid82/joker/issues/491 hash random order ==> ./kernel-source/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230414 BTF = https://lore.kernel.org/linux-doc/33018311-0bdf-4258-b0c0-428a548c710d@suse.... doc variation from Sphinx -j auto ==> ./lapackpp/.rb.notes <== #=> SR 1194839 => https://github.com/icl-utk-edu/lapackpp/pull/68 hostname in /usr/include/lapack/defines.h from cmake + defines.h.in ==> ./less/.rb.notes <== #=> less https://github.com/gwsw/less/pull/567 date ==> ./libcamera/.rb.notes <== = https://bugs.libcamera.org/show_bug.cgi?id=233 = https://bugzilla.opensuse.org/show_bug.cgi?id=1217690 random: private key signatures of modules /usr/lib64/libcamera-suse.so.3 differs in ELF section .rodata ==> ./libdb-4_8/.rb.notes <== #=> SR 1190247 1187675 /usr/share/java/db-4.8.30.jar mtimes from by pid=171773 dir=/home/abuild/rpmbuild/BUILD/db-4.8.30/build_nptl/classes exec="/usr/bin/jar", ["jar", "cfm", "../db.jar", "../../dist/../java/jarManifestEntries", "./com/sleepycat"] - started ==> ./libgtop/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230850 uname -r ==> ./libsamplerate/.rb.notes <== => SR 1202178 random/PGO: libsamplerate.so has random diffs from .o ==> ./libxslt/.rb.notes <== => SR 1202234 = https://gitlab.gnome.org/GNOME/libxslt/-/issues/37 => https://gitlab.gnome.org/GNOME/libxslt/-/commit/c45ed81aeb50a7fb6799a166270d... ==> ./llvm17/.rb.notes <== ASLR left in .so = https://github.com/llvm/llvm-project/issues/72206 Marvin investigates. Partially from ASLR - uninitialized memory? ==> ./lua-lmod/.rb.notes <== => SR 1195841 => https://github.com/TACC/Lmod/pull/702 date+time from `date` /usr/share/lmod/8.7.37/libexec/Version.lua differs (ASCII text) ==> ./lua-luarepl/.rb.notes <== => SR 1204160 lua-luarepl FTBFS OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --noservice --clean --vm-type=kvm -M=lua51 standard ==> ./mayavi/.rb.notes <== tvtk_classes.zip varies (mtimes?) from python build_wheel ==> ./nfdump/.rb.notes <== #=> SR 1163778 #=> https://github.com/phaag/nfdump/pull/482 date, solved differently ==> ./nodejs22/.rb.notes <== ASLR+CPU? ==> ./nvidia-persistenced/.rb.notes <== nvidia-persistenced date in /usr/share/man/man1/nvidia-persistenced.1.gz ==> ./nvidia-xconfig/.rb.notes <== => SR 1203885 nvidia-xconfig https://github.com/NVIDIA/nvidia-xconfig/pull/3 date in /usr/share/man/man1/nvidia-xconfig.1.gz ==> ./openttd/.rb.notes <== ASLR into build-id ==> ./openwsman/.rb.notes <== = openwsman https://bugzilla.opensuse.org/show_bug.cgi?id=1231038 FTBFS ==> ./parboiled/.rb.notes <== minor jar, strip-nd ==> ./perl/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230137 embeds build machine kernel version => must build in KVM /usr/lib/perl5/5.40.0/x86_64-linux-thread-multi/CORE/config.h differs (C source, ASCII text) ==> ./pinentry/.rb.notes <== => SR 1202479 make efl droppable ==> ./python-PyGithub/.rb.notes <== => https://github.com/PyGithub/PyGithub/pull/3045 FTBFS 2024-11-25 found range good=1732496089 bad=1732498014 ==> ./python-chroma-hnswlib/.rb.notes <== => SR 1202316 python-chroma-hnswlib CPU ==> ./python-emoji/.rb.notes <== FTBFS [ 91s] tests/test_zwj_remove.py::test_non_rgi_zwj_replace PASSED [100%] ==> ./python-endesive/.rb.notes <== #=> https://github.com/m32/endesive/commit/47f751ad877e45d7954fdead5482ad164ae8f... #= https://github.com/m32/endesive/issues/173 FTBFS-2040 ==> ./python-gobject/.rb.notes <== python-gobject _gi.cpython-310-x86_64-linux-gnu.so mesonpy toolchain random filename => https://build.opensuse.org/request/show/1204522 ==> ./python-libcst/.rb.notes <== => https://github.com/Instagram/LibCST/pull/1213 codegen-units=1 rust/llvm /usr/lib64/python3.11/site-packages/libcst/native.cpython-311-x86_64-linux-gnu.so ==> ./python-meson-python/.rb.notes <== = https://github.com/mesonbuild/meson-python/issues/671 toolchain for python-contourpy python-gobject python-numpy python-pandas python-scikit-learn python-scipy ==> ./python-numpy/.rb.notes <== mesonpy toolchain random ==> ./python-orjson/.rb.notes <== ?#solved in 3.10.7? machine code diffs from rustc+gcc13 ==> ./python-pandas/.rb.notes <== mesonpy toolchain random; random .pyc - ref counters? ==> ./python-pyHanko/.rb.notes <== = https://github.com/MatthiasValvekens/pyHanko/issues/472 FTBFS-2030 found range good=1893454427 bad=1893456352 ==> ./python-pygraphviz/.rb.notes <== = https://github.com/pygraphviz/pygraphviz/issues/541 captures execution time ==> ./python-pyhanko-certvalidator/.rb.notes <== FTBFS-2031 [ 57s] E pyhanko_certvalidator.errors.ExpiredError: The path could not be validated because intermediate certificate 1 expired 2030-12-31 08:30:00Z ==> ./python-rdflib/.rb.notes <== FTBFS :doc (pinged packager in Slack) ==> ./python-scikit-learn/.rb.notes <== mesonpy toolchain random parallelism via cython+openmp - similar to https://github.com/yt-project/yt/issues/4611 - ?does it need JOBLIB_MULTIPROCESSING=0 ? ==> ./python-scipy/.rb.notes <== mesonpy toolchain random ==> ./python-yt/.rb.notes <== = https://github.com/yt-project/yt/issues/4611 parallelism # old: ==> ./python312/.rb.notes <== => SR 1204725 python312 drop .pyc from doc = https://github.com/python/cpython/issues/124851 FTBFS-2038 ==> ./python313/.rb.notes <== = https://github.com/python/cpython/issues/122433 FTBFS-j1 OSC_BUILD_ROOT=/var/tmp/build-root.$slot time osc build --noservice --vm-type=kvm --clean standard ==> ./qqc2-breeze-style6/.rb.notes <== parallelism - source diff in BreezeStyle_Button_qml.cpp from "/usr/libexec/qt6/qmlcachegen", "--bare", "--resource-path", "/qt/qml/org/kde/breeze/Button.qml", "-I", "/home/abuild/rpmbuild/BUILD/qqc2-breeze-style-6.1.5/build/bin/", "-I", "/usr/lib64/qt6/qml", "-i", "/home/abuild/rpmbuild/BUILD/qqc2-breeze-style-6.1.5/build/bin/org/kde/breeze/qmldir", "--resource", "/home/abuild/rpmbuild/BUILD/qqc2-breeze-style-6.1.5/build/style/qtquickcontrols/.qt/rcc/qmake_org_kde_breeze.qrc", "--resource", "/home/abuild/rpmbuild/BUILD/qqc2-breeze-style-6.1.5/build/style/qtquickcontrols/.qt/rcc/BreezeStyle_raw_qml_0.qrc", "-o", "/home/abuild/rpmbuild/BUILD/qqc2-breeze-style-6.1.5/build/style/qtquickcontrols/.rcc/qmlcache/BreezeStyle_Button_qml.cpp", "/home/abuild/rpmbuild/BUILD/qqc2-breeze-style-6.1.5/style/qtquickcontrols/Button.qml" ==> ./resources/.rb.notes <== unknown rust/llvm ==> ./rmw/.rb.notes <== #=> https://github.com/theimpossibleastronaut/rmw/pull/442 = https://github.com/theimpossibleastronaut/rmw/issues/439 FTBFS-2038 found range 2147482293 -> 2147484218 ==> ./rpm/.rb.notes <== => https://github.com/rpm-software-management/rpm/pull/2762 = https://github.com/rpm-software-management/rpm/issues/932 Cannot set Build Date from changelog anymore ==> ./runawk/.rb.notes <== => SR 1199220 nocheck ==> ./scap-security-guide/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1230361 date ==> ./scheme48/.rb.notes <== FTBFS (from gcc14?) unreproducible dump ==> ./seahorse/.rb.notes <== = https://gitlab.gnome.org/GNOME/seahorse/-/issues/394 parallelism, bug Binary files /var/tmp/build-root.12/.mount/home/abuild/rpmbuild/BUILD/seahorse-43.0+22/x86_64-suse-linux/src/seahorse.p/meson-generated_application.c.o and /var/tmp/build-root.12b/.mount/home/abuild/rpmbuild/BUILD/seahorse-43.0+22/x86_64-suse-linux/src/seahorse.p/meson-generated_application.c.o differ ==> ./starship/.rb.notes <== maybe some low-entropy (filesys) issue left => https://github.com/starship/starship/pull/5352 filesys readdir, rust ==> ./subversion/.rb.notes <== => SR 1203785 minor jar mtimes /usr/lib64/svn-javahl/svn-javahl.jar ==> ./taigo/.rb.notes <== = taigo https://bugzilla.opensuse.org/show_bug.cgi?id=1231041 FTBFS ==> ./tycho/.rb.notes <== FTBFS-2038 osc build --noservice --clean --vm-type=kvm --build-opt=--vm-custom-opt="-rtc base=2038-01-25T00:00:00" -M=bootstrap java maven jar ==> ./wine/.rb.notes <== parallelism ==> ./xen/.rb.notes <== => SR 1202206 = https://bugzilla.opensuse.org/show_bug.cgi?id=1230856 date+time in debugsource ==> ./xindy/.rb.notes <== random xindy.mem CLISP memory image data = https://sourceforge.net/p/xindy/bugs/65/ report FTBFS-2036 ==> ./xmlsec1/.rb.notes <== FTBFS-2034-07-16 found range good=2036670842 bad=2036672767