Hi, This month I did plenty patches again, including one for the python interpreter's gzip lib itself https://github.com/python/cpython/pull/125261 to make it deterministic by default so we don't have to patch an infinite number of places. Also worth noting that the timebombs file got several updates as I re-tested all entries from ring1. I did several more patches to ring1 packages this month so that it is as bit-reproducible as it can be. The pbuild stats say pbuild --result|grep ": " Using default preset: standard broken: 9 succeeded: 3569 failed: 19 unresolvable: 8 excluded: 193 So nearly everything could be built and with the exception of emacs and a few other known issues, everything built perfectly reproducible. https://en.opensuse.org/openSUSE:Reproducible_openSUSE/Part2#remaining_issue... Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/3... Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2024-09/ I uploaded https://rb.zq1.de/compare.factory-20241030/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 15808 (+2) build-tried: 15744 (+3) build-failed: 18 (+6) build-n-a: 178 (-2) build-succeeded: 15548 (-1) build-official-failed+na: 378 (+326) build-compare-failed: 145 (-8) build-compare-succeeded: 15403 (+7) verify-failed: 255 (+0) verified-semi-reproducible: 15112 (+6) verified-bit-identical: 13760 (+126) bit-by-bit-identical: 15308 (+16) not-bit-by-bit-identical: 239 (-19) not-bit-by-bit-identicalcheck: 240 (-17) https://rb.zq1.de/compare.factory-20241030/graph.png shows the change over time https://rb.zq1.de/compare.factory-20241030/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 4 were in ring0 21 were in ring1 That makes it 25/4043 => 0.62 % which is below the overall average of 145/15548 => 0.93 % 239/15548 => 1.54 % of packages are not perfectly reproducible package notes: ==> ./ccache/.rb.notes <== = https://github.com/ccache/ccache/issues/1524 FTBFS-2038 found range good=2145915653 bad=2145917578 ==> ./ceph/.rb.notes <== = https://tracker.ceph.com/issues/68778 2038 date, boost toolchain = https://github.com/boostorg/build/issues/760 ==> ./certmonger/.rb.notes <== FTBFS-2038 found range good=2145900002 bad=2145901927 ==> ./cockpit/.rb.notes <== FTBFS-2030-07-14 SSL found range good=1910252845 bad=1910254770 ==> ./colord/.rb.notes <== = https://github.com/hughsie/colord/issues/174 https://bugzilla.opensuse.org/show_bug.cgi?id=1217747 .icc only-CPU ; toolchain from lcms2? ==> ./contrast/.rb.notes <== unknown rust/llvm => SR 1203242 contrast FTBFS-nocheck ==> ./descent3/.rb.notes <== => descent3 https://github.com/DescentDevelopers/Descent3/pull/606 mtime/date = archive with embedded mtimes from ["../tools/HogMaker", "/home/abuild/rpmbuild/BUILD/Descent3-616f921e97b0fb4745a2d36de149c737bf720214/build/Descent3/d3-linux.hog", "/home/abuild/rpmbuild/BUILD/Descent3-616f921e97b0fb4745a2d36de149c737bf720214/scripts/data/fullhog/d3-linux-fullhog.txt", "/home/abuild/rpmbuild/BUILD/Descent3-616f921e97b0fb4745a2d36de149c737bf720214/build/scripts"] ==> ./difftastic/.rb.notes <== rust/llvm ==> ./efivar/.rb.notes <== => SR 1206197 = https://bugzilla.opensuse.org/show_bug.cgi?id=1231368 CPU/march=native ==> ./emacs/.rb.notes <== = https://mail.gnu.org/archive/html/emacs-devel/2024-10/msg00004.html .pdmp from ["./temacs" "--__aslr-disabled" "-batch" "--no-build-details" "-l" "loadup" "--temacs=pdump" "--bin-dest" "/usr/bin/" "--eln-dest" "/usr/lib64/emacs/29.4/"] cd ~/rpmbuild/BUILD/emacs-29.4/src && ./temacs "--__aslr-disabled" "-batch" "--no-build-details" "-l" "loadup" "--temacs=pdump" "--bin-dest" "/usr/bin/" "--eln-dest" "/usr/lib64/emacs/29.4/" && md5sum emacs.pdmp ==> ./examine/.rb.notes <== rust/llvm toolchain random -N+inf-inf+NaN-NaNDarkKeyAKeyBKeyCKeyDKeyEKeyFKeyGKeyHKeyIKeyJKeyKKeyLKeyMKeyNKeyOKeyPKeyQKeyRKeySKeyTKeyUKeyVKeyWKeyXKeyYKeyZHelpHomeMetaCopyFindOpenUndoRedoAttnPlayEisuSaveCallExitInfoLinkLockWinkUserpathReadXlibnamegapscardDRI3viewlogoarchhashdate ==> ./forecast/.rb.notes <== low-entropy cosmic issue ==> ./gri/.rb.notes <== ps timestamps: %DVIPSSource: TeX output ... ==> ./groff/.rb.notes <== groff:full postscript toolchain date+time in /usr/share/doc/packages/groff/pdf/pdfmark.pdf ==> ./gsl/.rb.notes <== => SR 1206278 nocheck ==> ./ibmtss/.rb.notes <== # => https://github.com/kgoldman/ibmtss/commit/3a17ac01bea73d3568272d61b895a16a0b... = https://sourceforge.net/p/ibmtpm20tss/tickets/49/ FTBFS-2026-02-20 found range good=1771613260 bad=1771615185 (2026-02-20T19:19:45) ==> ./java-21-openjdk/.rb.notes <== date influence in libjvm.so debuginfo = https://bugzilla.opensuse.org/show_bug.cgi?id=1221224 java-21-openjdk parallelism +?ASLR? ==> ./kdevelop-plugin-php/.rb.notes <== = https://invent.kde.org/kdevelop/kdevelop-pg-qt/-/issues/2 https://bugzilla.opensuse.org/show_bug.cgi?id=1231627 ASLR, toolchain from kdevelop-pg-qt https://invent.kde.org/kdevelop/kdevelop-pg-qt/-/blame/master/src/kdev-pg-co... ==> ./kf6-kirigami/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228131 parallelism via /home/abuild/rpmbuild/BUILD/kirigami-6.4.0/build/src/dialogs/.rcc/qmlcache/KirigamiDialogs_PromptDialog_qml.cpp also /usr/src/debug/kirigami-6.4.0/build/src/delegates/.rcc/qmlcache/KirigamiDelegates_IconTitleSubtitle_qml.cpp ==> ./libcamera/.rb.notes <== year/date in tracepoints.h (with debuginfo) ==> ./libplasma6/.rb.notes <== parallelism/race variation in /usr/lib64/qt6/qml/org/kde/plasma/extras/libplasmaextracomponentsplugin.so from /home/abuild/rpmbuild/BUILD/libplasma-6.2.2/build/src/declarativeimports/plasmaextracomponents/.rcc/qmlcache/plasmaextracomponentsplugin_qml/PlasmoidHeading_qml.cpp ==> ./libvirt/.rb.notes <== FTBFS-2030-03-17 found range good=1899999549 bad=1900001474 ==> ./linutil/.rb.notes <== = https://github.com/ChrisTitusTech/linutil/pull/869 date ==> ./llvm17/.rb.notes <== ASLR left in .so = https://github.com/llvm/llvm-project/issues/72206 Marvin investigates. Partially from ASLR - uninitialized memory? ==> ./lsof/.rb.notes <== => SR 1218747 = https://bugzilla.opensuse.org/show_bug.cgi?id=1232425 host kernel version in LSOF_VSTR vmtype=--vm-type=podman rbk ==> ./lua-luarepl/.rb.notes <== => SR 1204160 lua-luarepl FTBFS OSC_BUILD_ROOT=/var/tmp/build-root.$slot osc build --noservice --clean --vm-type=kvm -M=lua51 standard ==> ./nodejs22/.rb.notes <== ASLR+CPU? ==> ./nvidia-persistenced/.rb.notes <== nvidia-persistenced date in /usr/share/man/man1/nvidia-persistenced.1.gz ==> ./nvidia-xconfig/.rb.notes <== => SR 1203885 nvidia-xconfig https://github.com/NVIDIA/nvidia-xconfig/pull/3 date in /usr/share/man/man1/nvidia-xconfig.1.gz ==> ./octave-forge-nan/.rb.notes <== random path in debuginfo --- old /usr/lib/debug/.dwz/octave-forge-nan-3.7.0-1.1.x86_64 (objdump) ==> ./openssl-1_1/.rb.notes <== => SR 1208152 = https://bugzilla.opensuse.org/show_bug.cgi?id=1231667 openssl-1_1 date(copyright year) = https://bugzilla.opensuse.org/show_bug.cgi?id=1185637 FTBFS-2022-06-01 ==> ./openttd/.rb.notes <== ASLR into build-id ==> ./openwsman/.rb.notes <== = openwsman https://bugzilla.opensuse.org/show_bug.cgi?id=1231038 FTBFS ==> ./perl-IO-Socket-SSL/.rb.notes <== FTBFS-2032 SSL found range good=1986146944 bad=1986148869 ==> ./perl-Net-SSLeay/.rb.notes <== TODO FTBFS-2038 found range good=2145915609 bad=2145917534 ==> ./pinentry/.rb.notes <== => SR 1202479 make efl droppable ==> ./postgresql17/.rb.notes <== = https://www.postgresql.org/message-id/ba8e1bc0-8a99-45b7-8397-3f2e94415e03%4... FTBFS-2024-11-03 found range good=1730573413 bad=1730576847 ==> ./python-emoji/.rb.notes <== FTBFS [ 91s] tests/test_zwj_remove.py::test_non_rgi_zwj_replace PASSED [100%] ==> ./python-gevent/.rb.notes <== FTBFS-2037-07-16 SSL found range good=2131351221 bad=2131353146 ==> ./python-gobject/.rb.notes <== python-gobject _gi.cpython-310-x86_64-linux-gnu.so mesonpy toolchain random filename => https://build.opensuse.org/request/show/1204522 ==> ./python-libcst/.rb.notes <== => https://github.com/Instagram/LibCST/pull/1213 codegen-units=1 rust/llvm /usr/lib64/python3.11/site-packages/libcst/native.cpython-311-x86_64-linux-gnu.so ==> ./python-meson-python/.rb.notes <== = https://github.com/mesonbuild/meson-python/issues/671 toolchain for python-contourpy python-gobject python-numpy python-pandas python-scikit-learn python-scipy ==> ./python-mitmproxy/.rb.notes <== FTBFS-2032-10-26 found range good=1982383545 bad=1982385470 ==> ./python-moto/.rb.notes <== => https://github.com/getmoto/moto/pull/8218 => https://github.com/python/cpython/pull/125261 gzip mtime variation in jwks-private.json.gz in RECORD from setup.py target.write(gzip.compress(source.read())) ==> ./python-numpy/.rb.notes <== mesonpy toolchain random ==> ./python-pandas/.rb.notes <== = https://github.com/pandas-dev/pandas/issues/60078 ASLR+parallelism+other? mesonpy toolchain random; random .pyc - ref counters? ==> ./python-panel/.rb.notes <== = python-panel https://bugzilla.opensuse.org/show_bug.cgi?id=1231254 package-lock random port ==> ./python-propcache/.rb.notes <== => SR 1207574 random from packaging/pep517_backend/_backend.py: with TemporaryDirectory(prefix='.tmp-propcache-pep517-') as tmp_dir: ==> ./python-pyHanko/.rb.notes <== = https://github.com/MatthiasValvekens/pyHanko/issues/472 FTBFS-2030 found range good=1893454427 bad=1893456352 ==> ./python-pydata-sphinx-theme/.rb.notes <== date+time in /usr/lib/python3.10/site-packages/pydata_sphinx_theme/locale/en/LC_MESSAGES/sphinx.mo 00000080 6e 2d 44 61 74 65 3a 20 32 30 32 34 2d 31 30 2d |n-Date: 2024-10-| ==> ./python-pygraphviz/.rb.notes <== => https://github.com/pygraphviz/pygraphviz/pull/544 => https://github.com/sphinx-gallery/sphinx-gallery/pull/1385 python-pygraphviz toolchain, benchmark = https://github.com/pygraphviz/pygraphviz/issues/541 captures execution time ==> ./python-pyhanko-certvalidator/.rb.notes <== FTBFS-2031 [ 57s] E pyhanko_certvalidator.errors.ExpiredError: The path could not be validated because intermediate certificate 1 expired 2030-12-31 08:30:00Z ==> ./python-pymol/.rb.notes <== => https://github.com/schrodinger/pymol-open-source/pull/404 date #define _PyMOL_BUILD_DATE 1728879569 ==> ./python-rdflib/.rb.notes <== FTBFS :doc (pinged packager in Slack) ==> ./python-scikit-learn/.rb.notes <== mesonpy toolchain random parallelism via cython+openmp - similar to https://github.com/yt-project/yt/issues/4611 - ?does it need JOBLIB_MULTIPROCESSING=0 ? ==> ./python-scipy/.rb.notes <== mesonpy toolchain random ==> ./python-tornado6/.rb.notes <== FTBFS-SSL 2033-10-30 found range good=2014239045 bad=2014240970 ==> ./python/.rb.notes <== FTBFS-2028-07-07 found range good=1846590951 bad=1846592876 ==> ./python312/.rb.notes <== => SR 1204725 python312 drop .pyc from doc = https://github.com/python/cpython/issues/124851 FTBFS-2038 ==> ./python313/.rb.notes <== => https://github.com/python/cpython/pull/125261 gzip mtime, toolchain = https://github.com/python/cpython/issues/122433 FTBFS-j1 ==> ./python314/.rb.notes <== /home/abuild/rpmbuild/BUILD/Python-3.14.0a1/jit_stencils.h varies ==> ./qt6-multimedia/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1228131 parallelism --- /var/tmp/build-root.1/.mount/home/abuild/rpmbuild/BUILD/qtmultimedia-everywhere-src-6.7.3/build/examples/multimedia/video/qmlvideo/qmlvideo/.rcc/qmlcache/qmlvideo_CameraItem_qml.cpp 2024-10-04 07:24:58.989999999 +0000 ==> ./rakudo/.rb.notes <== = https://github.com/rakudo/rakudo/issues/5427 EPOCH date/time in /usr/share/perl6/runtime/perl6.moarvm ==> ./rusty_v8/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1231548 rust random .o files ==> ./scapy/.rb.notes <== => SR 1205217 date ==> ./scheme48/.rb.notes <== FTBFS (from gcc14?) unreproducible dump ==> ./schily/.rb.notes <== => SR 1218925 = https://bugzilla.opensuse.org/show_bug.cgi?id=1232434 schily uname -r vmtype=--vm-type=podman rbk ==> ./seahorse/.rb.notes <== = https://gitlab.gnome.org/GNOME/seahorse/-/issues/394 parallelism, bug Binary files /var/tmp/build-root.12/.mount/home/abuild/rpmbuild/BUILD/seahorse-43.0+22/x86_64-suse-linux/src/seahorse.p/meson-generated_application.c.o and /var/tmp/build-root.12b/.mount/home/abuild/rpmbuild/BUILD/seahorse-43.0+22/x86_64-suse-linux/src/seahorse.p/meson-generated_application.c.o differ ==> ./subversion/.rb.notes <== => SR 1203785 minor jar mtimes /usr/lib64/svn-javahl/svn-javahl.jar ==> ./superlu/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1232550 FTBFS-nocheck ==> ./taigo/.rb.notes <== = taigo https://bugzilla.opensuse.org/show_bug.cgi?id=1231041 FTBFS ==> ./tycho/.rb.notes <== FTBFS-2038 osc build --noservice --clean --vm-type=kvm --build-opt=--vm-custom-opt="-rtc base=2038-01-25T00:00:00" -M=bootstrap java maven jar ==> ./ugrep/.rb.notes <== = https://bugzilla.suse.com/show_bug.cgi?id=1222752 1222752 ==> ./util-linux/.rb.notes <== = https://github.com/util-linux/util-linux/issues/3259 random test failure osc build --vm-type=kvm --noservice --clean -j1 standard ==> ./wine/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1231620 wine parallelism ==> ./xindy/.rb.notes <== random xindy.mem CLISP memory image data = https://sourceforge.net/p/xindy/bugs/65/ report FTBFS-2036 ==> ./zed/.rb.notes <== rust/llvm toolchain random? cosmic --- /var/tmp/build-root.11/.mount/home/abuild/rpmbuild/BUILD/zed-0.155.2/target/release/build/rav1e-7d061ae4d20f513a/out/built.rs 2024-10-16 05:45:35.693333331 +0000