Hi, Good news: after suffering from issues with python310, I finally managed to find the two required patches https://www.reddit.com/r/reproduciblebuilds/comments/1573d76/reproducible_py... and mcepl (our python packager) sent them towards Factory and SLE. I also worked with fstrba to make nice progress on xmvn UUIDs https://bugzilla.opensuse.org/show_bug.cgi?id=1162112 Meanwhile, the fix to ghc-rpm-macros is still waiting to be merged. https://github.com/opensuse-haskell/ghc-rpm-macros/pull/1 Add to that the doxygen toolchain fix https://build.opensuse.org/request/show/1099185 Once all four fixes above reached Factory, this should fix over half of the remaining issues. A few other major headaches like mono and javadoc still remain unpatched: https://github.com/mono/mono/issues/20172 https://bugzilla.opensuse.org/show_bug.cgi?id=1174795 Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/D... Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2023-06/ I uploaded https://rb.zq1.de/compare.factory-20230731/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 14812 (+69) build-tried: 14776 (+67) build-failed: 74 (+4) build-n-a: 200 (+19) build-succeeded: 14503 (+42) build-official-failed+na: 109 (+36) build-compare-failed: 411 (-51) build-compare-succeeded: 14092 (+93) verify-failed: 687 (-172) verified-semi-reproducible: 11455 (-1831) bit-by-bit-identical: 13931 (+43) not-bit-by-bit-identical: 564 (-8) not-bit-by-bit-identicalcheck: 572 (-1) https://rb.zq1.de/compare.factory-20230731/graph.png shows the change over time https://rb.zq1.de/compare.factory-20230731/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 4 were in ring0 43 were in ring1 That makes it 47/3206 => 1.47 % which is below the overall average of 411/14503 => 2.83 % 564/14503 => 3.89 % of packages are not perfectly reproducible package notes: ==> ./alex/.rb.notes <== ghc toolchain parallelism ==> ./c3p0/.rb.notes <== strip-nondeterminism, minor #= https://github.com/swaldman/c3p0/issues/163 date+time ; ?bug? order issues ; orphaned ==> ./cockpit-machines/.rb.notes <== => https://github.com/cockpit-project/cockpit/pull/18994 gzip mtime /usr/share/cockpit/machines/index.css.gz ==> ./deno/.rb.notes <== rust toolchain? binary order ==> ./fasterxml-oss-parent/.rb.notes <== mvn toolchain random UUID ==> ./gh/.rb.notes <== found range good=1704505952 bad=1704507877 (2024-01-06T02:24:37) #= https://github.com/cli/cli/issues/6259 FTBFS-2023 ==> ./git-interactive-rebase-tool/.rb.notes <== => https://github.com/MitMaro/git-interactive-rebase-tool/pull/881 date --- old /usr/bin/interactive-rebase-tool (objdump) ==> ./glucat/.rb.notes <== pdf /usr/share/doc/packages/glucat-doc/GluCat-API-reference-manual.pdf ==> ./gnutls/.rb.notes <== FTBFS --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed FAIL: gost.sh ==> ./golang-github-prometheus-prometheus/.rb.notes <== TODO: https://github.com/prometheus/promu => https://github.com/prometheus/prometheus/pull/12515 user host date ==> ./gstreamer-plugins-rs/.rb.notes <== rust/llvm asm diff variations in /usr/lib64/pkgconfig/gstwebrtchttp.pc ==> ./gtk2hs-buildtools/.rb.notes <== ghc toolchain parallelism ==> ./gutenprint/.rb.notes <== => https://sourceforge.net/p/gimp-print/source/merge-requests/2/ date+time +++ new//usr/lib64/gutenprint/5.3/config.summary 2023-04-22 00:00:00.000000000 +0000 ==> ./helix/.rb.notes <== rust: parallelism into machine code? +++ new /usr/lib64/helix/hx (disasm) ==> ./hledger-interest/.rb.notes <== ghc toolchain --- old /usr/bin/hledger-interest (objdump) ==> ./httpie/.rb.notes <== FTBFS-SSL ==> ./intel-graphics-compiler/.rb.notes <== ASLR order issues in /home/abuild/rpmbuild/BUILD/build/install_manifest.txt ==> ./jackson-parent/.rb.notes <== java maven xmvn random uuid ==> ./java-20-openjdk/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1213796 FTBFS-2032 found range good=1987887184 bad=1987889714 (2032-12-28T23:35:14) ==> ./kubernetes1.26/.rb.notes <== random go tmp build path ==> ./kxml/.rb.notes <== java maven random UUID, javadoc +++ new//usr/share/maven-metadata/kxml.xml 2019-05-03 12:00:00.000000000 +0000 ==> ./libarchive/.rb.notes <== => SR 1100254 version update with below fixes => https://github.com/libarchive/libarchive/pull/1836 FTBFS-2038 ==> ./libgav1/.rb.notes <== FTBFS-cpu [ 853s] 4 - common_avx2_test (ILLEGAL) ==> ./mage/.rb.notes <== => https://github.com/magefile/mage/pull/474 date+time ; golang from .goreleaser.yml: ldflags: -s -w -X github.com/magefile/mage/mage.timestamp={{.Date}} -X github.com/magefile/mage/mage.commitHash={{.Commit}} -X github.com/magefile/mage/mage.gitTag={{.Version}} ==> ./memtest86+/.rb.notes <== /boot/efi/EFI/memtest86/memtest.efi differs at offset '292' (Linux/x86 Kernel, Setup Version 0x20c, bzImage, Version Memtest86+ v6.10, RW-rootFS) --- old /boot/efi/EFI/memtest86/memtest.efi (hex) ==> ./meson/.rb.notes <== /usr/lib/rpm/macros.d/macros.meson smp ==> ./mhvtl/.rb.notes <== tar/gz /usr/lib/firmware/mhvtl/mhvtl_kernel.tgz ==> ./mumble/.rb.notes <== #=> https://github.com/mumble-voip/mumble/pull/6147 filesys (python listdir) in /home/abuild/rpmbuild/BUILD/mumble-1.5.517/build/src/mumble/mumble_client_object_lib_autogen/IZWCZGADZF/qrc_mumble_flags.cpp 2023-06-09 05:38:14.144000000 +0000 ==> ./nyxt/.rb.notes <== nyxt binary varies by pid=2536 dir=/home/abuild/rpmbuild/BUILD/nyxt-3.2.1/ exec="/usr/bin/sbcl" ["sbcl" "--dynamic-space-size" "3072" "--no-userinit" "--non-interactive" "--eval" "(require \"asdf\")" "--eval" "(when (string= \"true\" \"true\") (setf asdf:*default-source-registries* (list (quote asdf/source-registry:environment-source-registry))) (asdf:clear-configuration) (asdf:load-asd \"/home/abuild/rpmbuild/BUILD/nyxt-3.2.1//libraries/nasdf/nasdf.asd\") (asdf:load-asd \"/home/abuild/rpmbuild/BUILD/nyxt-3.2.1//nyxt.asd\") (asdf:load-system :nyxt/submodules))" "--eval" "(asdf:load-asd \"/home/abuild/rpmbuild/BUILD/nyxt-3.2.1//libraries/nasdf/nasdf.asd\")" "--eval" "(asdf:load-asd \"/home/abuild/rpmbuild/BUILD/nyxt-3.2.1//nyxt.asd\")" "--eval" "(format t \"~&; NASDF ~a: ~a~&\" (asdf:component-version (asdf:find-system :nasdf)) (nth-value 2 (asdf:locate-system :nasdf)))" "--eval" "(when (find-package :ql) (funcall (read-from-string \"ql:quickload\") :cffi))" "--eval" "(when (and (find-package :cffi) (uiop:getenv \"GUIX_ENVIRONMENT\")) (pushnew (pathname (format nil \"~a/lib/\" (uiop:getenv \"GUIX_ENVIRONMENT\"))) (symbol-value (read-from-string \"cffi:*foreign-library-directories*\" )) :test (quote equal)))" "--eval" "(asdf:load-system :nyxt/gi-gtk-application)" "--eval" "(asdf:make :nyxt/install)" "--eval" "(uiop:quit)"] - started ==> ./osgi-service-jdbc/.rb.notes <== java pom date+time ==> ./pcr-oracle/.rb.notes <== #=> https://github.com/okirch/pcr-oracle/pull/29 date (shell) in /usr/share/man/man8/pcr-oracle.8.gz ==> ./photoqt/.rb.notes <== Qt qrc mtimes /home/abuild/rpmbuild/BUILD/photoqt-1.7/build/qrc_lang.cpp ==> ./procmeter/.rb.notes <== parallelism race ==> ./pympress/.rb.notes <== date+time from python-Babel : fix in rb:test --- old /usr/lib/python3.11/site-packages/pympress/share/locale/zh_TW/LC_MESSAGES/pympress.mo (hex) ==> ./python3-pyside6/.rb.notes <== _smp_build_ncpus embedded in .src.rpm https://github.com/rpm-software-management/rpm/issues/2343 ASLR ==> ./python-bpython/.rb.notes <== => SR 1100750 drop sphinx doctrees ==> ./python-control/.rb.notes <== = https://github.com/python-control/python-control/issues/927 FTBFS-j4 osc build --noservice --vm-type=kvm -j4 standard ==> ./python-cx_Freeze/.rb.notes <== #=> SR 1100867 version upgrade for FTBFS-2038 #=> https://github.com/marcelotduarte/cx_Freeze/pull/1860 FTBFS-2038 ==> ./python-Django/.rb.notes <== #=> https://github.com/django/django/pull/16459 FTBFS-2038 #=> https://github.com/django/django/pull/10994 FTBFS-2028 to 2038 ==> ./python-enaml/.rb.notes <== #=> https://github.com/nucleic/enaml/pull/516 FTBFS-2038 osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-02-01T18:00:00" standard ==> ./python-flup/.rb.notes <== => SR 1101194 python-flup drop sphinx doctrees ==> ./python-gns3fy/.rb.notes <== FTBFS-20xx? ==> ./python-HyperKitty/.rb.notes <== toolchain issue? nothing helps from ["python3.11", "build_static_files/manage.py", "compress", "--force"] ==> ./python-iminuit/.rb.notes <== numba toolchain issue #=> https://github.com/scikit-hep/iminuit/pull/355 readdir (python glob for distutil) ==> ./python-librosa/.rb.notes <== unknown numba toolchain issue /usr/lib/python3.9/site-packages/librosa/util/__pycache__/utils.__shear_dense-1846.py39.nbi differs at offset '38' (XENIX 8086 relocatable or 80286 small model) ==> ./python-mpi4py/.rb.notes <== = https://bugzilla.suse.com/show_bug.cgi?id=1212698 lto+parallelism into debug # filterdiff hexdump\ -C R*/usr/lib64/python3.9/site-packages/mpi4py/MPI.cpython-39-x86_64-linux-gnu.so ==> ./python-mysqlclient/.rb.notes <== => SR 1100753 drop sphinx doctrees ==> ./python-pandas/.rb.notes <== random .pyc - ref counters? ==> ./python-pyface/.rb.notes <== = https://github.com/enthought/pyface/issues/1254 date+time ==> ./python-quantities/.rb.notes <== = https://github.com/python-quantities/python-quantities/issues/225 date+time in /usr/lib/python3.9/site-packages/quantities/constants/_codata.py ==> ./python-rdflib/.rb.notes <== toolchain? random Graph identifier ==> ./python-scikit-learn/.rb.notes <== parallelism via cython+openmp - similar to https://github.com/yt-project/yt/issues/4611 random FTBFS-j4 on bernhards ==> ./python-scipy/.rb.notes <== = https://github.com/scipy/scipy/issues/18987 date+time ; todo random mesonpy +++ new /usr/lib/hpc/gnu13/python-scipy/1.10.1/lib64/python3.9/site-packages/scipy/stats/_stats_pythran.cpython-39-x86_64-linux-gnu.so (objdump) ==> ./python-Telethon/.rb.notes <== => https://github.com/LonamiWebs/Telethon/pull/4163 sort order in /usr/lib/python3.9/site-packages/telethon/tl/alltlobjects.py from telethon_generator/generators/tlobject.py ==> ./python-toolz/.rb.notes <== other python3 pyc toolchain ASLR ; needs bpo-37596: Make `set` and `frozenset` marshalling deterministic (GH-27926) ==> ./python-waitress/.rb.notes <== => SR 1100756 Drop sphinx doctrees ==> ./python-WebOb/.rb.notes <== => SR 1100758 python-WebOb drop sphinx doctrees ==> ./python-WebTest/.rb.notes <== 1100760 python-WebTest drop sphinx doctrees ==> ./python-xarray-einstats/.rb.notes <== numba toolchain issue ==> ./python-yt/.rb.notes <== = https://github.com/yt-project/yt/issues/4611 parallelism => https://github.com/yt-project/yt/pull/4609 python filesys order ==> ./python-zeep/.rb.notes <== python .pyc ASLR (+readdir?) ==> ./python-zope.deprecation/.rb.notes <== => SR 1100560 drop sphinx doctrees ==> ./python-zope.event/.rb.notes <== => SR 1100761 python-zope.event drop sphinx doctrees ==> ./python-zope.hookable/.rb.notes <== => SR 1100764 python-zope.hookable drop sphinx doctrees ==> ./python-zope.i18nmessageid/.rb.notes <== => SR 1100765 python-zope.i18nmessageid drop sphinx doctrees ==> ./runc/.rb.notes <== minor: filesys+aslr ==> ./s3fs/.rb.notes <== #=> https://github.com/s3fs-fuse/s3fs-fuse/pull/2026 date -/usr/share/man/man1/s3fs.1.gz ==> ./shadowsocks-rust/.rb.notes <== => https://github.com/AlephAlpha/build-time/pull/5 toolchain date+time +++ new /usr/bin/sslocal (objdump) ==> ./sharpfont/.rb.notes <== mono dll ==> ./sonatype-plugins-parent/.rb.notes <== java xmvn random uuid ==> ./sonic-pi/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1208969 FTBFS-2038 from app/server/beam/tau/test/erlang_tau_osc_test.exs ErlangTauOSCTest ==> ./starship/.rb.notes <== => https://github.com/starship/starship/pull/5352 filesys readdir, rust in /usr/bin/starship ==> ./xindy/.rb.notes <== = https://sourceforge.net/p/xindy/bugs/65/ report FTBFS-2036 osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2036-02-07T06:30:00"