Hi, Did some more nice debugging and patches this month The ghc workaround is still pending to bring another 220+ reproducible packages. https://github.com/opensuse-haskell/ghc-rpm-macros/pull/1 Java still has some way to go, but is improving upstream as well. 313 is probably the lowest percentage of build-compare-failed (very-unreproducible) packages we ever had. Here are the autogenerated bits: last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/M... Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2023-07/ I uploaded https://rb.zq1.de/compare.factory-20230830/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 14840 (+28) build-tried: 14803 (+27) build-failed: 29 (-45) build-n-a: 229 (+29) build-succeeded: 14545 (+42) build-official-failed+na: 79 (-30) build-compare-failed: 313 (-98) build-compare-succeeded: 14232 (+140) verify-failed: 603 (-84) verified-semi-reproducible: 11432 (-23) bit-by-bit-identical: 14059 (+128) not-bit-by-bit-identical: 487 (-77) not-bit-by-bit-identicalcheck: 486 (-86) https://rb.zq1.de/compare.factory-20230830/graph.png shows the change over time https://rb.zq1.de/compare.factory-20230830/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 4 were in ring0 37 were in ring1 That makes it 41/3206 => 1.28 % which is below the overall average of 313/14545 => 2.15 % 487/14545 => 3.35 % of packages are not perfectly reproducible package notes: ==> ./alex/.rb.notes <== ghc toolchain parallelism ==> ./apache-arrow/.rb.notes <== = https://github.com/apache/arrow/issues/37276 FTBFS-j1 ==> ./apptainer/.rb.notes <== = https://github.com/apptainer/apptainer/issues/1623 random go buildid from mlocal/frags/go_normal_opts.mk:GO_LDFLAGS += -ldflags="-B 0x`head -c20 /dev/urandom|od -An -tx1|tr -d ' \n'`" ==> ./asymptote/.rb.notes <== remaining pdf issue - toolchain pdflatex #=> https://github.com/vectorgraphics/asymptote/pull/351 date ==> ./checkstyle/.rb.notes <== java toolchain date+time ; xmvn random uuid +++ new//usr/share/java/checkstylecompilation.properties 2020-03-09 23:52:37.968994653 +0000 ==> ./cockpit/.rb.notes <== #=> https://github.com/cockpit-project/cockpit/pull/14276 date ; FTBFS-2035 ==> ./deepin-desktop-schemas/.rb.notes <== # = https://github.com/linuxdeepin/developer-center/issues/5117 golang ordering #=> https://github.com/linuxdeepin/deepin-desktop-schemas/pull/98 ==> ./deno/.rb.notes <== rust toolchain? binary order ==> ./doxygen/.rb.notes <== => https://bugzilla.opensuse.org/show_bug.cgi?id=1201579 = https://github.com/doxygen/doxygen/issues/8158 toolchain, ASLR ==> ./elementary-xfce-icon-theme/.rb.notes <== => SR 1106231 = https://gitlab.gnome.org/GNOME/pango/-/issues/729 toolchain pango ASLR+readdir order ==> ./epy/.rb.notes <== time-based python .pyc, minor ==> ./ezmorph/.rb.notes <== toolchain javadoc ==> ./findbugs/.rb.notes <== java toolchain date+time gets injected into docs and .jar: ==> ./gegl/.rb.notes <== = https://gitlab.gnome.org/GNOME/gegl/-/issues/337 parallelism, memory +++ new//usr/share/gir-1.0/Gegl-0.4.gir 2023-06-26 00:00:00.000000000 +0000 ==> ./ghc-bootstrap/.rb.notes <== TODO filesys order in /usr/lib/ghc-8.2.1/package.conf.d/package.cache ==> ./gh/.rb.notes <== found range good=1704505952 bad=1704507877 (2024-01-06T02:24:37) #= https://github.com/cli/cli/issues/6259 FTBFS-2023 ==> ./ginac/.rb.notes <== pdf from TEXI2DVI ginac.texi => ginac.pdf ==> ./git-interactive-rebase-tool/.rb.notes <== => https://github.com/MitMaro/git-interactive-rebase-tool/pull/881 date --- old /usr/bin/interactive-rebase-tool (objdump) ==> ./glucat/.rb.notes <== pdflatex toolchain pdf /usr/share/doc/packages/glucat-doc/GluCat-API-reference-manual.pdf ==> ./gnutls/.rb.notes <== FTBFS --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed found range good=1943078121 bad=1943080046 (2031-07-29T08:27:26) ==> ./golang-github-prometheus-prometheus/.rb.notes <== #=> https://github.com/prometheus/promu/pull/267 user host #=> https://github.com/prometheus/prometheus/pull/12515 user host date ==> ./goldendict-ng/.rb.notes <== => https://codereview.qt-project.org/c/qt/qtbase/+/494174 qmake toolchain date+time = https://bugreports.qt.io/browse/QTBUG-115737 qmake toolchain date+time ==> ./google-noto-fonts/.rb.notes <== => https://github.com/notofonts/Arimo/pull/17 downstream patch fix caused mtime to creep into build results - timestamp - seconds via patch mtime /usr/share/fonts/truetype/Arimo-BoldItalic.ttf ==> ./grommunio-web/.rb.notes <== #=> https://github.com/grommunio/grommunio-web/pull/8 make/find filesys readdir order from ["/bin/sh", "-c", "cat client/third-party/tokenizr/tokenizr.js client/third-party/tinymce/TinyMceTextArea.js > deploy/client/third-party/ux-thirdparty-debug.js"] - started ==> ./grpc/.rb.notes <== => SR 1102203 grpc drop log ==> ./gstreamer-plugins-rs/.rb.notes <== rust/llvm asm diff variations in /usr/lib64/pkgconfig/gstwebrtchttp.pc ==> ./guile-parted/.rb.notes <== => SR 1102184 guile-parted parallelism = https://bugzilla.opensuse.org/show_bug.cgi?id=1170378 parallelism into guile object .go files ==> ./httpie/.rb.notes <== FTBFS-SSL found range good=2180619253 bad=2180621178 (2039-02-06T16:06:18) ==> ./icinga2/.rb.notes <== => SR 1104874 hostname in System.BuildHostName ==> ./icu/.rb.notes <== FTBFS-2023 osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2023-09-16T07:30:00" standard FTBFS-DST osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2022-10-30T00:30:00" standard ==> ./java-20-openjdk/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1213796 FTBFS-2032 found range good=1987887184 bad=1987889714 (2032-12-28T23:35:14) ==> ./kakoune/.rb.notes <== FTBFS-j1 osc build --vm-type=kvm --noservice -j1 standard ==> ./kernel-obs-build/.rb.notes <== => https://bugzilla.opensuse.org/show_bug.cgi?id=1189305 ==> ./libarchive/.rb.notes <== => SR 1100254 version update with below fixes => https://github.com/libarchive/libarchive/pull/1836 FTBFS-2038 ==> ./libgav1/.rb.notes <== FTBFS-cpu [ 853s] 4 - common_avx2_test (ILLEGAL) ==> ./libkolabxml/.rb.notes <== remaining order issue from CPU => SR 943530 ASLR ==> ./liquid-dsp/.rb.notes <== => SR 1107870 CPU autodetect => SR 630018 ==> ./mathgl/.rb.notes <== Qt qrc .png, .pdf random example .png ==> ./memcached/.rb.notes <== => https://github.com/memcached/memcached/pull/1074 FTBFS-2038 => https://github.com/memcached/memcached/pull/927 FTBFS-2038 ==> ./memtest86+/.rb.notes <== /boot/efi/EFI/memtest86/memtest.efi differs at offset '292' (Linux/x86 Kernel, Setup Version 0x20c, bzImage, Version Memtest86+ v6.10, RW-rootFS) --- old /boot/efi/EFI/memtest86/memtest.efi (hex) ==> ./mhvtl/.rb.notes <== tar/gz /usr/lib/firmware/mhvtl/mhvtl_kernel.tgz ==> ./mingw64-gcc/.rb.notes <== PE header timestamps? ==> ./mozc/.rb.notes <== => SR 1104227 FTBFS rpm %jobs ==> ./ocaml-camlp-streams/.rb.notes <== = https://github.com/ocaml/camlp-streams/issues/9 parallelism ==> ./openmpi5/.rb.notes <== => https://github.com/openpmix/openpmix/pull/3112 date + date/year sphinx toolchain? date into html/searchindex.js ==> ./orthanc-ohif/.rb.notes <== => https://orthanc.uclouvain.be/hg/orthanc-ohif/rev/154cb76a042f sent by mail to sebastien.jodogne@uclouvain : order, gzip timestamp from gzip.compress variations in /home/abuild/rpmbuild/BUILD/OrthancOHIF-1.0/build/AUTOGENERATED/StaticAssets.cpp ==> ./orthanc-volview/.rb.notes <== => https://orthanc.uclouvain.be/hg/orthanc-volview/rev/c50afec9f978 order, gzip mtime ==> ./ouch/.rb.notes <== # solved by version update: rust libzstd_sys read_dir order => https://github.com/gyscos/zstd-rs/commit/6fe67d6a ==> ./pandoc-cli/.rb.notes <== = ghc toolchain parallelism https://gitlab.haskell.org/ghc/ghc/-/issues/12935 ==> ./perl-Net-DNS/.rb.notes <== = https://rt.cpan.org/Ticket/Display.html?id=149456 FTBFS-2038 = https://github.com/jwilk-mirrors/perl-net-dns/issues/1 FTBFS-2038 ==> ./postgis/.rb.notes <== => SR 1101964 parallelism becomes reproducible with parallelism=1 parallelism2=1 multibuildrbk ==> ./pympress/.rb.notes <== date+time from python-Babel : fix in rb:test --- old /usr/lib/python3.11/site-packages/pympress/share/locale/zh_TW/LC_MESSAGES/pympress.mo (hex) ==> ./python3-pyside6/.rb.notes <== _smp_build_ncpus embedded in .src.rpm https://github.com/rpm-software-management/rpm/issues/2343 ASLR ==> ./python-bpython/.rb.notes <== => SR 1100750 drop sphinx doctrees ==> ./python-bson/.rb.notes <== .=> https://github.com/py-bson/bson/pull/117 FTBFS-2038 => https://github.com/pypa/setuptools/commit/5a0404fa3875a069f7a6436f508116e852... toolchain,filesys ==> ./python-control/.rb.notes <== = https://github.com/python-control/python-control/issues/927 FTBFS-j4 osc build --noservice --vm-type=kvm -j4 standard ==> ./python-cx_Freeze/.rb.notes <== #=> SR 1100867 version upgrade for FTBFS-2038 #=> https://github.com/marcelotduarte/cx_Freeze/pull/1860 FTBFS-2038 ==> ./python-DateTime/.rb.notes <== = https://github.com/zopefoundation/DateTime/issues/56 FTBFS-2038 found range good=2164298238 bad=2164300163 (2038-08-01T18:29:23) ==> ./python-django-graphql-jwt/.rb.notes <== = https://github.com/flavors/django-graphql-jwt/issues/318 FTBFS-2038 ==> ./python-Django/.rb.notes <== #=> https://github.com/django/django/pull/16459 FTBFS-2038 #=> https://github.com/django/django/pull/10994 FTBFS-2028 to 2038 ==> ./python-enaml/.rb.notes <== #=> https://github.com/nucleic/enaml/pull/516 FTBFS-2038 osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-02-01T18:00:00" standard ==> ./python-flup/.rb.notes <== => SR 1101194 python-flup drop sphinx doctrees ==> ./python-gns3fy/.rb.notes <== FTBFS-20xx? ==> ./python-grpcio/.rb.notes <== filesys-FTBFS? ==> ./python-HyperKitty/.rb.notes <== toolchain issue? nothing helps from ["python3.11", "build_static_files/manage.py", "compress", "--force"] ==> ./python-iminuit/.rb.notes <== numba toolchain issue #=> https://github.com/scikit-hep/iminuit/pull/355 readdir (python glob for distutil) ==> ./python-librosa/.rb.notes <== unknown numba toolchain issue /usr/lib/python3.9/site-packages/librosa/util/__pycache__/utils.__shear_dense-1846.py39.nbi differs at offset '38' (XENIX 8086 relocatable or 80286 small model) ==> ./python-mpi4py/.rb.notes <== = https://bugzilla.suse.com/show_bug.cgi?id=1212698 lto+parallelism into debug # filterdiff hexdump\ -C R*/usr/lib64/python3.9/site-packages/mpi4py/MPI.cpython-39-x86_64-linux-gnu.so ==> ./python-mysqlclient/.rb.notes <== => SR 1100753 drop sphinx doctrees ==> ./python-pandas/.rb.notes <== random .pyc - ref counters? ==> ./python-pyface/.rb.notes <== = https://github.com/enthought/pyface/issues/1254 date+time ==> ./python-quantities/.rb.notes <== = https://github.com/python-quantities/python-quantities/issues/225 date+time in /usr/lib/python3.9/site-packages/quantities/constants/_codata.py ==> ./python-rdflib/.rb.notes <== toolchain? random Graph identifier ==> ./python-scikit-learn/.rb.notes <== parallelism via cython+openmp - similar to https://github.com/yt-project/yt/issues/4611 random FTBFS-j4 on bernhards ==> ./python-scipy/.rb.notes <== => https://github.com/scipy/scipy/pull/19124 sort python readdir/os.walk => https://github.com/scipy/scipy/pull/19123 drop random path ==> ./python-snimpy/.rb.notes <== = https://github.com/vincentbernat/snimpy/issues/106 stuck tests/test_main.py::TestMain::test_loadfile : osc build --clean --noservice --vm-type=kvm --build-opt=--vm-custom-opt="-rtc base=2038-01-19T08:00:00" standard = https://github.com/vincentbernat/snimpy/issues/105 j1? stuck test after ==> ./python-tasklib/.rb.notes <== => https://github.com/GothenburgBitFactory/taskwarrior/pull/3052 fix FTBFS-2038 = https://github.com/GothenburgBitFactory/taskwarrior/issues/3050 FTBFS-2038 - toolchain from taskwarrior int overflow ==> ./python-Telethon/.rb.notes <== => https://github.com/LonamiWebs/Telethon/pull/4163 sort order in /usr/lib/python3.9/site-packages/telethon/tl/alltlobjects.py from telethon_generator/generators/tlobject.py ==> ./python-toolz/.rb.notes <== other python3 pyc toolchain ASLR ; needs bpo-37596: Make `set` and `frozenset` marshalling deterministic (GH-27926) ==> ./python-waitress/.rb.notes <== => SR 1100756 Drop sphinx doctrees ==> ./python-WebOb/.rb.notes <== => SR 1100758 python-WebOb drop sphinx doctrees ==> ./python-WebTest/.rb.notes <== 1100760 python-WebTest drop sphinx doctrees ==> ./python-xarray-einstats/.rb.notes <== numba toolchain issue ==> ./python-yt/.rb.notes <== = https://github.com/yt-project/yt/issues/4611 parallelism => https://github.com/yt-project/yt/pull/4609 python filesys order ==> ./python-zope.deprecation/.rb.notes <== => SR 1100560 drop sphinx doctrees ==> ./python-zope.event/.rb.notes <== => SR 1100761 python-zope.event drop sphinx doctrees ==> ./python-zope.hookable/.rb.notes <== => SR 1100764 python-zope.hookable drop sphinx doctrees ==> ./python-zope.i18nmessageid/.rb.notes <== => SR 1100765 python-zope.i18nmessageid drop sphinx doctrees ==> ./qtox/.rb.notes <== => SR 1103451 date ==> ./s3fs/.rb.notes <== #=> https://github.com/s3fs-fuse/s3fs-fuse/pull/2026 date -/usr/share/man/man1/s3fs.1.gz ==> ./scheme48/.rb.notes <== unreproducible dump ==> ./sharpfont/.rb.notes <== mono dll ==> ./sonic-pi/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1208969 FTBFS-2038 from app/server/beam/tau/test/erlang_tau_osc_test.exs ErlangTauOSCTest ==> ./starship/.rb.notes <== => https://github.com/starship/starship/pull/5352 filesys readdir, rust in /usr/bin/starship ==> ./tango-icon-theme/.rb.notes <== = https://gitlab.gnome.org/GNOME/librsvg/-/issues/1015 unknown toolchain from librsvg - probably same as https://gitlab.gnome.org/GNOME/pango/-/issues/729 --- old /usr/share/icons/Tango/128x128/apps/internet-news-reader.png (hex) ==> ./texlive/.rb.notes <== = rpm https://github.com/rpm-software-management/rpm/issues/2343#issuecomment-1678... => SR 966045 minor gzip mtime ==> ./time-api/.rb.notes <== javadoc ; copyright year ; filesys ==> ./tiny/.rb.notes <== rust unknown/nothing helps /usr/bin/tiny differs in assembler output ==> ./warp/.rb.notes <== = https://github.com/briansmith/ring/issues/1625 rust/ring filesys from ./vendor/ring/build.rs read_dir ==> ./WSL-DistroLauncher/.rb.notes <== PE timestamp ==> ./xindy/.rb.notes <== = https://sourceforge.net/p/xindy/bugs/65/ report FTBFS-2036 osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2036-02-07T06:30:00"