Hi, I spent some hours to report and fix many issues this month (so the Factory tree build got in good shape) and collected this list of packages that will soon start to fail: 2023-06-29 python-sunpy https://github.com/sunpy/sunpy/issues/6030 2023-08-07 mbedtls https://github.com/Mbed-TLS/mbedtls/issues/6978 2023-09-04 mozilla-nss https://bugzilla.mozilla.org/show_bug.cgi?id=1813401 2024-01-19 rnp https://github.com/rnpgp/rnp/issues/1961 2024-06-12 bmap-tools https://github.com/intel/bmap-tools/issues/116 unless someone defuses the timebomb. https://github.com/rpm-software-management/rpm/issues/2343 continues to be troublesome. The other big recurring issues are with 1. javadoc and xmvm e.g. https://bugzilla.opensuse.org/show_bug.cgi?id=1162112 got no love in 2y. 2. python .pyc files being influenced by ASLR and sometimes readdir order. e.g. https://bugs.python.org/issue37596 is only in 3.11, but we still have 3.8+ last month's status: https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/2... Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2022-12/ I uploaded https://rb.zq1.de/compare.factory-20230131/ today https://rb.zq1.de/spec/glossar.txt explains the meaning of below values: total-packages: 14741 (+63) build-tried: 14681 (+17) build-failed: 27 (-6) build-n-a: 226 (+1) build-succeeded: 14429 (+24) build-official-failed+na: 106 (+89) build-compare-failed: 474 (-88) build-compare-succeeded: 13955 (+112) verify-failed: 108 (-293) verified-semi-reproducible: 10667 (-1005) bit-by-bit-identical: 13695 (+454) not-bit-by-bit-identical: 734 (-431) not-bit-by-bit-identicalcheck: 734 (-430) https://rb.zq1.de/compare.factory-20230131/graph.png shows the change over time https://rb.zq1.de/compare.factory-20230131/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 4 were in ring0 59 were in ring1 That makes it 63/3316 => 1.90 % which is below the overall average of 474/14429 => 3.29 % 734/14429 => 5.09 % of packages are not perfectly reproducible package notes: ==> ./apache-commons-collections4/.rb.notes <== javadoc toolchain ==> ./b4/.rb.notes <== python3 toolchain .pyc ASLR old: parallelism in tar.xz from _service recompress at buildtime ==> ./bmap-tools/.rb.notes <== = https://github.com/intel/bmap-tools/issues/116 FTBFS-2024-06-12 osc build --clean --vm-type=kvm --noservice --build-opt=--vm-custom-opt="-rtc base=2024-06-13T00:00:00" standard ==> ./breezy/.rb.notes <== FTBFS-j1 osc build --clean --vm-type=kvm -j1 --noservice standard ==> ./endeavour/.rb.notes <== = https://gitlab.gnome.org/World/Endeavour/-/issues/503 report FTBFS-j1 osc build --vm-type=kvm -j1 --clean --noservice standard ==> ./form/.rb.notes <== date in html+pdf from doxygen / latex FTBFS-j1 : osc build --vm-type=kvm -j1 --noservice standard ==> ./galera-4/.rb.notes <== FTBFS-2037 found range good=2179559341 bad=2179561266 (2039-01-25T09:41:06) ==> ./grandorgue/.rb.notes <== => https://github.com/GrandOrgue/grandorgue/pull/1358 zip .htb help files vary is .zip with mtimes,atimes,ctimes ==> ./grass/.rb.notes <== filesys order /usr/lib64/grass78/bin/r3.mapcalc differs in assembler output ==> ./hylafax+/.rb.notes <== date in ./configure => SR 1060542 version upgrade, because tiff_450.diff touched CHANGES file and configure used that mtime ==> ./joda-time/.rb.notes <== javadoc toolchain ==> ./languagetool/.rb.notes <== java xmvn date+time +++ new//usr/share/languagetool/META-INF/org/languagetool/language-module.properties 2020-03-30 00:00:00.000000000 +0000 ==> ./libarchive/.rb.notes <== => https://github.com/libarchive/libarchive/pull/1836 FTBFS-2038 => https://github.com/libarchive/libarchive/pull/1838 FTBFS-2038 ==> ./librcc/.rb.notes <== => https://github.com/RusXMMS/librcc/pull/5 date --- old//usr/share/doc/packages/librcc-devel/examples/Makefile 2022-12-09 00:00:00.000000000 +0000 ==> ./libspf2/.rb.notes <== doxygen toolchain ==> ./libxmlb/.rb.notes <== FTBFS unknown reason [ 82s] Command '['/home/abuild/rpmbuild/BUILD/libxmlb-0.3.10/x86_64_v3-suse-linux/tmp-introspect0r88qt_v/Xmlb-2.0', '--introspect-dump=/home/abuild/rpmbuild/BUILD/libxmlb-0.3.10/x86_64_v3-suse-linux/tmp-introspect0r88qt_v/functions.txt,/home/abuild/rpmbuild/BUILD/libxmlb-0.3.10/x86_64_v3-suse-linux/tmp-introspect0r88qt_v/dump.xml']' died with <Signals.SIGILL: 4>. ==> ./lyx/.rb.notes <== FTBFS with home:bmwiedemann:reproducible for unknown reason lyx-2.2.3/src/frontends/qt4/Resources.cpp has random hex ==> ./maven-invoker/.rb.notes <== javadoc toolchain ==> ./mbedtls/.rb.notes <== = https://github.com/Mbed-TLS/mbedtls/issues/6978 FTBFS-2023 found range good=1691398940 bad=1691401091 (2023-08-07T09:38:11) ==> ./meson/.rb.notes <== /usr/lib/rpm/macros.d/macros.meson smp ==> ./mingw32-binutils/.rb.notes <== /usr/i686-w64-mingw32/sys-root/mingw/bin/readelf.exe differs at offset '137' (PE32 executable (console) Intel 80386, for MS Windows) ?timestamps? ==> ./mozilla-nss/.rb.notes <== = https://bugzilla.mozilla.org/show_bug.cgi?id=1813401 FTBFS-2023 found range good=1693787268 bad=1693789193 (2023-09-04T00:59:53) ==> ./musescore/.rb.notes <== ASLR in /usr/bin/mscore ==> ./mysql-connector-java/.rb.notes <== java jdbc order ; xmvn random uuid ; javadoc /usr/share/java/mysql-connector-j.jar/com/mysql/cj/jdbc/ClientPreparedStatement.class differs at offset '18905' (Cafe Babe compiled Java class data, version 52.0 (Java 1.8)) ==> ./nim/.rb.notes <== -j1 failed ==> ./openscap/.rb.notes <== = https://github.com/OpenSCAP/openscap/issues/1909 FTBFS => https://github.com/OpenSCAP/openscap/commit/8dec1bb5e9546e75ae6e7b7cf94cf001... ==> ./openssl-1_0_0/.rb.notes <== ordering issues in obj_dat.h = https://bugzilla.opensuse.org/show_bug.cgi?id=1201627 FTBFS 2023-05-26 ==> ./osc/.rb.notes <== => https://github.com/praiskup/argparse-manpage/pull/76 toolchain date # filterdiff zcat R*/usr/share/man/man1/osc.1.gz ==> ./osgi-service-log/.rb.notes <== java date+time +++ new//usr/share/java/osgi-service-log/org.osgi.service.log.jar/META-INF/maven/org.osgi/org.osgi.service.log/pom.properties 2023-01-29 07:14:54.625684100 +0000 ==> ./pdsh/.rb.notes <== minor rpm metadata filterdiff printrpmtags RPMS*/*src.rpm|grep ^- ==> ./perl-HTTP-Cookies/.rb.notes <== => https://github.com/libwww-perl/HTTP-Cookies/pull/72 FTBFS-2038 ==> ./python311/.rb.notes <== FTBFS osc build --clean --noservice --vm-type=kvm -j4 standard ==> ./python3-pyside6/.rb.notes <== _smp_build_ncpus embedded in .src.rpm https://github.com/rpm-software-management/rpm/issues/2343 ASLR ==> ./python-aiohttp/.rb.notes <== FTBFS-2038 --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed ==> ./python-aiosmtplib/.rb.notes <== => SR 1058352 toolchain python-trustme FTBFS-2038-01-01 SSL found range good=2145916590 bad=2145918515 (2038-01-01T00:28:35) ==> ./python-anyio/.rb.notes <== => SR 1058352 toolchain python-trustme FTBFS-2038-01-01 TLS ==> ./python-asciimatics/.rb.notes <== => https://github.com/peterbrittain/asciimatics/pull/365 FTBFS = https://github.com/peterbrittain/asciimatics/issues/359 report FTBFS-2038 ==> ./python-asyncpg/.rb.notes <== = https://github.com/MagicStack/asyncpg/issues/997 FTBFS-2032 found range good=1956864895 bad=1956866820 (2032-01-04T22:07:00) ==> ./python-cheroot/.rb.notes <== => SR 1058352 python-aiosmtplib/python-trustme toolchain FTBFS-2038-01-01 SSL ==> ./python-compileall2/.rb.notes <== => https://github.com/fedora-python/compileall2/pull/26 FTBFS-2038 ==> ./python-DateTime/.rb.notes <== found range good=2164298238 bad=2164300163 (2038-08-01T18:29:23) ==> ./python-devpi-server/.rb.notes <== = https://github.com/devpi/devpi/issues/948 FTBFS -j1 failed osc build --clean --vm-type=kvm -j1 --noservice standard ==> ./python-Django/.rb.notes <== => https://github.com/django/django/pull/16459 FTBFS-2038 => https://github.com/django/django/pull/10994 FTBFS-2028 to 2038 ==> ./python-enaml/.rb.notes <== => https://github.com/nucleic/enaml/pull/516 FTBFS-2038 osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-02-01T18:00:00" standard ==> ./python-gwpy/.rb.notes <== = failing from astropy leap second expiry warning https://github.com/astropy/astropy/issues/10228 --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed ==> ./python-mitmproxy/.rb.notes <== => https://github.com/mitmproxy/mitmproxy/pull/5852 FTBFS-2037 ==> ./python-pexpect/.rb.notes <== 1 test fails with -j2 - wrote email to Jeff Quast <contact@jeffquast.com> FAILED tests/test_async.py::AsyncTests::test_async_replwrap_multiline - pexpe... ==> ./python-Pint/.rb.notes <== python3 toolchain ASLR .pyc ==> ./python-pycsw/.rb.notes <== FTBFS-random? CPU? --build-opt=--vm-custom-opt=-rtc base=2038-01-25T00:00:00 failed ==> ./python-Pygments/.rb.notes <== python3 toolchain .pyc ASLR ==> ./python-pytest-httpbin/.rb.notes <== FTBFS-2038 SSL found range good=2179597065 bad=2179598990 (2039-01-25T20:09:50) ==> ./python-rich/.rb.notes <== python3 toolchain .pyc ASLR ==> ./python-softlayer-zeep/.rb.notes <== python3 toolchain ASLR .pyc ==> ./python-sunpy/.rb.notes <== = https://github.com/sunpy/sunpy/issues/6030 FTBFS-2023 found range good=1687994158 bad=1687997370 (2023-06-29T00:09:30) ==> ./python-tasklib/.rb.notes <== => https://github.com/GothenburgBitFactory/taskwarrior/pull/3052 fix FTBFS-2038 = https://github.com/GothenburgBitFactory/taskwarrior/issues/3050 FTBFS-2038 - toolchain from taskwarrior int overflow ==> ./python-trio/.rb.notes <== FTBFS-2038 toolchain trustme python3 pyc toolchain ASLR ==> ./qca/.rb.notes <== FTBFS-2031 found range good=1672719473 bad=1672721397 (2023-01-03T04:49:57) ==> ./qemu/.rb.notes <== /usr/share/qemu/slof.bin differs at offset '108' (data) ==> ./qt6-quick3d/.rb.notes <== TODO /usr/lib64/libQt6Quick3DHelpers.so.6.4.2 differs in ELF sections ==> ./qt-creator/.rb.notes <== /usr/lib64/qtcreator/libTracing.so.9.0.1 differs in ELF sections ?from mocs_compilation.cpp? ==> ./rnp/.rb.notes <== = https://github.com/rnpgp/rnp/issues/1961 FTBFS-2024-01-19 from src/lib/sec_profile.cpp SHA-1 depecation #found range good=1705629262 bad=1705631160 (2024-01-19T02:26:00) ==> ./rss2email/.rb.notes <== python3 toolchain .pyc ASLR ==> ./stax-ex/.rb.notes <== java maven random uuid ; javadoc ; other ==> ./thonny/.rb.notes <== python3 toolchain .pyc ASLR ==> ./tor/.rb.notes <== FTBFS-2037-04-20 found range good=2123823843 bad=2123825767 (2037-04-20T07:36:07) ==> ./uom-parent/.rb.notes <== java xmvn maven random UUID ==> ./velociraptor/.rb.notes <== = https://bugzilla.opensuse.org/show_bug.cgi?id=1207369 report date filterdiff strings RPMS*/usr/bin/velociraptor ==> ./weblate/.rb.notes <== = https://github.com/WeblateOrg/weblate/issues/8556 FTBFS-2038 toolchain mercurial ERROR: test_upstream_changes_rename (weblate.vcs.tests.test_vcs.VCSHgTest) ==> ./wrk/.rb.notes <== = https://github.com/wg/wrk/issues/507 hash order issue /usr/bin/wrk ==> ./xemacs/.rb.notes <== => SR 1058331 FTBFS-2038 stuck osc build --vm-type=kvm --noservice --clean --build-opt=--vm-custom-opt="-rtc base=2038-12-10T00:59:10" standard ==> ./zellij/.rb.notes <== /usr/bin/zellij differs in ELF sections from LTO? random?