Hi, last month's status: https://lists.opensuse.org/opensuse-factory/2019-03/msg00362.html Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/blog/posts/204/ https://reproducible-builds.org/blog/posts/205/ https://reproducible-builds.org/blog/posts/206/ They are currently switching to monthly reports with the 2019-04 one still in progress. I uploaded https://rb.zq1.de/compare.factory-20190429/ today and rbstats are: total-packages: 11721 build-tried: 11720 build-failed: 70 build-n-a: 111 build-succeeded: 11539 build-official-failed+na: 123 build-compare-failed: 290 build-compare-succeeded: 11249 verified-semi-reproducible: 9719 verified-bit-identical: 0 bit-by-bit-identical: 10933 not-bit-by-bit-identical: 598 https://rb.zq1.de/compare.factory-20190429/graph.png shows that the last weeks had pretty little change. The 'verified-semi-reproducible' number went up because I was able to build most packages with older Tumbleweed binaries thanks to https://lizards.opensuse.org/2019/04/03/experimental-opensuse-mirror-via-ipf... There are still some Tumbleweed binaries from before February that cannot be reproduced due to missing build deps. Of the badly unreproducible packages, 5 were in ring0 41 were in ring1 That makes it 46/2901 => 1.59 % which is below the overall average of 290/11539 => 2.51 % Fixed core packages since last month: gimp jing-trang from improved build-compare many python modules from restored python3 byte_compile sort patch Newly unreproducible core (DVD) packages: branding-openSUSE parallelism race causes 5120x3200.jpg to go missing silently already fixed in https://github.com/openSUSE/branding/pull/111 libqt5-qttools filesystem readdir order via qttools-everywhere-src-5.11.2/src/designer/src/lib/.rcc/qrc_SmartPhoneWithButtons.cpp maybe from cd src/designer/src/ && qmake -o Makefile src.pro Also, tchvatal looked into libreoffice and found some unreproducible .zip files (named .jar and .bau) Ciao Bernhard M.

Hi, last month's status: https://lists.opensuse.org/opensuse-factory/2019-04/msg00414.html Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-04/ The 2019-05 report is still a raw draft. I uploaded https://rb.zq1.de/compare.factory-20190528/ today and rbstats are: total-packages: 11911 build-tried: 11909 build-failed: 53 build-tried: 11909 build-n-a: 111 build-succeeded: 11745 build-official-failed+na: 182 build-compare-failed: 338 build-compare-succeeded: 11407 verified-semi-reproducible: 9937 verified-bit-identical: 0 bit-by-bit-identical: 11105 not-bit-by-bit-identical: 636 https://rb.zq1.de/compare.factory-20190528/graph.png shows the change over time https://rb.zq1.de/compare.factory-20190528/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 5 were in ring0 42 were in ring1 That makes it 47/2901 => 1.62 % which is below the overall average of 338/11745 => 2.88 % 636/11745 => 5.42 % of packages are not perfectly reproducible Nearly fixed: acl https://build.opensuse.org/request/show/705693 python-marshmallow https://build.opensuse.org/request/show/705891 already fixed upstream https://github.com/marshmallow-code/marshmallow/pull/679 Progress: libreoffice Some progress with .bau files - those are now reproducible thanks to a mtime-normalization patch in the zip package. Remaining variations in .thm, .sdg, overview-summary.html and segments_3 files Newly unreproducible: containerd /usr/sbin/containerd-shim has buildid variations from parallelism similar to many other go packages go1.12 (same problems as 1.10 and 1.11 have) variations in -a files and binaries: go, trace, pprof, objdump, nm, addr2line python-numpy .pyc file variation (maybe ref counters) python-tox .pyc files contain random /tmp/ path from pip install https://bugzilla.opensuse.org/show_bug.cgi?id=1094323 affects 27+ packages atm Ciao Bernhard M.

Hi, last month's status: https://lists.opensuse.org/opensuse-factory/2019-06/msg00429.html Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-06/ I made some nice progress with Firefox+Thunderbird (see below). This month also saw the enabling of Link Time Optimization (LTO) in Factory and this triggered a class of unreproducibility because the number of build CPUs was added to the CFLAGS value. As a quick workaround I added %_lto_cflags -flto=2 to https://build.opensuse.org/project/prjconf/home:bmwiedemann:reproducible so that it does not reduce visibility of other reproducibility issues. https://bugzilla.opensuse.org/show_bug.cgi?id=1141319 https://bugzilla.opensuse.org/show_bug.cgi?id=1141323 are related. A nicer solution still needs to be found. Nearly fixed core packages: MozillaFirefox/MozillaThunderbird: https://bugzilla.opensuse.org/show_bug.cgi?id=1137970 was fixed and https://bugzilla.mozilla.org/show_bug.cgi?id=1568145 contains a working patch newly unreproducible core packages: gstreamer-plugins-bad filesystem order similar to https://bugzilla.gnome.org/show_bug.cgi?id=794069 libqt5-qttools maybe was not really fixed last month? half-building packages that disturb my tools: python-nbconvert https://bugzilla.opensuse.org/show_bug.cgi?id=1136099 python-joblib https://github.com/joblib/joblib/issues/870 python-openstacksdk https://bugzilla.opensuse.org/show_bug.cgi?id=1107814 build stuck I uploaded https://rb.zq1.de/compare.factory-20190726/ today and rbstats (with comparison to last month) are: total-packages: 12096 (+82) build-tried: 12086 (+82) build-failed: 48 (+0) build-n-a: 119 (+13) build-succeeded: 11919 (+69) build-official-failed+na: 208 (+35) build-compare-failed: 352 (+4) build-compare-succeeded: 11567 (+65) verify-failed: 560 (+29) verified-semi-reproducible: 11279 (+33) verified-bit-identical: 0 (+0) bit-by-bit-identical: 11257 (+68) not-bit-by-bit-identical: 659 (+8) https://rb.zq1.de/compare.factory-20190726/graph.png shows the change over time https://rb.zq1.de/compare.factory-20190726/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 3 were in ring0 40 were in ring1 That makes it 43/2901 => 1.48 % which is below the overall average of 352/11919 => 2.95 % 659/11919 => 5.53 % of packages are not perfectly reproducible Ciao Bernhard M.

Hi, last month's status: https://lists.opensuse.org/opensuse-factory/2019-08/msg00186.html Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-08/ Builds this month were much cleaner again with several long-standing build issues fixed. This month, I also pushed https://github.com/python/cpython/pull/12341 for inclusion in our python and python3 packages to fix a whole class of reproducibility issues with python C bindings. Tested it with python-ephem - the latest (and last) newcomer in Factory to be affected. Another toolchain fix that helps several others to become reproducible is for OpenStack packages: https://build.opensuse.org/request/show/732328 I uploaded https://rb.zq1.de/compare.factory-20190926/ today and rbstats are: total-packages: 12270 build-tried: 12260 build-failed: 115 build-n-a: 125 build-succeeded: 12020 build-official-failed+na: 220 build-compare-failed: 408 build-compare-succeeded: 11612 verify-failed: 588 verified-semi-reproducible: 11396 verified-bit-identical: 0 bit-by-bit-identical: 11201 not-bit-by-bit-identical: 814 https://rb.zq1.de/compare.factory-20190926/graph.png shows the change over time https://rb.zq1.de/compare.factory-20190926/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 3 were in ring0 49 were in ring1 That makes it 52/2901 => 1.79 % which is below the overall average of 408/12020 => 3.39 % 814/12020 => 6.77 % of packages are not perfectly reproducible Nearly fixed core packages: MozillaThunderbird https://build.opensuse.org/request/show/732106 kernel-vanilla https://lists.opensuse.org/opensuse-kernel/2019-08/msg00000.html I also found it is possible to build a reproducible Firefox, but only when disabling PGO - and that was only possible when disabling LTO at the same time (build errors otherwise): https://build.opensuse.org/request/show/733089 So this is one of the places where one would have to make a trade-off between performance of the binary and reproducibility. I heard, upstream solved this by publishing the profiling data for other people to use as input to their builds. Not such a nice solution either. Then there is still one rpm update missing to use -flto=auto As can be seen in https://build.opensuse.org/package/revisions/openSUSE:Factory/rpm it was briefly there and then reverted the next day, because it was bundled with another (broken) change. Since I am keeping a history of all Tumbleweed binaries (see https://lizards.opensuse.org/2019/04/03/experimental-opensuse-mirror-via-ipf... ), I also noticed that the normal Factory update grew from ~4GB to ~8GB after LTO was enabled. Still need to do a closer investigation on why that is. Here is some raw data for x86 dirs with hardlinks: opensuse/history$ du -smc * 90090 20190630 4270 20190701 4967 20190702 5225 20190703 4493 20190704 7526 20190708 72674 20190713 # LTO rebuild 18845 20190716 7373 20190718 17633 20190721 7812 20190723 16653 20190724 7137 20190726 16724 20190730 20694 20190805 7951 20190806 8047 20190809 8182 20190810 9065 20190814 19168 20190815 11267 20190820 8955 20190822 8326 20190823 7129 20190824 32824 20190828 9543 20190829 9532 20190902 7183 20190904 7836 20190905 8484 20190907 9266 20190909 11367 20190916 6468 20190917 10443 20190918 8100 20190920 7868 20190921 519103 total Ciao Bernhard M. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, last month's status: https://lists.opensuse.org/opensuse-factory/2019-09/msg00244.html Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-09/ This month saw https://build.opensuse.org/request/show/732635 to update rpm with -flto=auto (since 2019-10-21). I already found that it reduced the size of the last 3 Tumbleweed updates to below 7GB. All previous updates since 20190704 were larger than 7GB, so this is going to save some disk space on mirrors and bandwidth for users and mirror operators. I uploaded https://rb.zq1.de/compare.factory-20191028/ today and rbstats are: total-packages: 12272 (+2) build-tried: 12264 (+4) build-failed: 74 (-41) build-n-a: 83 (-42) build-succeeded: 12107 (+87) build-official-failed+na: 219 (-1) build-compare-failed: 396 (-12) build-compare-succeeded: 11711 (+99) verify-failed: 578 (-10) verified-semi-reproducible: 11437 (+41) verified-bit-identical: 0 bit-by-bit-identical: 11318 (+117) not-bit-by-bit-identical: 781 (-33) https://rb.zq1.de/compare.factory-20191028/graph.png shows the change over time https://rb.zq1.de/compare.factory-20191028/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 2 were in ring0 48 were in ring1 That makes it 50/2901 => 1.72 % which is below the overall average of 396/12107 => 3.27 % 781/12107 => 6.45 % of packages are not perfectly reproducible newly unreproducible core packages: latex2html got one fix https://build.opensuse.org/request/show/733232 but now it has variations in /usr/share/doc/packages/latex2html-doc/tests/l2htest/WARNINGS perf probably has non-deterministic order from /home/abuild/rpmbuild/BUILD/tools/perf/pmu-events/pmu-events.c getting random filesystem order via a nftw call in linux/tools/perf/pmu-events/jevents.c MozillaThunderbird variations in zip file /usr/lib64/thunderbird/distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}.xpi newly reproducible core packages: openmpi{1,2} postgresql{95,96,10} via the rpm flto update Ciao Bernhard M.

Hi, last month's status: https://lists.opensuse.org/opensuse-factory/2019-10/msg00367.html Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-10/ I uploaded https://rb.zq1.de/compare.factory-20191129/ today and rbstats are: total-packages: 12424 (+152) build-tried: 12416 (+152) build-failed: 51 (-23) build-n-a: 81 (-2) build-succeeded: 12284 (+177) build-official-failed+na: 126 (-93) build-compare-failed: 415 (+19) build-compare-succeeded: 11869 (+158) verify-failed: 560 (-18) verified-semi-reproducible: 11608 (+171) verified-bit-identical: 0 bit-by-bit-identical: 11481 (+163) not-bit-by-bit-identical: 794 (+13) https://rb.zq1.de/compare.factory-20191129/graph.png shows the change over time https://rb.zq1.de/compare.factory-20191129/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD) Of the badly unreproducible packages, 2 were in ring0 48 were in ring1 That makes it 50/2901 => 1.72 % which is below the overall average of 415/12284 => 3.38 % 794/12284 => 6.46 % of packages are not perfectly reproducible This month, I added https://github.com/apache/felix/pull/209 as a test patch in https://build.opensuse.org/package/show/home:bmwiedemann:reproducible:test/m... and in addition with last month's https://build.opensuse.org/request/show/735873 that made it possible to build some dozen java packages reproducible by applying tweaks (e.g. use strip-nondeterminism and replace /dev/*random with /dev/zero to counter random maven UUIDs) On the unreproducible core package list, we have branding-openSUSE variations in .jpg files tomcat filesystem ordering issues and probably others python-python-xlib one .pyc file with timestamp newly reproducible core packages: nvme-cli from a gcc9 LTO patch = https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91307 python-tox from the https://bugzilla.opensuse.org/show_bug.cgi?id=1094323 toolchain patch And in the next week I am off to the reproducible-builds world summit. Ciao Bernhard M.