Hi,
last month's status:
https://lists.opensuse.org/opensuse-factory/2018-12/msg00171.html
Last months' r-b project updates (including my work): https://reproducible-builds.org/blog/posts/195/ https://reproducible-builds.org/blog/posts/194/ https://reproducible-builds.org/blog/posts/193/ https://reproducible-builds.org/blog/posts/192/
I uploaded https://rb.zq1.de/compare.factory-20190125/ today and rbstats are: total-packages: 11298 build-tried: 11297 build-failed: 37 build-n-a: 107 build-succeeded: 11153 build-official-failed+na: 85 build-compare-failed: 283 build-compare-succeeded: 10870 bit-by-bit-identical: 10605 not-bit-by-bit-identical: 538
Fixed core packages since last month: breeze5-icons dolphin libqt5-qtwebkit wireshark syntax-highlighting: Qt rcc/qrc fix merged in Factory https://build.opensuse.org/request/show/663360 also fixed dozens others like bitcoin.
opa-ff CPU-detection via -march=native https://build.opensuse.org/request/show/661771
gengetopt https://build.opensuse.org/request/show/661735
nearly fixed: pcre2 PGO/parallelism https://build.opensuse.org/request/show/668163
perl: ASLR-induced randomness https://build.opensuse.org/request/show/668211
--- ring0 --- acl gcc7 gcc8 --- ring1 --- MozillaFirefox MozillaThunderbird ant bsf colord emacs firebird gconf2-branding-openSUSE gdb gimp gnome-documents go1.10 go1.11 groff-full grub2 hamcrest installation-images java-11-openjdk java-1_8_0-openjdk jing-trang jtidy junit junitperf kernel-debug kernel-default kernel-vanilla kubernetes libkolabxml libqt5-qtscript libreoffice mono-core mozilla-nss pcp php7 python-base python3-base qdox release-notes-openSUSE ruby2.5 rubygem-gem2rpm rust scons transfig virtualbox xmlbeans yodl
If you are interested in why they vary: acl: date+time in /usr/share/locale/en@boldquot/LC_MESSAGES/acl.mo -PO-Revision-Date: 2018-10-23 01:45+0000 +PO-Revision-Date: 2033-11-24 15:01+0000
gcc7 gcc8: indeterminism from PGO/parallelism + mtime. See also https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/pgo
MozillaFirefox MozillaThunderbird: date+time, rust, other has bug around update.locale symlink /usr/lib64/firefox/libxul.so varies from https://github.com/rust-lang/rust/issues/57041 .../browser/extensions/langpack-ca@xxxxxxxxxxxxxxxxxxx/manifest.json - "version": "20181009013946" + "version": "20331110172641"
apache-commons-lang3 apache-pdfbox dom4j hsqldb icu4j javassist jing-trang jtidy junitperf jython objectweb-asm qdox tomcat xerces-j2 xpp2 xpp3: date + mtime in .jar + javadoc html can be normalized by strip-nondeterminism
bsh2 cglib: /usr/share/maven-metadata/bsh2.xml has ASLR order issues date + mtime in .jar + javadoc html
colord some randomness from uninitialized memory maybe from glib-compile-resources 16-byte random profileID in .icc files from cd-create-profile plus an uncommitted date/time fix in lib/colord/cd-it8.c: priv->enable_created = FALSE
ecj ASLR + date? /usr/share/maven-metadata/ecj.xml has alias entries in random order
emacs dumps lisp interpreter memory to create its binaries (similar to clisp) causes variations in /usr/bin/emacs-*
firebird: ships unreproducible database http://tracker.firebirdsql.org/browse/CORE-5548
gconf2-branding-openSUSE embeds mtime values in /etc/gconf/gconf.xml.vendor/%gconf-tree.xml
gdb contains testresults https://bugzilla.opensuse.org/show_bug.cgi?id=1110708
gimp ASLR 12 byte differ in bKGD header from gegl GEGL_USE_OPENCL=no GEGL_SWAP=ram /usr/bin/gegl ../../icons/Symbolic/64/gimp-texture.png -o 64/gimp-texture.png -- gegl:invert-gamma
gnome-documents gnome-documents-getting-started.pdf has random ID from inkscape
go1.10 go1.11: variations in /usr/lib64/go/1.10/pkg/obj/go-build/*/*-a
groff-full date+time + unknown reason? .ps files vary .html files have CreationDate
grub2 mtime + readdir https://savannah.gnu.org/bugs/index.php?54841
hamcrest java .class files vary from build/temp/hamcrest-core/generated-code /org/hamcrest/CoreMatchers.java written by ant javadoc html varies
installation-images many variations from mtimes + filesys/readdir and %post scripts
java-1_8_0-openjdk java-10-openjdk java-11-openjdk various .jar .zip .html .jmod ordering in /usr/lib64/jvm/java-10-openjdk-10/lib/classlist
kernel-debug kernel-default kernel-vanilla date+time ; random keys? "Build time autogenerated kernel key0 ..181009012108Z..21180915012108Z0.1,0"
kubernetes: fixed upstream: order in man-pages https://github.com/kubernetes/kubernetes/pull/68983 maybe fixed upstream: random build-ids
libkolabxml unknown/ASLR? from build/kolabformat-xcal-schema.cxx created by compiled/xsdbin.cxx
libqt5-qtscript filesys order
libreoffice various .jar .so .dat/.bau (.zip) javadoc_log.txt
mono-core date+time ; other
mozilla-nss DSA random temp-key from shlibsign https://bugzilla.opensuse.org/show_bug.cgi?id=1081723
pcp /var/lib/pcp/testsuite/perfevent/perfevent_coverage has random diffs from gcov / .gcno files causing diff in .o ?
php7 date / EPOCH timestamps e.g. in /usr/share/php7/PEAR/.channels/__uri.reg
python-base python3-base: PGO varies .o files that go into /usr/lib64/libpython3.6m.so.1.0 .pyc files vary python-base shows influence from date+hostname (in PGO?)
python-numpy 1 .pyc file varied
release-notes-openSUSE partial fix in https://github.com/openSUSE/daps/issues/482 date+time in .pdf ; needs work on fop java code random id values in .html
ruby2.5 2 gemspec files have date created.rid varies
rubygem-gem2rpm mtime ? /usr/lib64/ruby/gems/2.5.0/cache/checksums.yaml varies
rust asm diffs in cargo and others from llvm's llc - filed at https://github.com/rust-lang/rust/issues/57041 some progress made here with the switch to llvm7
scons hostname in various places
tigervnc date+time+randomness in /usr/share/vnc/classes/META-INF/TIGERVNC.RSA from SignJar.cmake calling jarsigner with temporary private key partial fix merged https://github.com/TigerVNC/tigervnc/pull/765
transfig date+time in sample-presentation.pdf /CreationDate /ModDate from pdflatex
virtualbox tar + random https://www.virtualbox.org/ticket/16854 .so files have random NT_GNU_BUILD_ID (unique build ID bitstring) maybe from out/linux.amd64/release/obj/webservice/gsoapH_from_gsoap.h that has date+time in comment /usr/share/virtualbox/extensions/VNC-5.2.16.vbox-extpack unhandled gzip content: POSIX tar archive (GNU) -13343137165.010505. 5... +16773576625.010527. 5...
xmlbeans index.xsb varies from filesystem created in org.apache.xmlbeans.impl.tool.SchemaCompiler
yodl pdf from latex/ghostscript/dvips/ps2pdf
additionally, on the SLE15 list ===============================
gnuplot date+time in pdf from pdflatex
golang-github-prometheus-prometheus date+time+random build-id
hawk2 https://bugzilla.opensuse.org/show_bug.cgi?id=1112159
ibus-libzhuyin libpinyin libzhuyin random bytes - maybe uninitialized memory
perf random filesys order from nftw call in ./linux-*/tools/perf/pmu-events/jevents.c
python-keystoneauth1 https://bugs.launchpad.net/keystoneauth/+bug/1796899
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-01/msg00393.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/blog/posts/199/ https://reproducible-builds.org/blog/posts/198/ https://reproducible-builds.org/blog/posts/197/ https://reproducible-builds.org/blog/posts/196/
I uploaded https://rb.zq1.de/compare.factory-20190228/ today and rbstats are: total-packages: 11405 build-tried: 11405 build-failed: 80 build-n-a: 108 build-succeeded: 11217 build-official-failed+na: 130 build-compare-failed: 287 build-compare-succeeded: 10930 bit-by-bit-identical: 10652 verified-semi-reproducible: 9106 verified-bit-identical: 0 not-bit-by-bit-identical: 569
The new "verified" lines are about comparison of the official build results with what I got through my local "nachbau" build.
Of the badly unreproducible packages, 5 were in ring0 46 were in ring1
That makes it 51/2744 => 1.86 % which is below the overall average of 287/11217 => 2.56 %
Fixed core packages since last month: ant
libqt5-qtscript
pcre2 PGO/parallelism https://build.opensuse.org/request/show/668163
rubygem-gem2rpm
And 4 newly semi-reproducible packages - that means build-compare can now tell that results are similar enough thanks to https://github.com/openSUSE/build-compare/pull/31 bsf jtidy junit junitperf
Newly unreproducible core (DVD) packages: bash llvm7 apache-commons-codec bsh2 cglib eigen3 gnuplot grep hsqldb libqt5-qtquickcontrols python-pip python-pycryptodome sac scons
If you are interested in why they vary: apache-commons-codec html doc variations autoclassify says: from low-entropy filesystem order non-determinism
bash PGO-related variations from PID and date and other influences in profiling run ; since upgrade to 5.0
bsh2 cglib /usr/share/maven-metadata/bsh2.xml has ASLR order issues
eigen3 latex .log files => https://build.opensuse.org/request/show/679669
gnuplot date+time in eg7.pdf from gnuplot+epstopdf
grep PGO again
hsqldb build-compare's javadoc filters do not apply to the files because their path does not match /usr/*/javadoc/*.html
llvm7 variations in .a files and some debuginfo
libqt5-qtquickcontrols 1-byte-variations in qmlc files from ASLR
python-pip python-pycryptodome unreproducible .pyc marshalling https://bugs.python.org/issue34033 low-entropy so may randomly produce same results twice
scons: hostname => https://github.com/SCons/scons/pull/3312
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-02/msg00599.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/blog/posts/200/ https://reproducible-builds.org/blog/posts/201/ https://reproducible-builds.org/blog/posts/202/ https://reproducible-builds.org/blog/posts/203/
I started https://raw.githubusercontent.com/bmwiedemann/reproducibleopensuse/devel/how... as a preparation for my openSUSE conference session.
I uploaded https://rb.zq1.de/compare.factory-20190326/ today and rbstats are: total-packages: 11547 build-tried: 11546 build-failed: 79 build-n-a: 107 build-succeeded: 11360 build-official-failed+na: 175
build-compare-failed: 289 build-compare-succeeded: 11071
verified-semi-reproducible: 9396 verified-bit-identical: 0
bit-by-bit-identical: 10777 not-bit-by-bit-identical: 579
The new "verified" lines are about comparison of the official build results with what I got through my local "nachbau" build. Official builds do not normalize mtimes, so no bit-identical reproduction is possible there. The number is a bit higher now, because old 2018 results have been re-checked. Need to build with the proper old binaries to get even better results.
Of the badly unreproducible packages, 5 were in ring0 47 were in ring1
That makes it 52/2811 => 1.85 % which is below the overall average of 289/11535 => 2.51 %
Fixed core packages since last month: pcp via https://build.opensuse.org/request/show/682435
Nearly fixed core packages since last month: libqt5-qtquickcontrols qt-5.12.3 will have the fix https://bugreports.qt.io/browse/QTBUG-74532
Newly unreproducible core (DVD) packages: libqt5-qtscript from filesystem order via Qt's rcc -name context2d context2d.qrc -o .rcc/qrc_context2d.cpp
many python modules because https://github.com/python/cpython/pull/8057 was dropped from our python3 package this is very annoying and steals focus from other issues.
Ciao Bernhard M.
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-03/msg00362.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/blog/posts/204/ https://reproducible-builds.org/blog/posts/205/ https://reproducible-builds.org/blog/posts/206/ They are currently switching to monthly reports with the 2019-04 one still in progress.
I uploaded https://rb.zq1.de/compare.factory-20190429/ today and rbstats are: total-packages: 11721 build-tried: 11720 build-failed: 70 build-n-a: 111 build-succeeded: 11539 build-official-failed+na: 123
build-compare-failed: 290 build-compare-succeeded: 11249
verified-semi-reproducible: 9719 verified-bit-identical: 0
bit-by-bit-identical: 10933 not-bit-by-bit-identical: 598
https://rb.zq1.de/compare.factory-20190429/graph.png shows that the last weeks had pretty little change.
The 'verified-semi-reproducible' number went up because I was able to build most packages with older Tumbleweed binaries thanks to https://lizards.opensuse.org/2019/04/03/experimental-opensuse-mirror-via-ipf...
There are still some Tumbleweed binaries from before February that cannot be reproduced due to missing build deps.
Of the badly unreproducible packages, 5 were in ring0 41 were in ring1
That makes it 46/2901 => 1.59 % which is below the overall average of 290/11539 => 2.51 %
Fixed core packages since last month: gimp
jing-trang from improved build-compare
many python modules from restored python3 byte_compile sort patch
Newly unreproducible core (DVD) packages: branding-openSUSE parallelism race causes 5120x3200.jpg to go missing silently already fixed in https://github.com/openSUSE/branding/pull/111
libqt5-qttools filesystem readdir order via qttools-everywhere-src-5.11.2/src/designer/src/lib/.rcc/qrc_SmartPhoneWithButtons.cpp maybe from cd src/designer/src/ && qmake -o Makefile src.pro
Also, tchvatal looked into libreoffice and found some unreproducible .zip files (named .jar and .bau)
Ciao Bernhard M.
Newly unreproducible core (DVD) packages: branding-openSUSE parallelism race causes 5120x3200.jpg to go missing silently already fixed in https://github.com/openSUSE/branding/pull/111
Thanks a bunch for that submission, it saved me a lot of time ;) It fixed two issues, with gfxboot branding returning to 12.x branding, and that flickering wallpaper as mentioned.
LCP [Stasiek] https://lcp.world
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-04/msg00414.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-04/ The 2019-05 report is still a raw draft.
I uploaded https://rb.zq1.de/compare.factory-20190528/ today and rbstats are: total-packages: 11911 build-tried: 11909 build-failed: 53 build-tried: 11909 build-n-a: 111 build-succeeded: 11745 build-official-failed+na: 182 build-compare-failed: 338 build-compare-succeeded: 11407
verified-semi-reproducible: 9937 verified-bit-identical: 0
bit-by-bit-identical: 11105 not-bit-by-bit-identical: 636
https://rb.zq1.de/compare.factory-20190528/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20190528/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 5 were in ring0 42 were in ring1
That makes it 47/2901 => 1.62 % which is below the overall average of 338/11745 => 2.88 %
636/11745 => 5.42 % of packages are not perfectly reproducible
Nearly fixed: acl https://build.opensuse.org/request/show/705693
python-marshmallow https://build.opensuse.org/request/show/705891 already fixed upstream https://github.com/marshmallow-code/marshmallow/pull/679
Progress: libreoffice Some progress with .bau files - those are now reproducible thanks to a mtime-normalization patch in the zip package. Remaining variations in .thm, .sdg, overview-summary.html and segments_3 files
Newly unreproducible: containerd /usr/sbin/containerd-shim has buildid variations from parallelism similar to many other go packages
go1.12 (same problems as 1.10 and 1.11 have) variations in -a files and binaries: go, trace, pprof, objdump, nm, addr2line
python-numpy .pyc file variation (maybe ref counters)
python-tox .pyc files contain random /tmp/ path from pip install https://bugzilla.opensuse.org/show_bug.cgi?id=1094323 affects 27+ packages atm
Ciao Bernhard M.
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-05/msg00341.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-05/
This month, due to recent rebuilds, the number of verified packages grew so much that I could check that currently all verifiable Tumbleweed binaries contain what they should. Not verified are: 1: 348+ packages known to not build reproducibly 2: 16 efi and kmp packages - mostly because https://github.com/openSUSE/pesign-obs-integration/pull/14 was reverted 3: ~80 packages using _multibuild without empty FLAVOR (needs work in my tooling) 4: ~5 packages with unsubmitted local fixes - that number shrunk a lot last month
The other caveat here is that this is using build-compare and that already had 3 bugs that caused it to report different rpms as identical. To avoid the possibility of such bugs in the future, I'd like to see bit-identical verifiable rpms for Leap 16. This is tracked in https://bugzilla.opensuse.org/show_bug.cgi?id=1133809
Newly reproducible core packages: acl https://build.opensuse.org/request/show/705693
gnome-documents had unreproducible pdf from inkscape - file was dropped
gcc8 only reproducible when building with %do_profiling 0 That is not the default in Factory to gain 8% compiler performance. note: gcc9 is similar.
libqt5-qttools maybe from rcc toolchain fixes? or maybe there are still low-entropy order issues in rcc output
python-marshmallow https://build.opensuse.org/request/show/705891
Still unreproducible: MozillaFirefox+Thunderbird filed http://bugzilla.opensuse.org/show_bug.cgi?id=1137970
I uploaded https://rb.zq1.de/compare.factory-20190626/ today and rbstats (with comparison to last month) are: total-packages: 12014 (+103) build-tried: 12004 (+95) build-failed: 48 (-5) build-n-a: 106 (-5) build-succeeded: 11850 (+105) build-official-failed+na: 173 (-9)
build-compare-failed: 348 (+10) build-compare-succeeded: 11502 (+95)
verify-failed: 531 (-1184) verified-semi-reproducible: 11246 (+1309) verified-bit-identical: 0 (+0)
bit-by-bit-identical: 11189 (+84) not-bit-by-bit-identical: 651 (+15)
https://rb.zq1.de/compare.factory-20190626/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20190626/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 3 were in ring0 39 were in ring1
That makes it 42/2901 => 1.45 % which is below the overall average of 348/11850 => 2.94 %
651/11850 => 5.49 % of packages are not perfectly reproducible
Ciao Bernhard M.
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-06/msg00429.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-06/
I made some nice progress with Firefox+Thunderbird (see below).
This month also saw the enabling of Link Time Optimization (LTO) in Factory and this triggered a class of unreproducibility because the number of build CPUs was added to the CFLAGS value.
As a quick workaround I added %_lto_cflags -flto=2 to https://build.opensuse.org/project/prjconf/home:bmwiedemann:reproducible so that it does not reduce visibility of other reproducibility issues. https://bugzilla.opensuse.org/show_bug.cgi?id=1141319 https://bugzilla.opensuse.org/show_bug.cgi?id=1141323 are related. A nicer solution still needs to be found.
Nearly fixed core packages: MozillaFirefox/MozillaThunderbird: https://bugzilla.opensuse.org/show_bug.cgi?id=1137970 was fixed and https://bugzilla.mozilla.org/show_bug.cgi?id=1568145 contains a working patch
newly unreproducible core packages: gstreamer-plugins-bad filesystem order similar to https://bugzilla.gnome.org/show_bug.cgi?id=794069
libqt5-qttools maybe was not really fixed last month?
half-building packages that disturb my tools: python-nbconvert https://bugzilla.opensuse.org/show_bug.cgi?id=1136099 python-joblib https://github.com/joblib/joblib/issues/870 python-openstacksdk https://bugzilla.opensuse.org/show_bug.cgi?id=1107814 build stuck
I uploaded https://rb.zq1.de/compare.factory-20190726/ today and rbstats (with comparison to last month) are: total-packages: 12096 (+82) build-tried: 12086 (+82) build-failed: 48 (+0) build-n-a: 119 (+13) build-succeeded: 11919 (+69) build-official-failed+na: 208 (+35) build-compare-failed: 352 (+4) build-compare-succeeded: 11567 (+65)
verify-failed: 560 (+29) verified-semi-reproducible: 11279 (+33) verified-bit-identical: 0 (+0)
bit-by-bit-identical: 11257 (+68) not-bit-by-bit-identical: 659 (+8)
https://rb.zq1.de/compare.factory-20190726/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20190726/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 3 were in ring0 40 were in ring1
That makes it 43/2901 => 1.48 % which is below the overall average of 352/11919 => 2.95 %
659/11919 => 5.53 % of packages are not perfectly reproducible
Ciao Bernhard M.
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-07/msg00364.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-07/
This month I did a full double-rebuild (with rbk[1]) of all Factory packages with Link Time Optimization (using a constant -flto=2 to evade that other known issue (fixed in devel:gcc with -flto=auto)) and that found some new issues in pcc + perl-XML-Parser - http://bugzilla.opensuse.org/show_bug.cgi?id=1146634
yudit - still to be debugged - probably nondeterminism from ASLR and/or CPU-type did not show up without LTO
and 12 others - fixed in upsteam gcc via https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91307
Builds were less clean this month, so there are ~140 packages unaccounted for.
Nearly fixed core packages: MozillaFirefox/MozillaThunderbird: https://bugzilla.mozilla.org/show_bug.cgi?id=1568145 is still pending. Maybe Wolfgang Rosenauer will add the patch into our packages.
I uploaded https://rb.zq1.de/compare.factory-20190824/ today and rbstats (with comparison to last month) are:
total-packages: 12232 (+136) build-tried: 12222 (+136) build-failed: 222 (+174) build-n-a: 131 (+12) build-succeeded: 11869 (-50) build-official-failed+na: 219 build-compare-failed: 397 (+45) build-compare-succeeded: 11472 (-95)
verify-failed: 542 (-18) verified-semi-reproducible: 11394 (+115) verified-bit-identical: 0
bit-by-bit-identical: 10949 (-308) not-bit-by-bit-identical: 776 (+117) not-bit-by-bit-identicalcheck: 920
https://rb.zq1.de/compare.factory-20190824/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20190824/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 3 were in ring0 47 were in ring1
That makes it 50/2901 => 1.72 % which is below the overall average of 397/11869 => 3.34 %
776/11869 => 6.54 % of packages are not perfectly reproducible
Ciao Bernhard M.
[1] https://github.com/bmwiedemann/reproducibleopensuse/blob/master/rbk
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-08/msg00186.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-08/
Builds this month were much cleaner again with several long-standing build issues fixed. This month, I also pushed https://github.com/python/cpython/pull/12341 for inclusion in our python and python3 packages to fix a whole class of reproducibility issues with python C bindings. Tested it with python-ephem - the latest (and last) newcomer in Factory to be affected.
Another toolchain fix that helps several others to become reproducible is for OpenStack packages: https://build.opensuse.org/request/show/732328
I uploaded https://rb.zq1.de/compare.factory-20190926/ today and rbstats are: total-packages: 12270 build-tried: 12260 build-failed: 115 build-n-a: 125 build-succeeded: 12020 build-official-failed+na: 220 build-compare-failed: 408 build-compare-succeeded: 11612
verify-failed: 588 verified-semi-reproducible: 11396 verified-bit-identical: 0
bit-by-bit-identical: 11201 not-bit-by-bit-identical: 814
https://rb.zq1.de/compare.factory-20190926/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20190926/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 3 were in ring0 49 were in ring1
That makes it 52/2901 => 1.79 % which is below the overall average of 408/12020 => 3.39 %
814/12020 => 6.77 % of packages are not perfectly reproducible
Nearly fixed core packages: MozillaThunderbird https://build.opensuse.org/request/show/732106 kernel-vanilla https://lists.opensuse.org/opensuse-kernel/2019-08/msg00000.html
I also found it is possible to build a reproducible Firefox, but only when disabling PGO - and that was only possible when disabling LTO at the same time (build errors otherwise): https://build.opensuse.org/request/show/733089 So this is one of the places where one would have to make a trade-off between performance of the binary and reproducibility. I heard, upstream solved this by publishing the profiling data for other people to use as input to their builds. Not such a nice solution either.
Then there is still one rpm update missing to use -flto=auto As can be seen in https://build.opensuse.org/package/revisions/openSUSE:Factory/rpm it was briefly there and then reverted the next day, because it was bundled with another (broken) change.
Since I am keeping a history of all Tumbleweed binaries (see https://lizards.opensuse.org/2019/04/03/experimental-opensuse-mirror-via-ipf... ), I also noticed that the normal Factory update grew from ~4GB to ~8GB after LTO was enabled. Still need to do a closer investigation on why that is.
Here is some raw data for x86 dirs with hardlinks: opensuse/history$ du -smc * 90090 20190630 4270 20190701 4967 20190702 5225 20190703 4493 20190704 7526 20190708 72674 20190713 # LTO rebuild 18845 20190716 7373 20190718 17633 20190721 7812 20190723 16653 20190724 7137 20190726 16724 20190730 20694 20190805 7951 20190806 8047 20190809 8182 20190810 9065 20190814 19168 20190815 11267 20190820 8955 20190822 8326 20190823 7129 20190824 32824 20190828 9543 20190829 9532 20190902 7183 20190904 7836 20190905 8484 20190907 9266 20190909 11367 20190916 6468 20190917 10443 20190918 8100 20190920 7868 20190921 519103 total
Ciao Bernhard M.
Hello,
Am Donnerstag, 26. September 2019, 11:12:59 CEST schrieb Bernhard M. Wiedemann:
https://rb.zq1.de/compare.factory-20190926/graph.png shows the change over time
That link gives me a 404 error. Can you please check what happened to your graph?
Since I am keeping a history of all Tumbleweed binaries (see https://lizards.opensuse.org/2019/04/03/experimental-opensuse-mirror-v ia-ipfs/ ), I also noticed that the normal Factory update grew from ~4GB to ~8GB after LTO was enabled. Still need to do a closer investigation on why that is.
Sounds interesting[tm], and might be one of the reasons why download.o.o (which keeps the last 20 Tumbleweed snapshots in /history/) ran out of disk space yesterday...
Regards,
Christian Boltz
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-09/msg00244.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-09/
This month saw https://build.opensuse.org/request/show/732635 to update rpm with -flto=auto (since 2019-10-21). I already found that it reduced the size of the last 3 Tumbleweed updates to below 7GB. All previous updates since 20190704 were larger than 7GB, so this is going to save some disk space on mirrors and bandwidth for users and mirror operators.
I uploaded https://rb.zq1.de/compare.factory-20191028/ today and rbstats are: total-packages: 12272 (+2) build-tried: 12264 (+4) build-failed: 74 (-41) build-n-a: 83 (-42) build-succeeded: 12107 (+87) build-official-failed+na: 219 (-1) build-compare-failed: 396 (-12) build-compare-succeeded: 11711 (+99)
verify-failed: 578 (-10) verified-semi-reproducible: 11437 (+41) verified-bit-identical: 0
bit-by-bit-identical: 11318 (+117) not-bit-by-bit-identical: 781 (-33)
https://rb.zq1.de/compare.factory-20191028/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20191028/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 2 were in ring0 48 were in ring1
That makes it 50/2901 => 1.72 % which is below the overall average of 396/12107 => 3.27 %
781/12107 => 6.45 % of packages are not perfectly reproducible
newly unreproducible core packages: latex2html got one fix https://build.opensuse.org/request/show/733232 but now it has variations in /usr/share/doc/packages/latex2html-doc/tests/l2htest/WARNINGS
perf probably has non-deterministic order from /home/abuild/rpmbuild/BUILD/tools/perf/pmu-events/pmu-events.c getting random filesystem order via a nftw call in linux/tools/perf/pmu-events/jevents.c
MozillaThunderbird variations in zip file /usr/lib64/thunderbird/distribution/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}.xpi
newly reproducible core packages: openmpi{1,2} postgresql{95,96,10} via the rpm flto update
Ciao Bernhard M.
On 10/28/19 6:10 PM, Bernhard M. Wiedemann wrote:
newly unreproducible core packages: latex2html got one fix https://build.opensuse.org/request/show/733232 but now it has variations in /usr/share/doc/packages/latex2html-doc/tests/l2htest/WARNINGS
That sounds like something we shouldn't even package.
Greetings, Stephan
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-10/msg00367.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-10/
I uploaded https://rb.zq1.de/compare.factory-20191129/ today and rbstats are: total-packages: 12424 (+152) build-tried: 12416 (+152) build-failed: 51 (-23) build-n-a: 81 (-2) build-succeeded: 12284 (+177) build-official-failed+na: 126 (-93) build-compare-failed: 415 (+19) build-compare-succeeded: 11869 (+158)
verify-failed: 560 (-18) verified-semi-reproducible: 11608 (+171) verified-bit-identical: 0
bit-by-bit-identical: 11481 (+163) not-bit-by-bit-identical: 794 (+13)
https://rb.zq1.de/compare.factory-20191129/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20191129/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 2 were in ring0 48 were in ring1
That makes it 50/2901 => 1.72 % which is below the overall average of 415/12284 => 3.38 %
794/12284 => 6.46 % of packages are not perfectly reproducible
This month, I added https://github.com/apache/felix/pull/209 as a test patch in https://build.opensuse.org/package/show/home:bmwiedemann:reproducible:test/m... and in addition with last month's https://build.opensuse.org/request/show/735873 that made it possible to build some dozen java packages reproducible by applying tweaks (e.g. use strip-nondeterminism and replace /dev/*random with /dev/zero to counter random maven UUIDs)
On the unreproducible core package list, we have
branding-openSUSE variations in .jpg files
tomcat filesystem ordering issues and probably others
python-python-xlib one .pyc file with timestamp
newly reproducible core packages: nvme-cli from a gcc9 LTO patch = https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91307
python-tox from the https://bugzilla.opensuse.org/show_bug.cgi?id=1094323 toolchain patch
And in the next week I am off to the reproducible-builds world summit.
Ciao Bernhard M.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi,
last month's status: https://lists.opensuse.org/opensuse-factory/2019-11/msg00370.html
Last months' reproducible builds project updates (including my work): https://reproducible-builds.org/reports/2019-11/
Three weeks ago, I attended the reproducible builds summit and then wrote about it at https://lizards.opensuse.org/2019/12/13/opensuse-on-reproducible-builds-summ...
We also opened https://www.reddit.com/r/reproduciblebuilds/ .
I already found and reported or fixed several problems that would have caused problems in 2020. Some of these may be forgotten upstream:
https://rt.cpan.org/Public/Bug/Display.html?id=124509 perl-TimeDate https://rt.cpan.org/Public/Bug/Display.html?id=124513 perl-OLE-Storage_Lite https://rt.cpan.org/Public/Bug/Display.html?id=124514 perl-Parse-Win32Registry https://rt.cpan.org/Public/Bug/Display.html?id=124528 perl-File-Listing https://rt.cpan.org/Public/Bug/Display.html?id=124524 perl-Image-ExifTool ; 11.81 and later include the fix
I uploaded https://rb.zq1.de/compare.factory-20191223/ today and rbstats are: total-packages: 12558 (+134) build-tried: 12550 (+134) build-failed: 49 (-2) build-n-a: 79 (-2) build-succeeded: 12422 (+138) build-official-failed+na: 100 (-26)
build-compare-failed: 380 (-35) build-compare-succeeded: 12042 (+173)
verify-failed: 550 (-10) verified-semi-reproducible: 11756 (+148) verified-bit-identical: 0
bit-by-bit-identical: 11662 (+181) not-bit-by-bit-identical: 757 (-37)
https://rb.zq1.de/compare.factory-20191223/graph.png shows the change over time
https://rb.zq1.de/compare.factory-20191223/unreproduciblerings.txt lists very unreproducible core packages (bootstrap+DVD)
Of the badly unreproducible packages, 2 were in ring0 45 were in ring1
That makes it 47/2901 => 1.62 % which is below the overall average of 380/12422 => 3.06 %
757/12422 => 6.09 % of packages are not perfectly reproducible
Newly unreproducible core packages: cri-o: from an old go problem (go1.9+) where parallelism in multi-core builds cause variations in the .note.go.buildid ELF header
Newly reproducible core packages: fftw3: no more varies from -flto=%smp since it was rebuilt with new rpm that uses -flto=auto
python-magnumclient From sphinx template patch https://review.opendev.org/#/c/652669/ and https://review.openstack.org/#/c/568815/
rpmlint-mini: https://build.opensuse.org/request/show/754705
Happy Christmas Bernhard M.
-
Bernhard M. Wiedemann -
Bernhard M. Wiedemann -
Christian Boltz -
Stasiek Michalski -
Stephan Kulow