[opensuse-factory] reproducible builds test results
Hi, I did another round of rebuild testing for openSUSE Factory. All resulting build-compare diffs are in http://rb.zq1.de/compare.factory-20170406/ as well as a machine-readable summary http://rb.zq1.de/compare.factory-20170406/rbstats.txt current summary says: total-packages: 10971 build-tried: 10971 build-failed: 118 build-n-a: 114 build-succeeded: 10739 build-official-failed+na: 115 build-compare-failed: 1209 build-compare-succeeded: 9530 bit-by-bit-identical: 8133 not-bit-by-bit-identical: 2599 not-bit-by-bit-identicalcheck: 2606 meaning, 75% of packages already give bit-identical results. And half of the remaining problems are so bad that even build-compare filters do not help and thus trigger unnecessary rebuilds in OBS. and I have details about each of the 10971 packages http://rb.zq1.de/compare.factory-20170406/reproducible.json and a .diff file next to it, comparing it to the previous round The good news is, that nearly all of the packages that build in OBS, could be built locally as well, with 2 of 5 build bugs already fixed * https://bugzilla.opensuse.org/show_bug.cgi?id=1032257 * https://bugzilla.opensuse.org/show_bug.cgi?id=1031548 * https://bugzilla.opensuse.org/show_bug.cgi?id=1031549 * https://bugzilla.opensuse.org/show_bug.cgi?id=1031554 * https://bugzilla.opensuse.org/show_bug.cgi?id=1031556 Ciao Bernhard M. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/08/2017, 10:22 PM, Bernhard M. Wiedemann wrote:
More interesting would be to know which packages are actually unreproducible, I guess: http://www.fi.muni.cz/~xslaby/sklad/unreproducible.txt Used perl: ===== use JSON; foreach (@{from_json(join("", <>))}) { print "$$_{package}\n" if ($$_{status} eq "unreproducible"); } ===== thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/10/2017, 08:54 AM, Jiri Slaby wrote:
And even more to compare with packages maintained by myself: comm -12 unreproducible.txt <(osc my pkg |sed 's@.*/@@'|sort -u) thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/10/2017, 09:02 AM, Jiri Slaby wrote:
Hmm, and what do entries like this actually mean? "architecture" : "x86_64", "build_compare_status" : "reproducible", "build_date" : 1491391865, "package" : "sparse", "release" : "openSUSE:Factory", "status" : "unreproducible", "version" : "20161117-32" They are marked as unreproducible, but there is no diff file for them? thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 10.04.2017 um 09:09 schrieb Jiri Slaby:
They are marked as unreproducible, but there is no diff file for them?
I guess their unreproducability is not reproducable ;-) Greetings, Stephan -- Ma muaß weiterkämpfen, kämpfen bis zum Umfalln, a wenn die ganze Welt an Arsch offen hat, oder grad deswegn. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-10 09:09, Jiri Slaby wrote:
In this case, the build_compare_status value tells that our build_compare filters dropped the differences, so this is one of the trickier cases. For OBS it counts as 'the same' , is not re-published and depending builds are not re-triggered, but it is not a true bit-identical reproducible build. I made a patch to my scripts, so that in the future there will be a diff like this one: http://rb.zq1.de/compare.factory-20170410/sparse-compare.out that contains telling lines such as sparse-devel -/usr/lib64/libsparse.a 3bc3e1d236ec48bb645fece5bef2fbb9 0 +/usr/lib64/libsparse.a 3ee192c50ba353019f46f925d836f9c1 0 which might come from file-system ordering differences or embedded timestamps. In this case, it is the latter as can be seen with unrpm sparse-devel*.rpm ar vt usr/lib64/libsparse.a < rw-r--r-- 399/399 1800 Apr 3 11:23 2018 target.o
rw-r--r-- 399/399 1800 Apr 9 13:19 2017 target.o
In Factory and 42.3 we have strip-nondeterminism which can help getting rid of these timestamps. Ciao Bernhard M. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Apr 10, 2017 at 12:21 PM, Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
In this case builing binutils with --enable-deterministic-archives should also help. I did, unfortunately it was reverted because it broke some unspecified makefiles..:-| -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-11 14:34, Cristian Rodríguez wrote:
I think, we should bring enable-deterministic-archives back. https://bugzilla.redhat.com/show_bug.cgi?id=1124342 and the related 'make' bug https://bugzilla.redhat.com/show_bug.cgi?id=1136481 are closed since last September. I did a round of rebuild-testing with https://github.com/bmwiedemann/reproducible-faketools/commit/8e2246585be9a45... to do the same and http://rb.zq1.de/compare.factory-20170418/rbstats looks a lot better than http://rb.zq1.de/compare.factory-20170410/rbstats -bit-by-bit-identical: 8140 -not-bit-by-bit-identical: 2619 +bit-by-bit-identical: 9340 +not-bit-by-bit-identical: 1890 so we might get 700 to 1200 newly bit-identical packages (while only 500 packages were added to Factory in that time) Ciao Bernhard M. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-18 15:19, Bernhard M. Wiedemann wrote:
I now submitted https://build.opensuse.org/request/show/509329 after finding several packages that failed build-compare without it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 11-07-2017 a las 8:19, Bernhard M. Wiedemann escribió:
that's cool.. thanks. Now.. I do not know if it has been taken care of already.. but calls to cpio should also take the --reproducible flag whenever applicable.. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 12 Jul 2017 16:38, Cristian Rodríguez wrote:
Hmmm, has "--reproducible" any negative impact at all? (looks like similar to --ignore-devno --renumber-inodes) - If not, could "--reproducible" made the default behavior of cpio? At least for most packages I can't see a negative point. (exclusive packages with dev-inodes, but that's a extra fish to fry) - Yamaban.
Hi, I did another round of rebuild testing for openSUSE Factory. See https://en.opensuse.org/openSUSE:Reproducible_Builds for pointers, especially https://github.com/bmwiedemann/reproducibleopensuse with my scripts to test reproducibility (now with slightly extended docs). All resulting build-compare diffs are in http://rb.zq1.de/compare.factory-20171001/ machine-readable information on 11817 packages http://rb.zq1.de/compare.factory-20170406/reproducible.json as well as a machine-readable summary http://rb.zq1.de/compare.factory-20171001/rbstats.txt total-packages: 11821 build-tried: 11817 build-failed: 109 build-n-a: 114 build-succeeded: 11594 build-compare-failed: 250 build-compare-succeeded: 11344 bit-by-bit-identical: 11111 not-bit-by-bit-identical: 478 The 250 most interesting packages to fix are listed in http://rb.zq1.de/compare.factory-20171001/build-compare-differed-builds.txt While I have fixed dozens of the simpler issues over the past months (mostly upstream), there are some large classes left: ~102 indeterministic from javadoc output (from filesystem readdir order and date+time) and other java-related issues - I'm not touching it yet, because it is so hard to work with java ~22 packages are indeterministic .pdf .dvi .ps files from *latex ~12 packages from mono some issues from Qt like ~17 https://codereview.qt-project.org/202999 ~3 https://codereview.qt-project.org/203352 ~4 from xemacs .elc file comments some python and ruby issues such as https://github.com/python/cpython/pull/296 https://github.com/python/cpython/pull/1794 https://github.com/python/cpython/pull/2263 https://github.com/ruby/ruby/pull/1602 https://github.com/ruby/ruby/pull/1603 and likely more cases of packages that write uninitialized memory to packaged files like https://github.com/davidsd/lie/pull/1/files and 'gcin' (already fixed) If someone wants to help, you are most welcome. Ciao Bernhard M.
On 04/08/2017, 10:22 PM, Bernhard M. Wiedemann wrote:
More interesting would be to know which packages are actually unreproducible, I guess: http://www.fi.muni.cz/~xslaby/sklad/unreproducible.txt Used perl: ===== use JSON; foreach (@{from_json(join("", <>))}) { print "$$_{package}\n" if ($$_{status} eq "unreproducible"); } ===== thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/10/2017, 08:54 AM, Jiri Slaby wrote:
And even more to compare with packages maintained by myself: comm -12 unreproducible.txt <(osc my pkg |sed 's@.*/@@'|sort -u) thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/10/2017, 09:02 AM, Jiri Slaby wrote:
Hmm, and what do entries like this actually mean? "architecture" : "x86_64", "build_compare_status" : "reproducible", "build_date" : 1491391865, "package" : "sparse", "release" : "openSUSE:Factory", "status" : "unreproducible", "version" : "20161117-32" They are marked as unreproducible, but there is no diff file for them? thanks, -- js suse labs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 10.04.2017 um 09:09 schrieb Jiri Slaby:
They are marked as unreproducible, but there is no diff file for them?
I guess their unreproducability is not reproducable ;-) Greetings, Stephan -- Ma muaß weiterkämpfen, kämpfen bis zum Umfalln, a wenn die ganze Welt an Arsch offen hat, oder grad deswegn. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-10 09:09, Jiri Slaby wrote:
In this case, the build_compare_status value tells that our build_compare filters dropped the differences, so this is one of the trickier cases. For OBS it counts as 'the same' , is not re-published and depending builds are not re-triggered, but it is not a true bit-identical reproducible build. I made a patch to my scripts, so that in the future there will be a diff like this one: http://rb.zq1.de/compare.factory-20170410/sparse-compare.out that contains telling lines such as sparse-devel -/usr/lib64/libsparse.a 3bc3e1d236ec48bb645fece5bef2fbb9 0 +/usr/lib64/libsparse.a 3ee192c50ba353019f46f925d836f9c1 0 which might come from file-system ordering differences or embedded timestamps. In this case, it is the latter as can be seen with unrpm sparse-devel*.rpm ar vt usr/lib64/libsparse.a < rw-r--r-- 399/399 1800 Apr 3 11:23 2018 target.o
rw-r--r-- 399/399 1800 Apr 9 13:19 2017 target.o
In Factory and 42.3 we have strip-nondeterminism which can help getting rid of these timestamps. Ciao Bernhard M. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Apr 10, 2017 at 12:21 PM, Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
In this case builing binutils with --enable-deterministic-archives should also help. I did, unfortunately it was reverted because it broke some unspecified makefiles..:-| -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-11 14:34, Cristian Rodríguez wrote:
I think, we should bring enable-deterministic-archives back. https://bugzilla.redhat.com/show_bug.cgi?id=1124342 and the related 'make' bug https://bugzilla.redhat.com/show_bug.cgi?id=1136481 are closed since last September. I did a round of rebuild-testing with https://github.com/bmwiedemann/reproducible-faketools/commit/8e2246585be9a45... to do the same and http://rb.zq1.de/compare.factory-20170418/rbstats looks a lot better than http://rb.zq1.de/compare.factory-20170410/rbstats -bit-by-bit-identical: 8140 -not-bit-by-bit-identical: 2619 +bit-by-bit-identical: 9340 +not-bit-by-bit-identical: 1890 so we might get 700 to 1200 newly bit-identical packages (while only 500 packages were added to Factory in that time) Ciao Bernhard M. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-18 15:19, Bernhard M. Wiedemann wrote:
I now submitted https://build.opensuse.org/request/show/509329 after finding several packages that failed build-compare without it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 11-07-2017 a las 8:19, Bernhard M. Wiedemann escribió:
that's cool.. thanks. Now.. I do not know if it has been taken care of already.. but calls to cpio should also take the --reproducible flag whenever applicable.. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 12 Jul 2017 16:38, Cristian Rodríguez wrote:
Hmmm, has "--reproducible" any negative impact at all? (looks like similar to --ignore-devno --renumber-inodes) - If not, could "--reproducible" made the default behavior of cpio? At least for most packages I can't see a negative point. (exclusive packages with dev-inodes, but that's a extra fish to fry) - Yamaban.
Hi, I did another round of rebuild testing for openSUSE Factory. See https://en.opensuse.org/openSUSE:Reproducible_Builds for pointers, especially https://github.com/bmwiedemann/reproducibleopensuse with my scripts to test reproducibility (now with slightly extended docs). All resulting build-compare diffs are in http://rb.zq1.de/compare.factory-20171001/ machine-readable information on 11817 packages http://rb.zq1.de/compare.factory-20170406/reproducible.json as well as a machine-readable summary http://rb.zq1.de/compare.factory-20171001/rbstats.txt total-packages: 11821 build-tried: 11817 build-failed: 109 build-n-a: 114 build-succeeded: 11594 build-compare-failed: 250 build-compare-succeeded: 11344 bit-by-bit-identical: 11111 not-bit-by-bit-identical: 478 The 250 most interesting packages to fix are listed in http://rb.zq1.de/compare.factory-20171001/build-compare-differed-builds.txt While I have fixed dozens of the simpler issues over the past months (mostly upstream), there are some large classes left: ~102 indeterministic from javadoc output (from filesystem readdir order and date+time) and other java-related issues - I'm not touching it yet, because it is so hard to work with java ~22 packages are indeterministic .pdf .dvi .ps files from *latex ~12 packages from mono some issues from Qt like ~17 https://codereview.qt-project.org/202999 ~3 https://codereview.qt-project.org/203352 ~4 from xemacs .elc file comments some python and ruby issues such as https://github.com/python/cpython/pull/296 https://github.com/python/cpython/pull/1794 https://github.com/python/cpython/pull/2263 https://github.com/ruby/ruby/pull/1602 https://github.com/ruby/ruby/pull/1603 and likely more cases of packages that write uninitialized memory to packaged files like https://github.com/davidsd/lie/pull/1/files and 'gcin' (already fixed) If someone wants to help, you are most welcome. Ciao Bernhard M.
participants (5)
-
Bernhard M. Wiedemann -
Cristian Rodríguez -
Jiri Slaby -
Stephan Kulow -
Yamaban