[Bug 1049186] New: python-3.6 packages do not build reproducibly
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 Bug ID: 1049186 Summary: python-3.6 packages do not build reproducibly Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: SUSE Other Status: NEW Severity: Normal Priority: P5 - None Component: Other Assignee: bnc-team-screening@forge.provo.novell.com Reporter: bwiedemann@suse.com QA Contact: qa-bugs@suse.de CC: jmatejek@suse.com Found By: Development Blocker: --- e.g. python-simplejson has one-bit diffs in .pyc files See http://rb.zq1.de/compare.factory-20170713/python-simplejson-compare.out in python3-simplejson.rpm we get -00004e50 68 6f 72 5f 5f da 07 64 65 63 69 6d 61 6c 72 0c |hor__..decimalr.| +00004e50 68 6f 72 5f 5f 5a 07 64 65 63 69 6d 61 6c 72 0c |hor__Z.decimalr.| in python3-simplejson-test.rpm we get the opposite change -00000580 72 13 00 00 00 5a 07 64 65 63 69 6d 61 6c 72 03 |r....Z.decimalr.| +00000580 72 13 00 00 00 da 07 64 65 63 69 6d 61 6c 72 03 |r......decimalr.| and it seems to be related to filesystem ordering, since it built reproducibly when using a filesystem with sorted readdir using disorderfs via reproducible-faketools-filesys from https://build.opensuse.org/package/show/home:bmwiedemann:reproducible/reprod... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c1 Jan Matejek <jmatejek@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |CONFIRMED --- Comment #1 from Jan Matejek <jmatejek@suse.com> --- The difference is basically a matter of reference count. Why it would be different based on the filesystem ordering, that I don't know. How do I use the disorderfs? Do you have tools that I can use to trigger the problem locally without manually running "osc build" several times and then manually unpacking rpms? :) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c2 --- Comment #2 from Bernhard Wiedemann <bwiedemann@suse.com> --- You can try the 'rbk' (Reproducible Build with KVM) script from https://github.com/bmwiedemann/reproducibleopensuse I am using disorderfs in osc builds (in a hacky way) like this: https://github.com/bmwiedemann/reproducible-faketools/blob/master/extra/diso... sometimes it also reduces races like in https://build.opensuse.org/request/show/510387 When we produce .pyc files, would that be once python call taking a directory and producing many .pyc files in one go? Then maybe the order of processing them makes a difference and we could avoid that by always processing them in the same order. Is there asynchronous garbage collection that could matter for the reference counts? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c3 --- Comment #3 from Jan Matejek <jmatejek@suse.com> --- (In reply to Bernhard Wiedemann from comment #2)
Then maybe the order of processing them makes a difference and we could avoid that by always processing them in the same order.
that's a possibility, but...
Is there asynchronous garbage collection that could matter for the reference counts?
...but you're not supposed to GC references that live inside a deterministically compiled code block. It seems somewhat more likely to me that the filesystem ordering is a red herring and the problem is actually caused by hash randomization (which changes order of elements in dicts, this could lead to something like "first listed element has one more reference because it's first") but we'll see -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c4 --- Comment #4 from Bernhard Wiedemann <bwiedemann@suse.com> --- I found one simple fix: https://github.com/python/cpython/compare/master...distropatches:pycrb36 It even helps when I disable the deterministic hashes that we get via export PYTHONHASHSEED=0 from /etc/profile.d/suse-buildsystem.sh so do you want to try to understand it more or do we just upstream/merge the trivial 1-line patch? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c5 --- Comment #5 from Jan Matejek <jmatejek@suse.com> --- well of course that would help if disorderfs helps ;e) we can merge this and get reproducible builds. i'd hold off upstreaming though, because this doesn't address the underlying question of how a file gets a different bytecode based on its compilation order -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c10 --- Comment #10 from Bernhard Wiedemann <bwiedemann@suse.com> --- There has now been some upstream activity on this https://bugs.python.org/issue34033 https://github.com/python/cpython/pull/8057 and it seems it will take a bit longer, because they want to do a proper fix. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c11 Bernhard Wiedemann <bwiedemann@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |mcepl@suse.com Resolution|UPSTREAM |--- --- Comment #11 from Bernhard Wiedemann <bwiedemann@suse.com> --- Patch was dropped 2018-12 so this is open again. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c12 Matej Cepl <mcepl@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution|--- |FIXED --- Comment #12 from Matej Cepl <mcepl@suse.com> --- I don't understand. See https://build.opensuse.org/request/show/687278 which is accepted and it returned the patch back as we discussed on IRC. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 Hans-Peter Jansen <hpj@urpla.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |hpj@urpla.net -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c38 --- Comment #38 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1049186) was mentioned in https://build.opensuse.org/request/show/923499 Factory / python36 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1049186 http://bugzilla.opensuse.org/show_bug.cgi?id=1049186#c39 --- Comment #39 from OBSbugzilla Bot <bwiedemann+obsbugzillabot@suse.com> --- This is an autogenerated message for OBS integration: This bug (1049186) was mentioned in https://build.opensuse.org/request/show/926876 Factory / python36 -- You are receiving this mail because: You are on the CC list for the bug.
participants (2)
-
bugzilla_noreply＠novell.com -
bugzilla_noreply＠suse.com