Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/
It seems Debian and Fedora are already part of the project, and the advantages are quite compelling, not just from a security perspective, but also due to the potential savings in storage and network consumption:
https://hackweek.suse.com/13/projects/131
Dne Pá 11. prosince 2015 10:43:51, Adam Spiers napsal(a):
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/It seems Debian and Fedora are already part of the project, and the advantages are quite compelling, not just from a security perspective, but also due to the potential savings in storage and network consumption:
https://hackweek.suse.com/13/projects/131
Even if we are not directly involved in this we actually mandate our packages to always generate the same output and bugs reported against this are always treated properly...
The hackweek I see is more about reducing the rebuilds that get tossed away.
Ie. we now rebuild libreoffice with each update of tumbleweed (because it is beast) and we rarely "update" it simply it is rebuilt, verified it is the same and the new packages are thrown away.
Sortly put: report bugs where the package does not generate same stuff every time and it will be fixed for stuff maintained internaly and should be fixed for community based stuff.
Tom
Hi,
On Fri, 11 Dec 2015, Adam Spiers wrote:
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/
We have that since about forever as far as easily possible. The hard part is changing packages to not depend on things like build time (e.g. encoding build date/time into strings into executables). That's not something you can do generally in a build system, but must be changed in each and every individual package.
Ciao, Michael.
On 11.12.2015 11:51, Michael Matz wrote:
Hi,
On Fri, 11 Dec 2015, Adam Spiers wrote:
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/We have that since about forever as far as easily possible. The hard part is changing packages to not depend on things like build time (e.g. encoding build date/time into strings into executables). That's not something you can do generally in a build system, but must be changed in each and every individual package.
Unfortunately most of my bug reports are gently rotting ;(
(check summaries starting with build-compare)
Greetings, Stephan
Michael Matz wrote:
On Fri, 11 Dec 2015, Adam Spiers wrote:
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/We have that since about forever as far as easily possible. The hard part is changing packages to not depend on things like build time (e.g. encoding build date/time into strings into executables). That's not something you can do generally in a build system, but must be changed in each and every individual package.
It's probably worth to explain the difference between our reproducible builds and this new interpretation. SUSE and openSUSE distributions have always had reproducible builds, for something like 20 years now. Reproducible in the sense that a packager never builds binaries on his own system in some magic way and then uploads binaries.
We always build sources server side (nowadays OBS, previously autobuild). How the build environment has to look like is defined via BuildRequires in the spec file and settings in the project config on server side. Moreover, we don't allow packagers to directly build packages in the distribution's project. There's always a review step (four eyes principle). Some distributions don't have that and only have reviews when a package is accepted for the first time.
OBS always re-creates the build environment from scratch for each package and automatically uses other packages in the same project to set up that build environment. Ie there's no magic base build system, the distribution bootstraps itself. Not only on request or mass rebuilds but fully automatic. So even packages that haven't been submitted for years are rebuilt with current compilers and libraries. Additionally every binary rpm produced by obs contains a back reference to the used sources (in the disturl).
IOW our process and infrastructure guarantees that our packages can reproducibly be built from source. Everyone can reproduce that by firing up their own build service and linking to OBS. In that sense _our build process is reproducible_ and has always been. The terrifying news here is that other distributions still have to do homework to even get there.
However, that reproducible-builds.org project wants to achieve an addition goal and that is to have bit identical rpm results of two build runs. We trust our process and infrastructure so it wasn't important to us so far.
The mechanism we have in place to detect unchanged binary rpms (build-compare) only partially gets us there as it solves a different problem. build-compare is basically a heuristic to avoid triggering rebuilds of other packages and publishing of packages with only trivial changes (like time stamps). It does that by "normalizing" some content and throwing away results with trivial changes. It does not necessarily prevent trivial changes. Some of the measurements to please build-compare are also useful for bit identical results though, like eliminating time stamps in build results in the first place.
So if anyone wants to pursue that reproducible-builds.org idea, fixing the issues with build-compare is a good first step.
Also, it needs to be defined what a bit identical rpm result actually is. Only the rpm payload isn't sufficient as the header includes e.g. scripts. The full header can't be compared as it contains time stamps, build host names, signatures etc. So again a script similar to what build-compare does would be needed to do the actual comparison, md5sums of rpm files won't suffice.
Another project related to that might be to verify source rpms can be rebuilt. So far our assumption is that a build service is available. So everything is centered around build service packages containers rather than srpms.
cu Ludwig
On Wed, 16 Dec 2015, Ludwig Nussel wrote:
Michael Matz wrote:
On Fri, 11 Dec 2015, Adam Spiers wrote:
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/We have that since about forever as far as easily possible. The hard part is changing packages to not depend on things like build time (e.g. encoding build date/time into strings into executables). That's not something you can do generally in a build system, but must be changed in each and every individual package.
It's probably worth to explain the difference between our reproducible builds and this new interpretation. SUSE and openSUSE distributions have always had reproducible builds, for something like 20 years now. Reproducible in the sense that a packager never builds binaries on his own system in some magic way and then uploads binaries.
We always build sources server side (nowadays OBS, previously autobuild). How the build environment has to look like is defined via BuildRequires in the spec file and settings in the project config on server side. Moreover, we don't allow packagers to directly build packages in the distribution's project. There's always a review step (four eyes principle). Some distributions don't have that and only have reviews when a package is accepted for the first time.
OBS always re-creates the build environment from scratch for each package and automatically uses other packages in the same project to set up that build environment. Ie there's no magic base build system, the distribution bootstraps itself. Not only on request or mass rebuilds but fully automatic. So even packages that haven't been submitted for years are rebuilt with current compilers and libraries. Additionally every binary rpm produced by obs contains a back reference to the used sources (in the disturl).
IOW our process and infrastructure guarantees that our packages can reproducibly be built from source. Everyone can reproduce that by firing up their own build service and linking to OBS. In that sense _our build process is reproducible_ and has always been. The terrifying news here is that other distributions still have to do homework to even get there.
Well, I don't see how you can "easily" reproduce a build of an "old" version of a package without re-bootstrapping the whole old state of the repository to build against. I understand that we eventually have enough information in the package to actually re-create that repository "soruce" state and we keep old sources around but not binaries.
Not sure if reproducible-builds.org intends to make that any easier than what we have though.
Richard.
Richard Biener wrote:
On Wed, 16 Dec 2015, Ludwig Nussel wrote:
[...] IOW our process and infrastructure guarantees that our packages can reproducibly be built from source. Everyone can reproduce that by firing up their own build service and linking to OBS. In that sense _our build process is reproducible_ and has always been. The terrifying news here is that other distributions still have to do homework to even get there.
Well, I don't see how you can "easily" reproduce a build of an "old" version of a package without re-bootstrapping the whole old state of the repository to build against. I understand that we eventually have enough information in the package to actually re-create that repository "soruce" state and we keep old sources around but not binaries.
Depends on whether one wants to reproduce some Tumbleweed package from a random point in the past or one from a released distribution. For the latter we do keep binaries of released maintenance updates. Nevertheless it should be possible to reproduce binaries in any case, even though it might require some time and computing power.
cu Ludwig
Hello Ludwig,
I think your great explanation about the SUSE and openSUSE build process should be made obvious for everybody out there.
Perhaps it could be placed on some of our web front pages at least on https://build.opensuse.org/ and then linked on other web front pages like https://en.opensuse.org/Main_Page
I would be a shame if not everybody out there knows about what On Dec 16 11:18 Ludwig Nussel wrote:
... SUSE and openSUSE distributions have always had reproducible builds, for something like 20 years now. Reproducible in the sense that a packager never builds binaries on his own system in some magic way and then uploads binaries.
We always build sources server side (nowadays OBS, previously autobuild). How the build environment has to look like is defined via BuildRequires in the spec file and settings in the project config on server side. Moreover, we don't allow packagers to directly build packages in the distribution's project. There's always a review step (four eyes principle). Some distributions don't have that and only have reviews when a package is accepted for the first time.
OBS always re-creates the build environment from scratch for each package and automatically uses other packages in the same project to set up that build environment. Ie there's no magic base build system, the distribution bootstraps itself. Not only on request or mass rebuilds but fully automatic. So even packages that haven't been submitted for years are rebuilt with current compilers and libraries. Additionally every binary rpm produced by obs contains a back reference to the used sources (in the disturl).
IOW our process and infrastructure guarantees that our packages can reproducibly be built from source. Everyone can reproduce that by firing up their own build service and linking to OBS. In that sense _our build process is reproducible_ and has always been. The terrifying news here is that other distributions still have to do homework to even get there.
Kind Regards Johannes Meixner
Hi!
Ludwig Nussel:
Michael Matz wrote:
On Fri, 11 Dec 2015, Adam Spiers wrote:
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/We have that since about forever as far as easily possible. The hard part is changing packages to not depend on things like build time (e.g. encoding build date/time into strings into executables). That's not something you can do generally in a build system, but must be changed in each and every individual package.
It's probably worth to explain the difference between our reproducible builds and this new interpretation. SUSE and openSUSE distributions have always had reproducible builds, for something like 20 years now. Reproducible in the sense that a packager never builds binaries on his own system in some magic way and then uploads binaries. […]
Thanks Ludwig for your explanations as we unfortunately use some common words to convey different ideas. I want to try to elaborate on them so we can better colaborate between projects as we—in Tor, Debian and many other projects—strongly believe that these issues concern all free software projects.
I believe we have trouble understanding one another because, from my understanding, we have been working on solving different problems:
The problem openSUSE has solved with OBS is “Can a software be rebuilt from source?”
The problem Bitcoin and Tor Browser have solved is “Can users verify that the binary has been built using the source distributed with it?” That's the problem which prompted efforts in Debian and other projects, and which lead to the creation of reproducible-builds.org.
From the point of view of an openSUSE developper, it might look the same. As only source code can be submitted to OBS, as long as one trust the infrastructure, they can be confident than the binary produced by OBS is coming from the source code that lands in the archive.
But end users might want to verify by themselves that binaries match a source code they can audit. When a build process doesn't produce bit-for-bit identical results, this become much harder.
Having the ability to have third parties assess binary packages also relieve the infrastructure maintainers from potential pressures. It's easier to resist by saying “if we change that binary, someone is going to notice”.
The mechanism we have in place to detect unchanged binary rpms (build-compare) only partially gets us there as it solves a different problem. build-compare is basically a heuristic to avoid triggering rebuilds of other packages and publishing of packages with only trivial changes (like time stamps). It does that by "normalizing" some content and throwing away results with trivial changes. It does not necessarily prevent trivial changes. Some of the measurements to please build-compare are also useful for bit identical results though, like eliminating time stamps in build results in the first place.
Speaking of build-compare, the tool that we built to debug differences between two builds, diffoscope, is getting wider exposure and becoming quite versatile. It's in Python and designed to be as modular as possible. The hope was to have something more maintainable and extensible than a single shell script.
One feature diffoscope is missing to become a proper alternative to build-compare is the ability to filter differences. It's on my list of things to implement, and I'll eventually get to it, but contributions would be great if anyone is interested.
In any cases, don't hesitate to get in touch. We, people behind Debian reproducible builds efforts and reproducible-builds.org, want to help! :)
On Thu, Dec 17, Jérémy Bobbio wrote:
Speaking of build-compare, the tool that we built to debug differences between two builds, diffoscope, is getting wider exposure and becoming quite versatile. It's in Python and designed to be as modular as possible. The hope was to have something more maintainable and extensible than a single shell script.
The (unwritten) goal of build-compare is that it must be usable in every released product. One just '_link's that package into his build project and it will just work. Perhaps a replacement in python may be equally compatible across the various python variants.
Olaf
Jérémy Bobbio wrote:
Ludwig Nussel:
Michael Matz wrote:
On Fri, 11 Dec 2015, Adam Spiers wrote:
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/We have that since about forever as far as easily possible. The hard part is changing packages to not depend on things like build time (e.g. encoding build date/time into strings into executables). That's not something you can do generally in a build system, but must be changed in each and every individual package.
It's probably worth to explain the difference between our reproducible builds and this new interpretation. SUSE and openSUSE distributions have always had reproducible builds, for something like 20 years now. Reproducible in the sense that a packager never builds binaries on his own system in some magic way and then uploads binaries. […]
Thanks Ludwig for your explanations as we unfortunately use some common words to convey different ideas.
Exactly. The Maximum RPM book already advertised "Reproducible Builds" in 1997 without promising bit identical results :-)
I want to try to elaborate on them so we can better colaborate between projects as we—in Tor, Debian and many other projects—strongly believe that these issues concern all free software projects.
I believe we have trouble understanding one another because, from my understanding, we have been working on solving different problems:
The problem openSUSE has solved with OBS is “Can a software be rebuilt from source?”
Yes, we take that for granted.
[...] But end users might want to verify by themselves that binaries match a source code they can audit. When a build process doesn't produce bit-for-bit identical results, this become much harder.
No doubt about that.
Speaking of build-compare, the tool that we built to debug differences between two builds, diffoscope, is getting wider exposure and becoming quite versatile. It's in Python and designed to be as modular as possible. The hope was to have something more maintainable and extensible than a single shell script.
One feature diffoscope is missing to become a proper alternative to build-compare is the ability to filter differences. It's on my list of things to implement, and I'll eventually get to it, but contributions would be great if anyone is interested.
To be usable within build roots it would have to be sufficiently standalone though. Due to the automatic boot strapping and always fresh build roots we have to be careful to avoid polluting build environments and causing build cycles.
cu Ludwig
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2015-12-11 11:43, Adam Spiers wrote:
Is anyone working on (or thinking of working on) making our build process reproducible?
https://reproducible-builds.org/
It seems Debian and Fedora are already part of the project, and the advantages are quite compelling, not just from a security perspective, but also due to the potential savings in storage and network consumption:
now I had some time to look into how reproducible our builds are and had a VM (with 4 cores) build all Leap 42.1 packages named a*
using some helper scripts, which I uploaded to https://github.com/bmwiedemann/reproducibleopensuse
with those helpers, I just had to do rebuildmany a* comparemany a*
btw: the rebuild of those 217 packages took 382 minutes
of those 217 (a small subset of Leap's 7830), build-compare reported a diff for 36 packages
and if you wonder, those are a2ps acct aegisub aespipe allegro alpine amanda amor anjuta anthy antlr apache-commons-cli apache-commons-codec apache-commons-collections apache-commons-email apache-commons-io apache-commons-lang apache-portlet-1_0-api apel appframework aqbanking argyllcms arts asl asm3 aspell aspell-en atmel-firmware autoconf-el autogen autoyast2 avfs avogadro avrdude awesome axis
The other thing I did was to look how common the use of the better SOURCE_DATE_EPOCH is. For that I used find openSUSE:Factory/ -name *.gz -o -name *.xz -o -name *.bz2 |\ grep -v ".osc" | xargs zgrep -l SOURCE_DATE_EPOCH
to find that SOURCE_DATE_EPOCH is already used in doxygen deja-dup help2man u-boot
another thing I found is that when you use osc getbinaries you get a file named _buildenv and that contains unique IDs of all packages used for building this, so it should be possible to archive and later fetch the exact versions of everything needed.
Ciao Bernhard M.
Hello,
Am Mittwoch, 17. Februar 2016 schrieb Bernhard M. Wiedemann:
now I had some time to look into how reproducible our builds are and had a VM (with 4 cores) build all Leap 42.1 packages named a*
using some helper scripts, which I uploaded to https://github.com/bmwiedemann/reproducibleopensuse
with those helpers, I just had to do rebuildmany a* comparemany a*
btw: the rebuild of those 217 packages took 382 minutes
of those 217 (a small subset of Leap's 7830), build-compare reported a diff for 36 packages
and if you wonder, those are a2ps acct aegisub aespipe allegro alpine amanda amor anjuta anthy antlr apache-commons-cli apache-commons-codec apache-commons-collections apache-commons-email apache-commons-io apache-commons-lang apache-portlet-1_0-api apel appframework aqbanking argyllcms arts asl asm3 aspell aspell-en atmel-firmware autoconf-el autogen autoyast2 avfs avogadro avrdude awesome axis
I just checked the Debian result for some of them on https://tests.reproducible-builds.org/reproducible.html and interestingly, several of the packages you listed above are reproducible on Debian: acct, aegisub, aespipe, anthy, apel, asl, aspell, autogen, avogadro, avrdude.
I know the reproducible builds team at Debian is actively pushing patches to make the build reproducible, but I'm not sure if they were able to fix that many packages already (10 out of 36 - actually I should probably say 10 out of ~25 because some of the packages you listed don't exist in Debian, at least not with this name).
Can you upload the differences for the packages you tested somewhere so that interested people don't need to rebuild everything just to get the diff?
Oh, BTW - maybe having some variables in your scripts instead of hardcoded paths would be nice ;-)
I was curious and tried with aspell. The compare result is:
/dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell/binaries /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell Comparing aspell-0.60.6.1-18.4.x86_64.rpm to aspell-0.60.6.1-0.x86_64.rpm comparing rpmtags --- /tmp/tmp.2P0iOiEtHL/tmp.jsMGUlvr4J 2016-02-17 21:34:31.060962628 +0100 +++ /tmp/tmp.2P0iOiEtHL/tmp.5h81k6Lq48 2016-02-17 21:34:31.068962571 +0100 @@ -9,7 +9,7 @@ has many other technical enhancements over Ispell, such as using shared memory for dictionaries and intelligently handling personal dictionaries when more than one Aspell process is open at once. - openSUSE openSUSE Leap 42.1 obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications + openSUSE openSUSE Leap 42.1 (none) GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications Productivity/Text/Spell http://aspell.net/ (none) (none) (none) (none) 4.11.2 x86_64-suse-linux cpio lzma 5 comparing RELEASE comparing PROVIDES comparing scripts comparing filelist comparing file checksum creating rename script RPM meta information is identical Extracting packages Package content is identical
[...same for other subpackages...]
It seems the only difference is in the rpmtags - (none) vs. obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell
Is this really a relevant difference, or should it be whitelisted in build-compare?
Hmm, actually $?=0, so on my system it _is_ reproducible. What did I do wrong? ;-)
I also tried aespipe, and get a real difference. Maybe someone should compare the openSUSE package with the Debian package to find out what they do different to make the package reproducible ;-)
Regards,
Christian Boltz
Doing these builds on OBS wouldn't they always differ? I mean the release tag in .rpm's is always incremented which already results in a diff. In my opinion only the actual files contained in the rpm and maybe the scriptlets contained should be compared
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2016-02-17 22:07, Christian Boltz wrote:
Hello,
Am Mittwoch, 17. Februar 2016 schrieb Bernhard M. Wiedemann:
now I had some time to look into how reproducible our builds are and had a VM (with 4 cores) build all Leap 42.1 packages named a*
using some helper scripts, which I uploaded to https://github.com/bmwiedemann/reproducibleopensuse
with those helpers, I just had to do rebuildmany a* comparemany a*
btw: the rebuild of those 217 packages took 382 minutes
of those 217 (a small subset of Leap's 7830), build-compare reported a diff for 36 packages
and if you wonder, those are a2ps acct aegisub aespipe allegro alpine amanda amor anjuta anthy antlr apache-commons-cli apache-commons-codec apache-commons-collections apache-commons-email apache-commons-io apache-commons-lang apache-portlet-1_0-api apel appframework aqbanking argyllcms arts asl asm3 aspell aspell-en atmel-firmware autoconf-el autogen autoyast2 avfs avogadro avrdude awesome axis
I just checked the Debian result for some of them on https://tests.reproducible-builds.org/reproducible.html and interestingly, several of the packages you listed above are reproducible on Debian: acct, aegisub, aespipe, anthy, apel, asl, aspell, autogen, avogadro, avrdude.
I know the reproducible builds team at Debian is actively pushing patches to make the build reproducible, but I'm not sure if they were able to fix that many packages already (10 out of 36 - actually I should probably say 10 out of ~25 because some of the packages you listed don't exist in Debian, at least not with this name).
I think, they are very active and use latest upstream versions, possibly with extra patches, while I tested with Leap, which is an older codebase, but Factory was too fast-moving for me to test with atm.
Some diffs could also be caused by the fact that we do not always rebuild all dependent packages on changes. That could be avoided by doing it like the debian guys - rebuilding twice in two very different contexts.
Can you upload the differences for the packages you tested somewhere so that interested people don't need to rebuild everything just to get the diff?
sure. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/
Oh, BTW - maybe having some variables in your scripts instead of hardcoded paths would be nice ;-)
yep, it is an early alpha and certainly on the TODO list.
I was curious and tried with aspell. The compare result is:
/dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell/binaries /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell Comparing aspell-0.60.6.1-18.4.x86_64.rpm to aspell-0.60.6.1-0.x86_64.rpm comparing rpmtags --- /tmp/tmp.2P0iOiEtHL/tmp.jsMGUlvr4J 2016-02-17 21:34:31.060962628 +0100 +++ /tmp/tmp.2P0iOiEtHL/tmp.5h81k6Lq48 2016-02-17 21:34:31.068962571 +0100 @@ -9,7 +9,7 @@ has many other technical enhancements over Ispell, such as using shared memory for dictionaries and intelligently handling personal dictionaries when more than one Aspell process is open at once. - openSUSE openSUSE Leap 42.1 obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell
GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications + openSUSE openSUSE Leap 42.1 (none) GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications Productivity/Text/Spell http://aspell.net/ (none) (none) (none) (none) 4.11.2 x86_64-suse-linux cpio lzma 5 comparing RELEASE comparing PROVIDES comparing scripts comparing filelist comparing file checksum creating rename script RPM meta information is identical Extracting packages Package content is identical
[...same for other subpackages...]
It seems the only difference is in the rpmtags - (none) vs. obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell
Is this really a relevant difference, or should it be whitelisted in build-compare?
that is what I did with https://github.com/bmwiedemann/reproducibleopensuse/blob/master/build-co mpare.diff but not yet in a nice upstreamable way.
for aspell (and only this 1 of 36) I found, that is is reproducible when building with osc build --vm-type=kvm but when using the default chroot, it shows
comparing filelist @@ -1,8 +1,8 @@ /usr/bin/aspell 0 (none) 100755 root root 0 4294967295 /usr/bin/aspell-import 0 (none) 100755 root root 0 4294967295 - -/usr/bin/precat 0 (none) 100755 root root 0 4294967295 - -/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 precat - -/usr/bin/prezip 0 (none) 120777 root root 0 4294967295 precat +/usr/bin/precat 0 (none) 120777 root root 0 4294967295 prezip +/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 prezip +/usr/bin/prezip 0 (none) 100755 root root 0 4294967295 /usr/bin/prezip-bin 0 (none) 100755 root root 0 4294967295 /usr/bin/run-with-aspell 0 (none) 100755 root root 0 4294967295 /usr/bin/word-list-compress 0 (none) 100755 root root 0 4294967295 comparing file checksum creating rename script RPM file checksum differs. Extracting packages @@ -1 +0,0 @@ - -precat symlink target for /usr/bin/prezip differs
probably from the brp-symlink doing things differently in that context.
Hmm, actually $?=0, so on my system it _is_ reproducible. What did I do wrong? ;-)
I also tried aespipe, and get a real difference. Maybe someone should compare the openSUSE package with the Debian package to find out what they do different to make the package reproducible ;-)
e.g. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/alpine-c ompare.out shows an embedded ASCII date in 2 binaries.
Ciao Bernhard M.
Christian Boltz opensuse@cboltz.de writes:
It seems the only difference is in the rpmtags - (none) vs. obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell
This is the the DISTURL, if that changes you have different sources. But since you didn't build in OBS you have no DISTURL.
Andreas.
On 18.02.2016 09:33, Andreas Schwab wrote:
Christian Boltz opensuse@cboltz.de writes:
It seems the only difference is in the rpmtags - (none) vs. obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell
This is the the DISTURL, if that changes you have different sources. But since you didn't build in OBS you have no DISTURL.
I guess we need to special case (none) then for use outside of OBS.
Greetings, Stephan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2016-02-17 22:07, Christian Boltz wrote:
Hello,
Am Mittwoch, 17. Februar 2016 schrieb Bernhard M. Wiedemann:
now I had some time to look into how reproducible our builds are and had a VM (with 4 cores) build all Leap 42.1 packages named a*
using some helper scripts, which I uploaded to https://github.com/bmwiedemann/reproducibleopensuse
with those helpers, I just had to do rebuildmany a* comparemany a*
btw: the rebuild of those 217 packages took 382 minutes
of those 217 (a small subset of Leap's 7830), build-compare reported a diff for 36 packages
and if you wonder, those are a2ps acct aegisub aespipe allegro alpine amanda amor anjuta anthy antlr apache-commons-cli apache-commons-codec apache-commons-collections apache-commons-email apache-commons-io apache-commons-lang apache-portlet-1_0-api apel appframework aqbanking argyllcms arts asl asm3 aspell aspell-en atmel-firmware autoconf-el autogen autoyast2 avfs avogadro avrdude awesome axis
I just checked the Debian result for some of them on https://tests.reproducible-builds.org/reproducible.html and interestingly, several of the packages you listed above are reproducible on Debian: acct, aegisub, aespipe, anthy, apel, asl, aspell, autogen, avogadro, avrdude.
I know the reproducible builds team at Debian is actively pushing patches to make the build reproducible, but I'm not sure if they were able to fix that many packages already (10 out of 36 - actually I should probably say 10 out of ~25 because some of the packages you listed don't exist in Debian, at least not with this name).
I think, they are very active and use latest upstream versions, possibly with extra patches, while I tested with Leap, which is an older codebase, but Factory was too fast-moving for me to test with atm.
Some diffs could also be caused by the fact that we do not always rebuild all dependent packages on changes. That could be avoided by doing it like the debian guys - rebuilding twice in two very different contexts.
Can you upload the differences for the packages you tested somewhere so that interested people don't need to rebuild everything just to get the diff?
sure. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/
Oh, BTW - maybe having some variables in your scripts instead of hardcoded paths would be nice ;-)
yep, it is an early alpha and certainly on the TODO list.
I was curious and tried with aspell. The compare result is:
/dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell/binaries /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell /dev/shm/reproducibleopensuse/openSUSE:Leap:42.1/aspell Comparing aspell-0.60.6.1-18.4.x86_64.rpm to aspell-0.60.6.1-0.x86_64.rpm comparing rpmtags --- /tmp/tmp.2P0iOiEtHL/tmp.jsMGUlvr4J 2016-02-17 21:34:31.060962628 +0100 +++ /tmp/tmp.2P0iOiEtHL/tmp.5h81k6Lq48 2016-02-17 21:34:31.068962571 +0100 @@ -9,7 +9,7 @@ has many other technical enhancements over Ispell, such as using shared memory for dictionaries and intelligently handling personal dictionaries when more than one Aspell process is open at once. - openSUSE openSUSE Leap 42.1 obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell
GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications
GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications + openSUSE openSUSE Leap 42.1 (none) GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications GFDL-1.1+ and LGPL-2.1 and HPND and SUSE-BSD-Mark-Modifications Productivity/Text/Spell http://aspell.net/ (none) (none) (none) (none) 4.11.2 x86_64-suse-linux cpio lzma 5 comparing RELEASE comparing PROVIDES comparing scripts comparing filelist comparing file checksum creating rename script RPM meta information is identical Extracting packages Package content is identical
[...same for other subpackages...]
It seems the only difference is in the rpmtags - (none) vs. obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0
ae04047dd14a214-aspell
Is this really a relevant difference, or should it be whitelisted
in build-compare?
that is what I did with https://github.com/bmwiedemann/reproducibleopensuse/blob/master/build-co mpare.diff but not yet in a nice upstreamable way.
for aspell (and only this 1 of 36) I found, that is is reproducible when building with osc build --vm-type=kvm but when using the default chroot, it shows
comparing filelist @@ -1,8 +1,8 @@ /usr/bin/aspell 0 (none) 100755 root root 0 4294967295 /usr/bin/aspell-import 0 (none) 100755 root root 0 4294967295 - -/usr/bin/precat 0 (none) 100755 root root 0 4294967295 - -/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 precat - -/usr/bin/prezip 0 (none) 120777 root root 0 4294967295 precat +/usr/bin/precat 0 (none) 120777 root root 0 4294967295 prezip +/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 prezip +/usr/bin/prezip 0 (none) 100755 root root 0 4294967295 /usr/bin/prezip-bin 0 (none) 100755 root root 0 4294967295 /usr/bin/run-with-aspell 0 (none) 100755 root root 0 4294967295 /usr/bin/word-list-compress 0 (none) 100755 root root 0 4294967295 comparing file checksum creating rename script RPM file checksum differs. Extracting packages @@ -1 +0,0 @@ - -precat symlink target for /usr/bin/prezip differs
probably from the brp-symlink doing things differently in that context.
Hmm, actually $?=0, so on my system it _is_ reproducible. What did I do wrong? ;-)
I also tried aespipe, and get a real difference. Maybe someone should compare the openSUSE package with the Debian package to find out what they do different to make the package reproducible ;-)
e.g. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/alpine-c ompare.out (and some others) shows an embedded ASCII date in 2 binaries.
Ciao Bernhard M.
Hi,
One thing, do we keep a list of packages which do not reproducible build(-compare) that can be processed manually at leisure?
Because I think there is some manual work to do.
Ciao, Marcus
On Thu, Feb 18, 2016 at 09:57:46AM +0100, Stephan Kulow wrote:
On 18.02.2016 09:33, Andreas Schwab wrote:
Christian Boltz opensuse@cboltz.de writes:
It seems the only difference is in the rpmtags - (none) vs. obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7fe0ae04047dd14a214-aspell
This is the the DISTURL, if that changes you have different sources. But since you didn't build in OBS you have no DISTURL.
I guess we need to special case (none) then for use outside of OBS.
Greetings, Stephan
"Bernhard M. Wiedemann" bernhardout@lsmod.de writes:
probably from the brp-symlink doing things differently in that context.
It probably depends on how files are ordered in the directory, which differs between filesystems.
Andreas.
On Thursday 2016-02-18 08:35, Damian Ivanov wrote:
Doing these builds on OBS wouldn't they always differ? I mean the release tag in .rpm's is always incremented which already results in a diff.
That is why B_CNT should be ignored during comparison. One can also ignore CI_CNT (and therefore the Release field), since the source checksum is in disturl.
On 18.02.2016 10:15, Marcus Meissner wrote:
Hi,
One thing, do we keep a list of packages which do not reproducible build(-compare) that can be processed manually at leisure?
Because I think there is some manual work to do.
Just do a bugzilla search for build-compare - and also look for WONTFIXes ;(
Greetings, Stephan
On 18.02.2016 10:55, Andreas Schwab wrote:
"Bernhard M. Wiedemann" bernhardout@lsmod.de writes:
probably from the brp-symlink doing things differently in that context.
It probably depends on how files are ordered in the directory, which differs between filesystems.
I think this is https://bugzilla.opensuse.org/show_bug.cgi?id=906154
Greetings, Stephan
On Wed, Feb 17, Bernhard M. Wiedemann wrote:
and had a VM (with 4 cores) build all Leap 42.1 packages named a*
Released versions have a fixed code base, so it does not matter much if a rebuild changes binaries. Noone cares about that. The resulting binaries are not republished.
What matters is the result in Factory because its a moving target and the same codebase will be built and possibly republished. I'm sure all of the packages in your list (and the other 66% of TW) are still "broken" from build-compare POV.
If it really bothers you, run SUSE:Factory:Head and check why most of the installed packages get updated after a rebuild, even if just glibc got updated. I'm sure, many pkgs can be fixed if a fixed link order is used and if the documentation is not built with random id="N" tags.
Olaf
On Thu, Feb 18, Marcus Meissner wrote:
One thing, do we keep a list of packages which do not reproducible build(-compare) that can be processed manually at leisure?
There is home:coolo:bc, which used have >200 failures. I think it requires too much resources to keep it active. And people to actually work on the reported packages.
Olaf
Hello,
Am Donnerstag, 18. Februar 2016 schrieb Bernhard M. Wiedemann:
On 2016-02-17 22:07, Christian Boltz wrote:
Am Mittwoch, 17. Februar 2016 schrieb Bernhard M. Wiedemann:
now I had some time to look into how reproducible our builds are and had a VM (with 4 cores) build all Leap 42.1 packages named a*
I just checked the Debian result for some of them on https://tests.reproducible-builds.org/reproducible.html and interestingly, several of the packages you listed above are reproducible on Debian: acct, aegisub, aespipe, anthy, apel, asl, aspell, autogen, avogadro, avrdude.
I know the reproducible builds team at Debian is actively pushing patches to make the build reproducible, but I'm not sure if they were able to fix that many packages already (10 out of 36 - actually I should probably say 10 out of ~25 because some of the packages you listed don't exist in Debian, at least not with this name).
I think, they are very active and use latest upstream versions,
BTW: when it comes to Debian, I wouldn't always assume that Leap packages are older ;-)
possibly with extra patches,
Yes, they are actively providing patches to make the builds reproducible (which will first be in Debian of course), but they also send those patches upstream, so we should get them sooner or later.
while I tested with Leap, which is an older codebase, but Factory was too fast-moving for me to test with atm.
That's understandable - checking a fixed target makes things much easier.
Leap is probably a good starting point - mass-rebuild all packages, get the list of packages that differ - and then check only those packages in Factory. A list of some hundred packages is easier to handle on a moving target than some thousand packages ;-)
(In the long term, I'd love to have Factory reproducible - but we need to start somewhere.)
Some diffs could also be caused by the fact that we do not always rebuild all dependent packages on changes. That could be avoided by doing it like the debian guys - rebuilding twice in two very different contexts.
They always rebuild with exactly the same versions of the BuildRequires both times (but with different timezone, locale, whatever).
Can you upload the differences for the packages you tested somewhere so that interested people don't need to rebuild everything just to get the diff?
sure. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/
Thanks!
I was curious and tried with aspell.
...
It seems the only difference is in the rpmtags - (none) vs. obs://build.opensuse.org/openSUSE:Leap:42.1/standard/dd3d6bcef5a4f7f e0ae04047dd14a214-aspell
Is this really a relevant difference, or should it be whitelisted in build-compare?
that is what I did with https://github.com/bmwiedemann/reproducibleopensuse/blob/master/build-%3E compare.diff but not yet in a nice upstreamable way.
Ah, I didn't look at that patch ;-)
for aspell (and only this 1 of 36) I found, that is is reproducible when building with osc build --vm-type=kvm but when using the default chroot, it shows
comparing filelist @@ -1,8 +1,8 @@ /usr/bin/aspell 0 (none) 100755 root root 0 4294967295 /usr/bin/aspell-import 0 (none) 100755 root root 0 4294967295 -/usr/bin/precat 0 (none) 100755 root root 0 4294967295 -/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 precat -/usr/bin/prezip 0 (none) 120777 root root 0 4294967295 precat +/usr/bin/precat 0 (none) 120777 root root 0 4294967295 prezip +/usr/bin/preunzip 0 (none) 120777 root root 0 4294967295 prezip +/usr/bin/prezip 0 (none) 100755 root root 0 4294967295 /usr/bin/prezip-bin 0 (none) 100755 root root 0 4294967295 /usr/bin/run-with-aspell 0 (none) 100755 root root 0 4294967295 /usr/bin/word-list-compress 0 (none) 100755 root root 0 4294967295 comparing file checksum creating rename script RPM file checksum differs. Extracting packages @@ -1 +0,0 @@ -precat symlink target for /usr/bin/prezip differs
probably from the brp-symlink doing things differently in that context.
I used osc build, but didn't see those differences. Maybe the relevant part is that I'm building in /dev/shm
The takeaway from the filelist diff is that (probably) brp-symlink needs a "sort" somewhere to get a stable order (including LC_ALL=C to ensure sort behaves the same all the time). I wouldn't be surprised if this little change fixes filelist diffs in lots of packages.
I also tried aespipe, and get a real difference. Maybe someone should compare the openSUSE package with the Debian package to find out what they do different to make the package reproducible ;-)
e.g. https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/alpine -c ompare.out shows an embedded ASCII date in 2 binaries.
Indeed, embedding the build date is the cause for differences in several of the packages you tested. Not only in compiled binaries - I even found the build date in *.png files in awesome-compare.out ;-)
Embedded build dates aren't too hard to fix - either completely leave it out (easiest solution) or use a fixed date that is still helpful. That fixed date can be something like the date of the last *.changes entry, which would IMHO be more useful than the information when OBS finally had some time to build the package. (We probably need to discuss which date to choose - I'd vote for the last *.changes entry.)
BTW: axis-compare.out shows a big difference in the filelist - it seems like your rebuild is missing half of the files in /usr/share/javadoc/. Also, /usr/share/javadoc/axis/javax/xml/namespace/class-use/QName.html in your rebuild seems to be completely empty.
I have no idea what caused this, but it's clearly more than some "random" change during the rebuild. I'm especially surprised that those missing files didn't cause the build to fail - which I'd call a bug.
Regards,
Christian Boltz
On 2016-02-18 22:39, Christian Boltz wrote:
(In the long term, I'd love to have Factory reproducible - but we need to start somewhere.)
I started by fixing bind, libzypp and beaver via SRs to their devel projects.
meanwhile I have rebuilt and compared a lot more packages with diffs available at https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/
and found that on Leap at least these packages use embedded build date strings:
apache-commons-collections apache-commons-io apache-commons-lang awesome bash #beaver berkeleydb #bind blitz blktrace bpython bsh2 busybox busybox-static bzflag calibre ckermit clamav clisp cobbler colord compat-openssl098 converseen courier-imap cpptasks crash crawl crmsh crosstool-ng csound daps doomsday dos2unix dosemu drbd dvbsnoop dvgt (via TeX) dynamips eciadsl eigen3 (via TeX) elfutils emacs epic fence-agents ffado flim-xemacs fvwm gcin gcolor2 geany getdp gfs2-utils ghostscript-fonts-grops git gnucash gnu-free-fonts gnugk gnugo gnuplot golang-org-x-tools go-mango-doc gpick graphviz-plugins gri groff grub2 gzip hdf5 hercules hsqldb hwinfo hylafax+ i4l-base igerman98 imagej infinipath-psm inkscape jack jakarta-commons-pool java-1_7_0-openjdk-plugin java-1_8_0-openjdk-plugin jedit jimtcl joe judy kdelibs4-apidocs kde-oxygen-fonts keepalived kernel-docs keyutils kile ... libjpeg-turbo libopenssl0_9_8 libusnic_verbs-rdmav2 libvisual #libzypp licq LiE lilypond ltrace lv2 machinery malaga man marisa mcrypt modglue nested pam_ldap pandoc pavuk pcp pdns pdns-recursor perl perl-RPC-XML php5 pidentd pilot-link plasma5-desktop? plowshare po4a presage psqlODBC pstoedit putty pythia python3-doc python3-gunicorn python3-numpy-doc python3-psycopg2 python3-virtualenv-doc python3-xlwt python-4Suite-XM python-amqp python-caja python-cliff python-configshell python-doc python-efl python-FormEncode python-fudge python-gnome-extras python-numpy-doc python-oslo.config python-oslo.messaging python-Paste python-psycopg2 python-pysqlite python-sqlalchemy python-stevedore python-translationstring python-utidy python-virtualenv qca qemu qpid-proton qt-creator quvi qwt racket R-base release-notes-openSUSE rkward rockdodger rocksndiamonds rrdtool rubygem-rice salt samba sane-backends scilab sendmail siege simh sitar snd snorenotify-qt5 sqliteodbc (from doxygen) srecord swipl systemtap-docs tango-icon-theme thai-fonts tipcutils tse3 tuxpaint uClibc update-test-trival uperf ushare uudeview v4l-utils vifm vips VirtualGL vpcs weechat wine xemacs yudit zpaq zynaddsubfx
Hello Bernhard,
This is quite usefull. Could you run it against Factory and just reporting it in form of a bugs, you can even create trackerbug so we can be sure on how we are stading?
Cheers
Tom
Bernhard M. Wiedemann píše v Po 29. 02. 2016 v 19:02 +0100:
On 2016-02-18 22:39, Christian Boltz wrote:
(In the long term, I'd love to have Factory reproducible - but we need to start somewhere.)
I started by fixing bind, libzypp and beaver via SRs to their devel projects.
meanwhile I have rebuilt and compared a lot more packages with diffs available at https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/
and found that on Leap at least these packages use embedded build date strings:
apache-commons-collections apache-commons-io apache-commons-lang awesome bash #beaver berkeleydb #bind blitz blktrace bpython bsh2 busybox busybox-static bzflag calibre ckermit clamav clisp cobbler colord compat-openssl098 converseen courier-imap cpptasks crash crawl crmsh crosstool-ng csound daps doomsday dos2unix dosemu drbd dvbsnoop dvgt (via TeX) dynamips eciadsl eigen3 (via TeX) elfutils emacs epic fence-agents ffado flim-xemacs fvwm gcin gcolor2 geany getdp gfs2-utils ghostscript-fonts-grops git gnucash gnu-free-fonts gnugk gnugo gnuplot golang-org-x-tools go-mango-doc gpick graphviz-plugins gri groff grub2 gzip hdf5 hercules hsqldb hwinfo hylafax+ i4l-base igerman98 imagej infinipath-psm inkscape jack jakarta-commons-pool java-1_7_0-openjdk-plugin java-1_8_0-openjdk-plugin jedit jimtcl joe judy kdelibs4-apidocs kde-oxygen-fonts keepalived kernel-docs keyutils kile ... libjpeg-turbo libopenssl0_9_8 libusnic_verbs-rdmav2 libvisual #libzypp licq LiE lilypond ltrace lv2 machinery malaga man marisa mcrypt modglue nested pam_ldap pandoc pavuk pcp pdns pdns-recursor perl perl-RPC-XML php5 pidentd pilot-link plasma5-desktop? plowshare po4a presage psqlODBC pstoedit putty pythia python3-doc python3-gunicorn python3-numpy-doc python3-psycopg2 python3-virtualenv-doc python3-xlwt python-4Suite-XM python-amqp python-caja python-cliff python-configshell python-doc python-efl python-FormEncode python-fudge python-gnome-extras python-numpy-doc python-oslo.config python-oslo.messaging python-Paste python-psycopg2 python-pysqlite python-sqlalchemy python-stevedore python-translationstring python-utidy python-virtualenv qca qemu qpid-proton qt-creator quvi qwt racket R-base release-notes-openSUSE rkward rockdodger rocksndiamonds rrdtool rubygem-rice salt samba sane-backends scilab sendmail siege simh sitar snd snorenotify-qt5 sqliteodbc (from doxygen) srecord swipl systemtap-docs tango-icon-theme thai-fonts tipcutils tse3 tuxpaint uClibc update-test-trival uperf ushare uudeview v4l-utils vifm vips VirtualGL vpcs weechat wine xemacs yudit zpaq zynaddsubfx
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2016-03-02 13:12, Tomas Chvatal wrote:
This is quite useful. Could you run it against Factory and just reporting it in form of a bugs, you can even create trackerbug so we can be sure on how we are standing?
I am re-building those on Factory now, but I am not sure if we really want to patch each individual package (with patches that in many cases cannot go upstream)
or if we should go the more general approach of patching gcc and javadoc to use the https://reproducible-builds.org/specs/source-date-epoch/ variable when a .c file uses __TIME__ and __DATE__ and set it in OBS/osc/rpmbuild/build to the epoch of the newest rpm involved in the build. Adrian said, if they implement it, it would be a project option.
doxygen and 3 others already use that variable.
Ciao Bernhard M.
29.02.2016 21:02, Bernhard M. Wiedemann пишет:
meanwhile I have rebuilt and compared a lot more packages with diffs available at https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare/
and found that on Leap at least these packages use embedded build date strings:
grub2
beta3 submitted yesterday should be more reproducible builds friendly.
On Wed, Mar 02, Tomas Chvatal wrote:
This is quite usefull. Could you run it against Factory and just
Some stats about the current state of the 8833 pkgs in Factory:
good: 7014 (no differences found by build-compare) fail: 30 (build failed for some reason) new: 123 (no rebuild happend since last checkin) diff: 1575 (build-compare found some differences to the former build) odd: 91 (either its excluded or a core pkg, so build-comare did not run)
For short: 80% done.
Olaf
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2016-03-05 10:41, Olaf Hering wrote:
On Wed, Mar 02, Tomas Chvatal wrote:
This is quite usefull. Could you run it against Factory and just
Some stats about the current state of the 8833 pkgs in Factory:
good: 7014 (no differences found by build-compare)
that could miss some bad packages that use year in copyright
I attached my current 190 entries of the findings which is verified on Factory and includes filenames containing build date/time
and there are more detailed diffs for those packages on https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare.factory/
For a few, I already filed bugs or SRs, which are then linked in the list.
Ciao Bernhard M.
On Mon, Mar 7, 2016 at 7:01 AM, Bernhard M. Wiedemann bernhardout2@lsmod.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2016-03-05 10:41, Olaf Hering wrote:
On Wed, Mar 02, Tomas Chvatal wrote:
This is quite usefull. Could you run it against Factory and just
Some stats about the current state of the 8833 pkgs in Factory:
good: 7014 (no differences found by build-compare)
that could miss some bad packages that use year in copyright
I attached my current 190 entries of the findings which is verified on Factory and includes filenames containing build date/time
and there are more detailed diffs for those packages on https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare.factory/
For a few, I already filed bugs or SRs, which are then linked in the list.
for libopenssl0_9_8 # /usr/lib64/libcrypto.so.0.9.8 = https://build.opensuse.org/request/show/365748 you have to fill a drop request instead. "The 0.9.8 and 1.0.0 versions are now out of support and should not be used."
Hi,
* Dne Sobota 12. březen 2016, 04:33:33 [CET] Cristian Rodríguez napsal:
On Mon, Mar 7, 2016 at 7:01 AM, Bernhard M. Wiedemann bernhardout2@lsmod.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2016-03-05 10:41, Olaf Hering wrote:
On Wed, Mar 02, Tomas Chvatal wrote:
This is quite usefull. Could you run it against Factory and just
Some stats about the current state of the 8833 pkgs in Factory:
good: 7014 (no differences found by build-compare)
that could miss some bad packages that use year in copyright
I attached my current 190 entries of the findings which is verified on Factory and includes filenames containing build date/time
and there are more detailed diffs for those packages on https://www.zq1.de/~bernhard/linux/reproducibleopensuse/compare.factory/
For a few, I already filed bugs or SRs, which are then linked in the list.
for libopenssl0_9_8 # /usr/lib64/libcrypto.so.0.9.8 = https://build.opensuse.org/request/show/365748 you have to fill a drop request instead. "The 0.9.8 and 1.0.0 versions are now out of support and should not be used."
Correct. This compatibility library isn't needed anymore.
I filed drop request for Factory as well as for its devel project: https://build.opensuse.org/request/show/370455 https://build.opensuse.org/request/show/370458
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2016-02-29 19:02, Bernhard M. Wiedemann wrote:
On 2016-02-18 22:39, Christian Boltz wrote:
(In the long term, I'd love to have Factory reproducible - but we need to start somewhere.)
I started by fixing bind, libzypp and beaver via SRs to their devel projects.
Many more packages were fixed now via SRs to their devel projects
I am still working on reproducible builds for openSUSE. Last weekend, I managed the first time to build rpms with 'osc build' that were every bit identical using the rpm+build versions from https://build.opensuse.org/project/show/home:bmwiedemann:reproducible for that to work, you also have to export SOURCE_DATE_EPOCH with a UNIX-timestamp number and add it to /etc/sudoers in the line with Defaults env_keep
Python packages are tricky to get right, because .pyc and .pyo files contain timestamps of their source file and will not be used unless it matches exactly. Why do we (and redhat) even include them? In Debian packages have only .py files and the precompiled .pyc files get added upon package installation.
It is even more tricky to get fully reproducible builds In OBS, because the build host name and signature time will vary. So for now I guess, I will continue working on fixing build-compare failures (e.g. from embedded timestamps, rebuild-counters or compile-time CPU detection)?
Ciao Bernhard M.
Hi,
On Mon, 14 Mar 2016, Bernhard M. Wiedemann wrote:
Python packages are tricky to get right, because .pyc and .pyo files contain timestamps of their source file and will not be used unless it matches exactly. Why do we (and redhat) even include them? In Debian packages have only .py files and the precompiled .pyc files get added upon package installation.
I wonder if .pyc or .pyo bring any advantages at all at this time?
It is even more tricky to get fully reproducible builds In OBS, because the build host name and signature time will vary.
I'm really not sure if we should strive for this. IMHO, if the contents of containers are provably the same, then the container itself doesn't matter much. In this case it seems to me that all files and scripts inside the rpm should be the same (and perhaps a selection of other rpm tags), not necessarily everything in the .rpm file.
If the goal is that others can reproduce a bit-identical .rpm file, then it seems reasonable to require them to have to adjust their build process (by e.g. setting the wanted build host name and fiddling with the signature process, they can't produce bit identical signatures anyway, as they don't have our secret key) ...
So for now I guess, I will continue working on fixing build-compare failures (e.g. from embedded timestamps, rebuild-counters or compile-time CPU detection)?
... so I think this is exactly the right way forward.
Ciao, Michael.
Hello,
On Mar 14 14:15 Michael Matz wrote (excerpt):
On Mon, 14 Mar 2016, Bernhard M. Wiedemann wrote:
Python packages are tricky to get right, because .pyc and .pyo files contain timestamps of their source file and will not be used unless it matches exactly. Why do we (and redhat) even include them? In Debian packages have only .py files and the precompiled .pyc files get added upon package installation.
I wonder if .pyc or .pyo bring any advantages at all at this time?
I am not all all a Python expert but I had some trouble with precompiled .pyc and .pyo files in the hplip package that lead in the end to the following "current solution":
--------------------------------------------------------------------- $ osc cat Printing hplip hplip.spec | less ... # Make and install Python compiled bytecode files # (.pyc compiled python and .pyo optimized compiled python) # because normal users do not have write permissions # to the install location /usr/share/hplip/ so that # for normal users Python would recompile the sources # every time which results longer program startup time # and waste of CPU for compiling, # see https://en.opensuse.org/openSUSE:Packaging_Python#Byte_Compiled_Files # and http://lists.opensuse.org/opensuse-packaging/2014-10/msg00028.html # Make and install .pyc files: %py_compile %{buildroot}%{_datadir}/hplip # Make and install .pyo files: %py_compile -O %{buildroot}%{_datadir}/hplip # Hardlink .pyc and .pyo when they have same content. # Do not run "fdupes buildroot/_datadir/hplip" because # fdupes will link any files with same content there # which can have unexpected side-effects, compare # https://bugzilla.opensuse.org/show_bug.cgi?id=784670 for pyc in $( find %{buildroot}%{_datadir}/hplip -name '*.pyc' ) do pyo="${pyc%.pyc}.pyo" if test -f $pyo && cmp -s $pyc $pyo then echo hardlinking $pyc and $pyo because both have same content ln -f $pyc $pyo fi done ---------------------------------------------------------------------
Kind Regards Johannes Meixner
On Mon, 14 Mar 2016 14:15, Michael Matz wrote:
On Mon, 14 Mar 2016, Bernhard M. Wiedemann wrote:
Python packages are tricky to get right, because .pyc and .pyo files contain timestamps of their source file and will not be used unless it matches exactly. Why do we (and redhat) even include them? In Debian packages have only .py files and the precompiled .pyc files get added upon package installation.
I wonder if .pyc or .pyo bring any advantages at all at this time?
On slow HW, esp with rotating rust, YES. But "compile on install" ala debian - best controlled by config setting - seems the the way to go forward fast in this aspect. This will become more imporatant, as other script languages will follow with precompiled files, esp. for system-libs, say hello to the debates on perl6 doing such a thing, and ruby devs also looks into that, never mind what php7 tries.
It is even more tricky to get fully reproducible builds In OBS, because the build host name and signature time will vary.
I'm really not sure if we should strive for this. IMHO, if the contents of containers are provably the same, then the container itself doesn't matter much. In this case it seems to me that all files and scripts inside the rpm should be the same (and perhaps a selection of other rpm tags), not necessarily everything in the .rpm file.
If the goal is that others can reproduce a bit-identical .rpm file, then it seems reasonable to require them to have to adjust their build process (by e.g. setting the wanted build host name and fiddling with the signature process, they can't produce bit identical signatures anyway, as they don't have our secret key) ...
IMHO: 1: diff on the whole rpm-file is wrong, diff the payload cpio only, the rpm header has to be handled extra.
2: the sig-hash should be done on the payload, if one wants a sig on the header, do that extra.
3: payload hash gen is either by the per-file-content and a hash over all of the per-file-hashes , or the file and dir timestamps in the payload will ruin your fun, too.
That way build-compare can compare the payload at a first glance by the payload-hash-of-per-file-hashes
So for now I guess, I will continue working on fixing build-compare failures (e.g. from embedded timestamps, rebuild-counters or compile-time CPU detection)?
... so I think this is exactly the right way forward.
Ciao, Michael.
And thank you for the work invested. - Yamaban
Hi,
On Mon, 14 Mar 2016, Yamaban wrote:
contain timestamps of their source file and will not be used unless it matches exactly. Why do we (and redhat) even include them? In Debian packages have only .py files and the precompiled .pyc files get added upon package installation.
I wonder if .pyc or .pyo bring any advantages at all at this time?
On slow HW, esp with rotating rust, YES.
Any numbers to convince me? :)
But "compile on install" ala debian - best controlled by config setting - seems the the way to go forward fast in this aspect.
Actually I would do such compilation in a deferred scheme, not directly at install time (or well, maybe that's acceptable as well if not taking too much time).
This will become more imporatant, as other script languages will follow with precompiled files, esp. for system-libs, say hello to the debates on perl6 doing such a thing, and ruby devs also looks into that, never mind what php7 tries.
Yeah, true. I actually loath "script" languages that have a compiled form, that seems a work-around for some fundamental mis-design of them ("whohooo, our parser is so slow" - "yeah, well, it's a script language, right?" - "but ... but, I want it to be fast" - "and I want a rosa pony, go dying").
Ciao, Michael.
On Monday 2016-03-14 16:02, Michael Matz wrote:
On Mon, 14 Mar 2016, Yamaban wrote:
contain timestamps of their source file and will not be used unless it matches exactly. Why do we (and redhat) even include them? In Debian packages have only .py files and the precompiled .pyc files get added upon package installation.
I wonder if .pyc or .pyo bring any advantages at all at this time?
On slow HW, esp with rotating rust, YES.
Any numbers to convince me? :)
On rotating rust it might just be the opposite - spending the time saved on compilation on seeks as directories now have thrice the amount of files.
But "compile on install" ala debian - best controlled by config setting - seems the the way to go forward fast in this aspect.
Actually I would do such compilation in a deferred scheme, not directly at install time (or well, maybe that's acceptable as well if not taking too much time).
Generally you do not want to increase install times, as they are already big by inevitably system-dependent things like mkinitrd and fonts-config. pyc files however are not so much system dependent as they are architecture dependent, so it's better to spend that build time in .spec files. That would also make it easier for them to be tracked by rpm.
On Mon, 14 Mar 2016 16:02, Michael Matz wrote:
On Mon, 14 Mar 2016, Yamaban wrote:
contain timestamps of their source file and will not be used unless it matches exactly. Why do we (and redhat) even include them? In Debian packages have only .py files and the precompiled .pyc files get added upon package installation.
I wonder if .pyc or .pyo bring any advantages at all at this time?
On slow HW, esp with rotating rust, YES.
Any numbers to convince me? :)
For a good storage compare try this: HW: raspi2 -- for the slow / puny / frail cpu Test 1: slow / older USB2 hdd w/ 5400rpm, little cache (e.g. 20MByte/s ~ 180Mbit/s Test 2: very fast USB3 - SSD / Flash (yes, the interface is 480Mbit/s only)
This 2.5x difference in through-put and the much bigger difference in 'seek'ing time becomes significant during the search for the dependencies.
put the python files on that, without the .pyc and .pyo,
load a bigger (using many libs / py files) app on both setups, see the difference wrt the storage, and why it makes the difference rust vs. solid
On my old Atom330 (Dual Core, HT, 1600Mhz, 4GB Ram) with openSUSE 12.3, I saw the difference between HDD and SSD, too, but also the difference between available compiled python and not. E.g. loading calibre was a pain in waiting without doing a precompile beforehand (My notes speak of 75 sec on rust w/o precompile vs 15 sec on SSD with precompile).
May that box rest in peace, the fans got loud at the end.
Now I'm working on a Haswell i3-4330T Passive Cooling Box w/ Leap 42.1 (3Ghz, 16GB Ram, Samsung Evo 250GB mSata, 2.5k Display)
On the new Box I can not tell the difference between with precompile and not -- the Storage is FAST, the Ram is BIG, the cpu has POWER.
But "compile on install" ala debian - best controlled by config setting - seems the the way to go forward fast in this aspect.
Actually I would do such compilation in a deferred scheme, not directly at install time (or well, maybe that's acceptable as well if not taking too much time).
That is what I meant with "config setting": 1: at install, put into /usr/lib..., 2: deferered cache /var/cache/python/... (per user would be the most easy), 3: no precompile at all, fresh every time, as "noarch" as possible
opt 1 for physically limited machines, pay the penalty once, opt 2 for those that, start python apps all the time, but have the machine power, opt 3 for those that use phython very seldom, or are heavy into python dev, or simply have the fast storage and cpu power that they do not care.
IMHO, bet would be a cli-tool, lets call it "python-precomile" with a few options: --remove :: either with additional lib/program name, or implied all --install :: either with additional lib/program name, or implied all --clear-cache :: either with system, user, or implied "all" --prewarm-cache :: (it this is wanted or needed) --respect-config :: read config, remove or install as configured
That called as root during postinstall with "python-precomile --respect-config <package>" should reduce the hassle for the packager as much as possible, and allow the admin to change his/her mind later on.
This will become more imporatant, as other script languages will follow with precompiled files, esp. for system-libs, say hello to the debates on perl6 doing such a thing, and ruby devs also looks into that, never mind what php7 tries.
Yeah, true. I actually loath "script" languages that have a compiled form, that seems a work-around for some fundamental mis-design of them ("whohooo, our parser is so slow" - "yeah, well, it's a script language, right?" - "but ... but, I want it to be fast" - "and I want a rosa pony, go dying").
*Grin* Yep, wanna bake the cake and eat the batter beforhand.
I know when to use "script" and when "compile", but then, my training started around 1985, during school, the teacher was a 50yo 'retired programming sysadmin', she knew her stuff from the pike up. (had fascinating, and hair-rising stories, too)
- Yamaban
On 14.03.2016 16:13, Jan Engelhardt wrote:
Generally you do not want to increase install times, as they are already big by inevitably system-dependent things like mkinitrd and fonts-config. pyc files however are not so much system dependent as they are architecture dependent, so it's better to spend that build time in .spec files. That would also make it easier for them to be tracked by rpm.
On one hand you argue that python is so quick to compile that you don't need to cache it and then you don't want it at install time as it's too slow? Pick your poison.
Greetings, Stephan
On Tuesday 2016-03-15 13:31, Stephan Kulow wrote:
On 14.03.2016 16:13, Jan Engelhardt wrote:
Generally you do not want to increase install times, as they are already big by inevitably system-dependent things like mkinitrd and fonts-config. pyc files however are not so much system dependent as they are architecture dependent, so it's better to spend that build time in .spec files. That would also make it easier for them to be tracked by rpm.
On one hand you argue that python is so quick to compile that you don't need to cache it and then you don't want it at install time as it's too slow? Pick your poison.
^^
In the course of that particular argument, not the compile step would be the slow one; having the extra files present on disk would be.
-
Adam Spiers -
Andreas Schwab -
Andrei Borzenkov -
Bernhard M. Wiedemann -
Bernhard M. Wiedemann -
Christian Boltz -
Cristian Rodríguez -
Damian Ivanov -
Jan Engelhardt -
Johannes Meixner -
Jérémy Bobbio -
Ludwig Nussel -
Marcus Meissner -
Michael Matz -
Olaf Hering -
Richard Biener -
Stephan Kulow -
Tomas Chvatal -
Tomáš Chvátal
-
Vitezslav Cizek -
Yamaban