[opensuse-packaging] Reproducible .tar.gz compression?
I have a package that "tar czf"s a directory in the build process. This breaks build-compare since every compression generates a different file. A "tar cjf" (bz2) doesn't suffer from the same problem... Any idea? Can "tar czf" produce reproducible results? TIA -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2011/3/7 Cristian Morales Vega <cmorve69@yahoo.es>:
I have a package that "tar czf"s a directory in the build process. This breaks build-compare since every compression generates a different file. A "tar cjf" (bz2) doesn't suffer from the same problem... Any idea? Can "tar czf" produce reproducible results?
In case it helps someone... The problem is the modification time of the intermediate tar file. This, tar -c <dir> | gzip -n > <output.tar.gz> , solves the problem. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
El 07/03/11 15:42, Cristian Morales Vega escribió:
2011/3/7 Cristian Morales Vega <cmorve69@yahoo.es>:
I have a package that "tar czf"s a directory in the build process. This breaks build-compare since every compression generates a different file.
a file with different timestamps or one with different checksum ? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2011/3/7 Cristian Rodríguez <crrodriguez@opensuse.org>:
El 07/03/11 15:42, Cristian Morales Vega escribió:
2011/3/7 Cristian Morales Vega <cmorve69@yahoo.es>:
I have a package that "tar czf"s a directory in the build process. This breaks build-compare since every compression generates a different file.
a file with different timestamps or one with different checksum ?
No in the original files. Otherwise the bzip2 file would also be different ;-) The problem was the timestamp of the temporal tar file... but now I have a different problem. I can create reproducible gzip files from the tar, but now I'm unable to create reproducible tar files. In my local system everything is OK. Using "osc build" I get reproducible results. If I manually create .tar.gz files from different copies of the directory I get reproducible results. If I uncompress the .tar.gz files created in the build service and create new .tar.gz from them I get reproducible results... But the builds in the (Packman) Build Service are not reproducible. I'm going to look through the tar file format specs, but any help is welcome. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2011/3/8 Cristian Morales Vega <cmorve69@yahoo.es>:
2011/3/7 Cristian Rodríguez <crrodriguez@opensuse.org>:
El 07/03/11 15:42, Cristian Morales Vega escribió:
2011/3/7 Cristian Morales Vega <cmorve69@yahoo.es>:
I have a package that "tar czf"s a directory in the build process. This breaks build-compare since every compression generates a different file.
a file with different timestamps or one with different checksum ?
No in the original files. Otherwise the bzip2 file would also be different ;-) The problem was the timestamp of the temporal tar file... but now I have a different problem. I can create reproducible gzip files from the tar, but now I'm unable to create reproducible tar files.
In my local system everything is OK. Using "osc build" I get reproducible results. If I manually create .tar.gz files from different copies of the directory I get reproducible results. If I uncompress the .tar.gz files created in the build service and create new .tar.gz from them I get reproducible results... But the builds in the (Packman) Build Service are not reproducible.
I'm going to look through the tar file format specs, but any help is welcome.
The problem was tar was adding the files in a different order each time. Finally I had to convert tar cvvzf skins2/default.vlt --exclude .svn -C $(srcdir)/skins2 default/ into (cd $(srcdir)/skins2; find default -print0 | sort -z | tar cvv --exclude .svn --no-recursion --null -T -) | gzip -n > skins2/default.vlt Works. But if someone knows of a simpler way... -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 8.3.2011 15:31, Cristian Morales Vega wrote:
2011/3/8 Cristian Morales Vega <cmorve69@yahoo.es>:
2011/3/7 Cristian Rodríguez <crrodriguez@opensuse.org>:
El 07/03/11 15:42, Cristian Morales Vega escribió:
2011/3/7 Cristian Morales Vega <cmorve69@yahoo.es>:
I have a package that "tar czf"s a directory in the build process. This breaks build-compare since every compression generates a different file.
a file with different timestamps or one with different checksum ?
No in the original files. Otherwise the bzip2 file would also be different ;-) The problem was the timestamp of the temporal tar file... but now I have a different problem. I can create reproducible gzip files from the tar, but now I'm unable to create reproducible tar files.
In my local system everything is OK. Using "osc build" I get reproducible results. If I manually create .tar.gz files from different copies of the directory I get reproducible results. If I uncompress the .tar.gz files created in the build service and create new .tar.gz from them I get reproducible results... But the builds in the (Packman) Build Service are not reproducible.
I'm going to look through the tar file format specs, but any help is welcome.
The problem was tar was adding the files in a different order each time. Finally I had to convert
tar cvvzf skins2/default.vlt --exclude .svn -C $(srcdir)/skins2 default/
into
(cd $(srcdir)/skins2; find default -print0 | sort -z | tar cvv --exclude .svn --no-recursion --null -T -) | gzip -n > skins2/default.vlt
Works. But if someone knows of a simpler way...
You should exclude directories in the find command, otherwise their content will be packaged twice. Here is a more complete version that also handles timestamps of files being packaged (in case the files are generated): http://gitorious.org/opensuse/kernel-source/blobs/master/scripts/tar-up.sh, function stable_tar(). Michal -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
2011/3/8 Michal Marek <mmarek@suse.cz>:
On 8.3.2011 15:31, Cristian Morales Vega wrote:
The problem was tar was adding the files in a different order each time. Finally I had to convert
tar cvvzf skins2/default.vlt --exclude .svn -C $(srcdir)/skins2 default/
into
(cd $(srcdir)/skins2; find default -print0 | sort -z | tar cvv --exclude .svn --no-recursion --null -T -) | gzip -n > skins2/default.vlt
Works. But if someone knows of a simpler way...
You should exclude directories in the find command, otherwise their content will be packaged twice. Here is a more complete version that also handles timestamps of files being packaged (in case the files are generated): http://gitorious.org/opensuse/kernel-source/blobs/master/scripts/tar-up.sh, function stable_tar().
tar's "--no-recursion" should handle the directories problem. But that scripts is indeed better, thanks. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (3)
-
Cristian Morales Vega -
Cristian Rodríguez -
Michal Marek