[opensuse-kernel] sequence-patch: Add support for xz-compressed archives and patches
Hi all, While kernel.org still offers bz2 archives for download, the discussion about dropping them and only leaving gz and xz is coming back every now and then. So I believe we should anticipate a future removal of bz2 archives and get our scripts ready to deal with xz-compressed archives now. If nothing else, that would encourage everyone to use xz-compressed archives whenever possible, which is a great saving of network bandwidth and storage space, plus decompression is twice as fast as with bz2. Thus I'd like to add support for xz-compressed files to sequence-patch for a starter. I'm attaching the patch. It's only tested locally (outside of the suse network) so not all code paths were tested. Some more testing and review would be welcome. We could also consider using xz to compress all tar files in tar-up.sh (and update the spec files accordingly.) This may make submission somewhat slower, as compressing with xz is significantly slower than with bzip2, but this would save some upload time, storage space in IBS/OBS and should also improve the build times by a small bit. -- Jean Delvare Suse L3 Support
On 8.11.2013 14:19, Jean Delvare wrote:
Thus I'd like to add support for xz-compressed files to sequence-patch for a starter. I'm attaching the patch. It's only tested locally (outside of the suse network) so not all code paths were tested. Some more testing and review would be welcome.
I tested it with our mirror that only has linux-*.bz2 and it still works. I'll change it now to also fetch *.xz.
--- a/scripts/sequence-patch.sh +++ b/scripts/sequence-patch.sh @@ -135,6 +135,7 @@ apply_patches() { case $PATCH in *.gz) exec < <(gzip -cd $PATCH) ;; *.bz2) exec < <(bzip2 -cd $PATCH) ;; + *.xz) exec < <(xz -cd $PATCH) ;; *) exec < $PATCH ;; esac patch -d $PATCH_DIR --backup --prefix=$backup_dir/ -p1 -E $fuzz \ @@ -149,6 +150,7 @@ apply_patches() { case $PATCH in *.gz) exec < <(gzip -cd $PATCH) ;; *.bz2) exec < <(bzip2 -cd $PATCH) ;; + *.xz) exec < <(xz -cd $PATCH) ;; *) exec < $PATCH ;; esac
I'd just drop support for compressed patches. We never had them in the tree. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hi Michal, Le Monday 11 November 2013 à 11:05 +0100, Michal Marek a écrit :
On 8.11.2013 14:19, Jean Delvare wrote:
Thus I'd like to add support for xz-compressed files to sequence-patch for a starter. I'm attaching the patch. It's only tested locally (outside of the suse network) so not all code paths were tested. Some more testing and review would be welcome.
I tested it with our mirror that only has linux-*.bz2 and it still works. I'll change it now to also fetch *.xz.
Thanks, this will start the transition period nicely. Ultimately it would be great if we could stop fetching *.bz2 and only use *.xz. But I have to admit I'm not too sure how to achieve that. Switching only new products to use *.xz would require adding some magic to several of our scripts. Switching them all would be even more work with a risk of regression. None of these options is really appealing, but maybe I'm missing an easier path.
--- a/scripts/sequence-patch.sh +++ b/scripts/sequence-patch.sh @@ -135,6 +135,7 @@ apply_patches() { case $PATCH in *.gz) exec < <(gzip -cd $PATCH) ;; *.bz2) exec < <(bzip2 -cd $PATCH) ;; + *.xz) exec < <(xz -cd $PATCH) ;; *) exec < $PATCH ;; esac patch -d $PATCH_DIR --backup --prefix=$backup_dir/ -p1 -E $fuzz \ @@ -149,6 +150,7 @@ apply_patches() { case $PATCH in *.gz) exec < <(gzip -cd $PATCH) ;; *.bz2) exec < <(bzip2 -cd $PATCH) ;; + *.xz) exec < <(xz -cd $PATCH) ;; *) exec < $PATCH ;; esac
I'd just drop support for compressed patches. We never had them in the tree.
Fine with me, I'll do that in a separate patch. Thanks for the review and testing, -- Jean Delvare Suse L3 Support -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On Wed, Nov 13, 2013 at 09:13:55AM +0100, Jean Delvare wrote:
Thanks, this will start the transition period nicely. Ultimately it would be great if we could stop fetching *.bz2 and only use *.xz. But I have to admit I'm not too sure how to achieve that. Switching only new products to use *.xz would require adding some magic to several of our scripts. Switching them all would be even more work with a risk of regression. None of these options is really appealing, but maybe I'm missing an easier path.
This worked for me, can you test it? If it works, then I'll commit the script changes and maybe switch master and SLE12 right away to use linux-*.xz. Michal diff --git a/rpm/kernel-source.spec.in b/rpm/kernel-source.spec.in index c94be7e..8d7956d 100644 --- a/rpm/kernel-source.spec.in +++ b/rpm/kernel-source.spec.in @@ -45,7 +45,7 @@ Requires(post): coreutils sed Provides: multiversion(kernel) Provides: linux Provides: %name = %version-%source_rel -Source0: @TARBALL_URL@linux-%srcversion.tar.bz2 +Source0: @TARBALL_URL@linux-%srcversion.tar.xz Source2: source-post.sh Source3: kernel-source.rpmlintrc Source8: devel-pre.sh diff --git a/rpm/mkspec b/rpm/mkspec index 48805ac..b5df39c 100755 --- a/rpm/mkspec +++ b/rpm/mkspec @@ -47,9 +47,9 @@ $rpmversion =~ s/-/./g; $rpmrelease =~ s/-/./g; my $sources = join("", $templates{source} =~ /\nSource\d+:[^\n]*/mg); -# Find all SourceN: foo.tar.bz2 lines and generate the NoSource: +# Find all SourceN: foo.tar.(bz2|xz) lines and generate the NoSource: # lines and the %setup line -my @tarballs = ($sources =~ /\nSource(\d+):[^\n]*\.tar\.bz2/mg); +my @tarballs = ($sources =~ /\nSource(\d+):[^\n]*\.tar\.(?:bz2|xz)/mg); my $nosource = join("\n", map { "NoSource: $_" } @tarballs); # Source0 (the linux tarball) is unpacked manually @tarballs = grep { $_ > 0 } @tarballs; diff --git a/scripts/tar-up-old.sh b/scripts/tar-up-old.sh index 080e9b8..d1d7f5c 100755 --- a/scripts/tar-up-old.sh +++ b/scripts/tar-up-old.sh @@ -154,7 +154,7 @@ done mkdir -p "$build_dir" if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" + get_tarball "$SRCVERSION" "tar.bz2" "$build_dir" fi # list of patches to include. diff --git a/scripts/tar-up.sh b/scripts/tar-up.sh index 543e71a..4238c88 100755 --- a/scripts/tar-up.sh +++ b/scripts/tar-up.sh @@ -109,19 +109,20 @@ check_for_merge_conflicts() { fi } +suffix=$(sed -rn 's/^Source0:.*\.(tar\.[a-z0-9]*)$/\1/p' rpm/kernel-source.spec.in) # Dot files are skipped by intention, in order not to break osc working # copies. The linux tarball is not deleted if it is already there for f in "$build_dir"/*; do case "$f" in - */"linux-$SRCVERSION.tar.bz2") + */"linux-$SRCVERSION.$suffix") continue esac rm -f "$f" done mkdir -p "$build_dir" -if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then - echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" +if test ! -e "$build_dir/linux-$SRCVERSION.$suffix"; then + echo "linux-$SRCVERSION.$suffix" + get_tarball "$SRCVERSION" "$suffix" "$build_dir" fi # list of patches to include. diff --git a/scripts/wd-functions.sh b/scripts/wd-functions.sh index e1f1484..78327b9 100644 --- a/scripts/wd-functions.sh +++ b/scripts/wd-functions.sh @@ -41,7 +41,7 @@ get_branch_name() _find_tarball() { - local version=$1 xz_ok=$2 dir subdir major + local version=$1 force_suffix=$2 dir subdir major suffixes suffix set -- ${version//[.-]/ } major=$1.$2 @@ -49,16 +49,19 @@ _find_tarball() 3.*) major=3.x esac + if test -n "$force_suffix"; then + suffixes=("$force_suffix") + else + suffixes=(tar.xz tar.bz2) + fi for dir in . $MIRROR {/mounts,/labs,}/mirror/kernel; do for subdir in "" "/v$major" "/testing" "/v$major/testing"; do - if test -n "$xz_ok" -a -r "$dir$subdir/linux-$version.tar.xz"; then - echo "$dir$subdir/linux-$version.tar.xz" - return - fi - if test -r "$dir$subdir/linux-$version.tar.bz2"; then - echo "$dir$subdir/linux-$version.tar.bz2" - return - fi + for suffix in "${suffixes[@]}"; do + if test -r "$dir$subdir/linux-$version.$suffix"; then + echo "$_" + return + fi + done done done } @@ -101,22 +104,33 @@ _get_tarball_from_git() get_tarball() { - local version=$1 dest=$2 tarball + local version=$1 suffix=$2 dest=$3 tarball compress - tarball=$(_find_tarball "$version") + tarball=$(_find_tarball "$version" "$suffix") if test -n "$tarball"; then - cp "$tarball" "$dest/linux-$version.tar.bz2.part" || exit - mv "$dest/linux-$version.tar.bz2.part" "$dest/linux-$version.tar.bz2" + cp "$tarball" "$dest/linux-$version.$suffix.part" || exit + mv "$dest/linux-$version.$suffix.part" "$dest/linux-$version.$suffix" return fi - echo "Warning: could not find linux-$version.tar.bz2, trying to create it from git" >&2 + echo "Warning: could not find linux-$version.$suffix, trying to create it from git" >&2 + case "$suffix" in + tar.bz2) + compress="bzip2 -9" + ;; + tar.xz) + compress="xz" + ;; + *) + echo "Unknown compression format: $suffix" >&2 + exit 1 + esac set -o pipefail - _get_tarball_from_git "$version" | bzip2 -9 \ - >"$dest/linux-$version.tar.bz2.part" + _get_tarball_from_git "$version" | $compress \ + >"$dest/linux-$version.$suffix.part" if test $? -ne 0; then exit 1 fi - mv "$dest/linux-$version.tar.bz2.part" "$dest/linux-$version.tar.bz2" + mv "$dest/linux-$version.$suffix.part" "$dest/linux-$version.$suffix" set +o pipefail } @@ -124,7 +138,7 @@ unpack_tarball() { local version=$1 dest=$2 tarball - tarball=$(_find_tarball "$version" zx_ok) + tarball=$(_find_tarball "$version") mkdir -p "$dest" if test -n "$tarball"; then echo "Extracting $tarball" -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hi Michal, Le Wednesday 13 November 2013 à 15:49 +0100, Michal Marek a écrit :
On Wed, Nov 13, 2013 at 09:13:55AM +0100, Jean Delvare wrote:
Thanks, this will start the transition period nicely. Ultimately it would be great if we could stop fetching *.bz2 and only use *.xz. But I have to admit I'm not too sure how to achieve that. Switching only new products to use *.xz would require adding some magic to several of our scripts. Switching them all would be even more work with a risk of regression. None of these options is really appealing, but maybe I'm missing an easier path.
This worked for me, can you test it? If it works, then I'll commit the script changes and maybe switch master and SLE12 right away to use linux-*.xz.
My first testing 10 days ago seemed wrong, however I tested it again today and it worked just fine. So I'm not sure what I did last time, but let's call it working and commit it. If the problem reappears (assuming it was not just my mistake - might as well be) we'll fix it then. I think you want to split this patch in two parts: the infrastructure itself which should go to the scripts branch and be pulled to all other branches from there, and the actual switch to tar.xz format (in rpm/kernel-source.spec.in) which should be committed to recent branches only. Overall the code looks great, nice job. Just two comment:
--- a/scripts/tar-up-old.sh +++ b/scripts/tar-up-old.sh @@ -154,7 +154,7 @@ done mkdir -p "$build_dir" if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" + get_tarball "$SRCVERSION" "tar.bz2" "$build_dir" fi
# list of patches to include.
Can't we get rid of tar-up-old.sh now? Anyone still using it?
--- a/scripts/wd-functions.sh +++ b/scripts/wd-functions.sh @@ -41,7 +41,7 @@ get_branch_name()
_find_tarball() { - local version=$1 xz_ok=$2 dir subdir major + local version=$1 force_suffix=$2 dir subdir major suffixes suffix
set -- ${version//[.-]/ } major=$1.$2 @@ -49,16 +49,19 @@ _find_tarball() 3.*) major=3.x esac + if test -n "$force_suffix"; then + suffixes=("$force_suffix") + else + suffixes=(tar.xz tar.bz2) + fi
If "suffixes" is supposed to be an array then it should have been declared as such with "local -a". But given that you are never accessing it by index, and items will never contain a space, I think you could make it a simple string. That will be easier and probably slightly faster, as the code can then be simplified to: local ... suffixes=$2 ... suffix ... test -z "$suffixes" && suffixes="tar.xz tar.bz2" ... for suffix in $suffixes; do
for dir in . $MIRROR {/mounts,/labs,}/mirror/kernel; do for subdir in "" "/v$major" "/testing" "/v$major/testing"; do - if test -n "$xz_ok" -a -r "$dir$subdir/linux-$version.tar.xz"; then - echo "$dir$subdir/linux-$version.tar.xz" - return - fi - if test -r "$dir$subdir/linux-$version.tar.bz2"; then - echo "$dir$subdir/linux-$version.tar.bz2" - return - fi + for suffix in "${suffixes[@]}"; do + if test -r "$dir$subdir/linux-$version.$suffix"; then + echo "$_" + return + fi + done done done }
-- Jean Delvare Suse L3 Support -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Dne 27.11.2013 17:37, Jean Delvare napsal(a):
Hi Michal,
Le Wednesday 13 November 2013 à 15:49 +0100, Michal Marek a écrit : My first testing 10 days ago seemed wrong, however I tested it again today and it worked just fine. So I'm not sure what I did last time, but let's call it working and commit it. If the problem reappears (assuming it was not just my mistake - might as well be) we'll fix it then.
OK. Thanks for testing.
I think you want to split this patch in two parts: the infrastructure itself which should go to the scripts branch and be pulled to all other branches from there, and the actual switch to tar.xz format (in rpm/kernel-source.spec.in) which should be committed to recent branches only.
Definitely, that's the plan.
Overall the code looks great, nice job. Just two comment:
--- a/scripts/tar-up-old.sh +++ b/scripts/tar-up-old.sh @@ -154,7 +154,7 @@ done mkdir -p "$build_dir" if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" + get_tarball "$SRCVERSION" "tar.bz2" "$build_dir" fi
# list of patches to include.
Can't we get rid of tar-up-old.sh now? Anyone still using it?
It's needed for SLES10 and SLES9. BTW, you are supposed to always run scripts/tar-up.sh, it hands over to tar-up-old.sh when needed.
+ if test -n "$force_suffix"; then + suffixes=("$force_suffix") + else + suffixes=(tar.xz tar.bz2) + fi
If "suffixes" is supposed to be an array then it should have been declared as such with "local -a". But given that you are never accessing it by index, and items will never contain a space, I think you could make it a simple string. That will be easier and probably slightly faster, as the code can then be simplified to:
local ... suffixes=$2 ... suffix
... test -z "$suffixes" && suffixes="tar.xz tar.bz2" ... for suffix in $suffixes; do
OK, I'll change that. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Le Wednesday 27 November 2013 à 18:00 +0100, Michal Marek a écrit :
Dne 27.11.2013 17:37, Jean Delvare napsal(a):
Overall the code looks great, nice job. Just two comment:
--- a/scripts/tar-up-old.sh +++ b/scripts/tar-up-old.sh @@ -154,7 +154,7 @@ done mkdir -p "$build_dir" if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" + get_tarball "$SRCVERSION" "tar.bz2" "$build_dir" fi
# list of patches to include.
Can't we get rid of tar-up-old.sh now? Anyone still using it?
It's needed for SLES10 and SLES9. BTW, you are supposed to always run scripts/tar-up.sh, it hands over to tar-up-old.sh when needed.
Oh I didn't know that. Alright then let's keep it. -- Jean Delvare Suse L3 Support -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Looks like we were right in time: https://www.kernel.org/happy-new-year-and-good-bye-bzip2.html Jean Le Friday 08 November 2013 à 14:19 +0100, Jean Delvare a écrit :
Hi all,
While kernel.org still offers bz2 archives for download, the discussion about dropping them and only leaving gz and xz is coming back every now and then. So I believe we should anticipate a future removal of bz2 archives and get our scripts ready to deal with xz-compressed archives now. If nothing else, that would encourage everyone to use xz-compressed archives whenever possible, which is a great saving of network bandwidth and storage space, plus decompression is twice as fast as with bz2.
Thus I'd like to add support for xz-compressed files to sequence-patch for a starter. I'm attaching the patch. It's only tested locally (outside of the suse network) so not all code paths were tested. Some more testing and review would be welcome.
We could also consider using xz to compress all tar files in tar-up.sh (and update the spec files accordingly.) This may make submission somewhat slower, as compressing with xz is significantly slower than with bzip2, but this would save some upload time, storage space in IBS/OBS and should also improve the build times by a small bit.
-- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 2014-01-06 15:40, Jean Delvare wrote:
Looks like we were right in time:
https://www.kernel.org/happy-new-year-and-good-bye-bzip2.html
Yes, that was good timing indeed. The master branch is using linux-*.tar.xz now. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (2)
-
Jean Delvare -
Michal Marek