On 8.11.2013 14:19, Jean Delvare wrote:

Thus I'd like to add support for xz-compressed files to sequence-patch for a starter. I'm attaching the patch. It's only tested locally (outside of the suse network) so not all code paths were tested. Some more testing and review would be welcome.

I tested it with our mirror that only has linux-*.bz2 and it still works. I'll change it now to also fetch *.xz.

--- a/scripts/sequence-patch.sh +++ b/scripts/sequence-patch.sh @@ -135,6 +135,7 @@ apply_patches() { case $PATCH in *.gz) exec < <(gzip -cd $PATCH) ;; *.bz2) exec < <(bzip2 -cd $PATCH) ;; + *.xz) exec < <(xz -cd $PATCH) ;; *) exec < $PATCH ;; esac patch -d $PATCH_DIR --backup --prefix=$backup_dir/ -p1 -E $fuzz \ @@ -149,6 +150,7 @@ apply_patches() { case $PATCH in *.gz) exec < <(gzip -cd $PATCH) ;; *.bz2) exec < <(bzip2 -cd $PATCH) ;; + *.xz) exec < <(xz -cd $PATCH) ;; *) exec < $PATCH ;; esac

I'd just drop support for compressed patches. We never had them in the tree. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

On Wed, Nov 13, 2013 at 09:13:55AM +0100, Jean Delvare wrote:

Thanks, this will start the transition period nicely. Ultimately it would be great if we could stop fetching *.bz2 and only use *.xz. But I have to admit I'm not too sure how to achieve that. Switching only new products to use *.xz would require adding some magic to several of our scripts. Switching them all would be even more work with a risk of regression. None of these options is really appealing, but maybe I'm missing an easier path.

This worked for me, can you test it? If it works, then I'll commit the script changes and maybe switch master and SLE12 right away to use linux-*.xz. Michal diff --git a/rpm/kernel-source.spec.in b/rpm/kernel-source.spec.in index c94be7e..8d7956d 100644 --- a/rpm/kernel-source.spec.in +++ b/rpm/kernel-source.spec.in @@ -45,7 +45,7 @@ Requires(post): coreutils sed Provides: multiversion(kernel) Provides: linux Provides: %name = %version-%source_rel -Source0: @TARBALL_URL@linux-%srcversion.tar.bz2 +Source0: @TARBALL_URL@linux-%srcversion.tar.xz Source2: source-post.sh Source3: kernel-source.rpmlintrc Source8: devel-pre.sh diff --git a/rpm/mkspec b/rpm/mkspec index 48805ac..b5df39c 100755 --- a/rpm/mkspec +++ b/rpm/mkspec @@ -47,9 +47,9 @@ $rpmversion =~ s/-/./g; $rpmrelease =~ s/-/./g; my $sources = join("", $templates{source} =~ /\nSource\d+:[^\n]*/mg); -# Find all SourceN: foo.tar.bz2 lines and generate the NoSource: +# Find all SourceN: foo.tar.(bz2|xz) lines and generate the NoSource: # lines and the %setup line -my @tarballs = ($sources =~ /\nSource(\d+):[^\n]*\.tar\.bz2/mg); +my @tarballs = ($sources =~ /\nSource(\d+):[^\n]*\.tar\.(?:bz2|xz)/mg); my $nosource = join("\n", map { "NoSource: $_" } @tarballs); # Source0 (the linux tarball) is unpacked manually @tarballs = grep { $_ > 0 } @tarballs; diff --git a/scripts/tar-up-old.sh b/scripts/tar-up-old.sh index 080e9b8..d1d7f5c 100755 --- a/scripts/tar-up-old.sh +++ b/scripts/tar-up-old.sh @@ -154,7 +154,7 @@ done mkdir -p "$build_dir" if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" + get_tarball "$SRCVERSION" "tar.bz2" "$build_dir" fi # list of patches to include. diff --git a/scripts/tar-up.sh b/scripts/tar-up.sh index 543e71a..4238c88 100755 --- a/scripts/tar-up.sh +++ b/scripts/tar-up.sh @@ -109,19 +109,20 @@ check_for_merge_conflicts() { fi } +suffix=$(sed -rn 's/^Source0:.*\.(tar\.[a-z0-9]*)$/\1/p' rpm/kernel-source.spec.in) # Dot files are skipped by intention, in order not to break osc working # copies. The linux tarball is not deleted if it is already there for f in "$build_dir"/*; do case "$f" in - */"linux-$SRCVERSION.tar.bz2") + */"linux-$SRCVERSION.$suffix") continue esac rm -f "$f" done mkdir -p "$build_dir" -if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then - echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" +if test ! -e "$build_dir/linux-$SRCVERSION.$suffix"; then + echo "linux-$SRCVERSION.$suffix" + get_tarball "$SRCVERSION" "$suffix" "$build_dir" fi # list of patches to include. diff --git a/scripts/wd-functions.sh b/scripts/wd-functions.sh index e1f1484..78327b9 100644 --- a/scripts/wd-functions.sh +++ b/scripts/wd-functions.sh @@ -41,7 +41,7 @@ get_branch_name() _find_tarball() { - local version=$1 xz_ok=$2 dir subdir major + local version=$1 force_suffix=$2 dir subdir major suffixes suffix set -- ${version//[.-]/ } major=$1.$2 @@ -49,16 +49,19 @@ _find_tarball() 3.*) major=3.x esac + if test -n "$force_suffix"; then + suffixes=("$force_suffix") + else + suffixes=(tar.xz tar.bz2) + fi for dir in . $MIRROR {/mounts,/labs,}/mirror/kernel; do for subdir in "" "/v$major" "/testing" "/v$major/testing"; do - if test -n "$xz_ok" -a -r "$dir$subdir/linux-$version.tar.xz"; then - echo "$dir$subdir/linux-$version.tar.xz" - return - fi - if test -r "$dir$subdir/linux-$version.tar.bz2"; then - echo "$dir$subdir/linux-$version.tar.bz2" - return - fi + for suffix in "${suffixes[@]}"; do + if test -r "$dir$subdir/linux-$version.$suffix"; then + echo "$_" + return + fi + done done done } @@ -101,22 +104,33 @@ _get_tarball_from_git() get_tarball() { - local version=$1 dest=$2 tarball + local version=$1 suffix=$2 dest=$3 tarball compress - tarball=$(_find_tarball "$version") + tarball=$(_find_tarball "$version" "$suffix") if test -n "$tarball"; then - cp "$tarball" "$dest/linux-$version.tar.bz2.part" || exit - mv "$dest/linux-$version.tar.bz2.part" "$dest/linux-$version.tar.bz2" + cp "$tarball" "$dest/linux-$version.$suffix.part" || exit + mv "$dest/linux-$version.$suffix.part" "$dest/linux-$version.$suffix" return fi - echo "Warning: could not find linux-$version.tar.bz2, trying to create it from git" >&2 + echo "Warning: could not find linux-$version.$suffix, trying to create it from git" >&2 + case "$suffix" in + tar.bz2) + compress="bzip2 -9" + ;; + tar.xz) + compress="xz" + ;; + *) + echo "Unknown compression format: $suffix" >&2 + exit 1 + esac set -o pipefail - _get_tarball_from_git "$version" | bzip2 -9 \ - >"$dest/linux-$version.tar.bz2.part" + _get_tarball_from_git "$version" | $compress \ + >"$dest/linux-$version.$suffix.part" if test $? -ne 0; then exit 1 fi - mv "$dest/linux-$version.tar.bz2.part" "$dest/linux-$version.tar.bz2" + mv "$dest/linux-$version.$suffix.part" "$dest/linux-$version.$suffix" set +o pipefail } @@ -124,7 +138,7 @@ unpack_tarball() { local version=$1 dest=$2 tarball - tarball=$(_find_tarball "$version" zx_ok) + tarball=$(_find_tarball "$version") mkdir -p "$dest" if test -n "$tarball"; then echo "Extracting $tarball" -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org

Dne 27.11.2013 17:37, Jean Delvare napsal(a):

Hi Michal,

Le Wednesday 13 November 2013 à 15:49 +0100, Michal Marek a écrit : My first testing 10 days ago seemed wrong, however I tested it again today and it worked just fine. So I'm not sure what I did last time, but let's call it working and commit it. If the problem reappears (assuming it was not just my mistake - might as well be) we'll fix it then.

OK. Thanks for testing.

I think you want to split this patch in two parts: the infrastructure itself which should go to the scripts branch and be pulled to all other branches from there, and the actual switch to tar.xz format (in rpm/kernel-source.spec.in) which should be committed to recent branches only.

Definitely, that's the plan.

Overall the code looks great, nice job. Just two comment:

...

--- a/scripts/tar-up-old.sh +++ b/scripts/tar-up-old.sh @@ -154,7 +154,7 @@ done mkdir -p "$build_dir" if test ! -e "$build_dir/linux-$SRCVERSION.tar.bz2"; then echo "linux-$SRCVERSION.tar.bz2" - get_tarball "$SRCVERSION" "$build_dir" + get_tarball "$SRCVERSION" "tar.bz2" "$build_dir" fi

# list of patches to include.

Can't we get rid of tar-up-old.sh now? Anyone still using it?

It's needed for SLES10 and SLES9. BTW, you are supposed to always run scripts/tar-up.sh, it hands over to tar-up-old.sh when needed.

...

+ if test -n "$force_suffix"; then + suffixes=("$force_suffix") + else + suffixes=(tar.xz tar.bz2) + fi

If "suffixes" is supposed to be an array then it should have been declared as such with "local -a". But given that you are never accessing it by index, and items will never contain a space, I think you could make it a simple string. That will be easier and probably slightly faster, as the code can then be simplified to:

local ... suffixes=$2 ... suffix

... test -z "$suffixes" && suffixes="tar.xz tar.bz2" ... for suffix in $suffixes; do

OK, I'll change that. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org