[opensuse] Re: You probably already knew, but lzma is way superior for archiving
jdd a écrit :
Ken Schneider - openSUSE a écrit :
It's l(el)zma. :-)
very old stuff :-)
http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm jdd -- http://www.dodin.net http://valerie.dodin.org http://www.youtube.com/watch?v=t-eic8MSSfM http://www.facebook.com/profile.php?id=1412160445 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
jdd wrote:
jdd a écrit :
Ken Schneider - openSUSE a écrit :
It's l(el)zma. :-)
very old stuff :-)
http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm
jdd
Yeah, it's really quite old, but somehow lzma has been living a life in the unknown for quite some time. Well, unknown to me, anyway. /Per -- Per Jessen, Zürich (18.3°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
In
http://en.wikipedia.org/wiki/Lempel-Ziv-Markov_chain_algorithm Yeah, it's really quite old, but somehow lzma has been living a life in
jdd wrote: the unknown for quite some time. Well, unknown to me, anyway.
IIRC, it didn't make its way into BSD/Linux for a while because there were some outstanding patents using the underlying Lempel-Ziv algorithm for compression. -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2009-04-07 at 19:43 +0200, Per Jessen wrote:
Yeah, it's really quite old, but somehow lzma has been living a life in the unknown for quite some time. Well, unknown to me, anyway.
openSUSE switched to lzma in the rpm packages for 11.0, I think it was, so we have been using it. It caused a problem for those attempting to upgrade from 10.3 to 11.0 because the 10.3 rpm program could not uncompress those. I have also considered lzma, but, last time I tried, "less" could not open and display those files. Also, zgrep does not support it. Nor midnight commander - and those are tools I use to open and look into old logs. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAknbqdEACgkQtTMYHG2NR9VDOgCgiko/U6oNr8pniH+j1+LTl97e dqkAnRTERoRVT5pNPZQi+435YXJCLuqf =MoRp -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tuesday, 2009-04-07 at 19:43 +0200, Per Jessen wrote:
Yeah, it's really quite old, but somehow lzma has been living a life in the unknown for quite some time. Well, unknown to me, anyway.
openSUSE switched to lzma in the rpm packages for 11.0, I think it was, so we have been using it. It caused a problem for those attempting to upgrade from 10.3 to 11.0 because the 10.3 rpm program could not uncompress those.
Yep, that's when I first heard of lzma.
I have also considered lzma, but, last time I tried, "less" could not open and display those files. Also, zgrep does not support it. Nor midnight commander - and those are tools I use to open and look into old logs.
I don't know about less, but I've got an lzless and an lzgrep. To me the 50% space saving and the faster decompression is a big win, so I can easily live with lzless and lzgrep. /Per -- Per Jessen, Zürich (10.9°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2009-04-08 at 09:20 +0200, Per Jessen wrote:
I have also considered lzma, but, last time I tried, "less" could not open and display those files. Also, zgrep does not support it. Nor midnight commander - and those are tools I use to open and look into old logs.
I don't know about less, but I've got an lzless and an lzgrep. To me the 50% space saving and the faster decompression is a big win, so I can easily live with lzless and lzgrep.
New tools... I don't like that. I prefer the "standard" having support for all methods, and me not caring. For instance, I can launch a zgrep and not care that some files are bz2 and some gz. In fact, I switched back to gz because bz2 was very slow to grep (ie, decompress). The real problem with lzma is that it is so slow compressing, but if you have many logs to keep, it may be worth the effort in your case. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkncfmMACgkQtTMYHG2NR9WfeQCeOflKAdtfOI8/96T+mxWYfUpI GTkAnRr8zV5JGFMwmJVHaUGJrQVhDNRn =CI0H -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
I don't know about less, but I've got an lzless and an lzgrep. To me the 50% space saving and the faster decompression is a big win, so I can easily live with lzless and lzgrep.
New tools... I don't like that. I prefer the "standard" having support for all methods, and me not caring.
Alright Carlos, here you go - a patch for /usr/bin/zgrep to support lzma: http://jessen.ch/files/zgrep-plus-lzma.patch
The real problem with lzma is that it is so slow compressing, but if you have many logs to keep, it may be worth the effort in your case.
For me the compression time is not a problem - it happens at night, after logrotate, no-one is waiting for it to finish. Like I said, lzma is perfect when it comes archiving logs - they're large, mostly write-only, but even for reading lzma is fast. /Per -- Per Jessen, Zürich (15.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2009-04-08 at 14:23 +0200, Per Jessen wrote:
Carlos E. R. wrote:
I don't know about less, but I've got an lzless and an lzgrep. To me the 50% space saving and the faster decompression is a big win, so I can easily live with lzless and lzgrep.
New tools... I don't like that. I prefer the "standard" having support for all methods, and me not caring.
Alright Carlos, here you go - a patch for /usr/bin/zgrep to support lzma:
(saved) Wow, thanks, but there was no need. I mean officially suported, working by default on all machines... your patch is usefull indeed, but unfortunately not enough to convince me of switching. Not yet, anyway :-) [...] Hold on, zgrep is a script! I never noticed that. :-O Then we could easily open a bugzilla request for the change... I'll try later.
The real problem with lzma is that it is so slow compressing, but if you have many logs to keep, it may be worth the effort in your case.
For me the compression time is not a problem - it happens at night, after logrotate, no-one is waiting for it to finish. Like I said, lzma is perfect when it comes archiving logs - they're large, mostly write-only, but even for reading lzma is fast.
I know, it makes a lot of sense. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkncoGEACgkQtTMYHG2NR9WWFACeLupOeilYBSt0EKw4/0K390gj N2MAn20hUVgdPw3JHM0f8fM+2hzQTsHJ =qqSF -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2009-04-08 at 15:02 +0200, I wrote:
On Wednesday, 2009-04-08 at 14:23 +0200, Per Jessen wrote:
Alright Carlos, here you go - a patch for /usr/bin/zgrep to support lzma:
[...]
Hold on, zgrep is a script! I never noticed that. :-O
Then we could easily open a bugzilla request for the change... I'll try later.
My /usr/bin/zgrep must be different from yours. Your patch is: @@ -109,6 +109,9 @@ if test "${i##*.}" = "bz2" ; then uncompress="bzip2" fi + if test "${i##*.}" = "lzma" ; then + uncompress="lzma" + fi $uncompress -cdfq "$i" | if test $files_with_matches -eq 1; then $grep $opt "$pat" > /dev/null && printf "%s\n" "$i" but I can't find that section, both in 11.0 or 11.1. The code I have seems to be: for i do case $i in *.bz2) uncompress=bzip2 ;; *) uncompress=gzip ;; esac I assume that entering: case $i in *.bz2) uncompress=bzip2 ;; case $i in *.lzma) uncompress=lzma ;; *) uncompress=gzip ;; esac would work. Still, we would need to patch more utilities: * /usr/bin/zcat * /usr/bin/zcmp * /usr/bin/zdiff * /usr/bin/zegrep * /usr/bin/zfgrep * /usr/bin/zforce * /usr/bin/zgrep * /usr/bin/zless * /usr/bin/zmore * /usr/bin/znew (not knowing alll of them, just looked into the rpm list) - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkncpzcACgkQtTMYHG2NR9VEwQCbBiGjccfOc8KVAkhyIIWBbaUR YuAAniirKs/5pnsibPewntWf+lQwNd4R =0dge -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, 8 Apr 2009, Carlos E. R. wrote:-
I assume that entering:
case $i in *.bz2) uncompress=bzip2 ;; case $i in ^^^^^^^^^^ Not if you include this line. Remove it and it'll work.
*.lzma) uncompress=lzma ;;
<Snip>
Still, we would need to patch more utilities:
* /usr/bin/zcat
Only handles gzip files. bzcat and lzcat are the bzip2 and lzma equivalents.
* /usr/bin/zcmp
A wrapper for zdiff. bzcat and lzcat are the bzip2 and lzma equivalents.
* /usr/bin/zdiff
Only handles compress or gzip files. bzdiff and lzdiff are the bzip2 and lzma equivalents.
* /usr/bin/zegrep
A wrapper for zgrep. bzegrep and lzegrep are the bzip2 and lzma equivalents.
* /usr/bin/zfgrep
A wrapper for zgrep. bzfgrep and lzfgrep are the bzip2 and lzma equivalents.
* /usr/bin/zforce
Only handles compress or gzip files. No bzip2 or lzma equivalents.
* /usr/bin/zgrep
Only handles gzip or bzip2 files. The above lines will patch it to add lzma support. Also, bzgrep and lzgrep are the bzip2 and lzma equivalents.
* /usr/bin/zless
Only handles gzip files. bzless and lzless are the bzip2 and lzma equivalents.
* /usr/bin/zmore
Only handles gzip files. bzmore and lzmore are the bzip2 and lzma equivalents.
* /usr/bin/znew
Converts "compress" compressed files to gzip. While not quite the equivalent, bznew re-compresses a gzip using bzip2 compression, and optionally leaves behind the gzip2 if the resultant file is larger. Regards, David Bolt -- Team Acorn: http://www.distributed.net/ OGR-NG @ ~100Mnodes RC5-72 @ ~1Mkeys/s openSUSE 10.3 32b | openSUSE 11.0 32b | | openSUSE 10.3 64b | openSUSE 11.0 64b | openSUSE 11.1 64b | openSUSE 10.3 PPC | RISC OS 3.6 | RISC OS 3.11 | TOS 4.02 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. wrote:
My /usr/bin/zgrep must be different from yours. Your patch is:
Sorry, I used on old test-system, probably 10.3. /Per -- Per Jessen, Zürich (18.2°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wed, Apr 08, 2009 at 03:02:20PM +0200, Carlos E. R. wrote: [ 8< ]
Then we could easily open a bugzilla request for the change... I'll try later.
See https://bugzilla.novell.com/show_bug.cgi?id=493410 Lars -- Lars Müller [ˈlaː(r)z ˈmʏlɐ] Samba Team SUSE Linux, Maxfeldstraße 5, 90409 Nürnberg, Germany
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2009-04-08 at 23:34 +0200, Lars Müller wrote:
On Wed, Apr 08, 2009 at 03:02:20PM +0200, Carlos E. R. wrote: [ 8< ]
Then we could easily open a bugzilla request for the change... I'll try later.
Good! :-) - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkndI7cACgkQtTMYHG2NR9U0agCfSIylpTDUCTR3Htfl9iuvYP1a VbEAmgPjWTVVHahetaXCL7eZ66tUXFkz =/kXB -----END PGP SIGNATURE-----
participants (6)
-
Boyd Stephen Smith Jr. -
Carlos E. R. -
David Bolt -
jdd -
Lars Müller -
Per Jessen