[opensuse] You probably already knew, but lzma is way superior for archiving
A couple weeks ago I stumbled over a mention that "lzma was really good with large files". Well, I archive several gigabytes of logfiles every day, compressed with gzip. I deliberately chose gzip over bzip2 as it was faster decompressing, which is important when you trawling through the archives looking for something or other. I did a couple of small tests with lzma, and then decided to run a test on about 200G of data collected during Jan-Feb-Mar. I'm still not done with the data from March, but sofar lzma compresses twice as much as gzip, i.e. if gzip compresses to 10% of original size, lzma does 5%. Granted, compression IS awfully slow, but decompression is faster than gzip, so for archival and analysis purposes lzma compression is near perfect. But I'm sure you all knew that. /Per -- Per Jessen, Zürich (17.6°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Randall R Schulz pecked at the keyboard and wrote:
On Tuesday April 7 2009, Per Jessen wrote:
... But I'm sure you all knew that.
I don't know what Izma is, ma.
/Per
RRS
It's l(el)zma. :-) -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
A couple weeks ago I stumbled over a mention that "lzma was really good with large files". Well, I archive several gigabytes of logfiles every day, compressed with gzip. I deliberately chose gzip over bzip2 as it was faster decompressing, which is important when you trawling through the archives looking for something or other.
I did a couple of small tests with lzma, and then decided to run a test on about 200G of data collected during Jan-Feb-Mar. I'm still not done with the data from March, but sofar lzma compresses twice as much as gzip, i.e. if gzip compresses to 10% of original size, lzma does 5%. Granted, compression IS awfully slow, but decompression is faster than gzip, so for archival and analysis purposes lzma compression is near perfect. But I'm sure you all knew that.
If you are into graphs and comparisons, take a look at this article from 2005, it includes a lot of different algorithms: http://www.linuxjournal.com/article/8051 It also has a quite interesting and useful cpu-to-bandwidth trade-off graph, comparing the time needed to compress vs. transfer, at different cpu and link speeds: http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/080...
/Per
/Sylvester Lykkehus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (4)
-
Ken Schneider - openSUSE -
Per Jessen -
Randall R Schulz -
Sylvester Lykkehus