[opensuse] compression with recovery data and automatic archive split
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I have a large image file (159GiB) I want to compress and split before burning to several DVDs. I'm testing the shareware version of rar, which has automatic split on a given size, plus addition of error recovery data, which is nice. More or less, I use: rar a -rr -rv -v4589808k sda2 sda2.img which produces (slowly, about 45 minutes per 4.4GiB file) 8 files named like sda2.part01.rar to sda2.part08.rar, plus one sda2.part1.rev. It is slow, disk is busy at only about 2MB/s, or about 20..30MB/s on the empty regions of the image. It only uses one core of the cpu. I recogn I have not used rar since about 1997 (when I started using linux), but I don't remember it being slow. Of course, the file I'm compressing is very large... I wonder if there are opensource compressors with similar features? (I also intend to use dvdisaster recovery data on the iso images later) - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkv30igACgkQtTMYHG2NR9XIJACfQmewbqxnGoiU87NqucH8QqHp uVkAnRRzj3kD1jxUf0ZAUFMcg7gKQqME =igBk -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Carlos E. R. said the following on 05/22/2010 08:46 AM:
Of course, the file I'm compressing is very large...
Somehow I don't think compression algorithms are O(1). This is why I have LVM and lots of "small" file systems; "small" meaning they can go on a DVD without compression. I realise that isn't always practical, but I try, and sometimes solving 90% of the problem easily gives me the breathing space to deal with the exasperating remaining 10%. Regardless of the above, expect to loose hair over the years. -- "These are not books, lumps of lifeless paper, but minds alive on the shelves." -- Gilbert Highet -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Hi,
On Sat, May 22, 2010 at 3:46 PM, Carlos E. R.
rar a -rr -rv -v4589808k sda2 sda2.img
which produces (slowly, about 45 minutes per 4.4GiB file) 8 files named like sda2.part01.rar to sda2.part08.rar, plus one sda2.part1.rev.
It is slow, disk is busy at only about 2MB/s, or about 20..30MB/s on the empty regions of the image. It only uses one core of the cpu. I recogn I have not used rar since about 1997 (when I started using linux), but I don't remember it being slow. Of course, the file I'm compressing is very large...
I wonder if there are opensource compressors with similar features?
7-zip, probably? It does have -v (volume) option. I do not know whether it creates recovery data. I never used it for such a huge volumes, but on usual stuff it is mostly better than rar. (There is p7zip version for Linux and some GUI front-ends, like Q7Z). I would expect the compressor to be slow on huge data set. It has to do some preliminary pass or at least some level of "looking forward", has it not? -- Mark Goldstein -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2010-05-22 at 16:13 +0300, Mark Goldstein wrote:
On Sat, May 22, 2010 at 3:46 PM, Carlos E. R. <> wrote:
I wonder if there are opensource compressors with similar features?
7-zip, probably? It does have -v (volume) option. I do not know whether it creates recovery data. I never used it for such a huge volumes, but on usual stuff it is mostly better than rar. (There is p7zip version for Linux and some GUI front-ends, like Q7Z).
I'll have a look.
I would expect the compressor to be slow on huge data set. It has to do some preliminary pass or at least some level of "looking forward", has it not?
I suppose :-) I'm now testing another alternative: zisofs. First I split the original image file: time nice split --bytes=220M --numeric-suffixes --suffix-length=4 ../sda.img sda. It produced 738 files in 113 minutes (same hard disk) (cpu time 10s user, 5' sys). The next step is to compress the data: time nice mkzftree -p 4 splits/ splits.zisofs I see the HD churning at 30..50 MB/s, with 61 files so far of variying size (summing 8.0 GiB now). It has some time to run, so I'll get my siesta now :-) Next step will be to produce the iso images, which are compressed and readable directly by the system. Humm... what will "dvdisaster" say about those images, I wonder? It might not work :-? - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkv4ATsACgkQtTMYHG2NR9XWFQCeN1q2TouematlMN8yR+Dtbn6a V10AnRJsHd/kKl0mhTJfyhjd+6ZCxe32 =+2CF -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
* Carlos E. R.
I have a large image file (159GiB) I want to compress and split before burning to several DVDs. I'm testing the shareware version of rar, which has automatic split on a given size, plus addition of error recovery data, which is nice. More or less, I use:
rar a -rr -rv -v4589808k sda2 sda2.img
which produces (slowly, about 45 minutes per 4.4GiB file) 8 files named like sda2.part01.rar to sda2.part08.rar, plus one sda2.part1.rev.
It is slow, disk is busy at only about 2MB/s, or about 20..30MB/s on the empty regions of the image. It only uses one core of the cpu. I recogn I have not used rar since about 1997 (when I started using linux), but I don't remember it being slow. Of course, the file I'm compressing is very large...
I wonder if there are opensource compressors with similar features?
have a look at dar and kdar or dargui dar is excellent for archiving and splitting into dvd/cd sized chunks. I use it for all my local archival purposes. And it is available for other operating systems. -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2010-05-22 at 21:28 -0400, Patrick Shanahan wrote:
I wonder if there are opensource compressors with similar features?
have a look at dar and kdar or dargui
dar is excellent for archiving and splitting into dvd/cd sized chunks. I use it for all my local archival purposes. And it is available for other operating systems.
Unfortunately, it does not use reliable compression, ie, no forward recovery data. You have to use it with parchive to get that. On the other hand, dar is not a compression program: it uses gzip or bzip, when told. I can not get better results than by using those programs directly. What I'm looking for is for a compression program, that has forward recovery and split archive capability. Notice that I'm compressing a single huge file: tools like "gzip" or "zip" (linux) can not split a single file across multiple volumes. However, the original zip for dos/win can. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkv4irwACgkQtTMYHG2NR9W2ogCeIp6LZ5A0cpn9nf4cyh0fhXcp fkMAn36sPZNvRB2TbjQJqXad8su5HUdp =djGz -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (5)
-
Anton Aylward -
Carlos E. R. -
Carlos E. R. -
Mark Goldstein -
Patrick Shanahan