On 02/09/2010 05:37, Larry Stotler wrote:

On Wed, Sep 1, 2010 at 2:50 AM, Basil Chupin wrote:

...

Back in the early-mid 1990s there was an archiver written which not only beat all other archivers hands down re compression but was also very fast. I was using it for my data. But at that time zip etc "ruled" the world and this archiver was kinda by-passed..... And do you think I can remember its name? :-(

Back in the day, we had ARC, ARJ, LHA, ZIP and a couple others I don't remember. ARJ was my preference. It usually beat out ZIP for file size. Haven't used it in 15 years probably.......

Yeah....memory, memory :-( . Of course, none of the above strikes an accord. Event-u-a-ll-y, the name will probably come to me in a dream :-) . BC -- ...more people are driven insane through religious hysteria than by drinking alcohol. W C Fields -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 02/09/2010 18:17, Carlos E. R. wrote:

On 2010-09-02 09:39, Basil Chupin wrote:

...

On 02/09/2010 05:37, Larry Stotler wrote:

...

...

Back in the day, we had ARC, ARJ, LHA, ZIP and a couple others I don't remember. ARJ was my preference. It usually beat out ZIP for file size. Haven't used it in 15 years probably.......

Yeah....memory, memory :-( . Of course, none of the above strikes an accord.

Event-u-a-ll-y, the name will probably come to me in a dream :-) .

Rar?

It is a good one, shareware. Features things like error recovery.

I think you may have it - the name registers strongly when I read it. If only I could find at least one file in my archives to confirm this :-) . BC -- ...more people are driven insane through religious hysteria than by drinking alcohol. W C Fields -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Mark Goldstein wrote:

On Wed, Sep 1, 2010 at 9:50 AM, Per Jessen wrote:

...

Per Jessen wrote:

...

I'm looking for a compression tool (lzma, gzip, zip et al) with the following characteristics:

1. high compression ratio 2. multiple files per archive 3. very fast decompression/extraction of single files

One of the best from the compression rate point of view is 7z (well, I know it depends on the type of files you are compressing). It definitely satisfies 2. No idea about 3, but worth trying.

Thanks Mark - the last one I tried was indeed 7z (p7zip in the Linux variation), but it fails miserably on 3. AFAICT, when extracting a single file, it scans the complete archive. Wrt the files, they're all emails, so lots of of text, html and copious amounts of base64 coded content. lzma does a brilliant job on cpio archives, compressed down to 10% of the original, but the extraction is much too slow. -- Per Jessen, Zürich (10.2°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Wilfred van Velzen wrote:

...

...

...

On 2010-09-01 at 09:22, Per Jessen wrote:

...

I'm looking for a compression tool (lzma, gzip, zip et al) with the following characteristics:

1. high compression ratio 2. multiple files per archive 3. very fast decompression/extraction of single files

A compressed filesystem? squashfs? Sorry, don't know much about the field. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 2010-09-01 11:04, Per Jessen wrote:

Dave Howorth wrote:

...

A compressed filesystem? squashfs?

Sorry, don't know much about the field.

Thanks Dave - it's a valid suggestion. I've already looked at squashfs and I may have to take another look, but I think the problem was in the mount time as well as in single-file extraction/retrieval time.

A compressed filesystem is indeed interesting for email. The only only I tested is zisofs, used to create compressed CD/DVDs, directly mountable in linux - ie, access to a single file is direct. However, it does not seem very fast. And the creation is cumbersome. squashfs is also RO. The wikipedia says it uses gzip compression (lzma projected). http://en.wikipedia.org/wiki/Squashfs ]> SquashFS is used by the Live CD versions of Arch Linux, Debian, Fedora, Gentoo Linux, Ubuntu and on embedded distributions such as the OpenWRT and DD-WRT router firmware. It is often combined with a union mount filesystem, such as UnionFS or aufs, to provide a read-write environment for live Linux distributions. This takes advantage of both the SquashFS's high speed compression abilities with the ability to alter the distribution while running it from a live CD. Distributions such as Slax, Debian Live, Mandriva One and Puppy Linux use this combination. ]> ]> The on-disk format of SquashFS has stabilized enough that it has been merged into the 2.6.29 version of the Linux kernel.[3] In that process, the backward-compatibility code for older formats was removed.[4] And: ]> See also ]> ]> * Cloop is a compressed loopback device module for the Linux kernel ]> * Cramfs is another read-only compressed file system ]> * e2compr provides compression for ext2 http://en.wikipedia.org/wiki/Cloop ]> The compressed loopback device or cloop is a module for the Linux kernel. It adds support for transparently decompressed, read-only block devices. It is not a compressed file system in itself. http://en.wikipedia.org/wiki/Cramfs

The compressed ROM file system (or cramfs) is a free (GPL'ed) read-only Linux file system designed for simplicity and space-efficiency. It is mainly used in embedded systems and small-footprint systems.

Unlike a compressed image of a conventional file system, a cramfs image can be used as it is i.e. without the need to decompress the image first. For this reason, some Linux distributions also use cramfs for initrd images (Debian 3.1 in particular) and installation images (SUSE Linux in particular), where there are constraints on memory and image size.

http://en.wikipedia.org/wiki/E2compr#Compression_extension ]> e2compr is a modification to the ext2 file system driver in the Linux kernel to support online compression and decompression of files on file system level without any support by user applications. ]> ]> e2compr is a small patch against the ext2 file system that allows on-the-fly compression and decompression. It compresses only regular files; the administrative data (superblock, inodes, directory files etc.) are not compressed (mainly for safety reasons). Access to compressed blocks is provided for read and write operations. The compression algorithm and cluster size is specified on a per-file basis. Directories can also be marked for compression, in which case every newly created file in the directory will be automatically compressed with the same cluster size and the same algorithm that was specified for the directory. ]> ]> e2compr is not a new file system. It is only a patch to the ext2 file system made to support the EXT2_COMPR_FL flag. It does not require you to make a new partition, and will continue to read or write existing ext2 file systems. One can consider it as simply a way for the read and write routines to access files that could have been created by a simple utility similar to gzip or compress. Compressed and uncompressed files coexist nicely on ext2 partitions. ]> ]> The latest e2compr-branch is available for current releases of 2.6 and 2.4 Linux kernels, but development is stalled. There are also older branches for older 2.0 and 2.2 kernels, which are more stable. Pity that e2compr development has stalled, I love the R/W idea (to access a mail folder many MUAs want to write to it - meaning a DVD copy does not work, has to be copied to HD first. -- Cheers / Saludos, Carlos E. R. (from 11.2 x86_64 "Emerald" GM (Elessar))

Per Jessen wrote:

Wilfred van Velzen wrote:

...

...

...

...

On 2010-09-01 at 09:22, Per Jessen wrote:

...

I'm looking for a compression tool (lzma, gzip, zip et al) with the following characteristics:

1. high compression ratio 2. multiple files per archive 3. very fast decompression/extraction of single files

One of the best from the compression rate point of view is 7z (well, I know it depends on the type of files you are compressing). It definitely satisfies 2. No idea about 3, but worth trying.

Thanks Mark - the last one I tried was indeed 7z (p7zip in the Linux variation), but it fails miserably on 3. AFAICT, when extracting a single file, it scans the complete archive.

Try compressing with '-ms=off' option for p7zip and test decompression speed again...

Interesting, that was a significant improvement. Almost as good as zip and certainly good enough to make 7za a good candidate. Thanks.

I'm running a lengthy test right now, but my immediate impression is that 7zip is only marginally slower than zip. Zip typically needs slightly less than 0.02seconds for an extraction, where as 7zip typically needs a little more than that. If 7zip does better compression too (which it should), it's a winner. However, 7zip for Linux looks a bit, well, unfinished. It doesn't read filenames from stdin, it doesn't have a --quiet option, and it's still called 'beta'. Makes me a little wary. -- Per Jessen, Zürich (17.9°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 9/1/2010 11:11 AM, Wilfred van Velzen wrote:

...

...

...

On 2010-09-01 at 16:10, Per Jessen wrote: Try compressing with '-ms=off' option for p7zip and test decompression speed again...

Interesting, that was a significant improvement. Almost as good as zip and certainly good enough to make 7za a good candidate. Thanks.

I'm running a lengthy test right now, but my immediate impression is that 7zip is only marginally slower than zip. Zip typically needs slightly less than 0.02seconds for an extraction, where as 7zip typically needs a little more than that. If 7zip does better compression too (which it should), it's a winner. However, 7zip for Linux looks a bit, well, unfinished. It doesn't read filenames from stdin, it doesn't have a --quiet option,

I use '-bd -y' for "quiet" opperation in scripts.

...

and it's still called 'beta'. Makes me a little wary.

It's been called that for several years now (for as long as it exists really). During that time I have been using it on several servers for all kinds of archiving jobs, and never noticed any real problems. Like where an archive was damaged and files inside it couldn't be extracted anymore... That never happend. You should probably regard it as a CYA disclaimer from the author. ;)

xar * random access thanks to built in xml toc * lzma (should be xz aka lzma2 by now but I haven't looked lately) -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 9/1/2010 2:38 PM, Per Jessen wrote:

Brian K. White wrote:

...

xar

* random access thanks to built in xml toc * lzma (should be xz aka lzma2 by now but I haven't looked lately)

First thought: XML sounds a little wasteful in this context. After having built it, it doesn't appear to support lzma. I'll still be trying it out though.

The xar in the Archiving repo doesn't either. Probably just needs liblzma present at build time for ./configure to enable it. I'll take a whack at it in my own OBS repo and provide a link here if the result works. The web site clearly states lzma support, and the whole point of the util is to be extensible anyways. -- bkw -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Per Jessen wrote:

Brian K. White wrote:

...

On 9/1/2010 2:38 PM, Per Jessen wrote:

...

Brian K. White wrote:

...

xar

* random access thanks to built in xml toc * lzma (should be xz aka lzma2 by now but I haven't looked lately)

First thought: XML sounds a little wasteful in this context. After having built it, it doesn't appear to support lzma. I'll still be trying it out though.

The xar in the Archiving repo doesn't either. Probably just needs liblzma present at build time for ./configure to enable it. I'll take a whack at it in my own OBS repo and provide a link here if the result works. The web site clearly states lzma support, and the whole point of the util is to be extensible anyways.

Yeah, I did notice that. I then installed xz-development and rebuilt, that should have done it, but ...

Update - AFAICT, xar 1.5.2 does not support lzma, only gzip and bzip2 - the auto-config doesn't even look for the lzma include files. -- Per Jessen, Zürich (17.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

...

...

On 2010-09-02 at 09:00, Per Jessen wrote: I use '-bd -y' for "quiet" opperation in scripts.

I notice you put quiet in quotes - coz those two options don't exactly make 7za very quiet ...

It's good enough for script usage. Just use the output if there's something to report, otherwise dump it. Something like: OUTPUT=$(7za a -bd -y archive.7z files 2>&1) if [ $? -ne 0 ]; then echo $OUTPUT >>logfile fi -- Met vriendelijke groet / Best regards, Wilfred van Velzen -- SERCOM Regeltechniek B.V. Heereweg 9 2161 AB Lisse Nederland +31 (0)252 416530 (voice) +31 (0)252 419481 (fax) http://www.sercom.nl/ Op al onze offertes, op alle opdrachten aan ons en op alle met ons gesloten overeenkomsten zijn toepasselijk de METAALUNIEVOORWAARDEN, gedeponeerd ter Griffie van de Rechtbank te Rotterdam, zoals deze luiden volgens de laatstelijk aldaar neergelegde tekst. De leveringsvoorwaarden worden u op verzoek toegezonden. --- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Philipp Thomas wrote:

On Wed, 01 Sep 2010 08:34:02 +0200, Per Jessen wrote:

...

I'm looking for a compression tool (lzma, gzip, zip et al) with the following characteristics:

Have you look at 7zip? It uses lzma so the compression rate should be superior to zip.

Yep, I did have a look at at it, but at first it was too slow in unpacking. However, Wilfred van Velzen suggested using -ms=off which helped a lot. -- Per Jessen, Zürich (17.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org