On Friday 29 August 2008 09:36, James Knott wrote:

Randall R Schulz wrote:

...

Hi,

Does anyone know if there's a Linux counterpart to the Windows ability to treat Zip files as if they're just directories, ...

For what I'd like to be able to do, a KDE I/O slave will not suffice, it has to be system-wide and transparent so arbitrary scripts and applications can access it.

...

I don't know if it'll do what you want, but Konqueror lets you browse the contents of zip files.

I thought I covered that by saying that KDE I/O slave-based solutions won't work for me and that I need to be able to access contents from "arbitrary scripts and applications." Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Friday 29 August 2008 12:25, Amedee Van Gasse wrote:

Randall R Schulz schreef:

...

Hi,

Does anyone know if there's a Linux counterpart to the Windows ability to treat Zip files as if they're just directories, transparently deriving directory information and automatically extracting their contents on demand?

...

Sounds like you are asking for a Filesystem in USErspace aka FUSE.

... http://fuse.sourceforge.net/wiki/index.php/ArchiveFileSystems

I think you'll find what you need over there.

I tried archivemount. It works beautifully. I just had to install libarchive (available via Guru's Packages for 10.0) and, for some reason, manually issue the "insmod" call for the FUSE kernel module. I wrote a little script, "am" to facilitate easy mounting and unmounting of archive files. What's more, while the capsule summary on the page to which you referred me says that archivemount handles cpio, tar.gz, tar.bz2, I find that it also handles .zip and uncompressed .tar, too. I guess it's up to libarchive which formats are handled, so presumably all those mentioned on the man page installed with libarchive are also handled: · old-style tar archives, · most variants of the POSIX “ustar” format, · the POSIX “pax interchange” format, · GNU-format tar archives, · most common cpio archive formats, · ISO9660 CD images (with or without RockRidge extensions), · Zip archives. The library automatically detects archives compressed with gzip(1), bzip2(1), or compress(1) and decompresses them transparently. I confirmed that ISO images work with archivemount. To me (and with my helper script), this is a nice convenience by comparison to mounting ISOs via a loopback device. Thanks again for the pointer. This is very useful.

-- Amedee

Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Friday 29 August 2008 06:09:50 pm Randall R Schulz wrote:

On Friday 29 August 2008 16:50, John Andersen wrote:

...

On Fri, Aug 29, 2008 at 9:28 AM, Randall R Schulz

wrote:

...

...

Hi,

Does anyone know if there's a Linux counterpart to the Windows ability to treat Zip files as if they're just directories, ...

Others have provided answers on this thread, I just wanted to ask a question:

Are you sure Vista provides this as transparently as you are requesting for Linux?

I've never used Vista, nor did I mention it. I only said "Windows."

Nonetheless, I don't recall the extent of this capability, but would we be surprised if Linux did it better?

Nah, they're pretty much on equal par. I have vista on one of my desktops at work (Vista, openSUSE 10.3, Win2K, WinXP) and as a guest OS in my laptop. On both KDE and Vista you can open compressed files as folders, browse the contents, edit in native programs (word, openoffice..) and then save back without having to manipulate the archive file. Neither seem to allow for opening an archive file as a folder from within a file open/save dialog box. -- kai www.filesite.org || www.4thedadz.com || www.perfectreign.com remember - a turn signal is a statement, not a request -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

----- Original Message ----- From: "John Andersen" Cc: Sent: Friday, August 29, 2008 7:50 PM Subject: Re: [opensuse] Transparent Archive (TAR or Zip) Access for Linux?

On Fri, Aug 29, 2008 at 9:28 AM, Randall R Schulz wrote:

...

Hi,

Does anyone know if there's a Linux counterpart to the Windows ability to treat Zip files as if they're just directories, transparently deriving directory information and automatically extracting their contents on demand?

For what I'd like to be able to do, a KDE I/O slave will not suffice, it has to be system-wide and transparent so arbitrary scripts and applications can access it.

Others have provided answers on this thread, I just wanted to ask a question:

Are you sure Vista provides this as transparently as you are requesting for Linux?

Because if you open a cmd prompt and CD to a zip name Vista will slap you down. Its really only explorer that can do that, same as Konqueror. And I don't know of a Fuse extension for Vista.

Same on XP. The request is a nice idea but it was wrong to claim that Windows does it and expect to find it in Linux because it existed in Windows when in fact it does not. I laughed when I read that because I knew it was only an application, explorer, not magic-everywhere, exactly like konqueror. But color me surprised when something more than midnight commander and other explorer-alikes turned up after all. There is a thig called compressed folder in Windows but it's not a zip file. -- Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Randall R Schulz wrote:

...

... I laughed when I read that ...

I'm pleased to have indebted you to me for the provision of laughter.

Sidenote: :-) -- there really isn't enough laughter in life these days... Ever noticed how much of our 'humor' involves putting someone or something else "down"? Certainly "clowns" exploit this easy source of getting laughs, but it's odd that a small percentage of what we find humorous doesn't involve someone or something else's discomfiture or mistake. But that's a philosophical issue for another place, perhaps. (* look at the silly linda getting all serious over philosophical roots of what we find humorous...*) ... (* ok, some are more easily amused than others...*) (* but being easily amused can be a good thing: you are more often amused? *) :-) The real Question (missing feature in Linux?): Windows (NT-based) does have transparent, on-the-fly compression that can be enabled on NTFS. The transparent compress/decompress works with with all apps. If one enables compression for a volume, NT will look for "opportunities" to save space on on fixed-block sizes that are 16 times the Cluster size up to a max "compression unit", "CU", of 64k. This implicitly makes 4k the largest cluster size that will still allow file compression to be enabled on NTFS. NT requires that compression save at least 1 cluster, or it doesn't bother, but a file of all one's, for example, could be stored in about 1/16th the cluster space of a non-compressed file (a file of all 'zero's, can get better savings with "sparse files"). But this means each integral "CU" is checked for possible compression savings -- so data files might have some ranges compressed, but not others. The fact that NT implements compression in "CU"-sized chunks means there is very little hit in speed for random-access. Obviously, the speed hit, will "vary" by the ratio of CPU-power:disk speed...with a speed hit being more noticeable if the drive is a 15K RPM SAS(scsi)-based RAID compared to a slower IDE-based non-raid systems. On linux, I don't know of an implementation that does this as transparently as NT does with NTFS. Since the clusters are marked 'compressed' in the file system, there likely has to be some flag available on a "per-CU" basis in the NTFS meta information. On a data-read of a compressed section, NTFS decompresses the contents of the CU AND allocates the full, uncompressed file-space needed by the "CU" as *backing store* for the data that's decompressed into the block buffer. If the data isn't modified, the backing store doesn't get written to, so the CU stays compressed. If the user modifies the data, the modified data can be automatically written back to disk into the sectors that were allocated for backing store of the memory mapped file. At *some* point, (in writing), NT will try to compress the data in the CU to check for space savings of a cluster or more. Anyway -- I suppose the actual linux implementation is left as an exercise for the user...? :-) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

----- Original Message ----- From: "Linda Walsh" To: Cc: "Randall R Schulz" Sent: Saturday, August 30, 2008 3:28 PM Subject: [opensuse] Re: Transparent content compression for space savings on linux like on NTFS?

Randall R Schulz wrote:

...

...

... I laughed when I read that ...

I'm pleased to have indebted you to me for the provision of laughter.

---

Sidenote: :-) -- there really isn't enough laughter in life these days... Ever noticed how much of our 'humor' involves putting someone or something else "down"? Certainly "clowns" exploit this easy source of getting laughs, but it's odd that a small percentage of what we find humorous doesn't involve someone or something else's discomfiture or mistake. But that's a philosophical issue for another place, perhaps. (* look at the silly linda getting all serious over philosophical roots of what we find humorous...*) ... (* ok, some are more easily amused than others...*) (* but being easily amused can be a good thing: you are more often amused? *) :-)

The real Question (missing feature in Linux?):

Windows (NT-based) does have transparent, on-the-fly compression that can be enabled on NTFS. The transparent compress/decompress works with with all apps. If one enables compression for a volume, NT will look for "opportunities" to save space on on fixed-block sizes that are 16 times the Cluster size up to a max "compression unit", "CU", of 64k. This implicitly makes 4k the largest cluster size that will still allow file compression to be enabled on NTFS.

NT requires that compression save at least 1 cluster, or it doesn't bother, but a file of all one's, for example, could be stored in about 1/16th the cluster space of a non-compressed file (a file of all 'zero's, can get better savings with "sparse files"). But this means each integral "CU" is checked for possible compression savings -- so data files might have some ranges compressed, but not others. The fact that NT implements compression in "CU"-sized chunks means there is very little hit in speed for random-access. Obviously, the speed hit, will "vary" by the ratio of CPU-power:disk speed...with a speed hit being more noticeable if the drive is a 15K RPM SAS(scsi)-based RAID compared to a slower IDE-based non-raid systems.

On linux, I don't know of an implementation that does this as transparently as NT does with NTFS. Since the clusters are marked 'compressed' in the file system, there likely has to be some flag available on a "per-CU" basis in the NTFS meta information. On a data-read of a compressed section, NTFS decompresses the contents of the CU AND allocates the full, uncompressed file-space needed by the "CU" as *backing store* for the data that's decompressed into the block buffer. If the data isn't modified, the backing store doesn't get written to, so the CU stays compressed. If the user modifies the data, the modified data can be automatically written back to disk into the sectors that were allocated for backing store of the memory mapped file. At *some* point, (in writing), NT will try to compress the data in the CU to check for space savings of a cluster or more.

Anyway -- I suppose the actual linux implementation is left as an exercise for the user...? :-)

I think the OP already knew about compressed filesystems. The question was not about compressed filesystems. It was about treating a zip file as a directory. As for laughter, I laughed yet again when I saw how he chose to react to my admitting I was wrong for laughinging. And I am indebted to him as he says, not for the laughter so much as for the research, since he was the one who came up with the fuse answer in the first place. Although I don't think I have much use for mounting an archive as a filesystem, it's interesting to know about none the less. Maybe it's slightly fewer keystrokes for working on initrd's which are now cpio files instead of ext2 filesystems. -- Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

----- Original Message ----- From: "Randall R Schulz" To: Sent: Sunday, August 31, 2008 10:23 PM Subject: Re: [opensuse] Re: Transparent content compression for space savings on linux like on NTFS?

On Sunday 31 August 2008 18:22, Brian K. White wrote:

...

Although I don't think I have much use for mounting an archive as a filesystem, it's interesting to know about none the less. Maybe it's slightly fewer keystrokes for working on initrd's which are now cpio files instead of ext2 filesystems.

In my case, the need arose when I devised a set of scripts (additions to existing ones, actually) to facilitate large-scale experimentation [...] So I augmented the invocation scripts to maintain a log of compressed test results. Each run of the prover generally produces many files,

Wow interesting. So, is there a special reason why you want these in individual archives instead of simply in directories? Sounds to me like a compressed filesystem is your better answer after all. I have something similar where I have a lot of different automated EDI transaction scripts spawned many times a day by cron or by apache or postfix, or by local users in my application. Sometimes a script will involve importing or exporting or a several step process of both, and sometimes the other side is a big company with not-so-great IT staff, and sometimes the transactions involve big consequences, and sometimes disputes arise where the other side claims we never sent them a file or that we sent them a file with bad data etc.. Or that they sent us a file with such & such data in it that we failed to react to... Of course sometimes it is us and sometimes it is them and sometimes it is a weakness in the procedure they specified we use, sometimes the program and scripts all worked perfectly but the customer put in bad data, and sometimes it is the simple fact that the internet isn't perfect. So over time by now I have most of these scripts maintaining logs that are comprised of the entire working directory and all work files for the transction. The script starts by creating a new unique directory and then doing all work within it, never deleting any temp or data import/export files, capturing stdout and stderr and sometimes other diagnostic info from each command into seperate files. And then just leaving the whole directory there when done. So when such disputes arise now, there is a lot more evidense to look at. I have a cron job the just runs a simple find command to delete all files/dirs older than X days, or sometimes I have the find command right in the script so that script maintains it's own history and tims off any old data every time the script runs. The total data set is lots of files but it's all organized into trees and dirs and it's no hassle to go look at any particular transaction, or all the transactions for a given time frame etc, and the total data size isn't straining the disk space. But the point of all this was, if I had more data, or wanted to keep it longer or forever such that I wanted to compress it all, and, if I really needed immediate convenient random access to all of it (not just the most recent X days/months worth) then I think I would still want this to be a compressed filesystem instead of individual archive files. If I ever wanted one or some logical grouping of them in archive form to mail to someone, that's easy enough to do in one command. About the only reason I can see for really wanting them to be maintaned in the form of archives is if you had say an internal web/ftp/samba share, and you needed to provide a simple browseable list for other people to grab an archive file instead of browseable access to the equivalent directories. (maybe some are using interfaces that don't allow them to download a group of files or a directory all at once easily) Another approach if you don't need immediate/convenient random access to old files, even if you don't want to delete them, is just leave the dirs on the regular filesystem and and have a find command that turns old dirs into tar.bz2's instead of deleting them. Thats still a simple command and is maybe a simpler system than having a seperate cmpressed filesystem. Then you could use fuse or just traditional unpacking or a front-end like kio to access the old ones since it would be uncommon enough not to matter much, and the ones you access more often are just plain files in plain dirs, no special access hoops to jump though and full normal filesystem functionality a-la the df hitch you discovered. -- Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Sat, August 30, 2008 21:28, Linda Walsh wrote:

The real Question (missing feature in Linux?):

Windows (NT-based) does have transparent, on-the-fly compression that can be enabled on NTFS. The transparent compress/decompress works with with all apps. If one enables compression for a volume, NT will look for "opportunities" to save space on on fixed-block sizes that are 16 times the Cluster size up to a max "compression unit", "CU", of 64k. This implicitly makes 4k the largest cluster size that will still allow file compression to be enabled on NTFS.

NT requires that compression save at least 1 cluster, or it doesn't bother, but a file of all one's, for example, could be stored in about 1/16th the cluster space of a non-compressed file (a file of all 'zero's, can get better savings with "sparse files"). But this means each integral "CU" is checked for possible compression savings -- so data files might have some ranges compressed, but not others. The fact that NT implements compression in "CU"-sized chunks means there is very little hit in speed for random-access. Obviously, the speed hit, will "vary" by the ratio of CPU-power:disk speed...with a speed hit being more noticeable if the drive is a 15K RPM SAS(scsi)-based RAID compared to a slower IDE-based non-raid systems.

On linux, I don't know of an implementation that does this as transparently as NT does with NTFS. Since the clusters are marked 'compressed' in the file system, there likely has to be some flag available on a "per-CU" basis in the NTFS meta information. On a data-read of a compressed section, NTFS decompresses the contents of the CU AND allocates the full, uncompressed file-space needed by the "CU" as *backing store* for the data that's decompressed into the block buffer. If the data isn't modified, the backing store doesn't get written to, so the CU stays compressed. If the user modifies the data, the modified data can be automatically written back to disk into the sectors that were allocated for backing store of the memory mapped file. At *some* point, (in writing), NT will try to compress the data in the CU to check for space savings of a cluster or more.

Anyway -- I suppose the actual linux implementation is left as an exercise for the user...? :-)

You should take a look at ZFS which implements LZJB. From the Wikipedia article on ZFS: Variable block sizes ZFS uses variable-sized blocks of up to 128 kilobytes. The currently available code allows the administrator to tune the maximum block size used as certain workloads do not perform well with large blocks. Automatic tuning to match workload characteristics is contemplated.[citation needed] If data compression (LZJB) is enabled, variable block sizes are used. If a block can be compressed to fit into a smaller block size, the smaller size is used on the disk to use less storage and improve IO throughput (though at the cost of increased CPU use for the compression and decompression operations). ZFS is currently available in OpenSolaris, BSD and Mac OS X. Because of a license issue, ZFS cannot be implemented in the Linux kernel, but technically it would be a Simple Matter Of Programming. However there is a FUSE (yes, FUSE again) implementation of ZFS. -- Amedee -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org