dynamic compressed read/write filesystems?
Does anyone know offhand if any of the commonly supported filesystems support dynamic compression? I was looking at the size of my maildirs today, and thought to myself "text generally compresses well"... I guess I could archive stuff to a cramfs or squashfs periodically, as that'd give me a readable directory structure with compression, but I'd really like to trade some CPU time for some disk space on a couple of machines. If Ms can manage to get individually compressed files into NTFS, surely *someone* has gotten around to doing so under Linux. Right? I'll head off to google, but figured that someone here might know of something and maybe even have experience with it. --Danny, hoping not to run NTFS under Linux, even if RW support *is* marked "stable" someday :)
The Tuesday 2004-10-19 at 12:56 -0500, Danny Sauer wrote:
Does anyone know offhand if any of the commonly supported filesystems support dynamic compression? I was looking at the size of my maildirs today, and thought to myself "text generally compresses well"...
Me too...
I guess I could archive stuff to a cramfs or squashfs periodically, as that'd give me a readable directory structure with compression, but I'd really like to trade some CPU time for some disk space on a couple of machines. If Ms can manage to get individually compressed files into NTFS, surely *someone* has gotten around to doing so under Linux. Right?
Ha! Half wrong...
I'll head off to google, but figured that someone here might know of something and maybe even have experience with it.
Ok, I know that ext2 was thought, at some point, to get it. But I'm unable to find the man page that mentioned this time ago - it was "man attr", I think. It has not being implemented. Then, it is possible to create compressed cdroms and dvds: the kernel can read them on the fly, transparently. But of course, it is read-only. Pity. It seems that most developpers think that HD space is cheap. That may be so, but using less space is even cheaper. It does not make much sense to me to waste space... -- Cheers, Carlos Robinson
On Wednesday 20 October 2004 01:13, Carlos E. R. wrote:
The Tuesday 2004-10-19 at 12:56 -0500, Danny Sauer wrote:
Does anyone know offhand if any of the commonly supported filesystems support dynamic compression? I was looking at the size of my maildirs today, and thought to myself "text generally compresses well"...
Me too...
I guess I could archive stuff to a cramfs or squashfs periodically, as that'd give me a readable directory structure with compression, but I'd really like to trade some CPU time for some disk space on a couple of machines. If Ms can manage to get individually compressed files into NTFS, surely *someone* has gotten around to doing so under Linux. Right?
Ha! Half wrong...
I'll head off to google, but figured that someone here might know of something and maybe even have experience with it.
Ok, I know that ext2 was thought, at some point, to get it. But I'm unable to find the man page that mentioned this time ago - it was "man attr", I think. It has not being implemented.
Then, it is possible to create compressed cdroms and dvds: the kernel can read them on the fly, transparently. But of course, it is read-only.
Pity.
It seems that most developpers think that HD space is cheap. That may be so, but using less space is even cheaper. It does not make much sense to me to waste space...
On the 9.1 live CD, cloop.img is compressed. A kernelmodule, cloop.ko, is used to uncompress on the fly, like: insmod cloop.ko file=cloop.img && mount -r -t iso9660 /dev/cloop mount_point Found something with 'gg:cloop kernel': http://dag.wieers.com/packages/kernel-module-cloop/kernel-module-cloop.spec So Dag Wieers has an rpm package for a cloop kernel module. A bit further searching on http://rpm.pbone.net/, I learn that Mandrake has a cloop-utils package. From the src rpm, I discover that Klaus Knopper is the maintainer of the cloop tarball. See http://developer.linuxtag.net/knoppix/sources/cloop* Cheers, Leen
On Wednesday 20 October 2004 02:01, Leendert Meyer wrote:
On Wednesday 20 October 2004 01:13, Carlos E. R. wrote:
The Tuesday 2004-10-19 at 12:56 -0500, Danny Sauer wrote:
Does anyone know offhand if any of the commonly supported filesystems support dynamic compression? I was looking at the size of my maildirs today, and thought to myself "text generally compresses well"...
Me too...
I guess I could archive stuff to a cramfs or squashfs periodically, as that'd give me a readable directory structure with compression, but I'd really like to trade some CPU time for some disk space on a couple of machines. If Ms can manage to get individually compressed files into NTFS, surely *someone* has gotten around to doing so under Linux. Right?
Ha! Half wrong...
I'll head off to google, but figured that someone here might know of something and maybe even have experience with it.
Ok, I know that ext2 was thought, at some point, to get it. But I'm unable to find the man page that mentioned this time ago - it was "man attr", I think. It has not being implemented.
Then, it is possible to create compressed cdroms and dvds: the kernel can read them on the fly, transparently. But of course, it is read-only.
Pity.
It seems that most developpers think that HD space is cheap. That may be so, but using less space is even cheaper. It does not make much sense to me to waste space...
On the 9.1 live CD, cloop.img is compressed. A kernelmodule, cloop.ko, is used to uncompress on the fly, like:
insmod cloop.ko file=cloop.img && mount -r -t iso9660 /dev/cloop mount_point
Found something with 'gg:cloop kernel':
http://dag.wieers.com/packages/kernel-module-cloop/kernel-module-cloop.spec
So Dag Wieers has an rpm package for a cloop kernel module. A bit further searching on http://rpm.pbone.net/, I learn that Mandrake has a cloop-utils package. From the src rpm, I discover that Klaus Knopper is the maintainer of the cloop tarball. See http://developer.linuxtag.net/knoppix/sources/cloop*
Just discovered that the cloop module is part of the kotd kernel in the projects directory. And it is also in kernel-default-2.6.8-16.i586.rpm, that was part of the SuSE 9.1.42 (9.2 beta). Those rpms, including source, are still available on e.g.: ftp://ftp.nluug.nl/pub/os/Linux/distr/suse/suse/i386/update/9.1.42/ I'm thinking about trying to install those on my 9.1 system tomorrow... ;) <whisper> Seems that 9.2 can handle compressed filesystems? Would be nice! </whisper> Cheers, Leen
The Wednesday 2004-10-20 at 03:10 +0200, Leendert Meyer wrote:
Just discovered that the cloop module is part of the kotd kernel in the projects directory.
And it is also in kernel-default-2.6.8-16.i586.rpm, that was part of the SuSE 9.1.42 (9.2 beta). Those rpms, including source, are still available on e.g.:
ftp://ftp.nluug.nl/pub/os/Linux/distr/suse/suse/i386/update/9.1.42/
I'm thinking about trying to install those on my 9.1 system tomorrow... ;)
I'll be listening :-)
<whisper> Seems that 9.2 can handle compressed filesystems? Would be nice! </whisper>
Really nice! That's the way that compressed cdroms appeared between suse 7.3 and suse 8.1 -- Cheers, Carlos Robinson
On Wednesday 20 October 2004 11:56, Carlos E. R. wrote:
The Wednesday 2004-10-20 at 03:10 +0200, Leendert Meyer wrote:
Just discovered that the cloop module is part of the kotd kernel in the projects directory.
And it is also in kernel-default-2.6.8-16.i586.rpm, that was part of the SuSE 9.1.42 (9.2 beta). Those rpms, including source, are still available on e.g.:
ftp://ftp.nluug.nl/pub/os/Linux/distr/suse/suse/i386/update/9.1.42/
I'm thinking about trying to install those on my 9.1 system tomorrow... ;)
I'll be listening :-)
I did it shortly after, I couldn't resist ;P bootsplash is broken (again), but besides that no problem /so far/. [remember: 'rpm -i' keeps the current kernel ;) ]
<whisper> Seems that 9.2 can handle compressed filesystems? Would be nice! </whisper>
Really nice!
cloop.ko is there, but I could not locate the source file. Might be in another package. We'll see when 9.2 comes out. ;) Let's try the 9.1 LiveCD: # mount -o loop LiveCD-9.1-01.iso mnt # cd mnt # ls . .. boot.catalog bootlogo cloop.img initrd.gz isolinux.bin isolinux.cfg media.1 txtmsg vmlinuz # insmod /lib/modules/2.6.8-16-default/extra/cloop.ko file=cloop.img # mount -r -t iso9660 /dev/cloop0 ../mnt-01/ # ls ../mnt-01/ . .. .CD.real bin boot dev etc home lib media mnt opt proc root sbin srv sys tmp usr var # Yup. Works. Cheers, Leen
The Wednesday 2004-10-20 at 14:39 +0200, Leendert Meyer wrote:
I'm thinking about trying to install those on my 9.1 system tomorrow... ;)
I'll be listening :-)
I did it shortly after, I couldn't resist ;P
...
cloop.ko is there, but I could not locate the source file. Might be in another package. We'll see when 9.2 comes out. ;)
I'll do that. :-)
Let's try the 9.1 LiveCD:
# mount -o loop LiveCD-9.1-01.iso mnt # cd mnt # ls . .. boot.catalog bootlogo cloop.img initrd.gz isolinux.bin isolinux.cfg media.1 txtmsg vmlinuz # insmod /lib/modules/2.6.8-16-default/extra/cloop.ko file=cloop.img # mount -r -t iso9660 /dev/cloop0 ../mnt-01/ # ls ../mnt-01/ . .. .CD.real bin boot dev etc home lib media mnt opt proc root sbin srv sys tmp usr var #
Yup. Works.
You mean you mounted the livecd on your modified system, and it read, so it means that it can read compressed images, is that so? Good! However... are those "loops" writeable? Because read only CDs have existed for some time (zisofs/RockRidge). -- Cheers, Carlos Robinson
On Wednesday 20 October 2004 21:57, Carlos E. R. wrote:
The Wednesday 2004-10-20 at 14:39 +0200, Leendert Meyer wrote:
I'm thinking about trying to install those on my 9.1 system tomorrow... ;)
I'll be listening :-)
I did it shortly after, I couldn't resist ;P
...
cloop.ko is there, but I could not locate the source file. Might be in another package. We'll see when 9.2 comes out. ;)
I'll do that. :-)
Let's try the 9.1 LiveCD:
# mount -o loop LiveCD-9.1-01.iso mnt # cd mnt # ls . .. boot.catalog bootlogo cloop.img initrd.gz isolinux.bin isolinux.cfg media.1 txtmsg vmlinuz # insmod /lib/modules/2.6.8-16-default/extra/cloop.ko file=cloop.img # mount -r -t iso9660 /dev/cloop0 ../mnt-01/ # ls ../mnt-01/ . .. .CD.real bin boot dev etc home lib media mnt opt proc root sbin srv sys tmp usr var #
Yup. Works.
You mean you mounted the livecd on your modified system, and it read, so it means that it can read compressed images, is that so? Good!
Yes.
However... are those "loops" writeable? Because read only CDs have existed for some time (zisofs/RockRidge).
Well, I wouldn't spoil all the fun for you, would I? I've left a bit of testing to do for you. ;)) If it doesn't work, we'll probably need a kind of cloop-utils package. I'll try to create that rpm tomorrow if it is needed. Cheers, Leen
On Thursday 21 October 2004 02:16, Leendert Meyer wrote:
On Wednesday 20 October 2004 21:57, Carlos E. R. wrote:
The Wednesday 2004-10-20 at 14:39 +0200, Leendert Meyer wrote:
I'm thinking about trying to install those on my 9.1 system tomorrow... ;)
I'll be listening :-)
I did it shortly after, I couldn't resist ;P
...
cloop.ko is there, but I could not locate the source file. Might be in another package. We'll see when 9.2 comes out. ;)
I'll do that. :-)
Let's try the 9.1 LiveCD:
# mount -o loop LiveCD-9.1-01.iso mnt # cd mnt # ls . .. boot.catalog bootlogo cloop.img initrd.gz isolinux.bin isolinux.cfg media.1 txtmsg vmlinuz # insmod /lib/modules/2.6.8-16-default/extra/cloop.ko file=cloop.img # mount -r -t iso9660 /dev/cloop0 ../mnt-01/ # ls ../mnt-01/ . .. .CD.real bin boot dev etc home lib media mnt opt proc root sbin srv sys tmp usr var #
Yup. Works.
You mean you mounted the livecd on your modified system, and it read, so it means that it can read compressed images, is that so? Good!
Yes.
However... are those "loops" writeable? Because read only CDs have existed for some time (zisofs/RockRidge).
Well, I wouldn't spoil all the fun for you, would I? I've left a bit of testing to do for you. ;))
If it doesn't work, we'll probably need a kind of cloop-utils package. I'll try to create that rpm tomorrow if it is needed.
Here's a spec file: http://cc22149-a.groni1.gr.home.nl/~leen/cloop/cloop-utils.spec Get this tarball: http://developer.linuxtag.net/knoppix/sources/cloop_2.01-5.tar.gz Put cloop-utils.spec in /usr/src/packages/SPECS Put cloop_2.01-5.tar.gz in /usr/src/packages/SOURCES Execute 'rpmbuild -ba /usr/src/packages/SPECS/cloop-utils.spec' The rpm will be in /usr/src/packages/RPMS/i586 Build time: a few minutes. Cheers, Leen
The Thursday 2004-10-21 at 05:09 +0200, Leendert Meyer wrote:
Well, I wouldn't spoil all the fun for you, would I? I've left a bit of testing to do for you. ;))
:-)
If it doesn't work, we'll probably need a kind of cloop-utils package. I'll try to create that rpm tomorrow if it is needed.
Here's a spec file: http://cc22149-a.groni1.gr.home.nl/~leen/cloop/cloop-utils.spec Get this tarball: http://developer.linuxtag.net/knoppix/sources/cloop_2.01-5.tar.gz
Mmm, but I'm using the standard suse kernel, so I'm waiting till 9.2 for experimenting with that :-) -- Cheers, Carlos Robinson
On Thursday 21 October 2004 18:33, Carlos E. R. wrote:
The Thursday 2004-10-21 at 05:09 +0200, Leendert Meyer wrote:
Well, I wouldn't spoil all the fun for you, would I? I've left a bit of testing to do for you. ;)) : :-) : If it doesn't work, we'll probably need a kind of cloop-utils package. I'll try to create that rpm tomorrow if it is needed.
Here's a spec file: http://cc22149-a.groni1.gr.home.nl/~leen/cloop/cloop-utils.spec Get this tarball: http://developer.linuxtag.net/knoppix/sources/cloop_2.01-5.tar.gz
Mmm, but I'm using the standard suse kernel, so I'm waiting till 9.2 for experimenting with that :-)
Not necessary. That spec file is gone, but I made some packages you might be interested in: http://cc22149-a.groni1.gr.home.nl/~leen/cloop/ They run on 9.1 with any 2.6 kernel. I tried them with kernel-default-2.6.5-7.108. So far it works. Install both, goto /usr/src/kernel-modules/cloop, and follow the instructions in the INSTALL file. man compressedloop tells how to use it. For the insmod, use the full path to cloop.ko (e.g. /lib/modules/2.6.5-7.108-default/extra/cloop.ko). Cheers, Leen
Did you ever test the compressed filesystem (cloop) for read / write
support, or is it read-only?
On Thu, 21 Oct 2004 19:26:21 +0200, Leendert Meyer
On Thursday 21 October 2004 18:33, Carlos E. R. wrote:
The Thursday 2004-10-21 at 05:09 +0200, Leendert Meyer wrote:
Well, I wouldn't spoil all the fun for you, would I? I've left a bit of testing to do for you. ;)) : :-) : If it doesn't work, we'll probably need a kind of cloop-utils package. I'll try to create that rpm tomorrow if it is needed.
Here's a spec file: http://cc22149-a.groni1.gr.home.nl/~leen/cloop/cloop-utils.spec Get this tarball: http://developer.linuxtag.net/knoppix/sources/cloop_2.01-5.tar.gz
Mmm, but I'm using the standard suse kernel, so I'm waiting till 9.2 for experimenting with that :-)
Not necessary. That spec file is gone, but I made some packages you might be interested in:
http://cc22149-a.groni1.gr.home.nl/~leen/cloop/
They run on 9.1 with any 2.6 kernel. I tried them with kernel-default-2.6.5-7.108. So far it works.
Install both, goto /usr/src/kernel-modules/cloop, and follow the instructions in the INSTALL file. man compressedloop tells how to use it. For the insmod, use the full path to cloop.ko (e.g. /lib/modules/2.6.5-7.108-default/extra/cloop.ko).
Cheers,
Leen
-- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
On Friday 22 October 2004 21:11, Greg Freemyer wrote:
Did you ever test the compressed filesystem (cloop) for read / write support, or is it read-only?
The compressed loopback device (/dev/cloop1, ...) is read/only. But there are two manual pages in the cloop-utils package I created. rpm packages are easily browsed with e.g. Midnight Commander (mc). Those man pages are quite informative in howto create and use compressed filesystems. Cheers, Leen
Greg wrote regarding 'Re: [SLE] dynamic compressed read/write filesystems? - cloop-utils.spec (1.4k) inside' on Fri, Oct 22 at 14:12:
Did you ever test the compressed filesystem (cloop) for read / write support, or is it read-only?
Well, since the first result for a google search for "cloop" states that "Cloop is a kernel module to add support for filesystem- independent, transparently decompressed, read-only, block devices", I'm guessing the answer is "no"... You can already make transparently decompressed read only block devices with cramfs, squashfs, ziosfs, etc. Looks like I'm still waiting for Reiser4 to actually be stable, so the plugin architecture can be used to implement transparent compression 'n decompression. --Danny, who wants to trust Reiser4, but just can't do that yet
On Friday 22 October 2004 23:39, Danny Sauer wrote:
Greg wrote regarding 'Re: [SLE] dynamic compressed read/write filesystems? - cloop-utils.spec (1.4k) inside' on Fri, Oct 22 at 14:12:
Did you ever test the compressed filesystem (cloop) for read / write support, or is it read-only?
Well, since the first result for a google search for "cloop" states that "Cloop is a kernel module to add support for filesystem- independent, transparently decompressed, read-only, block devices", I'm guessing the answer is "no"...
Correct guess. Dag Wieers's rpms? ;) BTW, there is a 9.2 cloop-*.src.rpm now, so my 'hack' is already overhauled. It can be built on 9.1, but one needs to comment out the line 'BuildRequires' in the spec file.
You can already make transparently decompressed read only block devices with cramfs, squashfs, ziosfs, etc.
Looks like I'm still waiting for Reiser4 to actually be stable,
Hmmm, a year or so??? Just a guess.
so the plugin architecture
Yes, this is really interesting! :)
can be used to implement transparent compression 'n decompression.
--Danny, who wants to trust Reiser4, but just can't do that yet
Neither would I. Cheers, Leen
Carlos wrote regarding 'Re: [SLE] dynamic compressed read/write filesystems?' on Tue, Oct 19 at 18:29:
The Tuesday 2004-10-19 at 12:56 -0500, Danny Sauer wrote: [...]
I'll head off to google, but figured that someone here might know of something and maybe even have experience with it.
Ok, I know that ext2 was thought, at some point, to get it. But I'm unable to find the man page that mentioned this time ago - it was "man attr", I think. It has not being implemented.
http://e2compr.sourceforge.net/ Looks only half-alive, but it does exist. It's not alive enough to trust my data with, though. Perhaps if I was still running a 2.2 kernel... :) --Danny
On Friday 22 October 2004 23:42, Danny Sauer wrote:
Carlos wrote regarding 'Re: [SLE] dynamic compressed read/write filesystems?' on Tue, Oct 19 at 18:29:
The Tuesday 2004-10-19 at 12:56 -0500, Danny Sauer wrote:
[...]
I'll head off to google, but figured that someone here might know of something and maybe even have experience with it.
Ok, I know that ext2 was thought, at some point, to get it. But I'm unable to find the man page that mentioned this time ago - it was "man attr", I think. It has not being implemented.
Try 'man lsattr' and 'man chattr'. The 'c' attribute indicates file by file compression. These commands also work on Reiser on SuSE9.1 Pro i.e. they do not report an error when executed. Paul -- Paul Hewlett (Linux #359543) Email:`echo az.oc.evitcaten@ttelweh | rev` Tel: +27 21 852 8812 Cel : +27 72 719 2725 FAX: +27 866720563 --
The Saturday 2004-10-23 at 10:31 +0200, Paul Hewlett wrote:
Ok, I know that ext2 was thought, at some point, to get it. But I'm unable to find the man page that mentioned this time ago - it was "man attr", I think. It has not being implemented.
Try 'man lsattr' and 'man chattr'. The 'c' attribute indicates file by file compression. These commands also work on Reiser on SuSE9.1 Pro i.e. they do not report an error when executed.
That's the man page I was refering to, that I had forgotten which one it was. But read on that man: BUGS AND LIMITATIONS As of Linux 2.2, the c', 's', and u' attribute are not honored by the kernel filesystem code. These attributes will be implemented in a future ext2 fs version. There are also some mentions of "experimental compression patches" -- Cheers, Carlos Robinson
Paul wrote regarding 'Re: [SLE] dynamic compressed read/write filesystems?' on Sat, Oct 23 at 03:29:
On Friday 22 October 2004 23:42, Danny Sauer wrote:
Carlos wrote regarding 'Re: [SLE] dynamic compressed read/write filesystems?' on Tue, Oct 19 at 18:29:
The Tuesday 2004-10-19 at 12:56 -0500, Danny Sauer wrote:
[...]
I'll head off to google, but figured that someone here might know of something and maybe even have experience with it.
Ok, I know that ext2 was thought, at some point, to get it. But I'm unable to find the man page that mentioned this time ago - it was "man attr", I think. It has not being implemented.
Try 'man lsattr' and 'man chattr'. The 'c' attribute indicates file by file compression. These commands also work on Reiser on SuSE9.1 Pro i.e. they do not report an error when executed.
They also don't actualy *do* anything. :) newwww /var/log # du xferlog 2976 xferlog newwww /var/log # df . Filesystem 1k-blocks Used Available Use% Mounted on /dev/sdb2 5121884 1585656 3536228 31% /var newwww /var/log # chattr +c xferlog newwww /var/log # du xferlog 2976 xferlog newwww /var/log # df . Filesystem 1k-blocks Used Available Use% Mounted on /dev/sdb2 5121884 1585656 3536228 31% /var Note the total lack of any change in disk space used. Perhps I'm doing something wrong, but I'm guessing that they're unimplemented on the ext2, ext3, and reiser filesystems that I tried this upon... --Danny
participants (5)
-
Carlos E. R. -
Danny Sauer -
Greg Freemyer -
Leendert Meyer -
Paul Hewlett