[opensuse-factory] Squashfs
I just noticed that while squashfs 3.4 kernel module is used on the -default kernel, the squashfs tools on both PowerPC and x86 repos are stuck at 3.2 neko@frodo:~> mksquashfs -version mksquashfs version 3.2-r2 (2007/01/15) Is there any particular reason for this? Is there an update planned? Also, more for the kernel guys.. is there any reason why the mkinitrd etc. stuff relies on cpio archives when the SUSE kernel is built with squashfs (granted as a module), all the rescue/install images are squashfs? It does not provide any remarkably better compression - in fact I tested 100k larger filesystem with mksquashfs 3.2 but then mksquashfs 3.2 refuses to use large block sizes so the compression is lower than it could be, also mksquashfs 3.4 supposes a 10% improvement and unsquashfs supposes a 40% improvement in speed in creation and extraction (we will ignore the kernel improvements as, they are in the kernel already).. .. But a few tests in performance usually always show that accessing a squashfs initrd mounted to RAM is faster than a cpio initrd in RAM. It also has a few extra little advantages here and there but I do not know if they are worth it, but I do know that managing squashfs filesystems is a billion times easier than pushing cpio archives around. Since squashfs is going to be in the kernel mainline at some point (it's in Linus' tree) and everyone and his dog makes squashfs initrds, are we going to see a move to it at some point and have the thing built in with more use of the format? -- Matt -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Wed, Jan 14, 2009 at 03:12:32PM -0600, Matt Sealey wrote:
I just noticed that while squashfs 3.4 kernel module is used on the -default kernel, the squashfs tools on both PowerPC and x86 repos are stuck at 3.2
neko@frodo:~> mksquashfs -version mksquashfs version 3.2-r2 (2007/01/15)
Is there any particular reason for this? Is there an update planned?
Someone needs to kick the owner of the userspace squashfs tools to get them to update them. If you could be so kind as to file a bug against the package, that would help.
Also, more for the kernel guys.. is there any reason why the mkinitrd etc. stuff relies on cpio archives when the SUSE kernel is built with squashfs (granted as a module), all the rescue/install images are squashfs?
No specific reason, other than "that's just the way we've always done it." Do you really want to touch that mkinitrd code to change this? :)
It does not provide any remarkably better compression - in fact I tested 100k larger filesystem with mksquashfs 3.2 but then mksquashfs 3.2 refuses to use large block sizes so the compression is lower than it could be, also mksquashfs 3.4 supposes a 10% improvement and unsquashfs supposes a 40% improvement in speed in creation and extraction (we will ignore the kernel improvements as, they are in the kernel already)..
.. But a few tests in performance usually always show that accessing a squashfs initrd mounted to RAM is faster than a cpio initrd in RAM.
Are you sure? What kind of performance difference is there?
It also has a few extra little advantages here and there but I do not know if they are worth it, but I do know that managing squashfs filesystems is a billion times easier than pushing cpio archives around.
But these are generated "on-the-fly" and don't need to be managed by anything, right? What advantages do you know of?
Since squashfs is going to be in the kernel mainline at some point (it's in Linus' tree)
Yes, it will be in 2.6.29, and is already there in 2.6.29-rc1.
and everyone and his dog makes squashfs initrds,
Who else?
are we going to see a move to it at some point and have the thing built in with more use of the format?
I think the goal with mkinitrd is to work with the community to come up with a common tool that all the distros use. Dave Jones is currently working on a first cut at this, and then we will all get together to go from there. If it happens to use squashfs instead of cpio, well, that will be up to the community, not just us. thanks, greg k-h -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Greg KH wrote:
On Wed, Jan 14, 2009 at 03:12:32PM -0600, Matt Sealey wrote:
I just noticed that while squashfs 3.4 kernel module is used on the -default kernel, the squashfs tools on both PowerPC and x86 repos are stuck
Someone needs to kick the owner of the userspace squashfs tools to get them to update them. If you could be so kind as to file a bug against the package, that would help.
I am doing it now, but also going through another bunch of bugs.. also I'm a little wary of the actual category to put it in but, oh well.. Other can do for now.
Also, more for the kernel guys.. is there any reason why the mkinitrd etc. stuff relies on cpio archives when the SUSE kernel is built with squashfs (granted as a module), all the rescue/install images are squashfs?
No specific reason, other than "that's just the way we've always done it."
Do you really want to touch that mkinitrd code to change this? :)
I was sure at some point I was reading mkinitrd code and saw some option for initrd_fs which specified a type, which said "ext2" was the default. This is obviously dead wrong (since it's always a cpio). Looking at it again, it turns out mk_image (argh what's with these names, I just had a fight with mkimage from U-Boot and jigdo) supports squashfs through the Conv2Image perl module. Aren't these two codebases doing EXACTLY the same thing but in different ways, here? Some room for consolidation, I'd think.
It does not provide any remarkably better compression - in fact I tested 100k larger filesystem with mksquashfs 3.2 but then mksquashfs 3.2 refuses to use large block sizes so the compression is lower than it could be, also mksquashfs 3.4 supposes a 10% improvement and unsquashfs supposes a 40% improvement in speed in creation and extraction (we will ignore the kernel improvements as, they are in the kernel already)..
.. But a few tests in performance usually always show that accessing a squashfs initrd mounted to RAM is faster than a cpio initrd in RAM.
Are you sure? What kind of performance difference is there?
I'm trying to benchmark that now, I'm wondering what I can actually do.. maybe I will grab the 11.1 common and make a cpio out of it and loop mount them both and do something with it, but it is sort of beyond my means at the moment. I've run out of hardware that's not in permanent use.. benchmarking on a system running an rpm build is not fair :) I will endeavour to poke at it though. In the meantime I bugged the author, since he would know :D
It also has a few extra little advantages here and there but I do not know if they are worth it, but I do know that managing squashfs filesystems is a billion times easier than pushing cpio archives around.
But these are generated "on-the-fly" and don't need to be managed by anything, right?
Indeed..
What advantages do you know of?
Appending files is a breeze compared to a cpio archive. Replacing the same file in a cpio usually means dumping and remaking it. mksquashfs can get rid of the original file and relink the new one at the end.. it would mean you can do clever tricks like building half the initrd or filesystem image, doing something else based on that result, and then appending further processed data. I guess it's just as easy with cpio with clever scripting. There's also the potential for slightly better compression if there was some known sort order of the files so they could be grouped together. Perhaps in the future LZO (not as great compression as gz but much faster to decompress and FAR less memory intensive of course) or LZMA (better, but a bit heavier of course. Always a trade-off) sometime, perhaps, maybe. I'm not counting that but it would be a benefit worth risking a little work on.
Since squashfs is going to be in the kernel mainline at some point (it's in Linus' tree)
Yes, it will be in 2.6.29, and is already there in 2.6.29-rc1.
and everyone and his dog makes squashfs initrds,
Who else?
Gentoo, GoboLinux.. umm.. Knoppix et al. :) For no other good reason than it ends up that someone wants to loop mount a root filesystem at some point during the life of the install, be it just during install (as is on the SUSE DVD common/rescue images etc.), or using it to pre-seed the root filesystem and then union mounting a real root on top (I can't remember exactly where I saw this trick now, but I think it was GoboLinux. All the kernel modules are in the initrd, not on the main filesystem.. it hasn't the finesse of the SUSE "dynamically make an initrd based on your machine", but it's no more clunky than the SUSE "the installer initrd cpio actually contains a big squashfs of all the modules and loop mounts it and sometimes in the betas it's not there (bug 437796)" :) Granted it's nothing you can't do with cpio, but.. some places people decided, if you're going to make a filesystem archive which gets loaded or mounted, they may as well all be the same type. And SUSE is already using it in the installer.
are we going to see a move to it at some point and have the thing built in with more use of the format?
I think the goal with mkinitrd is to work with the community to come up with a common tool that all the distros use. Dave Jones is currently working on a first cut at this, and then we will all get together to go from there. If it happens to use squashfs instead of cpio, well, that will be up to the community, not just us.
I don't think you would get too many objections.. there's also no reason (like mk_image in installation-images) that it can't use a bunch of different image types. Who knows, squashfs could be superceded by something else in the future. You know how usually goes.. it's better to be flexible :) -- Matt Sealey Genesi, Manager, Developer Relations -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
2009/1/14 Matt Sealey
Greg KH wrote:
On Wed, Jan 14, 2009 at 03:12:32PM -0600, Matt Sealey wrote:
What advantages do you know of?
Appending files is a breeze compared to a cpio archive. Replacing the same file in a cpio usually means dumping and remaking it. mksquashfs can get rid of the original file and relink the new one at the end.. it would mean you can do clever tricks like building half the initrd or filesystem image, doing something else based on that result, and then appending further processed data.
That might help with an issue I've stumbled into. I wanted to change from using, the pata_pdc202xx_old driver, to using ATA/IDE pdc202xx_old, and the new mkinitrd script was relying on udev and what was used in the current system. The expert on the mkinitrd changes, said that there wasn't now a way to make this change. So the workround to avoid a particular driver, became a tortuous fresh install, then remake the initrd and copy over to the original installation. Similarly, I can imagine it being reasonable for a sysadmin to install using plain partitions, then with a burned in machine, want to move / and /boot over to a RAID array, or / in LVM managed partition when the box is live in production. Being able to manipulate the initrd's, with standard tools could help considerably in recovering a system, or where hardware change requires new driver support for /, but you don't yet have a running system to build the new initrd on.
I guess it's just as easy with cpio with clever scripting.
Mounting a filesystem and using standard tools is far more sysadmin friendly. The main drawback would seem to be, that the new format would be opaque to older kernels (minor?), but also need kernel support for the purpose. Having squashfs inside cpio/gz would just make things more complicated, adding an unecessary extra layer for little benefit on an installed system.
There's also the potential for slightly better compression if there was some known sort order of the files so they could be grouped together.
Perhaps in the future LZO (not as great compression as gz but much faster to decompress and FAR less memory intensive of course) or LZMA (better, but a bit heavier of course. Always a trade-off) sometime, perhaps, maybe. I'm not counting that but it would be a benefit worth risking a little work on.
It would seem wise to err on side of heavier compression and processing. bzImage has been used for a very long time, to decompress the kernel on older CPUs with less memory than is standard now, so it is likely pointless to go to a looser compression for decompression speed and lower memory consumption.
For no other good reason than it ends up that someone wants to loop mount a root filesystem at some point during the life of the install, be it just during install (as is on the SUSE DVD common/rescue images etc.), or using it to pre-seed the root filesystem and then union mounting a real root on top (I can't remember exactly where I saw this trick now, but I think it was GoboLinux.
All the kernel modules are in the initrd, not on the main filesystem.. it hasn't the finesse of the SUSE "dynamically make an initrd based on your machine", but it's no more clunky than the SUSE "the installer initrd cpio actually contains a big squashfs of all the modules and loop mounts it and sometimes in the betas it's not there (bug 437796)" :)
Can that support live installion of new modules? Logically the initrd has been a minimal size on typical running systems, and should ideally be the minimal set of modules to get / (and /lib/modules) mounted. Then "all" of the modules are in 1 place, and the initrd only needs changing when a kernel update, or hardware change occurs.
Granted it's nothing you can't do with cpio, but.. some places people decided, if you're going to make a filesystem archive which gets loaded or mounted, they may as well all be the same type. And SUSE is already using it in the installer.
That initrd is there to be loaded into RAM by the boot loader, to provide the kernel code to mount the real / filesystem. So the cpio gz format was what the kernel supported and needed. Installers may benefit from the extra layer of mounting a special purpose filesystem type, but if the kernel expects a cpio archive in RAM, and not the SqaushFS format, then you gain nothing. One reason perhaps for use of cpio, was that the format was absolutely stably fixed. If SquashFS got used in the suggested way, then it's format needs to be similarly stable, with a mature tool chain.
are we going to see a move to it at some point and have the thing built in with more use of the format?
I think the goal with mkinitrd is to work with the community to come up with a common tool that all the distros use. Dave Jones is currently working on a first cut at this, and then we will all get together to go from there. If it happens to use squashfs instead of cpio, well, that will be up to the community, not just us.
I don't think you would get too many objections.. there's also no reason (like mk_image in installation-images) that it can't use a bunch of different image types. Who knows, squashfs could be superceded by something else in the future. You know how usually goes.. it's better to be flexible
Being able to mount the initrd and alter, and add modules, not automatically chosen would solve some issues. Presumbably some FUSE scheme could do the same, with the current format? Better compression, and/or performance considerations of the initrd format would be secondary in my view. Does Dave Jone's consider altering the format? My impression was it was more an attempt to having a sane unification of code with many special cases, to reduce duplication of effort. Dracut - "Unlike existing initramfs's, this is an attempt at having as little as possible hard-coded into the initramfs as possible. The initramfs has (basically) one purpose in life -- getting the rootfs mounted so that we can transition to the real rootfs. This is all driven off of device availability." On a running system that approach would appear compatible with a retro-fit for older kernels, which might be a goal of a cross distro project, though the main 2.4 holdouts I came across last year, seem to have found a way round their permanent objects and now use 2.6. That might be a source of objections. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Rob OpenSuSE wrote:
2009/1/14 Matt Sealey
: Mounting a filesystem and using standard tools is far more sysadmin friendly. The main drawback would seem to be, that the new format would be opaque to older kernels (minor?), but also need kernel support for the purpose.
That's true but that's up to mkinitrd to decide which it makes or not. Since it's generated on the fly, based on the current system..
Having squashfs inside cpio/gz would just make things more complicated
Nobody is suggesting that this be a solution, although it IS the solution used for mkinstallinitrd in installation-images - it makes a cpio.gz initrd and stuffs in squashfs files full of modules and other stuff, and loop mounts them via Linuxrc (after loading the squashfs module and loop module which are the only ones in the cpio.gz). That's already overcomplicated.
It would seem wise to err on side of heavier compression and processing. bzImage has been used for a very long time, to decompress the kernel on older CPUs with less memory than is standard now, so it is likely pointless to go to a looser compression for decompression speed and lower memory consumption.
bzImage etc. only exist for bootloaders which could not load files bigger than a certain size, which was kind of a problem when the kernel and initrds were getting rather large. Sine you have to decompress the kernel to run it, bzImage doesn't save any memory at all (it just takes the bootloader longer to decompress it and bzip requires far more memory to decompress than gzip). Once it's decompressed, the original bzImage data is discarded and only the uncompressed kernel and (and compressed initrd data) are left in RAM when the kernel entry point is called.
Can that support live installion of new modules? Logically the initrd has been a minimal size on typical running systems, and should ideally be the minimal set of modules to get / (and /lib/modules) mounted. Then "all" of the modules are in 1 place, and the initrd only needs changing when a kernel update, or hardware change occurs.
As an example, you could build a standard initrd and then just append the modules to it with mksquashfs, or add config options to append a new module when a new kmp is installed which may be needed earlier in boot (for instance making sure a new raid controller is brought up). This is as easy with mksquashfs as it is with cpio because of the cpio format (iirc just concatenating cpio archives works), but mksquashfs is more flexible about it.
Installers may benefit from the extra layer of mounting a special purpose filesystem type, but if the kernel expects a cpio archive in RAM, and not the SqaushFS format, then you gain nothing.
Squashfs patch makes the kernel expect Squashfs as well as a bunch of other formats (cramfs, ext2 etc.) besides cpio that the kernel already supports. mkinitrd (or dracut) gets run in the kernel-flavor.arch.rpm post.sh script on INSTALLATION, which means basically you can tell when and if the right filesystem is available in the kernel (this decision is left up to the kernel maintainers who know the kernel version and RPM release the support started, and can check for it - hell, there's not even a need for that, CONFIG_SQUASHFS=y in /boot/config-VERSION-FLAVOR is enough notice that you can support that format.
One reason perhaps for use of cpio, was that the format was absolutely stably fixed. If SquashFS got used in the suggested way, then it's format needs to be similarly stable, with a mature tool chain.
Phillip Lougher is doing this right now which is why squashfs is going into mainline in 10.. 9.. 8... The reason I bring it up is because it will be in mainline, it will be available in every kernel-source after that without a patch, the format has to be stable at that point (4.0) and basically it is a better solution in a bunch of ways, some huge, some not so huge, some may even be negative, but it is in common use, it is a flexible format and a real filesystem (I remember when all our initrds were cramfs..) and has many other benefits outside of initrds (Linuxrc already relies on it for the root filesystem, common/rescue/etc. in the installer) and would provide a common filesystem compression format to be used across the whole distribution. If they ever manage to support LZMA in squashfs (which requires support in the kernel first, which people seem to be dilly-dallying about right now), then there is no reason why the DVD images (gnome/kde/x11/base) can't be compressed in it either. -- Matt Sealey -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
The problem that seems to be overlooked is that, SquashFS is only just
making it into the mainline kernel. initrd's are from kernel hacker
point of you, non-broken and functioning perfectly satisfactorily
using the cpio/gzip format.
So to utilise a SquashFS format initrd, requires this kernel change.
A distro version which then uses this format, is not going to be able
to support booting with older kernels, hence unsuitable for kernel
hackers and support, who may very well need to do such things, unless
the support is backported into their "stable" kernels.
It seems unlikely to me, that a initrd unification project could make
changes that might prove contraversial, more they're going to need to
build consensus, minimising changes that ppl may object to.
So perhaps a FUSE based initrd admin tool, or unpack and repack
scripts, is a more feasible solution to the difficulties disovered
with 11.1 mkinitrd for the sysadmin.
2009/1/15 Matt Sealey
Rob OpenSuSE wrote:
2009/1/14 Matt Sealey
:
Nobody is suggesting that this be a solution, although it IS the solution used for mkinstallinitrd in installation-images - it makes a cpio.gz initrd and stuffs in squashfs files full of modules and other stuff, and loop mounts them via Linuxrc (after loading the squashfs module and loop module which are the only ones in the cpio.gz).
That's already overcomplicated.
Agreed.
It would seem wise to err on side of heavier compression and processing. bzImage has been used for a very long time, to decompress the kernel on older CPUs with less memory than is standard now, so it is likely pointless to go to a looser compression for decompression speed and lower memory consumption.
bzImage etc. only exist for bootloaders which could not load files bigger than a certain size, which was kind of a problem when the kernel and initrds were getting rather large.
It was also nice, because you could boot a kernel off a floppy disk :)
Sine you have to decompress the kernel to run it, bzImage doesn't save any memory at all (it just takes the bootloader longer to decompress it and bzip requires far more memory to decompress than gzip). Once it's decompressed, the original bzImage data is discarded and only the uncompressed kernel and (and compressed initrd data) are left in RAM when the kernel entry point is called.
True, but gzip is the kernel decrompession and uses less memory than LZMA, and CPU. It's been used for a long time, so should not upset folk with embedded class low power hardware, whereas LZMA might be a problem on older CPUs. The "less memory" referred to systems where gzip method was successfully used, not the effect of using bzImage or gzip for initrd. We agree again there. The boot loader doesn't decompress the kernel, as far as I understand, it's the kernel that does that, effectively unpacking itself into it's final position in memory. That feature allowed the kernel to work with very simple minimal boot loaders on things like floppies.
Can that support live installion of new modules? Logically the initrd has been a minimal size on typical running systems, and should ideally be the minimal set of modules to get / (and /lib/modules) mounted. Then "all" of the modules are in 1 place, and the initrd only needs changing when a kernel update, or hardware change occurs.
As an example, you could build a standard initrd and then just append the modules to it with mksquashfs, or add config options to append a new module when a new kmp is installed which may be needed earlier in boot (for instance making sure a new raid controller is brought up). This is as easy with mksquashfs as it is with cpio because of the cpio format (iirc just concatenating cpio archives works), but mksquashfs is more flexible about
The current mkinitrd wraps up stuff in a non obvious way. I suspect FUSE, might provide a suitable alternative, to permit changes to the generated initrd, where udev's copying of currently booted system. But I much prefer the idea of being able to mount the initrd, like an iso file and make changes, to current unpack, and repack scheme. The real point is, the current mkinitrd in 11.1, has a flaw through using udev, which fixed other issues (not ones that are clear to me) that moving to a mountable format might ameliorate. It's a Pro reason for Novell/SuSE to support the SquashFS initrd idea, when Dave Jones is seeking input. Though I guess I could email him, pointing out the bug report involved, I am sure there are other Pro/Con that I am not fully party to.
Installers may benefit from the extra layer of mounting a special purpose filesystem type, but if the kernel expects a cpio archive in RAM, and not the SqaushFS format, then you gain nothing.
Squashfs patch makes the kernel expect Squashfs as well as a bunch of other formats (cramfs, ext2 etc.) besides cpio that the kernel already supports. mkinitrd (or dracut) gets run in the kernel-flavor.arch.rpm post.sh script on INSTALLATION, which means basically you can tell when and if the right filesystem is available in the kernel (this decision is left up to the kernel maintainers who know the kernel version and RPM release the support started, and can check for it - hell, there's not even a need for that, CONFIG_SQUASHFS=y in /boot/config-VERSION-FLAVOR is enough notice that you can support that format.
Great to make that point clear. But it means SqaushFS must be compiled in, and not a loadable module, like most filesystems.
One reason perhaps for use of cpio, was that the format was absolutely stably fixed. If SquashFS got used in the suggested way, then it's format needs to be similarly stable, with a mature tool chain.
Phillip Lougher is doing this right now which is why squashfs is going into mainline in 10.. 9.. 8...
The reason I bring it up is because it will be in mainline, it will be available in every kernel-source after that without a patch, the format has to be stable at that point (4.0) and basically it is a better solution in a bunch of ways, some huge, some not so huge, some may even be negative, but it is in common use, it is a flexible format and a real filesystem (I remember when all our initrds were cramfs..) and has many other benefits outside of initrds (Linuxrc already relies on it for the root filesystem, common/rescue/etc. in the installer) and would provide a common filesystem compression format to be used across the whole distribution.
If they ever manage to support LZMA in squashfs (which requires support in the kernel first, which people seem to be dilly-dallying about right now), then there is no reason why the DVD images (gnome/kde/x11/base) can't be compressed in it either.
Now that the SquashFS has been accepted, it'll be easier to add kernel support for LZMA, and then LZMA patches to SquashFS, if the change has technical merit and meets user requirements. Modern systems would likely benefit, without seeing the downsides, that low powered hardware might experience. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Thu, Jan 15, 2009 at 8:53 AM, Rob OpenSuSE
The problem that seems to be overlooked is that, SquashFS is only just making it into the mainline kernel. initrd's are from kernel hacker point of you, non-broken and functioning perfectly satisfactorily using the cpio/gzip format.
Squashfs support has been built in and built from every kernel package I've seen on SUSE since somewhere around version 8. It is not like it is not ready on peoples' systems *right now*. Only vanilla kernels won't see it, which may cause some problem for people who love to use vanilla kernels, but that gets fixed when squashfs is mainlined.
So to utilise a SquashFS format initrd, requires this kernel change.
A tiny one.. Check your /lib/modules directory, check the patches in kernel-source (read series.conf or so). The patch is there. The module is built. The only change is an "m" to a "y"..
A distro version which then uses this format, is not going to be able to support booting with older kernels, hence unsuitable for kernel hackers and support, who may very well need to do such things, unless the support is backported into their "stable" kernels.
mkinitrd is run at kernel install time, so it can pick and choose which filesystem to use dependant on the kernel configuration in use, versioning, and a bunch of other ways.
It seems unlikely to me, that a initrd unification project could make changes that might prove contraversial, more they're going to need to build consensus, minimising changes that ppl may object to.
So perhaps a FUSE based initrd admin tool, or unpack and repack scripts, is a more feasible solution to the difficulties disovered with 11.1 mkinitrd for the sysadmin.
The scripts are already there.. I don't remember any PROBLEMS for the sysadmin with the current system, it just seems opportune to move to consolidate all the root filesystem/installer filesystems and initramfs filesystems to a common format and further consolidate the scripts that build them - after all, they are all just fancy ways of collecting a list of files, stuffing them in a directory and making an archive. At the moment there are several tools and several packages and a lot of weird Perl scripts and shell scripts mixed in to do most of what dracut intends to do, just with a different file list. Another idea in the dracut wiki page is of course, to detect drivers at runtime instead of at buildtime - rather than pick drivers out from the running system, to load all the drivers into an initrd and pick the ones required. This increases the size of the initrd but it means a common initrd works on every system which runs the same kernel. With cpio archives, this is decompressed into ramfs/tmpfs or similar after the main kernel initialization is done, which leaves a copy of every possible kernel module in RAM, potentially 'forever'. Since the cpio is decompressed you basically have this glob of modules living there uncompressed. Loading them basically doubles the footprint of every loaded module. And not to mention, the footprint of every unloaded module stays around. A compressed filesystem such as squashfs basically eliminates some of that effect by not requiring the decompression step.
True, but gzip is the kernel decrompession and uses less memory than LZMA, and CPU
You're not doing anything else at the point that cpio archive is decompressed. CPU usage at that stage is moot. The fact is that LZMA uses less memory and has faster decompression than gzip. This is also moot, though, since lzma compressed cpio is not supported (and won't be until LZMA is in the kernel, which is also stopping squashfs from supporting LZMA)
folk with embedded class low power hardware, whereas LZMA might be a problem on older CPUs.
Not at all. The problem is that LZMA increases memory usage on COMPRESSION (just doing a 40MB tarball now, top is telling me %MEM for lzma is 50.9 (and it's taken out my disk buffers to do it) - for gzip it's 0.9. Decompression is much lighter - it's directly related to the block size used so for a 128KiB block, it's around 128KiB. For compression methods like LZO the decompression memory is 64KiB. Compression is not as good (actually it's very similar to gzip give or take 10-20% depending entirely on the data) but decompression is lighter and faster than gzip.
The current mkinitrd wraps up stuff in a non obvious way. I suspect FUSE, might provide a suitable alternative, to permit changes to the generated initrd, where udev's copying of currently booted system.
I don't think fuse works any better than loop mounting a squashfs here except that squashfs is read only and a fuse filesystem may be writable.. but this means the kernel needs to read this fuse-based filesystem... not a good plan.
But I much prefer the idea of being able to mount the initrd, like an iso file and make changes, to current unpack, and repack scheme.
mksquashfs, unsquashfs.
The real point is, the current mkinitrd in 11.1, has a flaw through using udev, which fixed other issues (not ones that are clear to me) that moving to a mountable format might ameliorate.
It's a Pro reason for Novell/SuSE to support the SquashFS initrd idea, when Dave Jones is seeking input. Though I guess I could email him, pointing out the bug report involved, I am sure there are other Pro/Con that I am not fully party to.
Bug report?
Great to make that point clear. But it means SqaushFS must be compiled in, and not a loadable module, like most filesystems.
You change one byte in config/arch/* config files in kernel-source and suddenly everyone has support. The patch is already there.
Now that the SquashFS has been accepted, it'll be easier to add kernel support for LZMA
Squashfs has no impact on the acceptance of LZMA in the kernel.
and then LZMA patches to SquashFS, if the change has technical merit and meets user requirements.
Modern systems would likely benefit, without seeing the downsides, that low powered hardware might experience.
I don't think there would be any downsides but let's stop talking
about LZMA now. If the kernel supports LZMA it will also support
cpio-compressed LZMA archives (and lzImage boot images :) so the
advantages are basically only really the extra flexibility of squashfs
being a real compressed filesystem.
--
Matt Sealey
2009/1/15 Matt Sealey
On Thu, Jan 15, 2009 at 8:53 AM, Rob OpenSuSE
wrote: The problem that seems to be overlooked is that, SquashFS is only just making it into the mainline kernel. initrd's are from kernel hacker point of you, non-broken and functioning perfectly satisfactorily using the cpio/gzip format.
Squashfs support has been built in and built from every kernel package I've seen on SUSE since somewhere around version 8.
It is not like it is not ready on peoples' systems *right now*. Only vanilla kernels won't see it, which may cause some problem for people who love to use vanilla kernels, but that gets fixed when squashfs is mainlined.
The linux kernel team "kernel hacker" use all kinds of distro's not just SuSE, and as Greg KH said "I think the goal with mkinitrd is to work with the community to come up with a common tool that all the distros use. Dave Jones is currently working on a first cut at this, and then we will all get together to go from there. If it happens to use squashfs instead of cpio, well, that will be up to the community, not just us." So the questions he asked, wanting numbers matter, as does that objection, the advocates of change need to be provide evidence. There are end user benefits to mountable initrd's, and would have been a much better workround for one bug, than what I actually had to do. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Thu, Jan 15, 2009 at 10:29 AM, Rob OpenSuSE
2009/1/15 Matt Sealey
: On Thu, Jan 15, 2009 at 8:53 AM, Rob OpenSuSE
wrote: The problem that seems to be overlooked is that, SquashFS is only just making it into the mainline kernel. initrd's are from kernel hacker point of you, non-broken and functioning perfectly satisfactorily using the cpio/gzip format.
Squashfs support has been built in and built from every kernel package I've seen on SUSE since somewhere around version 8.
It is not like it is not ready on peoples' systems *right now*. Only vanilla kernels won't see it, which may cause some problem for people who love to use vanilla kernels, but that gets fixed when squashfs is mainlined.
The linux kernel team "kernel hacker" use all kinds of distro's not just SuSE, and as Greg KH said
This is an openSUSE mailing list. I'm not talking about all kinds of other distros. Although if I was I'd note that a bunch of them do use squashfs for initrd (and root filesystems unioned under hard disks) It's only Fedora and Debian which don't but this is most of a factor of "we've been using mkinitramfs since 1997 anyway, and didn't want to change it to a system which needed a kernel patch". Time has changed! No patch required for modern kernels. And the current kernels are already patched anyway. If you loop mount a squashfs kernel, too, you don't need any special tools to extract files from it - this is how the LiveCD installers work. You just copy the files to the target.
So the questions he asked, wanting numbers matter, as does that objection, the advocates of change need to be provide evidence.
There are end user benefits to mountable initrd's, and would have been a much better workround for one bug, than what I actually had to do.
I don't think mountable initrds is the point (there are very few
situations where you need to mount it), it's the CONSOLIDATION of
cpio/squashfs as currently used in initrd/roots on the LiveCDs and
potentially the DVD images for the standard installation, to use a
single build system and a single (loop mountable) filesystem which
offers several benefits.
You're simply not paying attention to what is going on in the
discussion. I don't know why you are talking about "sysadmin
problems", because you never qualified it. Or mountable initrds (I was
talking about the fact that the installer ALREADY uses squashfs files
INSIDE cpio initrds). They aren't the plus points I'm trying to push
here.
--
Matt Sealey
2009/1/15 Matt Sealey
On Thu, Jan 15, 2009 at 10:29 AM, Rob OpenSuSE
The linux kernel team "kernel hacker" use all kinds of distro's not just SuSE, and as Greg KH said
This is an openSUSE mailing list. I'm not talking about all kinds of other distros. Although if I was I'd note that a bunch of them do use squashfs for initrd (and root filesystems unioned under hard disks)
Ignoring other distributions, and current work in progress is not going to benefit the discussion.
It's only Fedora and Debian which don't but this is most of a factor
Their needs are going to be a consideration in a joint project.
If you loop mount a squashfs kernel, too, you don't need any special tools to extract files from it - this is how the LiveCD installers work. You just copy the files to the target.
That feature is a user benefit, that will IMO ease support in cases similar to problem I found.
So the questions he asked, wanting numbers matter, as does that objection, the advocates of change need to be provide evidence.
There are end user benefits to mountable initrd's, and would have been a much better workround for one bug, than what I actually had to do.
I don't think mountable initrds is the point (there are very few situations where you need to mount it), it's the CONSOLIDATION of cpio/squashfs as currently used in initrd/roots on the LiveCDs and potentially the DVD images for the standard installation, to use a single build system and a single (loop mountable) filesystem which offers several benefits.
You're simply not paying attention to what is going on in the discussion. I don't know why you are talking about "sysadmin problems", because you never qualified it. Or mountable initrds (I was talking about the fact that the installer ALREADY uses squashfs files INSIDE cpio initrds). They aren't the plus points I'm trying to push here
That's rich. I actually emailed in a request relating out of a specific bug report. The exact details are not particularly relevant to the discussion, just supporting the support benefits of being able to fix up, initrd's conveniently rather than suffer the consequences, of mkinitrd utilising the currently booted system. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Greg KH wrote:
On Wed, Jan 14, 2009 at 03:12:32PM -0600, Matt Sealey wrote:
I just noticed that while squashfs 3.4 kernel module is used on the -default kernel, the squashfs tools on both PowerPC and x86 repos are stuck at 3.2
neko@frodo:~> mksquashfs -version mksquashfs version 3.2-r2 (2007/01/15)
Is there any particular reason for this? Is there an update planned?
Someone needs to kick the owner of the userspace squashfs tools to get them to update them. If you could be so kind as to file a bug against the package, that would help.
https://bugzilla.novell.com/show_bug.cgi?id=466228 Done :D -- Matt -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
participants (3)
-
Greg KH
-
Matt Sealey
-
Rob OpenSuSE