Re: [opensuse-kernel] Squashfs
2009/1/15 Jeff Mahoney <jeffm@suse.de>:
Rob OpenSuSE wrote:
2009/1/15 Matt Sealey <matt@genesi-usa.com>:
On Thu, Jan 15, 2009 at 8:53 AM, Rob OpenSuSE <rob.opensuse.linux@googlemail.com> wrote:
There are end user benefits to mountable initrd's, and would have been a much better workround for one bug, than what I actually had to do.
Actually you would have been just as stuck with squashfs. It's a read-only file system. You can't loopback mount it and change it. In the end, you end up having to do the same thing you'd have to do with cpio - you just replace "zcat initrd | cpio -id" with "mount -o loop".
If it's RO and you can't alter it, then there wouldn't seem to be any end user benefit apart from casual browsing, to overcome the "if it ain't broke don't fix it". Just developer benefit through supporting fewer formats. This is what led me to believe, SquashFS was permitting easier modification, I think I misunderstood that. 2009/1/14 Matt Sealey <matt@genesi-usa.com>:
What advantages do you know of?
Appending files is a breeze compared to a cpio archive. Replacing the same file in a cpio usually means dumping and remaking it. mksquashfs can get rid of the original file and relink the new one at the end.. it would mean you can do clever tricks like building half the initrd or filesystem image, doing something else based on that result, and then appending further processed data.
Though mounting, copying to disk, and then re-building the formatted file, is more like making iso files. "zcat initrd | cpio -id" and then rebuilding, wasn't obvious enough to get suggested when I had an issue with replacing the driver with mkinintrd on a live system. It is still a nice to have, as udev generating it's own module list, to be combined with INITRD_MODULES, doesn't make the change methodology clear to the sysadmin. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On Thu, Jan 15, 2009 at 11:12 AM, Rob OpenSuSE <rob.opensuse.linux@googlemail.com> wrote:
2009/1/15 Jeff Mahoney <jeffm@suse.de>:
Rob OpenSuSE wrote:
2009/1/15 Matt Sealey <matt@genesi-usa.com>:
On Thu, Jan 15, 2009 at 8:53 AM, Rob OpenSuSE <rob.opensuse.linux@googlemail.com> wrote:
There are end user benefits to mountable initrd's, and would have been a much better workround for one bug, than what I actually had to do.
Actually you would have been just as stuck with squashfs. It's a read-only file system. You can't loopback mount it and change it. In the end, you end up having to do the same thing you'd have to do with cpio - you just replace "zcat initrd | cpio -id" with "mount -o loop".
If it's RO and you can't alter it, then there wouldn't seem to be any end user benefit apart from casual browsing, to overcome the "if it ain't broke don't fix it". Just developer benefit through supporting fewer formats.
This is what led me to believe, SquashFS was permitting easier modification, I think I misunderstood that.
Well, the extract-add-archive step is reduced to add. You don't need to extract a squashfs archive to add anything to it - mksquashfs will append files by default, although this can come at the cost of reduced compression (since it just adds extra inodes, metadata and files at the end of the current one). So appending is basically "mksquashfs source/ destination.squashfs" instead of "gunzip -c cpio.gz | cpio -idmv" then copying in new files and "cpio -idmv | gzip -c >cpio.gz" kind of stuff. Actually mounting the file is something done by Linuxrc or so during the installer - in the current case, boot/arch/initrd32 or so is a cpio.gz archive with several squashfs files contained in it. the cpio.gz archive contains ONLY squashfs.ko and loop.ko - enough to mount these squashfs files. One of the squashfs files is the lib/modules directory (which has all the modules from the different `uname -r`/kernel/* directories flattened into a single directory which is linked into the current root /lib/modules/`uname-r` as "initrd -> /mounts/00_lib/" or something similar. There are several others in there which get mounted by some means or another by Linuxrc. It's quirky, seems to me to be overly complex to be honest. I'm not sure why you would want the modules seperate in another filesystem (ease of updating? I dunno.. mkinstallinitrd does this by using a list of interesting modules from installation-images/etc), but anyway. This is another discussion. The point is squashfs gets used during the installer, and from installation-images to build the installer, and is part of the kernel-source default configuration (currently as a module) on all architectures anyway through the usual patch process. So it's in good use right now.
2009/1/14 Matt Sealey <matt@genesi-usa.com>:
Though mounting, copying to disk, and then re-building the formatted file, is more like making iso files.
Not so. Rob, again, you've overcomplicated it by looking at it from your skewed view of things, and not what I actually said or what is under discussion.
"zcat initrd | cpio -id" and then rebuilding, wasn't obvious enough to get suggested when I had an issue with replacing the driver with mkinintrd on a live system.
You can't replace a file in the initrd on a live filesystem that's *mounted*. If you want to add files to the initrd, grab your kernel module name, add it to the list in /etc/sysconfig/kernel INITRD_MODULES like you said below, and run mkinitrd again (or if you're being lazy, --force reinstall your kernel package).
It is still a nice to have, as udev generating it's own module list, to be combined with INITRD_MODULES, doesn't make the change methodology clear to the sysadmin.
udev captures the modules "coldplugged" by the system (usually USB and Firewire) and this gets added to the list in INITRD_MODULES for the initrd. INITRD_MODULES is basically the list of modules that *has* to be loaded to get to mount the disk that udev is on (plus anything else you care to add). This is usually exactly your libata driver and your root filesystem module (on my system, "pata_mpc52xx xfs", on my Pegasos, "pata_via", on my VirtualBox systems, "ahci") The "problem" with adding stuff to the initrd is, is not a problem at all. Apart from the fact that INITRD_MODULES is generated during the install? If it installed the wrong driver (via82cxxx for example) for the disk then it glues that in there and it's not that easy to get it to automatically fix it. -- Matt Sealey <matt@genesi-usa.com> Genesi, Manager, Developer Relations -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
2009/1/15 Matt Sealey <matt@genesi-usa.com>:
So appending is basically "mksquashfs source/ destination.squashfs" instead of "gunzip -c cpio.gz | cpio -idmv" then copying in new files and "cpio -idmv | gzip -c >cpio.gz" kind of stuff.
That would make a simple add easier, more frequent probably would be delete & add files though.
2009/1/14 Matt Sealey <matt@genesi-usa.com>:
Though mounting, copying to disk, and then re-building the formatted file, is more like making iso files.
Not so. Rob, again, you've overcomplicated it by looking at it from your skewed view of things, and not what I actually said or what is under discussion.
"zcat initrd | cpio -id" and then rebuilding, wasn't obvious enough to get suggested when I had an issue with replacing the driver with mkinintrd on a live system.
You can't replace a file in the initrd on a live filesystem that's *mounted*. If you want to add files to the initrd, grab your kernel module name, add it to the list in /etc/sysconfig/kernel INITRD_MODULES like you said below, and run mkinitrd again (or if
Well Matt, if you read back, you'll find as explained, doing that did not work.
udev captures the modules "coldplugged" by the system (usually USB and Firewire) and this gets added to the list in INITRD_MODULES for the initrd. INITRD_MODULES is basically the list of modules that *has* to be loaded to get to mount the disk that udev is on (plus anything else you care to add). This is usually exactly your libata driver and your root filesystem module (on my system, "pata_mpc52xx xfs", on my Pegasos, "pata_via", on my VirtualBox systems, "ahci")
That was not the case. INITRD_MODULES was not containing the expected information, and replacing the root filesystem driver, from the udev list was not possible. The broken driver was always loaded into the initrd. If INITRD_MODULES was working like that, as it used to pre-11.1 I don't think I'd have had an issue. It would have been a simple edit, and re-run mkinitrd.
The "problem" with adding stuff to the initrd is, is not a problem at all. Apart from the fact that INITRD_MODULES is generated during the install? If it installed the wrong driver (via82cxxx for example) for the disk then it glues that in there and it's not that easy to get it to automatically fix it.
It's actually removing things, as well as adding what you actually want, that appears necessary. As it stood, running mkinitrd by default updates for all kernels and initrd, which can caused the system to become unbootable, when attempting to force usage of the pdc202xx_old driver. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
udev captures the modules "coldplugged" by the system (usually USB and Firewire) and this gets added to the list in INITRD_MODULES for the initrd. INITRD_MODULES is basically the list of modules that *has* to be loaded to get to mount the disk that udev is on (plus anything else you care to add). This is usually exactly your libata driver and your root filesystem module (on my system, "pata_mpc52xx xfs", on my Pegasos, "pata_via", on my VirtualBox systems, "ahci")
That was not the case. INITRD_MODULES was not containing the expected information
It doesn't get updated when you run mkinitrd - what I meant was, INITRD_MODULES is taken, then udev is poked to give up a list of loaded modules, which gets combined and passed to mkinitrd. INITRD_MODULES is fairly static as far as I have ever seen.. nothing has ever automatically updated it. The sole purpose of INITRD_MODULES is to load in drivers which CANNOT be coldplugged by udev because they have to be loaded to get to udev in the first place. Since udev can't know about them it can't be relied on to suggest them. There is no way to stop udev from doing this except to blacklist the driver from loading, and then udev won't bother (since it's not loaded at the time of mkinitrd, it will not add it to the list of coldplugged devices). If a broken driver is still being loaded, brokenmodules= on the kernel command line is your first step.
If INITRD_MODULES was working like that, as it used to pre-11.1 I don't think I'd have had an issue. It would have been a simple edit, and re-run mkinitrd.
Which works fine here. I don't see it on MY systems :D
The "problem" with adding stuff to the initrd is, is not a problem at all. Apart from the fact that INITRD_MODULES is generated during the install? If it installed the wrong driver (via82cxxx for example) for the disk then it glues that in there and it's not that easy to get it to automatically fix it.
It's actually removing things, as well as adding what you actually want, that appears necessary. As it stood, running mkinitrd by default updates for all kernels and initrd, which can caused the system to become unbootable, when attempting to force usage of the pdc202xx_old driver.
brokenmodules=pdc202xx_old, remove it from INITRD_MODULES if it's there, and blacklist it in /etc/modprobe.d/blacklist, then rebewt.. that's the real failsafe way (that way it can never be loaded from the initrd, it will never be put in the initrd next time it's built because neither the initrd init nor udev will load it, and if you ever try, modprobe is going to whine at you). Now make a new initrd and check it and rebewt again. Once you have a working initrd without it, and once you have it blacklisted from modprobe, you can remove the brokenmodules= line. It would be nicer to consolidate the stuff, that's for sure. There are no such problems with this stuff on FreeBSD etc. where the development is far more strict around kernel development. -- Matt Sealey <matt@genesi-usa.com> Genesi, Manager, Developer Relations -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
2009/1/15 Matt Sealey <matt@genesi-usa.com>:
udev captures the modules "coldplugged" by the system (usually USB and Firewire) and this gets added to the list in INITRD_MODULES for the initrd. INITRD_MODULES is basically the list of modules that *has* to be loaded to get to mount the disk that udev is on (plus anything else you care to add). This is usually exactly your libata driver and your root filesystem module (on my system, "pata_mpc52xx xfs", on my Pegasos, "pata_via", on my VirtualBox systems, "ahci")
That was not the case. INITRD_MODULES was not containing the expected information
It doesn't get updated when you run mkinitrd - what I meant was, INITRD_MODULES is taken, then udev is poked to give up a list of loaded modules, which gets combined and passed to mkinitrd. INITRD_MODULES is fairly static as far as I have ever seen.. nothing has ever automatically updated it. The sole purpose of INITRD_MODULES is to load in drivers which CANNOT be coldplugged by udev because they have to be loaded to get to udev in the first place. Since udev can't know about them it can't be relied on to suggest them.
There is no way to stop udev from doing this except to blacklist the driver from loading, and then udev won't bother (since it's not loaded at the time of mkinitrd, it will not add it to the list of coldplugged devices).
If a broken driver is still being loaded, brokenmodules= on the kernel command line is your first step.
Didn't work! It didn't work with Live CD installs either. broken_modules=pata_pdc202xx_old does work with Net Install and DVD install methods.
brokenmodules=pdc202xx_old, remove it from INITRD_MODULES if it's there, and blacklist it in /etc/modprobe.d/blacklist, then rebewt.. that's the real failsafe way (that way it can never be loaded from the initrd, it will never be put in the initrd next time it's built
I might be able to check that tomorrow, by disabling a pata_ module, despite being non-broken. It would appear to be a better workround if it does work. As I was including a kernel patch, for another problem on another system, I actually compiled in the driver I wanted, and left out the broken one, initially. Then later I confirmed the recommended workround via re-install with 11.1 after release.
because neither the initrd init nor udev will load it, and if you ever try, modprobe is going to whine at you). Now make a new initrd and check it and rebewt again. Once you have a working initrd without it, and once you have it blacklisted from modprobe, you can remove the brokenmodules= line.
It would be nicer to consolidate the stuff, that's for sure. There are no such problems with this stuff on FreeBSD etc. where the development is far more strict around kernel development.
The problem I had, was the lack of visibility into the process. In past, at some point during SuSE 7.x (I think) I was using a patched mkinitrd, until different changes were incorporated which supported my requirements. I couldn't see where in script INITRD_MODULES was used, and that was my first port of call, as INITRD_MODULES was the place to add modules for RAID and/or LVM support in the past. Being able to mount the initrd and dig around, would I think have helped the situation. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (2)
-
Matt Sealey -
Rob OpenSuSE