[opensuse-kernel] kernel/initrd loading delays
Felix Miata composed on 2015-10-12 12:29 (UTC-0400): (originally to initramfs@vger.kernel.org) https://www.mail-archive.com/initramfs@vger.kernel.org/msg04168.html
On more and more installations since most distros have replaced mkinitrd with dracut, on selecting a bootloader menu choice the "root (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd" message stays on screen 40 seconds or more before the time-stamped boot messages begin scrolling the screen. IIRC, these delays first began appearing last January or February, and appear on multiple machines. All kernel and initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all with native 512b sector size, manufactured in 2011 or earlier). All installations are configured with static IP. All have Plymouth either not installed, or are booted with noplymouth included on kernel cmdline.
e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1 (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is about 80 seconds following that 40 second delay, with last time stamp showing 36.844484.
Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964; initramfs-tools 0.120) delays start of boot messages nearly 80 seconds, reaching DM greeter ready nearly 140 seconds after stanza selection.
Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360) exhibits no perceptible delay, reaching multi-user.target shell prompt in under 47 seconds from boot stanza selection.
Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays start of boot messages about 40 seconds, reaching multi-user.target bash prompt about 90 seconds after stanza selection.
Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no delay, reaching multi-user.target bash prompt in about 45 seconds.
Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits no delay, reaching DM greeter in less than about 45 seconds.
Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches multi-user.target shell prompt in about 75 seconds after unknown kernel/initrd delay (puts display into sleep mode until boot messages appear).
Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting (second, on sda28, vs. other on sda23) openSUSE Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay, reaching multi-user.target bash prompt in about 100 seconds.
Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608) exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90 seconds.
Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no delay, reaching multi-user.target bash prompt in less than 50 seconds.
Any ideas what could cause these delays, or how to eliminate them?
Booting host hpg33 openSUSE Tumbleweed 20160307 64 bit kernel-default-4.4.3, 6194336 bytes, and initrd 38456340 bytes, just took 9.4 minutes for the last Grub message to clear, and extended that to ~10.2 minutes to reach a shell prompt. Prior 4.3.3-5 kernel-default's initrd is 9589212 bytes, 25% the size of the current. 4.3.3-5 took 2 2/3 minutes for the Grub message to clear, about 2/3 more minute to reach shell prompt, and 4.3.3-1 about the same. 3rd prior, kernel-desktop-4.2.1 cleared the Grub message in too few seconds to count, well under a minute to reach shell prompt. kernel-default-4.5.0 cleared the Grub in 130 seconds, reached shell prompt at 174, has 8664600 byte initrd, smallest since 4.1.6's 8663972. Mount reported /dev/mapper/* names with 4.4.3 and 4.5.0, while 4.2.1 and 4.3.3 reported device names. What's taking so long booting with kernel-default? P.S. Are long-winded /dev/mapper/* mount reports a new normal? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Felix Miata composed on 2016-03-22 04:57 (UTC-0400):
Felix Miata composed on 2015-10-12 12:29 (UTC-0400):
(originally to initramfs@vger.kernel.org) https://www.mail-archive.com/initramfs@vger.kernel.org/msg04168.html
On more and more installations since most distros have replaced mkinitrd with dracut, on selecting a bootloader menu choice the "root (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd" message stays on screen 40 seconds or more before the time-stamped boot messages begin scrolling the screen. IIRC, these delays first began appearing last January or February, and appear on multiple machines. All kernel and initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all with native 512b sector size, manufactured in 2011 or earlier). All installations are configured with static IP. All have Plymouth either not installed, or are booted with noplymouth included on kernel cmdline.
e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1 (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is about 80 seconds following that 40 second delay, with last time stamp showing 36.844484.
Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964; initramfs-tools 0.120) delays start of boot messages nearly 80 seconds, reaching DM greeter ready nearly 140 seconds after stanza selection.
Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360) exhibits no perceptible delay, reaching multi-user.target shell prompt in under 47 seconds from boot stanza selection.
Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays start of boot messages about 40 seconds, reaching multi-user.target bash prompt about 90 seconds after stanza selection.
Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no delay, reaching multi-user.target bash prompt in about 45 seconds.
Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits no delay, reaching DM greeter in less than about 45 seconds.
Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches multi-user.target shell prompt in about 75 seconds after unknown kernel/initrd delay (puts display into sleep mode until boot messages appear).
Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting (second, on sda28, vs. other on sda23) openSUSE Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay, reaching multi-user.target bash prompt in about 100 seconds.
Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608) exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90 seconds.
Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no delay, reaching multi-user.target bash prompt in less than 50 seconds.
Any ideas what could cause these delays, or how to eliminate them?
Booting host hpg33 openSUSE Tumbleweed 20160307 64 bit kernel-default-4.4.3, 6194336 bytes, and initrd 38456340 bytes, just took 9.4 minutes for the last Grub message to clear, and extended that to ~10.2 minutes to reach a shell prompt. Prior 4.3.3-5 kernel-default's initrd is 9589212 bytes, 25% the size of the current. 4.3.3-5 took 2 2/3 minutes for the Grub message to clear, about 2/3 more minute to reach shell prompt, and 4.3.3-1 about the same. 3rd prior, kernel-desktop-4.2.1 cleared the Grub message in too few seconds to count, well under a minute to reach shell prompt. kernel-default-4.5.0 cleared the Grub in 130 seconds, reached shell prompt at 174, has 8664600 byte initrd, smallest since 4.1.6's 8663972.
Mount reported /dev/mapper/* names with 4.4.3 and 4.5.0, while 4.2.1 and 4.3.3 reported device names.
What's taking so long booting with kernel-default?
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
Felix, I wonder one thing, could you just check if any of them patch the cpu on boot. If patching occur, a delay will be shown. -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Bruno Friedmann composed on 2016-03-23 06:47 (UTC+0100):
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
Felix, I wonder one thing, could you just check if any of them patch the cpu on boot. If patching occur, a delay will be shown.
What does "patch the cpu on boot" mean? How would I check? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On Wed, Mar 23, 2016 at 9:39 AM, Felix Miata
Bruno Friedmann composed on 2016-03-23 06:47 (UTC+0100):
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
Felix, I wonder one thing, could you just check if any of them patch the cpu on boot. If patching occur, a delay will be shown.
What does "patch the cpu on boot" mean? How would I check?
dmesg should contain something like microcode updated early to revision 0x%x, date = %04x-%02x-%02x\n -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Andrei Borzenkov composed on 2016-03-23 09:53 (UTC+0300):
Felix Miata wrote:
Bruno Friedmann composed on 2016-03-23 06:47 (UTC+0100):
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
Felix, I wonder one thing, could you just check if any of them patch the cpu on boot. If patching occur, a delay will be shown.
What does "patch the cpu on boot" mean? How would I check?
dmesg should contain something like
microcode updated early to revision 0x%x, date = %04x-%02x-%02x\n
204 seconds for Grub's initrd message to clear, 235 seconds to see
a shell prompt with 4.5.0 on TW host big 41.
[ 0.000000] microcode: CPU0 microcode updated early to revision 0xa0b, date = 2010-09-28
[ 0.008000] microcode: CPU1 microcode updated early to revision 0xa0b, date = 2010-09-28
[ 0.866318] microcode: CPU0 sig=0x1067a, pf=0x1, revision=0xa0b
[ 0.866459] microcode: CPU1 sig=0x1067a, pf=0x1, revision=0xa0b
[ 0.866624] microcode: Microcode Update Driver: v2.01
Felix Miata wrote:
Andrei Borzenkov composed on 2016-03-23 09:53 (UTC+0300):
Felix Miata wrote:
Bruno Friedmann composed on 2016-03-23 06:47 (UTC+0100):
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
Felix, I wonder one thing, could you just check if any of them patch the cpu on boot. If patching occur, a delay will be shown.
What does "patch the cpu on boot" mean? How would I check?
dmesg should contain something like
microcode updated early to revision 0x%x, date = %04x-%02x-%02x\n
204 seconds for Grub's initrd message to clear, 235 seconds to see a shell prompt with 4.5.0 on TW host big 41.
Hi Felix there's a lot of information here, too much to dig through. Have you managed to identify where the delays are happening? If it's not in the logs, it must be the initrd loading which you would presumably see. lilo has an option "compact" which merges reads of consecutive sectors into one read, I don't know if grub does that by default. -- Per Jessen, Zürich (10.8°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Per Jessen composed on 2016-04-01 10:13 (UTC+0200):
204 seconds for Grub's initrd message to clear, 235 seconds to see a shell prompt with 4.5.0 on TW host big 41.
Hi Felix
there's a lot of information here, too much to dig through. Have you managed to identify where the delays are happening? If it's not in the
Only one delay apparent each.
logs, it must be the initrd loading which you would presumably see.
You seem to be right. The delay is obviously at vmlinuz/initrd load time, as the Grub loading messages stay on screen for the entirety of the delay. The problem is finding a way to troubleshoot. Until this started last year, I had no idea this kind of problem existed. Googling has turning up nothing like it so far. Today I spent time cataloging which have the problem. It's very random as to which kernels/initrds, but so far it's limited to 3 HDs, one of which is a clone of one of the others, made to determine if the disk hardware could be where the fault lies. It's not the disk hardware. The clone has the same problem kernels/initrds, and they go with the disk when moved to a different machine. All problem disks are .5TB or less. Two are old enough that likelihood of 4k sectors is remote. The other, st3500411sv, is plenty new enough that it could have 4k sectors, but fdisk and spec sheet report 512b. So far, the problem has been exclusive to 64 bit installations on EXT4 / filesystems, with one exception. One 32 kernel/initrd has the problem, on the only 64 bit AMD machine I have, using a pair of PATA disks. All others with the problem have only a single HD, and all others are on Intel CPUs and single SATA disks. What may be a new clue is that the problem disks each have at least one installation on which a bunch of dm* modules load, and on which df and mount report partitions by /dev/mapper names instead of by device names. http://fm.no-ip.com/Tmp/Linux/Delays/ contains the day's data collection. big31-linux.txt gx62b-linux.txt hpg33-linux.txt Above list which kernels/initrds do or don't suffer the delays. The others contain the mount, df and lsmod outputs from several installations, among which are two that are outputting /dev/mapper/* instead of /dev/sd*, plus the partitioning logs. None of the problem disk installations include RAID or LVM. However, because I do so much cloning, imaging and hardware shuffling around here, I suspect it's possible some kind of dm or md info escaped somewhere onto the disks with the delays. I don't yet have any ideas how to locate such info if it exists. /proc/mdstat does not exist on any where I've looked for it. Suggestions and ideas welcome. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Am 02.04.2016 um 09:14 schrieb Felix Miata:
Per Jessen composed on 2016-04-01 10:13 (UTC+0200):
204 seconds for Grub's initrd message to clear, 235 seconds to see a shell prompt with 4.5.0 on TW host big 41.
Hi Felix
there's a lot of information here, too much to dig through. Have you managed to identify where the delays are happening? If it's not in the
Only one delay apparent each.
logs, it must be the initrd loading which you would presumably see.
You seem to be right. The delay is obviously at vmlinuz/initrd load time, as the Grub loading messages stay on screen for the entirety of the delay. The problem is finding a way to troubleshoot.
That's relatively easy (with grub2): Just edit /boot/grub2/grub.cfg (it will be overwritten by grub2-mkconfig). Find places like this: echo 'Loading Linux 4.5.0-4.g3d86af7-default ...' linux /boot/vmlinuz-4.5.0-4.g3d86af7-default root=/dev/sda1 quiet echo 'Loading initial ramdisk ...' initrd /boot/initrd-4.5.0-4.g3d86af7-default and add the line after "initrd": echo 'done! booting now.' Then you'll see this additional line before the kernel starts booting. If you also remove "quiet", then you'll probably see kernel messages earlier than you do now. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Stefan Seyfried composed on 2016-04-03 09:15 (UTC+0200):
Felix Miata Composed: ...
The delay is obviously at vmlinuz/initrd load time, as the Grub loading messages stay on screen for the entirety of the delay. The problem is finding a way to troubleshoot.
That's relatively easy (with grub2): Just edit /boot/grub2/grub.cfg (it will be overwritten by grub2-mkconfig). Find places like this:
echo 'Loading Linux 4.5.0-4.g3d86af7-default ...' linux /boot/vmlinuz-4.5.0-4.g3d86af7-default root=/dev/sda1 quiet echo 'Loading initial ramdisk ...' initrd /boot/initrd-4.5.0-4.g3d86af7-default
and add the line after "initrd":
echo 'done! booting now.'
Then you'll see this additional line before the kernel starts booting. If you also remove "quiet", then you'll probably see kernel messages earlier than you do now.
Good idea! I adapted your suggestion to the installation currently vexing my attempts to fix, host big31's 13.2, thus: title openSUSE 13.2 defkernel root (hd0,12) pause press any key to begin kernel /boot/vmlinuz <yada> pause post-kernel, pre-initrd initrd /boot/initrd pause post-initrd Between first and second pause took 85 seconds. Between second and third was too short to measure. So, Grub's apparently having a problem loading the 3.16.7-35 kernel (5686472 bytes), a problem which as noted up-thread disappears if moving the installation from EXT4 to EXT3. On same host, I tried same with TW: 90 seconds between first and second pauses, 100 seconds between second and third pauses (kernel 4.4.1-1; 6182432 bytes; initrd 6978588 bytes). :-( -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Felix Miata composed on 2016-03-22 22:10 (UTC-0400):
Felix Miata composed on 2016-03-22 04:57 (UTC-0400):
Felix Miata composed on 2015-10-12 12:29 (UTC-0400):
(originally to initramfs@vger.kernel.org) https://www.mail-archive.com/initramfs@vger.kernel.org/msg04168.html
On more and more installations since most distros have replaced mkinitrd with dracut, on selecting a bootloader menu choice the "root (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd" message stays on screen 40 seconds or more before the time-stamped boot messages begin scrolling the screen. IIRC, these delays first began appearing last January or February, and appear on multiple machines. All kernel and initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all with native 512b sector size, manufactured in 2011 or earlier). All installations are configured with static IP. All have Plymouth either not installed, or are booted with noplymouth included on kernel cmdline.
e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1 (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is about 80 seconds following that 40 second delay, with last time stamp showing 36.844484.
Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964; initramfs-tools 0.120) delays start of boot messages nearly 80 seconds, reaching DM greeter ready nearly 140 seconds after stanza selection.
Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360) exhibits no perceptible delay, reaching multi-user.target shell prompt in under 47 seconds from boot stanza selection.
Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays start of boot messages about 40 seconds, reaching multi-user.target bash prompt about 90 seconds after stanza selection.
Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no delay, reaching multi-user.target bash prompt in about 45 seconds.
Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits no delay, reaching DM greeter in less than about 45 seconds.
Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches multi-user.target shell prompt in about 75 seconds after unknown kernel/initrd delay (puts display into sleep mode until boot messages appear).
Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting (second, on sda28, vs. other on sda23) openSUSE Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay, reaching multi-user.target bash prompt in about 100 seconds.
Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608) exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90 seconds.
Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no delay, reaching multi-user.target bash prompt in less than 50 seconds.
Any ideas what could cause these delays, or how to eliminate them?
Booting host hpg33 openSUSE Tumbleweed 20160307 64 bit kernel-default-4.4.3, 6194336 bytes, and initrd 38456340 bytes, just took 9.4 minutes for the last Grub message to clear, and extended that to ~10.2 minutes to reach a shell prompt. Prior 4.3.3-5 kernel-default's initrd is 9589212 bytes, 25% the size of the current. 4.3.3-5 took 2 2/3 minutes for the Grub message to clear, about 2/3 more minute to reach shell prompt, and 4.3.3-1 about the same. 3rd prior, kernel-desktop-4.2.1 cleared the Grub message in too few seconds to count, well under a minute to reach shell prompt. kernel-default-4.5.0 cleared the Grub in 130 seconds, reached shell prompt at 174, has 8664600 byte initrd, smallest since 4.1.6's 8663972.
Mount reported /dev/mapper/* names with 4.4.3 and 4.5.0, while 4.2.1 and 4.3.3 reported device names.
What's taking so long booting with kernel-default?
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
I had dracut rebuild big41's 4.5.0 without using my custom adds and omits, no customizing at all. That speeded up boot by about 30 seconds. Then I had dracut rebuild 4.5.0 with hostonly=no. That caused a 11.5 minute delay between stanza selection and initrd loading message being replaced by init messages, total time to shell prompt, 12.25 minutes. I had dracut build another with only deviation from default the omission of /etc/dracut.conf.d/20-early-microcode.conf. That caused 3m45s delay, 4m25s net to shell prompt. Big41's 4.4.1 reaches shell prompt in well under a minute. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Felix Miata composed on 2016-03-25 17:06 (UTC-0400):
Felix Miata composed on 2016-03-22 22:10 (UTC-0400):
Felix Miata composed on 2016-03-22 04:57 (UTC-0400):
Felix Miata composed on 2015-10-12 12:29 (UTC-0400):
(originally to initramfs@vger.kernel.org) https://www.mail-archive.com/initramfs@vger.kernel.org/msg04168.html
On more and more installations since most distros have replaced mkinitrd with dracut, on selecting a bootloader menu choice the "root (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd" message stays on screen 40 seconds or more before the time-stamped boot messages begin scrolling the screen. IIRC, these delays first began appearing last January or February, and appear on multiple machines. All kernel and initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all with native 512b sector size, manufactured in 2011 or earlier). All installations are configured with static IP. All have Plymouth either not installed, or are booted with noplymouth included on kernel cmdline.
e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1 (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is about 80 seconds following that 40 second delay, with last time stamp showing 36.844484.
Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964; initramfs-tools 0.120) delays start of boot messages nearly 80 seconds, reaching DM greeter ready nearly 140 seconds after stanza selection.
Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360) exhibits no perceptible delay, reaching multi-user.target shell prompt in under 47 seconds from boot stanza selection.
Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays start of boot messages about 40 seconds, reaching multi-user.target bash prompt about 90 seconds after stanza selection.
Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no delay, reaching multi-user.target bash prompt in about 45 seconds.
Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits no delay, reaching DM greeter in less than about 45 seconds.
Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches multi-user.target shell prompt in about 75 seconds after unknown kernel/initrd delay (puts display into sleep mode until boot messages appear).
Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting (second, on sda28, vs. other on sda23) openSUSE Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay, reaching multi-user.target bash prompt in about 100 seconds.
Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608) exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90 seconds.
Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no delay, reaching multi-user.target bash prompt in less than 50 seconds.
Any ideas what could cause these delays, or how to eliminate them?
Booting host hpg33 openSUSE Tumbleweed 20160307 64 bit kernel-default-4.4.3, 6194336 bytes, and initrd 38456340 bytes, just took 9.4 minutes for the last Grub message to clear, and extended that to ~10.2 minutes to reach a shell prompt. Prior 4.3.3-5 kernel-default's initrd is 9589212 bytes, 25% the size of the current. 4.3.3-5 took 2 2/3 minutes for the Grub message to clear, about 2/3 more minute to reach shell prompt, and 4.3.3-1 about the same. 3rd prior, kernel-desktop-4.2.1 cleared the Grub message in too few seconds to count, well under a minute to reach shell prompt. kernel-default-4.5.0 cleared the Grub in 130 seconds, reached shell prompt at 174, has 8664600 byte initrd, smallest since 4.1.6's 8663972.
Mount reported /dev/mapper/* names with 4.4.3 and 4.5.0, while 4.2.1 and 4.3.3 reported device names.
What's taking so long booting with kernel-default?
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
I had dracut rebuild big41's 4.5.0 without using my custom adds and omits, no customizing at all. That speeded up boot by about 30 seconds. Then I had dracut rebuild 4.5.0 with hostonly=no. That caused a 11.5 minute delay between stanza selection and initrd loading message being replaced by init messages, total time to shell prompt, 12.25 minutes. I had dracut build another with only deviation from default the omission of /etc/dracut.conf.d/20-early-microcode.conf. That caused 3m45s delay, 4m25s net to shell prompt. Big41's 4.4.1 reaches shell prompt in well under a minute.
Preliminary indication seems to be that I have narrowed this down to some EXT4 related problem. I cloned big41's HD to another physical disk, which didn't help, so not likely it is hardware fault. Then, I created a fresh EXT3 partition of identical size as the original on the clone, rsync'd to it, adjusted bootloader and fstab accordingly, and the EXT3 copy boots 4.5.0 just as fast as 4.4.1, in well under a minute. Fedora 24's 4.5rc6 was giving identical trouble, so I repeated the switch to EXT3 with it. Problem solved there too. How can I narrow down what (Dracut's?) problem with EXT4 seems to be? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 03/31/2016 09:13 PM, Felix Miata wrote:
Felix Miata composed on 2016-03-25 17:06 (UTC-0400):
Felix Miata composed on 2016-03-22 22:10 (UTC-0400):
Felix Miata composed on 2016-03-22 04:57 (UTC-0400):
Felix Miata composed on 2015-10-12 12:29 (UTC-0400):
(originally to initramfs@vger.kernel.org) https://www.mail-archive.com/initramfs@vger.kernel.org/msg04168.html
On more and more installations since most distros have replaced mkinitrd with dracut, on selecting a bootloader menu choice the "root (hd0,...filesystemtype..[Linux-bzImage, setup, ... initrd /boot/initrd" message stays on screen 40 seconds or more before the time-stamped boot messages begin scrolling the screen. IIRC, these delays first began appearing last January or February, and appear on multiple machines. All kernel and initrd images are on smallish EXT3 or EXT4 (mostly the latter) filesystems using 1024 blocksize on BIOS logical partitions on rotating rust (IIRC, all with native 512b sector size, manufactured in 2011 or earlier). All installations are configured with static IP. All have Plymouth either not installed, or are booted with noplymouth included on kernel cmdline.
e.g, on my fastest system, host msi85, a 3.0 GHz dual core Haswell, total time from boot stanza select on BIOS host msi85 to Rawhide 4.3.0-0.rc3.git4.1 (dracut initramfs.img 11,326,223) multi-user.target shell prompt ready is about 80 seconds following that 40 second delay, with last time stamp showing 36.844484.
Same msi85 system booting LMDE 2 3.16.0-4 (non-dracut? initrd.img 27,362,964; initramfs-tools 0.120) delays start of boot messages nearly 80 seconds, reaching DM greeter ready nearly 140 seconds after stanza selection.
Same system booting openSUSE Leap 42.1 4.1.8 (dracut initrd 8,082,360) exhibits no perceptible delay, reaching multi-user.target shell prompt in under 47 seconds from boot stanza selection.
Same system booting openSUSE 13.2 3.16.7 (dracut initrd 5,351,996) delays start of boot messages about 40 seconds, reaching multi-user.target bash prompt about 90 seconds after stanza selection.
Same system booting Mageia 5 4.1.8 (dracut initrd.img 9,391,664) exhibits no delay, reaching multi-user.target bash prompt in about 45 seconds.
Same system booting Wheezy 3.2.0 (non-dracut initrd.img 10,537,321) exhibits no delay, reaching DM greeter in less than about 45 seconds.
Same system booting openSUSE 13.1 3.12.44 (non-dracut initrd 7,679,390) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting Fedora 23 4.2.2 (dracut initramfs.img 11,255,214) reaches multi-user.target shell prompt in about 75 seconds after unknown kernel/initrd delay (puts display into sleep mode until boot messages appear).
Same system booting openSUSE Tumbleweed 4.2.1 (dracut initrd 8,970,760) exhibits no delay, reaching multi-user.target bash prompt in less than 45 seconds.
Same system booting (second, on sda28, vs. other on sda23) openSUSE Tumbleweed 4.2.1 (dracut initrd 8,965,124) exhibits ~40 second delay, reaching multi-user.target bash prompt in about 100 seconds.
Same system booting Kubuntu 14.10 3.16.0 (non-dracut initrd 20,550,608) exhibits no initial (linux/initrd) delay, reaching KDM greeter in about 90 seconds.
Same system booting Fedora 22 4.1.8 (dracut initrd 10,788,392) exhibits no delay, reaching multi-user.target bash prompt in less than 50 seconds.
Any ideas what could cause these delays, or how to eliminate them?
Booting host hpg33 openSUSE Tumbleweed 20160307 64 bit kernel-default-4.4.3, 6194336 bytes, and initrd 38456340 bytes, just took 9.4 minutes for the last Grub message to clear, and extended that to ~10.2 minutes to reach a shell prompt. Prior 4.3.3-5 kernel-default's initrd is 9589212 bytes, 25% the size of the current. 4.3.3-5 took 2 2/3 minutes for the Grub message to clear, about 2/3 more minute to reach shell prompt, and 4.3.3-1 about the same. 3rd prior, kernel-desktop-4.2.1 cleared the Grub message in too few seconds to count, well under a minute to reach shell prompt. kernel-default-4.5.0 cleared the Grub in 130 seconds, reached shell prompt at 174, has 8664600 byte initrd, smallest since 4.1.6's 8663972.
Mount reported /dev/mapper/* names with 4.4.3 and 4.5.0, while 4.2.1 and 4.3.3 reported device names.
What's taking so long booting with kernel-default?
On host big41's TW, 4.5.0 delay is roughly the same as with hpg33's 4.5.0, while big41's previous kernel-default-4.4.1 has no delay.
I had dracut rebuild big41's 4.5.0 without using my custom adds and omits, no customizing at all. That speeded up boot by about 30 seconds. Then I had dracut rebuild 4.5.0 with hostonly=no. That caused a 11.5 minute delay between stanza selection and initrd loading message being replaced by init messages, total time to shell prompt, 12.25 minutes. I had dracut build another with only deviation from default the omission of /etc/dracut.conf.d/20-early-microcode.conf. That caused 3m45s delay, 4m25s net to shell prompt. Big41's 4.4.1 reaches shell prompt in well under a minute.
Preliminary indication seems to be that I have narrowed this down to some EXT4 related problem. I cloned big41's HD to another physical disk, which didn't help, so not likely it is hardware fault.
Then, I created a fresh EXT3 partition of identical size as the original on the clone, rsync'd to it, adjusted bootloader and fstab accordingly, and the EXT3 copy boots 4.5.0 just as fast as 4.4.1, in well under a minute.
Fedora 24's 4.5rc6 was giving identical trouble, so I repeated the switch to EXT3 with it. Problem solved there too.
How can I narrow down what (Dracut's?) problem with EXT4 seems to be?
You might try bisecting the kernel source between 4.5 and 4.4 to see if you can identify which kernel commit is causing the problem. Larry -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Larry Finger composed on 2016-03-31 21:28 (UTC-0500):
Felix Miata wrote:
Preliminary indication seems to be that I have narrowed this down to some EXT4 related problem. I cloned big41's HD to another physical disk, which didn't help, so not likely it is hardware fault.
Then, I created a fresh EXT3 partition of identical size as the original on the clone, rsync'd to it, adjusted bootloader and fstab accordingly, and the EXT3 copy boots 4.5.0 just as fast as 4.4.1, in well under a minute.
Fedora 24's 4.5rc6 was giving identical trouble, so I repeated the switch to EXT3 with it. Problem solved there too.
How can I narrow down what (Dracut's?) problem with EXT4 seems to be?
You might try bisecting the kernel source between 4.5 and 4.4 to see if you can identify which kernel commit is causing the problem.
What would I be looking for, assuming I had any idea how to read source in the first place, which I don't? This started as far back as a year ago IIRC, with what seemed way back when to be random installations and kernel versions. It never happened on 32 bit installations, and I wasn't keeping track of which 64 bit were on EXT3 and which on EXT4. These have or not the problem on host big41's clone HD on EXT4 installations: Fedora 24: 4.5.rc6git2.1 bad 4.5.rc5git0.1 OK 4.4.rc6git1.1 OK 4.3.rc5git0.1 bad 4.2.rc8git0.1 bad Fedora 23: 4.1.rc8git0.2 OK TW: 4.5.0-1 (hd0,9) bad 4.5.0-1 (hd0,22) bad 4.4.3-1 (hd0,9) bad 4.4.1-1 (hd0,22) OK 4.3.3-1 (hd0,22) bad 4.2.1 (hd0,9) bad 4.1.3 (hd0,9) bad 13.2: 3.16.7-35 OK 3.16.7-29 OK (Dec 1 2015 initrd) 3.16.7-7 bad (Dec 31 2014 initrd) 13.1: 3.12.53-40 OK (Feb 26 2016 initrd) 3.12.51-1 OK (Dec 31 2015 initrd) 3.11.10-28 OK (Apr 27 2015 initrd) Host p5bse TW (old big41 HD from which big41's prior HD was cloned to newer, then moved without modifications to p5bse): 4.4.1-1 OK (26 Feb 2016 initrd) 4.3.3-5 OK (10 Jan 2016 initrd) 4.2.1-1 OK (15 Oct 2015 initrd) 4.1.6-1 OK (28 Aug 2015 initrd) 4.0.5-1 OK (17 Jun 2015 initrd) Host p5bse 42.1: 4.1.15-8 OK 4.1.12-1 OK -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (6)
-
Andrei Borzenkov -
Bruno Friedmann -
Felix Miata -
Larry Finger -
Per Jessen -
Stefan Seyfried