[Bug 379745] New: Bad handling of md* in /etc/init.d/halt
https://bugzilla.novell.com/show_bug.cgi?id=379745 Summary: Bad handling of md* in /etc/init.d/halt Product: openSUSE 10.3 Version: Final Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem AssignedTo: werner@novell.com ReportedBy: suse-beta@cboltz.de QAContact: qa@suse.de CC: guenter.ohmer@gmx.de Found By: --- This section of /etc/init.d/halt has a bug: if test "$HALT_DISKS" = "yes" ; then while read line ; do case "$line" in md.*active) HALT_DISKS=no; break esac done < /proc/mdstat The problem is the "md.*active)" line. This is a bash script, not a regular expression - the dot should not be there. Pseudo-patch: - md.*active) HALT_DISKS=no; break + md*active) HALT_DISKS=no; break Even with this fixed in the script, the raid seems to be shutdown unclean. /var/log/boot.msg shows md0: bitmap file is out of date (4144 < 4145) -- forcing full recovery which means the raid is resynced on each boot. mdadm --grow /dev/md0 --bitmap=none solves (or hides?) this problem. If you need more details, please ask Günter (mail address in CC) - it's his machine that has the problem. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c1 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |mmarek@novell.com --- Comment #1 from Dr. Werner Fink <werner@novell.com> 2008-04-15 03:31:48 MST --- Thanks for the report. Adding Michal as the maintainer mdadm tools of to the CC list. Michal? Do you know what goes wrong even if the disks are not touched? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c2 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |werner@novell.com AssignedTo|werner@novell.com |mmarek@novell.com Status|NEEDINFO |NEW Info Provider|mmarek@novell.com | --- Comment #2 from Dr. Werner Fink <werner@novell.com> 2008-04-15 03:53:11 MST --- The halt script is fixed. Nevertheless the problem with the resync still remains. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User mmarek@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c3 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |guenter.ohmer@gmx.de --- Comment #3 from Michal Marek <mmarek@novell.com> 2008-06-02 08:28:06 MDT --- Where kind of bitmap do you use and where is it stored? Can you post the output of mdadm --detail /dev/md0 ? Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User guenter.ohmer@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c4 Günter Ohmer <guenter.ohmer@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|guenter.ohmer@gmx.de | --- Comment #4 from Günter Ohmer <guenter.ohmer@gmx.de> 2008-06-09 11:34:31 MDT --- Created an attachment (id=221083) --> (https://bugzilla.novell.com/attachment.cgi?id=221083) raid-details, boot.log, etc Hi Michal, I added also my boot.log and some other files. Guenter -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User mmarek@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c5 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nfbrown@novell.com --- Comment #5 from Michal Marek <mmarek@novell.com> 2008-07-01 07:06:51 MDT --- I tried to replicate your setup - root on raid1 with an internal bitmap, using XFS, but the bitmap still OK in my case. Neil, do you have an idea what's going on here..? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User nfbrown@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c6 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |NEEDINFO Info Provider| |suse-beta@cboltz.de --- Comment #6 from Neil Brown <nfbrown@novell.com> 2008-07-07 19:51:22 MDT --- (sorry, this fell off my radar as it wasn't needinfo). Is it possible to get the output of mdadm -E /dev/sda1 mdadm -X /dev/sda1 as well? Also mdadm -V uname -a just to be on the safe side. I did have one bug with the bitmap event counters getting wrong, but it involved one drive failing just before the array was shut down. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 Christian Boltz <suse-beta@cboltz.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Info Provider|suse-beta@cboltz.de |guenter.ohmer@gmx.de -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User guenter.ohmer@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c7 --- Comment #7 from Günter Ohmer <guenter.ohmer@gmx.de> 2008-07-09 09:57:50 MDT --- Created an attachment (id=226785) --> (https://bugzilla.novell.com/attachment.cgi?id=226785) mdadm -E /dev/sda1 etc -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User nfbrown@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c8 --- Comment #8 from Neil Brown <nfbrown@novell.com> 2008-07-09 21:13:18 MDT --- Thanks. The evidence seems to point to the very last write to the device is getting lost. md writes out the bitmap header and then the raid metadata. So you the event counter in the bitmap header should normally be the same as that in the raid metadata, or one more if the bitmap header write succeeded but the metadata write didn't. We are seeing the bitmap metadata with 4144 and the raid metadata with 4145. There is a case where the event count goes backwards though. When a dirty array is being marked clean we sometimes step the event count backwards rather than forwards (there is a good reason...). It seems like that is happening here. This suggests that the update of the bitmap header works, but the update of the array metadata doesn't. I can't think of a really good way to get these writes to happen earlier. Possibly putting mount -o remount,ro / sync ; sync; sleep 1 in /etc/init.d/halh just before the last line: exec $command -d -f -n $opts might help. If md doesn't see any write activity for 200ms it marks the array clean. The above should allow that to happen 800ms before the machine is stopped. Can you try that? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User guenter.ohmer@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c9 Günter Ohmer <guenter.ohmer@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |ASSIGNED Info Provider|guenter.ohmer@gmx.de | --- Comment #9 from Günter Ohmer <guenter.ohmer@gmx.de> 2008-07-11 22:52:13 MDT --- Hi Neil, it works with this changes. The first change of Werner puts the -h option to the halt command, but ... read man halt: ========= man halt ============= The -h flag puts all harddisks in standby mode just before halt or poweroff. Right now this is only implemented for IDE drives. A side effect of putting the drive in standby mode is that the write cache on the disk is flushed. This is important for IDE drives, since the kernel doesn't flush the write-cache itself before poweroff. The halt program uses /proc/ide/hd* to find all IDE disk devices, which means that /proc needs to be mounted when halt or poweroff is called or the -h switch will do nothing. ================================= The /proc filesystem is not mounted anymore in newer SUSE releases. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c10 --- Comment #10 from Dr. Werner Fink <werner@novell.com> 2008-07-14 03:20:24 MDT --- This remark of the manual page is outdated. The new libata interface *does* flush the write-cache of the IDE disks. There is a patch in halt.c included to check if the kernel does a flush for not only IDE but *all* disks and skip the flush in halt.c its self for that case. See patch sysvinit-2.86-hddown.patch in sysvinit source rpm and see that not /proc/ide/hd* but /sys/block/ and /sys/class/scsi_disk/ are used to determine the capabilities of the linux kernel drivers (compare with bug #386487 and bug #229210). Beside this the option -h is *not* performed if a md raid is found in the system, see line 78ff in /etc/init.d/halt -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P3 - Medium -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User mmarek@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c11 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mmarek@novell.com AssignedTo|mmarek@novell.com |werner@novell.com Status|ASSIGNED |NEW --- Comment #11 from Michal Marek <mmarek@novell.com> 2008-07-29 07:38:17 MDT --- So, what needs to be fixed here? The halt init script? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c12 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |mmarek@novell.com --- Comment #12 from Dr. Werner Fink <werner@novell.com> 2008-07-29 07:51:39 MDT --- The dot in md.*active is arelady removed in /etc/init.d/halt therefore which change has to be done ... the mount -o remount,ro / sync ; sync; sleep 1 isn't very clean as we already have mount -no remount,ro / 2> /dev/null sync at line 170ff ... how about a `sleep 1' in case of md only or is there an mdadm command line option to solve this problem in a clean way? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User mmarek@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c13 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Info Provider|mmarek@novell.com |nfbrown@novell.com --- Comment #13 from Michal Marek <mmarek@novell.com> 2008-07-29 07:58:55 MDT --- I don't think there is, Neil? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c14 --- Comment #14 from Dr. Werner Fink <werner@novell.com> 2008-07-29 08:15:31 MDT --- I'm seeking for something like a check to know if an mdraid is dirty which is that the sync syscall is not ready and the md is busy. On the other side what is about lines like mdadm --stop --scan before executing the sync syscall. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c15 --- Comment #15 from Dr. Werner Fink <werner@novell.com> 2008-07-30 06:01:36 MDT --- Ping! Any news? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c16 --- Comment #16 from Dr. Werner Fink <werner@novell.com> 2008-07-30 07:29:50 MDT --- I'm now using in /etc/init.d/boot.halt HAS_MDRAID=no [...] mddev="" [...] # # Sometimes it is wanted to stop also the disks, but # do not do this for software raids. # while read line ; do case "$line" in md*:*active*) HAS_MDRAID=yes; break esac done < /proc/mdstat unset line # # If found a software raids, check out the devices # if test "$HAS_MDRAID" = "yes" ; then if test -e /etc/mdadm.conf ; then while read type dev rest; do case "$dev" in /dev/md*) mddev="${mddev:+$mddev }$dev" esac done < /etc/mdadm.conf unset type dev rest fi else opts="${opts:+$opts }-h" fi [...] # on umsdos fs this would lead to an error message, so direct errors to # /dev/null mount -no remount,ro / 2> /dev/null sync # for an software raid stop inactive devices and wait if test "$HAS_MDRAID" = "yes" ; then if test -n "$mddev" ; then mdadm --quiet --stop $mddev mdadm --quiet --wait $mddev else mdadm --quiet --stop --scan mdadm --quiet --wait /dev/md[0-9] /dev/md[0-9][0-9] fi fi which should be OK now. Btw. within boot.md the usage of $mdadm_BIN and mdadm is mixed. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c17 --- Comment #17 from Dr. Werner Fink <werner@novell.com> 2008-07-30 07:30:44 MDT --- Created an attachment (id=230765) --> (https://bugzilla.novell.com/attachment.cgi?id=230765) New /etc/init.d/halt Please test out if this works -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User nfbrown@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c18 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|nfbrown@novell.com | --- Comment #18 from Neil Brown <nfbrown@novell.com> 2008-07-30 23:01:26 MDT --- The need to wait has very little to do with mdadm. It is (I'm guessing) that the drives are configured with write-behind caching, and if you turn the power off before they flush their cache, you lose data. So what you really want is simply a "sleep 1" immediately before power is removed. I had put "mount -o ro ..." in my suggestion because I wasn't sure where this was being done and wanted to be sure that it gets done. I hope that clarified the situation (as I understand it). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c19 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |suse-beta@cboltz.de --- Comment #19 from Dr. Werner Fink <werner@novell.com> 2008-07-31 02:56:12 MDT --- Is there any other method to wait upto the point when the md raid is not busy anymore .... this because we want to avoid any vague and dumpy sleep where ever possible. We've done a lot of work to remove sleeps from the boot script last year. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c20 --- Comment #20 from Dr. Werner Fink <werner@novell.com> 2008-07-31 03:07:39 MDT --- AFAIK the current kernel does flush the write-behind cache for IDE, SATA, and SCSI disk. This was the reason why I've written the sysvinit-2.86-hddown.patch which enables halt(8) to use the data found at /sys/class/scsi_disk and /sys/block to determine if there is any disk which will not handled by the e.g. libata. The option -h of halt then enforce flushing for those remaining devices (e.g. usb sticks). Why this method does not work with any md raid is an interesting question .. or is this not true anymore? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Info Provider|suse-beta@cboltz.de |nfbrown@novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User nfbrown@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c21 Neil Brown <nfbrown@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|nfbrown@novell.com | --- Comment #21 from Neil Brown <nfbrown@novell.com> 2008-07-31 23:30:48 MDT --- md will mark an array as clean about 200ms after the last write, which will typcially happen when the filesystem is marked readonly. Alternately, md will mark the array as clean when it is stopped. This happens very late in the shutdown sequence by a reboot_notifier. This is normally enough to mark the array clean properly. If the reboot notifiers are not being run for some reason, or if devices aren't flushed after md's reboot notifier (which has a priority of INT_MAX and so should run first), then the marking of the array as clean might not get to the device. So it should all 'just work'. It isn't clear to me why it doesn't "just work" in this situation. You probably need to ask someone who understands cache flushing with the particular devices in question. If you send a SIGKILL to the md thread, e.g. killall -9 md0_raid1 it will cause the array to be marked clean immediately after the last write. This might help a little but isn't really a fix (more of value for experimentation). The real problem is that either the md rebot_notifier isn't being run, or the device cache isn't being flushed after that. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c22 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #230765|0 |1 is obsolete| | --- Comment #22 from Dr. Werner Fink <werner@novell.com> 2008-08-01 06:13:41 MDT --- Created an attachment (id=231283) --> (https://bugzilla.novell.com/attachment.cgi?id=231283) /etc/init.d/halt for openSuSE 11.0 I'm now using this /etc/init.d/halt for the next openSuSE 11.0 this stops all inactive md arrays after the umount of root and the following sync. Beside this the process table is scaned for md*_raid* kernel threads and those are signaled with SIGKILL to mark all arrays as clean. Hopefully this makes it now posssible to use the `-h' option of halt for flushing the hard disks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=379745 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=379745#c23 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED --- Comment #23 from Dr. Werner Fink <werner@novell.com> 2008-08-01 06:23:24 MDT --- fixed -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com