[Bug 398356] New: Booting from Software-RAID fails
https://bugzilla.novell.com/show_bug.cgi?id=398356 Summary: Booting from Software-RAID fails Product: openSUSE 11.0 Version: RC 1 Platform: x86-64 OS/Version: openSUSE 11.0 Status: NEW Severity: Normal Priority: P5 - None Component: Installation AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: mail@steffen-moser.de QAContact: jsrain@novell.com Found By: Beta-Customer YaST2 allows using a software RAID device (mdX) as a root partition when installing openSUSE-11.0-RC1 (without having a separate non-software-RAID partition for "/boot"). After then, the installed system is not bootable (tested with software RAID level 1). As far as I know, GRUB is not able to boot from MD devices in general. LILO, on the other hand, is able to boot from MD devices. So in my opinion, it would make sense to keep the permission of having "/" (or even a separate RAIDed "/boot") on a software RAID device, but also to inform the user that he would get an unbootable system. When selecting LILO as a boot loader in YaST2, it terminates with an error when having "/" (or the separate "/boot") on an MD device. I suppose that this is related to the problem described in [1]. So up to now, the only thing a user can do to allow booting from an MD device is to make an MD partition in RAID version 0.90 manually - as described in [1] and using LILO then. I understand that booting from RAID devices is not necessary, there are other solutions, to have a fail-tolerant system - e.g. generating separate non-RAID "/boot" partitions on both disks. But if "/" on RAID or "/" and "/boot" on RAID functions, it would make things like kernel updates easier. The advanced user is able to configure it by hand easily. But I think the not so advanced user should be warned that the partitioning he created will lead to an unbootable system as long as he doesn't think of some additional things manually (at the moment: using LILO instead of GRUB and setting up a RAID superblock in version 0.90 for the boot device). I personally would not disallow doing it such a "boot from RAID configuration", but warn the user (or, even better - if time permits, fix YaST2 in a way that it chooses a working solution automatically). [1] http://anti.teamidiot.de/nei/2006/10/softraid_lilo/ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 Andreas Jaeger <aj@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team-screening@forge.provo.novell.com |aschnell@novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User aschnell@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c1 Arvin Schnell <aschnell@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aschnell@novell.com AssignedTo|aschnell@novell.com |jsrain@novell.com Component|Installation |Booting --- Comment #1 from Arvin Schnell <aschnell@novell.com> 2008-06-08 11:39:58 MDT --- yast2-storage gives a warning for "/" on raid and no "/boot" partition but only for raid-type != 1. Changing that would be very easy. The other things are out of my reach. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User mail@steffen-moser.de added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c2 Steffen Moser <mail@steffen-moser.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mail@steffen-moser.de --- Comment #2 from Steffen Moser <mail@steffen-moser.de> 2008-06-08 11:44:17 MDT --- It seems that my initial statement regarding GRUB is wrong. I've just switched from LILO to GRUB and the system is still booting without any problems. In the meantime, I suppose that my initial problem is only related to the RAID superblock version. I will do a further test: I am going to make "/boot" to version 1.x (instead of 0.9) and look what GRUB says to it. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User aschnell@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c3 --- Comment #3 from Arvin Schnell <aschnell@novell.com> 2008-06-08 11:56:42 MDT --- In bug #343851 are some info about LILO and superblock version 1.0. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User mail@steffen-moser.de added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c4 --- Comment #4 from Steffen Moser <mail@steffen-moser.de> 2008-06-08 17:19:56 MDT --- I hope that I can clarify it a bit: - LILO has definitive the problem that it still cannot boot from MD devices that have the superblock version 1.0 (at least the version of LILO that comes with 11.0-RC1). MD devices created during the installation do have version 1.0, so they lead to an unbootable system. I think that in this case, we can warn the user and/or use superblock version 0.90 in this case for the boot device. - Despite to my initial posting, GRUB in general is able to boot from MD devices, no matter which superblock version they consist of. When installing openSUSE-11.0-RC1 on a system partitioned as follows: /dev/sda1 \ ---- /dev/md0 -> SWAP /dev/sdb1 / /dev/sda2 \ ---- /dev/md1 -> /boot /dev/sdb2 / /dev/sda3 \ ---- /dev/md2 -> / /dev/sdb3 / the system may be unbootable. The problem is that in this case, GRUB gets only installed to the MBR of "/dev/sdb", but not to the MBR of "/dev/sda". Now it depends on the system BIOS, which disk is selected for booting. In my test environment, the BIOS only tried to boot from "/dev/sda" which had an empty MBR, as it was a completely fresh install. After telling the BIOS to boot from "/dev/sdb", the system came up cleanly. Installing GRUB also to "/dev/sda" manually made the system bootable from both disks. So I think, that when using GRUB, YaST2 (or "grub-install") should take care of installing the boot code to the MBRs of all disks which are part of the MD device that is used for booting. I have only tested RAID level 1. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User jsrain@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c5 Jiri Srain <jsrain@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|jsrain@novell.com |juhliarik@novell.com --- Comment #5 from Jiri Srain <jsrain@novell.com> 2008-06-09 03:16:45 MDT --- Could you, please, attach the logs? My guess is that BIOS ID detection failed, which caused the error. However, without the logs I can only guess. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 Jiri Srain <jsrain@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |mail@steffen-moser.de -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User mail@steffen-moser.de added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c6 --- Comment #6 from Steffen Moser <mail@steffen-moser.de> 2008-06-09 13:32:15 MDT --- Attachted you'll find an archive of "/var/log/YaST2". The BIOS I use comes from "VMware Workstation" as I did this beta test within VMware. "(hd0)" seems to get mapped to "/dev/sdb" and "(hd1)" seems to get mapped to "/dev/sda". Nevertheless, when doing a | grep stage2 y2log I find the following matches: | 'setup --stage2=/boot/grub/stage2 (hd0,1) (hd1,1)' | ' setup --stage2=/boot/grub/stage2 (hd1,1) (hd1,1)' | ' setup --stage2=/boot/grub/stage2 (hd0) (hd1,1)' | 'setup --stage2=/boot/grub/stage2 (hd1,1) (hd1,1)' | ' setup --stage2=/boot/grub/stage2 (hd0) (hd1,1)' There is no line that indicates the installation of the GRUB code to the MBR of "(hd1)". This confirms my observation: "/dev/sdb" (which is "(hd0)") is bootable, but "/dev/sda" (which is "(hd1)") isn't. If you need more information or other logs, feel free to ask for. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User mail@steffen-moser.de added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c7 Steffen Moser <mail@steffen-moser.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|mail@steffen-moser.de | --- Comment #7 from Steffen Moser <mail@steffen-moser.de> 2008-06-09 13:33:31 MDT --- Created an attachment (id=221106) --> (https://bugzilla.novell.com/attachment.cgi?id=221106) Log files from YaST2 ("/var/log/YaST2") after installation of openSUSE-11.0-RC1 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User rhg@marxmeier.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c8 Roland Genske <rhg@marxmeier.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rhg@marxmeier.com --- Comment #8 from Roland Genske <rhg@marxmeier.com> 2008-06-21 11:19:03 MDT --- I think I can contribute. I get the same problem while trying a network installation. My partition scheme is (this is a VMware test installation before installing the real thing [hosted root server in a remote data center]): Disk /dev/sda: 5368 MB, 5368709120 bytes Device Boot Start End Blocks Id System /dev/sda1 * 1 9 72261 fd Linux raid autodetect /dev/sda2 10 42 265072+ fd Linux raid autodetect /dev/sda3 43 652 4899825 fd Linux raid autodetect Disk /dev/sdb: 5368 MB, 5368709120 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 9 72261 fd Linux raid autodetect /dev/sdb2 10 42 265072+ fd Linux raid autodetect /dev/sdb3 43 652 4899825 fd Linux raid autodetect Raid arrays: /dev/md0 mounted as /boot consisting of /dev/sda1 and /dev/sdb1 /dev/md1 mounted as swap consisting of /dev/sda2 and /dev/sdb2 /dev/md2 mounted as / consisting of /dev/sda3 and /dev/sdb3 During installation the error message below appears on the shell where I did the ssh login and started yast: --- snip --- *** Starting YaST2 *** error: cannot open Packages index using db3 - No such file or directory (2) Use of uninitialized value $extended_dev in string eq at /usr/lib/perl5/vendor_perl/5.10.0/Bootloader/Core/GRUB.pm line 852 (#1) (W uninitialized) An undefined value was used as if it were already defined. It was interpreted as a "" or a 0, but maybe it was a mistake. To suppress this warning assign a defined value to your variables. To help you figure out what was undefined, perl will try to tell you the name of the variable (if any) that was undefined. In some cases it cannot do this, so it also tells you what operation you used the undefined value in. Note, however, that perl optimizes your program and the operation displayed in the warning may not necessarily appear literally in your program. For example, "that $foo" is usually optimized into "that " . $foo, and the warning will refer to the concatenation (.) operator, even though there is no . in your program. --- snap --- The first reboot then fails with an "invalid boot sector" error message. With the help of the Rescue System I found that /etc/grub.conf was setup incorrectly: setup --stage2=/boot/grub/stage2 (hd1,0) (hd0,0) quit This should have been: setup --stage2=/boot/grub/stage2 (hd0,0) (hd0,0) setup --stage2=/boot/grub/stage2 (hd1,0) (hd0,0) quit The problem can be fixed by correcting /etc/grub.conf and then invoking grub-install. After that, the system successfully boots and the installation can continue with /usr/lib/YaST2/startup/YaST2.ssh. Good that I tried this in a test installation before using this on the remote server :-) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User rhg@marxmeier.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c9 Roland Genske <rhg@marxmeier.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Normal |Blocker Found By|Beta-Customer |Third Party Developer/Partner Priority|P5 - None |P2 - High Version|RC 1 |Final --- Comment #9 from Roland Genske <rhg@marxmeier.com> 2008-06-22 05:15:20 MDT --- Happens with the final openSUSE 11.0. Trying to fix this during installation does not work, at least I did not find a workaround yet. What I tried: Created /tmp/YaST2_keep_sshd_running so that I get a chance to fix this before the first reboot. However, the steps that fixed the problem in the Rescue System did not work. Here is what I did: mount /dev/md2 /mnt mount /dev/md0 /mnt/boot mount -t proc /proc /mnt/proc mount -t sysfs /sys /mnt/sys mknod /mnt/dev/sda b 8 0 mknod /mnt/dev/sda1 b 8 1 mknod /mnt/dev/sda2 b 8 2 mknod /mnt/dev/sda3 b 8 3 mknod /mnt/dev/sdb b 8 16 mknod /mnt/dev/sdb1 b 8 17 mknod /mnt/dev/sdb2 b 8 18 mknod /mnt/dev/sdb3 b 8 19 chroot /mnt cat >/etc/grub.conf <<eof. setup --stage2=/boot/grub/stage2 (hd0,0) (hd0,0) setup --stage2=/boot/grub/stage2 (hd1,0) (hd0,0) quit eof. grub-install Here is what grub-install outputs: grub> setup --stage2=/boot/grub/stage2 (hd0,0) (hd0,0) Checking if "/boot/grub/stage1" exists... yes Checking if "/boot/grub/stage2" exists... yes Checking if "/boot/grub/e2fs_stage1_5" exists... yes Running "embed /boot/grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal) Running "embed /boot/grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal) Running "install --stage2=/boot/grub/stage2 /boot/grub/stage1 (hd0,0) /boot/grub/stage2 p /boot/grub/menu.lst "... succeeded Done. grub> setup --stage2=/boot/grub/stage2 (hd1,0) (hd0,0) Checking if "/boot/grub/stage1" exists... yes Checking if "/boot/grub/stage2" exists... yes Checking if "/boot/grub/e2fs_stage1_5" exists... yes Running "embed /boot/grub/e2fs_stage1_5 (hd1,0)"... failed (this is not fatal) Running "embed /boot/grub/e2fs_stage1_5 (hd0,0)"... failed (this is not fatal) Running "install --stage2=/boot/grub/stage2 /boot/grub/stage1 d (hd1,0) /boot/grub/stage2 p /boot/grub/menu.lst "... succeeded Done. Result: On reboot I get a "invalid partition table" BIOS message. When I try the same in the Rescue System it works. The grub-install messages are identical and afterwards the system boots correctly. Unfortunately, using the Rescue System on the remote server is not an option, therefore please allow me to raise the priority and set the severity to "blocker". -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User juhliarik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c10 Jozef Uhliarik <juhliarik@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED --- Comment #10 from Jozef Uhliarik <juhliarik@novell.com> 2008-06-23 04:56:47 MDT --- to commnent #6: - I can write that hd0 is /dev/sdb and hd1 is /dev/sda (from logs comment #7) - there is missing "write to MBR of the second disk (hd1)" (perl-Bootloader) - yast2-bootloader use wrong device (/dev/md) for analyse MBR -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User juhliarik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c11 --- Comment #11 from Jozef Uhliarik <juhliarik@novell.com> 2008-06-23 06:45:03 MDT --- to comment #4: - your raid 1 is version:"01.00.00". The version 1.0... is not supported by LILO. - yast2-bootloader should check version of raid for LILO and it should write error message: "Because of the partitioning, the boot loader cannot be installed properly." (there are missing logs from installation where was selected LILO) bnc #357897 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User juhliarik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c12 Jozef Uhliarik <juhliarik@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|juhliarik@novell.com |aosthof@novell.com Status|ASSIGNED |NEW --- Comment #12 from Jozef Uhliarik <juhliarik@novell.com> 2008-06-23 08:18:21 MDT --- Result: - the order of disks seems to be wrong (hwinfo) - the perl-Bootloader writes GRUB only to one disk (MBR) - yast2-bootloader use wrong disk for analyse MBR The perl-Bootloader is responsible for creating /etc/grub.conf and support to write GRUB to both disk also in MBR (RAID1) yast2-bootloader needs update of analyse MBR. It should be done soon. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 Alexander Osthof <aosthof@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |ASSIGNED -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User juhliarik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c13 Jozef Uhliarik <juhliarik@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |juhliarik@novell.com --- Comment #13 from Jozef Uhliarik <juhliarik@novell.com> 2008-06-26 09:03:33 MDT --- yast2-bootloader 2.17.1 includes support for analyse correct MBR resp. it is selected correct device if soft-raid is used. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User volker3204@paradise.net.nz added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c14 Volker Kuhlmann <volker3204@paradise.net.nz> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |volker3204@paradise.net.nz --- Comment #14 from Volker Kuhlmann <volker3204@paradise.net.nz> 2008-06-28 17:44:59 MDT --- It's unfortunate that 11.0 final is shipping with yast2-bootloader-2.16.20. This means yast doens't manage to install a boot loader and half way through the installation the reboot stops dead. Nice GAU, but it's getting worse. For 10.3 yast was at least able to install a functional boot loader *if* one selected the right options, which is no longer possible. This was already reported for beta3 (#391971) and something similar for 10.3 final (#341309). Why are things like this shipped? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User bsouthey@gmail.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c15 Bruce S <bsouthey@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bsouthey@gmail.com --- Comment #15 from Bruce S <bsouthey@gmail.com> 2008-07-01 09:09:39 MDT --- I think that I may have also met this bug with using opensuse-11 KDE live CD x86_64. I was eventually able to get around it by formating my / partition (includes /boot) and explicitly setting the inode size to 128 rather than leaving it as default - this was from a post by Tamas Sarga (http://www.nabble.com/inode-size-256-cause-a-pain-td17290645.html). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User volker3204@paradise.net.nz added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c16 --- Comment #16 from Volker Kuhlmann <volker3204@paradise.net.nz> 2008-07-01 15:03:05 MDT --- Bruce (comment #15), this bug has nothing to do with filesystem or ext2 inode size (btw I'm using reiser). Yast/perl-bootloader are simply creating the wrong grub commands for installing the boot loader. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User aosthof@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c17 Alexander Osthof <aosthof@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|aosthof@novell.com |jreidinger@novell.com Status|ASSIGNED |NEW --- Comment #17 from Alexander Osthof <aosthof@novell.com> 2008-07-14 07:31:23 MDT --- Reassigning to new maintainer of perl-Bootloader. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User juhliarik@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c18 Jozef Uhliarik <juhliarik@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |franz.x.maier@aon.at --- Comment #18 from Jozef Uhliarik <juhliarik@novell.com> 2008-07-15 11:47:51 MDT --- *** Bug 227377 has been marked as a duplicate of this bug. *** https://bugzilla.novell.com/show_bug.cgi?id=227377 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=398356 User jreidinger@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=398356#c19 Josef Reidinger <jreidinger@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FEATURE --- Comment #19 from Josef Reidinger <jreidinger@novell.com> 2008-07-22 01:26:28 MDT --- Part with badly selected disk is fixed by juhliarik. Part which want recognize if MD array cover disks with same partition table and write same MBR to it is cover by feature request. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com