[Bug 1160604] New: BTRFS related kernel NULL pointer dereference
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 Bug ID: 1160604 Summary: BTRFS related kernel NULL pointer dereference Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: aarch64 OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-maintainers@forge.provo.novell.com Reporter: guillaume.gardet@arm.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Created attachment 827267 --> http://bugzilla.opensuse.org/attachment.cgi?id=827267&action=edit kernel 5.4 traces On OBS while building JeOS images based on BTRFS filesystem, kernel crashes with a kernel NULL pointer dereference. There was no problem kernel 5.3, only with 5.4. Traces are in attachment. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 Guillaume GARDET <guillaume.gardet@arm.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |afaerber@suse.com, | |dmueller@suse.com, | |guillaume.gardet@arm.com, | |mbrugger@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c1 --- Comment #1 from Guillaume GARDET <guillaume.gardet@arm.com> --- It happens at least on obs-arm-1,2,3 workers (ThunderX machines). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 Matthias Brugger <mbrugger@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|kernel-maintainers@forge.pr |jeffm@suse.com |ovo.novell.com | -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 Anthony Iliopoulos <ailiopoulos@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|jeffm@suse.com |rgoldwyn@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c4 --- Comment #4 from Guillaume GARDET <guillaume.gardet@arm.com> --- Could we have this fix https://lore.kernel.org/linux-btrfs/20191212181934.GA33645@dennisz-mbp.dhcp.... backported to Tumbleweed (stable and master branches), please? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c5 --- Comment #5 from Guillaume GARDET <guillaume.gardet@arm.com> --- Fix is merged in 5.5-rc6. Could we get it for 5.4 (stable branch)? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c6 Matthias Brugger <mbrugger@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |wqu@suse.com Flags| |needinfo?(wqu@suse.com) --- Comment #6 from Matthias Brugger <mbrugger@suse.com> --- (In reply to Guillaume GARDET from comment #5)
Fix is merged in 5.5-rc6. Could we get it for 5.4 (stable branch)?
that's upstream commit: 46bcff2bfc5e ("btrfs: fix compressed write bio blkcg attribution") and maybe as well: 7b62e66cbbfb ("btrfs: punt all bios created in btrfs_submit_compressed_write()") Wenruo can you take care of this? This issue blocks tumbleweed update on aarch64/arm64. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c7 David Sterba <dsterba@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dsterba@suse.com --- Comment #7 from David Sterba <dsterba@suse.com> --- The fix pointed in comment #4 is related to patches that are new in 5.5 (and unreleased by now). To reporduce that the cgroups2 and io controler need to be enabled. Also, the stacktraces are different so I don't think it's the same bug. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c8 --- Comment #8 from David Sterba <dsterba@suse.com> --- The differences, this bugreport: submit_compressed_extents+0x50 Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028 upstream: kernel NULL pointer dereference, address: 0000000000000510 bio_associate_blkg_from_css+0x1e btrfs_submit_compressed_write+0x137 submit_compressed_extents+0x40f Though it's on different arches, the offsets to page should be same or close, the crash happens in quite different functions and even if they get inlined, the offset into the function for aarch64 is right at the beginning of submit_compressed_extents (0x50, with inlines this would be much larger number). -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c11 Guillaume GARDET <guillaume.gardet@arm.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(wqu@suse.com) | --- Comment #11 from Guillaume GARDET <guillaume.gardet@arm.com> --- (In reply to Guillaume GARDET from comment #10)
It would need more tests, but it seems to be better with kernel 5.5.
It still happens with kernel 5.5.11 in current Tumbleweed on ThunderX1 machines in OBS while building aarch64 kiwi images. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c12 --- Comment #12 from Guillaume GARDET <guillaume.gardet@arm.com> --- Created attachment 833733 --> http://bugzilla.opensuse.org/attachment.cgi?id=833733&action=edit kernel 5.5.11 traces (from OBS build log) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 Dominique Leuenberger <dimstar@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dimstar@opensuse.org -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c13 --- Comment #13 from Guillaume GARDET <guillaume.gardet@arm.com> --- Created attachment 838532 --> http://bugzilla.opensuse.org/attachment.cgi?id=838532&action=edit Kernel 5.6.14-1-default log Happened right now on obs-arm-1 (ThunderX1) with kernel 5.6.14-1-default. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c14 --- Comment #14 from Guillaume GARDET <guillaume.gardet@arm.com> --- Happen again today on obs-arm-1 while building openSUSE:Factory:ARM:Live/JeOS:GNOME-efi.aarch64. This is with kernel 5.7.1-1-default. Kernel traces: [ 1768s] [ DEBUG ]: 01:27:54 | EXEC: [rsync -a -H -X -A --one-file-system --exclude /image --exclude /.profile --exclude /.kconfig --exclude /.buildenv --exclude /var/cache/kiwi --exclude /boot/efi/* --exclude /boot/efi/.* /usr/src/packages/KIWI-oem/build/image-root/ /tmp/kiwi_volumes.6fgf71g3/@/.snapshots/1/snapshot] [ 1986s] [ 1959.198156] Unable to handle kernel paging request at virtual address fffffffffffffdc0 [ 1986s] [ 1959.206746] Mem abort info: [ 1986s] [ 1959.208354] ESR = 0x96000004 [ 1986s] [ 1959.210177] EC = 0x25: DABT (current EL), IL = 32 bits [ 1986s] [ 1959.213101] SET = 0, FnV = 0 [ 1986s] [ 1959.215869] EA = 0, S1PTW = 0 [ 1986s] [ 1959.218844] Data abort info: [ 1986s] [ 1959.224086] ISV = 0, ISS = 0x00000004 [ 1986s] [ 1959.229306] CM = 0, WnR = 0 [ 1986s] [ 1959.234868] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000416a2000 [ 1986s] [ 1959.244463] [fffffffffffffdc0] pgd=0000000000000000 [ 1986s] [ 1959.249975] Internal error: Oops: 96000004 [#1] SMP [ 1986s] [ 1959.254779] Modules linked in: e1000 nls_iso8859_1 nls_cp437 vfat fat virtio_rng virtio_blk virtio_mmio xfs btrfs blake2b_generic xor xor_neon raid6_pq libcrc32c reiserfs squashfs fuse dm_snapshot dm_bufio dm_crypt dm_mod binfmt_misc loop sg [ 1986s] [ 1959.272691] CPU: 3 PID: 7830 Comm: kworker/u12:8 Not tainted 5.7.1-1-default #1 openSUSE Tumbleweed (unreleased) [ 1986s] [ 1959.281383] Hardware name: linux,dummy-virt (DT) [ 1986s] [ 1959.285885] Workqueue: btrfs-delalloc btrfs_work_helper [btrfs] [ 1986s] [ 1959.291117] pstate: 60000005 (nZCv daif -PAN -UAO) [ 1986s] [ 1959.295586] pc : submit_compressed_extents+0x54/0x414 [btrfs] [ 1986s] [ 1959.300917] lr : async_cow_submit+0x68/0xbc [btrfs] [ 1986s] [ 1959.305298] sp : ffff800018e93c10 [ 1986s] [ 1959.308280] x29: ffff800018e93c20 x28: ffff000057795148 [ 1986s] [ 1959.312972] x27: ffff800011d5c0a8 x26: ffff000057795130 [ 1986s] [ 1959.317666] x25: dead000000000100 x24: 0000000000000000 [ 1986s] [ 1959.322304] x23: ffff000057795108 x22: ffff000057795a48 [ 1986s] [ 1959.326939] x21: 0000000000000000 x20: ffff000057795108 [ 1986s] [ 1959.331546] x19: ffff000057795148 x18: 0000000000fffc18 [ 1986s] [ 1959.336237] x17: 00000000fffffffd x16: 0000000000000803 [ 1986s] [ 1959.340858] x15: 00000000fffffffe x14: b5845f6418f42f0a [ 1986s] [ 1959.345569] x13: 19ad3d2019bd3d0a x12: 0a2e29730226788c [ 1986s] [ 1959.350190] x11: 980a19b92dd0d8d9 x10: 70236c6d72654701 [ 1986s] [ 1959.354841] x9 : ffff800008ef31ec x8 : ffff00005829e000 [ 1986s] [ 1959.359476] x7 : ffff800011269000 x6 : 0000000000000002 [ 1986s] [ 1959.364053] x5 : 0000000000000000 x4 : ffff0000f7889518 [ 1986s] [ 1959.368257] x3 : 0000000000000000 x2 : ffff0000f78892e8 [ 1986s] [ 1959.371847] x1 : fffffffffffffdc0 x0 : ffff000057795130 [ 1986s] [ 1959.375286] Call trace: [ 1986s] [ 1959.377240] submit_compressed_extents+0x54/0x414 [btrfs] [ 1986s] [ 1959.381136] async_cow_submit+0x68/0xbc [btrfs] [ 1986s] [ 1959.384279] run_ordered_work+0xc8/0x300 [btrfs] [ 1986s] [ 1959.387450] btrfs_work_helper+0x98/0x2a0 [btrfs] [ 1986s] [ 1959.390466] process_one_work+0x1e4/0x490 [ 1986s] [ 1959.393172] worker_thread+0x170/0x440 [ 1986s] [ 1959.395604] kthread+0x11c/0x120 [ 1986s] [ 1959.397747] ret_from_fork+0x10/0x18 [ 1986s] [ 1959.400056] Code: d2800001 d10902a1 f90043e1 f94016e0 (f9400038) [ 1986s] [ 1959.404102] ---[ end trace 611c9ec75144d130 ]--- -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c15 Matthias Brugger <mbrugger@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(guillaume.gardet@ | |arm.com) --- Comment #15 from Matthias Brugger <mbrugger@suse.com> --- Can you get the memory dump from the crashed kernel. Maybe that would help to analyze the problem. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c16 Guillaume GARDET <guillaume.gardet@arm.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(guillaume.gardet@ | |arm.com) | --- Comment #16 from Guillaume GARDET <guillaume.gardet@arm.com> --- (In reply to Matthias Brugger from comment #15)
Can you get the memory dump from the crashed kernel. Maybe that would help to analyze the problem.
AFAIK, OBS does not produce it and I have no ThunderX1 to reproduce locally. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 Matthias Brugger <mbrugger@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ro@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c17 --- Comment #17 from Matthias Brugger <mbrugger@suse.com> --- Rudi, could we get a memory dump? I understand that the crash happens in the guest OS. Do we have a possibility to create a crash dump in the OBS guest at all? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 Matthias Brugger <mbrugger@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |http://bugzilla.opensuse.or | |g/show_bug.cgi?id=1155888 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c18 --- Comment #18 from Guillaume GARDET <guillaume.gardet@arm.com> --- The problem is back on obs-arm-2 host in OBS, with kernel 5.14.6 in Tumbleweed: [ 2380s] [ 2287.115388][T20685] Unable to handle kernel paging request at virtual address fffffffffffffdd0 [ 2380s] [ 2287.122121][T20685] Mem abort info: [ 2380s] [ 2287.125116][T20685] ESR = 0x96000004 [ 2380s] [ 2287.128094][T20685] EC = 0x25: DABT (current EL), IL = 32 bits [ 2380s] [ 2287.153836][T20685] SET = 0, FnV = 0 [ 2380s] [ 2287.156704][T20685] EA = 0, S1PTW = 0 [ 2380s] [ 2287.160310][T20685] FSC = 0x04: level 0 translation fault [ 2380s] [ 2287.164863][T20685] Data abort info: [ 2380s] [ 2287.167726][T20685] ISV = 0, ISS = 0x00000004 [ 2380s] [ 2287.192746][T20685] CM = 0, WnR = 0 [ 2380s] [ 2287.197319][T20685] swapper pgtable: 4k pages, 48-bit VAs, pgdp=0000000041a9d000 [ 2380s] [ 2287.206677][T20685] [fffffffffffffdd0] pgd=0000000000000000, p4d=0000000000000000 [ 2380s] [ 2287.216024][T20685] Internal error: Oops: 96000004 [#1] SMP [ 2380s] [ 2287.242512][T20685] Modules linked in: sha2_ce sha256_arm64 qemu_fw_cfg overlay e1000 nls_iso8859_1 nls_cp437 vfat fat virtio_rng virtio_blk virtio_mmio xfs btrfs blake2b_generic xor xor_neon raid6_pq libcrc32c reiserfs ext4 mbcache jbd2 squashfs fuse dm_snapshot dm_bufio dm_crypt essiv authenc trusted asn1_encoder tee dm_mod binfmt_misc loop sg [ 2380s] [ 2287.293948][T20685] CPU: 0 PID: 20685 Comm: kworker/u8:9 Not tainted 5.14.6-1-default #1 openSUSE Tumbleweed c1872505db63fbf00d9a52001e44bfff60e82ba4 [ 2380s] [ 2287.310914][T20685] Hardware name: linux,dummy-virt (DT) [ 2380s] [ 2287.317742][T20685] Workqueue: btrfs-delalloc btrfs_work_helper [btrfs] [ 2380s] [ 2287.326723][T20685] pstate: 80400005 (Nzcv daif +PAN -UAO -TCO BTYPE=--) [ 2380s] [ 2287.335554][T20685] pc : submit_compressed_extents+0x4c/0x40c [btrfs] [ 2380s] [ 2287.344185][T20685] lr : async_cow_submit+0x50/0xd0 [btrfs] [ 2380s] [ 2287.351864][T20685] sp : ffff8000184d3c50 [ 2380s] [ 2287.357041][T20685] x29: ffff8000184d3c60 x28: 0000000000000000 x27: ffffbda48d635000 [ 2380s] [ 2287.367200][T20685] x26: fffffffffffffdd0 x25: dead000000000100 x24: ffff000271672808 [ 2380s] [ 2287.377535][T20685] x23: ffff0000c7808410 x22: ffff000271672b48 x21: 0000000000000000 [ 2380s] [ 2287.387308][T20685] x20: ffff0000ca476000 x19: 0000000000000001 x18: 0000000000000000 [ 2380s] [ 2287.397152][T20685] x17: 0000000000000020 x16: ffffbda48bf5c1c0 x15: 08010a0304fe1201 [ 2380s] [ 2287.407122][T20685] x14: 0201040112011001 x13: ffff00027ebe8480 x12: 0000000000000020 [ 2380s] [ 2287.417248][T20685] x11: ffffbda48d650ca0 x10: ffff425df1505000 x9 : ffffbda45b008b50 [ 2380s] [ 2287.427189][T20685] x8 : fffffc00030b8048 x7 : fffffc00030b8040 x6 : 0000000000000000 [ 2380s] [ 2287.437034][T20685] x5 : dead000000000122 x4 : 0000000000000000 x3 : ffff0000c7808420 [ 2380s] [ 2287.447076][T20685] x2 : 0000000000000000 x1 : ffff000271672830 x0 : ffff000271672830 [ 2380s] [ 2287.457424][T20685] Call trace: [ 2380s] [ 2287.461571][T20685] submit_compressed_extents+0x4c/0x40c [btrfs 6c5d395590a945b218fd40a8a3d4ca1900960144] [ 2380s] [ 2287.474127][T20685] async_cow_submit+0x50/0xd0 [btrfs 6c5d395590a945b218fd40a8a3d4ca1900960144] [ 2380s] [ 2287.485617][T20685] btrfs_work_helper+0x134/0x4bc [btrfs 6c5d395590a945b218fd40a8a3d4ca1900960144] [ 2380s] [ 2287.497624][T20685] process_one_work+0x1e4/0x4a0 [ 2380s] [ 2287.503716][T20685] worker_thread+0x174/0x56c [ 2380s] [ 2287.509504][T20685] kthread+0x12c/0x140 [ 2380s] [ 2287.514475][T20685] ret_from_fork+0x10/0x18 [ 2380s] [ 2287.519980][T20685] Code: f9005fe2 d2800002 d108c2ba f9401700 (f9400357) [ 2380s] [ 2287.528698][T20685] ---[ end trace f6440f612969c083 ]--- -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c19 Wenruo Qu <wqu@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(guillaume.gardet@ | |arm.com) --- Comment #19 from Wenruo Qu <wqu@suse.com> --- Any code line number and context for "submit_compressed_extents+0x4c"? Just from the function name, it looks like the async_chunk->extents is NULL? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c20 Guillaume GARDET <guillaume.gardet@arm.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(guillaume.gardet@ | |arm.com) | --- Comment #20 from Guillaume GARDET <guillaume.gardet@arm.com> --- (gdb) l *submit_compressed_extents+0x4c 0x42748 is in submit_compressed_extents (../fs/btrfs/inode.c:845). 840 * and send them down to the disk. 841 */ 842 static noinline void submit_compressed_extents(struct async_chunk *async_chunk) 843 { 844 struct btrfs_inode *inode = BTRFS_I(async_chunk->inode); 845 struct btrfs_fs_info *fs_info = inode->root->fs_info; 846 struct async_extent *async_extent; 847 u64 alloc_hint = 0; 848 struct btrfs_key ins; 849 struct extent_map *em; -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c21 Guillaume GARDET <guillaume.gardet@arm.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(wqu@suse.com) --- Comment #21 from Guillaume GARDET <guillaume.gardet@arm.com> --- Anything I could do to help to move forward? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c22 Wenruo Qu <wqu@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(wqu@suse.com) | --- Comment #22 from Wenruo Qu <wqu@suse.com> --- I guess Nik has pinned down the root cause, and the upstream fix is here: https://patchwork.kernel.org/project/linux-btrfs/patch/20211102124916.433836... Mind to test if this patch solves the problem? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c23 --- Comment #23 from Guillaume GARDET <guillaume.gardet@arm.com> --- (In reply to Wenruo Qu from comment #22)
I guess Nik has pinned down the root cause, and the upstream fix is here:
https://patchwork.kernel.org/project/linux-btrfs/patch/20211102124916.433836... 1-nborisov@suse.com/
Mind to test if this patch solves the problem?
This patch is in kernel 5.15.5 which is currently tested in staging, and thus, should land in :Factory:ARM in the coming days. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160604 http://bugzilla.opensuse.org/show_bug.cgi?id=1160604#c24 Guillaume GARDET <guillaume.gardet@arm.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #24 from Guillaume GARDET <guillaume.gardet@arm.com> --- No issue since kernel 5.15.5 has been merged. So let's close it! -- You are receiving this mail because: You are on the CC list for the bug.
participants (2)
-
bugzilla_noreply＠novell.com -
bugzilla_noreply＠suse.com