http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c1 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fcrozat@suse.com, | |nborisov@suse.com Flags| |needinfo?(fcrozat@suse.com) --- Comment #1 from Takashi Iwai --- FWIW, it's Leap 42.2, thus corresponding to SLE12-SP2. The commit id should be c1231a7a1efbc7b3f9efc602e80cc0f45b031bba. Just to be sure, the bug is still present in the kernel in OBS Kernel:openSUSE-42.2, too? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c3 Nikolay Borisov changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(fcrozat@suse.com) --- Comment #3 from Nikolay Borisov --- So while reading the chunk tree the mount code finds discrepancy between the size written in the super block and the size of the underlying device. It's interesting that the difference is negligible - 3584 bytes. Could someone with affected system (I haven't been able to reproduce this) attach the output of the following two commands: btrfs inspect-internal dump-super -af /dev/vdb btrfs inspect-internal dump-tree -d /dev/vdb they will dump respectively the all superblocks and the the device-related trees. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c5 Frederic Crozat changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fcrozat@suse.com) | --- Comment #5 from Frederic Crozat --- Created attachment 728738 --> http://bugzilla.opensuse.org/attachment.cgi?id=728738&action=edit dump tree for affected partition -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c7 Frederic Crozat changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fcrozat@suse.com) | --- Comment #7 from Frederic Crozat --- (In reply to Nikolay Borisov from comment #6)

Can you also provide the output of lsblk -b /dev/sdb.

# lsblk -b /dev/sdb NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 0 1000204886016 0 disk └─sdb1 8:17 0 1000203820544 0 part -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c9 --- Comment #9 from Frederic Crozat --- (In reply to Nikolay Borisov from comment #8)

So all the provided information points that the actual size of the underlying device is 1000203820544, which is 3.5kb larger than what the super block thinks. This is also correlated by both the info in the device extent tree and in the multiple copies of the super block. Independently of the filesystem, lsblk also confirms that the device size is 1000203820544.This is odd indeed, so if I understand correctly this volume worked fine before you upgraded to 42.2?

Correction. This volume is working fine with 42.2 release kernel and all maintenance kernel until 4.4.62-18.6-default (included). It is no longer mountable with latest maintenance kernel

If so could you show what number is reported by lsblkid on the old kernel, where this used to work?

with 4.4.62-18.6-default kernel: # lsblk -b /dev/sdb NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sdb 8:16 0 1000204886016 0 disk └─sdb1 8:17 0 1000203820544 0 part /mnt/cache

Also have you performed any device replace operations?

No, this is a disk I added to my system a long time ago (either with openSUSE 13.1 or 42.1, I'm not 100% sure), I don't remember doing any replace operations on it. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c11 --- Comment #11 from Frederic Crozat --- (In reply to Nikolay Borisov from comment #10)

Right, then it's apparent why it doesn't want to mount - the check which fails the mount was not present until rpm-4.4.63-1. It's only a consistency check. However, the more pressing question the volume got in a situation where device size doesn't equal to what is recorded in the superblock. I guess there are 2 courses of action:

1) Revert the offending patch, but it's supposed to harden the the kernel against such "corruption" ( the quotes are due to my not being sure that it's actually a corruption)

Would a online resize of the FS be enough to "fix" the issue ?

2) If you have enough freespace on a different drive, try to recreate the filesystem via btrfs send/receive. Obviously this might work in your case but we don't know how widespread this bug could be, it seems so far no one has complained.

I'm really concern we might see this from our SP2 customers at some point as L3 (we never know when they will apply maintenance update)..

David, Jeff what do you make out of this ?

-- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c15 --- Comment #15 from Frederic Crozat --- (In reply to Nikolay Borisov from comment #14)

I've built a 42.2-based kernel with possible patches which fix the issue here: https://build.suse.de/project/show/home:nikbor:bsc1043912

Can you please test it and confirm that they work before I push-them for merging?

I will be able to test it only next Sunday. Will report then -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c19 --- Comment #19 from Frederic Crozat --- (In reply to Nikolay Borisov from comment #18)

I've built yet another kernel off latest 42.2, it can be found here: https://build.suse.de/project/show/home:nikbor:bsc1043912

It contains a patch which rounds down the values used in the check and should hopefully allow Frederic to mount his system. If everything checks out I will commit it.

I can confirm kernel 4.4.73-1.g16a5f2a-default properly mounts my "problematic" btrfs partition However, I got the following warning (not sure if it is related): [ 54.116369] WARNING: CPU: 0 PID: 278 at ../fs/btrfs/ctree.h:1513 btrfs_update_device+0x1ad/0x1c0 [btrfs]() [ 54.116398] ath9k_hw gpio_ich snd_hda_core snd_hwdep snd_pcm_oss iTCO_wdt iTCO_vendor_support lrw gf128mul ath r8169 snd_pcm mac80211 joydev glue_helper mii ablk_helper lpc_ich mfd_core snd_seq cfg80211 rfkill fjes snd_seq_device snd_timer snd_mixer_oss processor i2c_i801 snd cryptd mei_me shpchp mei soundcore pcspkr btrfs xor hid_microsoft hid_generic usbhid raid6_pq sr_mod cdrom sd_mod crc32c_intel ahci libahci i915 libata video i2c_algo_bit xhci_pci drm_kms_helper ehci_pci syscopyarea sysfillrect xhci_hcd sysimgblt ehci_hcd fb_sys_fops usbcore drm usb_common button sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod efivarfs autofs4 [ 54.116438] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs] [ 54.116490] [<ffffffffa04c66fd>] btrfs_update_device+0x1ad/0x1c0 [btrfs] [ 54.116512] [<ffffffffa04cbf91>] btrfs_finish_chunk_alloc+0x151/0x590 [btrfs] [ 54.116521] [<ffffffffa04875c7>] btrfs_create_pending_block_groups+0x137/0x280 [btrfs] [ 54.116539] [<ffffffffa049ff53>] __btrfs_end_transaction+0x93/0x350 [btrfs] [ 54.116548] [<ffffffffa0488b6f>] flush_space+0xef/0x5f0 [btrfs] [ 54.116557] [<ffffffffa048918b>] btrfs_async_reclaim_metadata_space+0x11b/0x4b0 [btrfs] [ 54.117794] WARNING: CPU: 0 PID: 278 at ../fs/btrfs/ctree.h:1513 btrfs_update_device+0x1ad/0x1c0 [btrfs]() [ 54.117819] gpio_ich snd_hda_core snd_hwdep snd_pcm_oss iTCO_wdt iTCO_vendor_support lrw gf128mul ath r8169 snd_pcm mac80211 joydev glue_helper mii ablk_helper lpc_ich mfd_core snd_seq cfg80211 rfkill fjes snd_seq_device snd_timer snd_mixer_oss processor i2c_i801 snd cryptd mei_me shpchp mei soundcore pcspkr btrfs xor hid_microsoft hid_generic usbhid raid6_pq sr_mod cdrom sd_mod crc32c_intel ahci libahci i915 libata video i2c_algo_bit xhci_pci drm_kms_helper ehci_pci syscopyarea sysfillrect xhci_hcd sysimgblt ehci_hcd fb_sys_fops usbcore drm usb_common button sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod efivarfs autofs4 [ 54.117855] Workqueue: events_unbound btrfs_async_reclaim_metadata_space [btrfs] [ 54.117884] [<ffffffffa04c66fd>] btrfs_update_device+0x1ad/0x1c0 [btrfs] [ 54.117896] [<ffffffffa04cbf91>] btrfs_finish_chunk_alloc+0x151/0x590 [btrfs] [ 54.117905] [<ffffffffa04875c7>] btrfs_create_pending_block_groups+0x137/0x280 [btrfs] [ 54.117915] [<ffffffffa049ff53>] __btrfs_end_transaction+0x93/0x350 [btrfs] [ 54.117925] [<ffffffffa0488b6f>] flush_space+0xef/0x5f0 [btrfs] [ 54.117933] [<ffffffffa048918b>] btrfs_async_reclaim_metadata_space+0x11b/0x4b0 [btrfs] -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c26 Frederic Crozat changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|FIXED |--- --- Comment #26 from Frederic Crozat --- I just upgraded my Leap 42.2 system to the latest kernel-default from maintenance update ( 4.4.74-18.20-default ) and problem is still there: [ 7.052652] BTRFS info (device sdb1): disk space caching is enabled [ 7.060731] BTRFS error (device sdb1): super_total_bytes 1000203816960 mismatch with fs_devices total_rw_bytes 1000203820544 [ 7.060733] BTRFS error (device sdb1): failed to read chunk tree: -22 [ 7.110461] BTRFS error (device sdb1): open_ctree failed I look like the fix didn't make it or isn't properly applied -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c28 --- Comment #28 from Nikolay Borisov --- (In reply to Frederic Crozat from comment #26)

I just upgraded my Leap 42.2 system to the latest kernel-default from maintenance update ( 4.4.74-18.20-default ) and problem is still there:

[ 7.052652] BTRFS info (device sdb1): disk space caching is enabled [ 7.060731] BTRFS error (device sdb1): super_total_bytes 1000203816960 mismatch with fs_devices total_rw_bytes 1000203820544 [ 7.060733] BTRFS error (device sdb1): failed to read chunk tree: -22 [ 7.110461] BTRFS error (device sdb1): open_ctree failed

I look like the fix didn't make it or isn't properly applied

So can yo perform the following steps: 1. Boot your system with the last kernel I provided you with: https://build.suse.de/project/show/home:nikbor:bsc1043912 2. You should be able to mount the system and will likely see the warnings your described before. 3. Run btrfs balance as before - this time you shouldn't see the warnings. At this point I'm expecting to see rounded-down values stored on disk. To verify this again provide a dump of the super block and the device tree for the affected partition with those 2 commands: btrfs inspect-internal dump-super -af /dev/vdb btrfs inspect-internal dump-tree -d /dev/vdb -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c30 Frederic Crozat changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #728737|0 |1 is obsolete| | --- Comment #30 from Frederic Crozat --- Created attachment 732171 --> http://bugzilla.opensuse.org/attachment.cgi?id=732171&action=edit dump super -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c32 --- Comment #32 from Nikolay Borisov --- (In reply to Frederic Crozat from comment #29)

(In reply to Nikolay Borisov from comment #28)

...

(In reply to Frederic Crozat from comment #26)

...

I just upgraded my Leap 42.2 system to the latest kernel-default from maintenance update ( 4.4.74-18.20-default ) and problem is still there:

[ 7.052652] BTRFS info (device sdb1): disk space caching is enabled [ 7.060731] BTRFS error (device sdb1): super_total_bytes 1000203816960 mismatch with fs_devices total_rw_bytes 1000203820544 [ 7.060733] BTRFS error (device sdb1): failed to read chunk tree: -22 [ 7.110461] BTRFS error (device sdb1): open_ctree failed

I look like the fix didn't make it or isn't properly applied

So can yo perform the following steps:

1. Boot your system with the last kernel I provided you with: https://build.suse.de/project/show/home:nikbor:bsc1043912

2. You should be able to mount the system and will likely see the warnings your described before.

I was able to mount the fs with this kernel but no warning at all.

I'm attaching both dump tree and super before running balance.

Can you attach logs AFTER you have run balance. What I'm seeing in the dumps provided is that the DEV_ITEM still has unrounded value: item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 3897 itemsize 98 devid 1 total_bytes 1000203820544 bytes_used 426351001600 My expectation would be that rebalance should store a rounded-down value. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c35 --- Comment #35 from Frederic Crozat --- Created attachment 732502 --> http://bugzilla.opensuse.org/attachment.cgi?id=732502&action=edit dump super after resize -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c39 --- Comment #39 from Frederic Crozat --- (In reply to Frederic Crozat from comment #38)

Latest Leap 42.2 maintenance kernel was still not able to mount the partition. I'll install a TW kernel and test later

Latest kernel for TW (Kernel:stable) 4.12.2-1.g1b6adc0-default doesn't mount the partition: [ 8.312756] BTRFS error (device sdb1): super_total_bytes 1000203808768 mismatch with fs_devices total_rw_bytes 1000203812864 [ 8.312796] BTRFS error (device sdb1): failed to read chunk tree: -22 [ 8.367709] BTRFS error (device sdb1): open_ctree failed -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1043912 http://bugzilla.opensuse.org/show_bug.cgi?id=1043912#c43 --- Comment #43 from Frederic Crozat --- (In reply to Nikolay Borisov from comment #42)

(In reply to Frederic Crozat from comment #41)

...

When asked to resize, I ran btrfs fi resize 1000203816448 /mnt/cache (which is 1000203820544 - 4096)

Okay this makes sense and it exposes a missed site which should have been patched my previous attempt at fixing it. I've updated the patches and pushed them to build at https://build.suse.de/project/show/home:nikbor:bsc1043912. When it's done can you please test by downlading that kernel and booting into it and then trying to grow the volume to its maximum capacity:

After rebooting with this kernel: [ 4.469419] BTRFS error (device sdb1): super_total_bytes 1000203808768 mismatch with fs_devices total_rw_bytes 1000203812864 but the partition was properly mounted

btrfs fi resize max /mnt/cache

[ 147.637427] BTRFS info (device sdb1): new size for /dev/sdb1 is 1000203816960

Then report the numbers from these two commands:

btrfs inspect dump-super /mnt/cache | grep total

(proper command needs device name, not mount path) total_bytes 1000203812864 dev_item.total_bytes 1000203816960

btrfs inspect dump-tree /mnt/cache | grep total devid 1 total_bytes 1000203816960 bytes_used 426351001600 total bytes 1000203812864

I then unmounted /mnt/cache and re-mounted it (without rebooting): [ 405.607741] BTRFS error (device sdb1): super_total_bytes 1000203812864 mismatch with fs_devices total_rw_bytes 1000203816960 (but partition was mounted) -- You are receiving this mail because: You are on the CC list for the bug.