Hello,
while running a git based kernel build from openSUSE-42.2 commit id
328b08be8a2f8a1d09c27e5f8038f9c7446669c3.
I'm no longer able to write more than 1 to 2 hours of data to my disk.
# dmesg:
------------[ cut here ]------------
WARNING: CPU: 5 PID: 6589 at fs/btrfs/extent-tree.c:2956
btrfs_run_delayed_refs+0x282/0x2e0 [btrfs]()
BTRFS: Transaction aborted (error -28)
Modules linked in: netconsole xt_multiport ipt_REJECT nf_reject_ipv4 xt_set
iptable_filter ip_tables x_tables ip_set_hash_net ip_set nfnetlink 8021q garp
bonding ipmi_devintf x86_pkg_temp_thermal coretemp ast kvm_intel ttm kvm
drm_kms_helper irqbypass drm crc32_pclmul ghash_clmulni_intel sb_edac
fb_sys_fops syscopyarea sysfillrect sysimgblt edac_core i40e(O) vxlan
ip6_udp_tunnel udp_tunnel shpchp wmi ipmi_si ipmi_msghandler button loop btrfs
dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid0 multipath linear usbhid raid1 md_mod sg sd_mod
xhci_pci ehci_pci xhci_hcd ehci_hcd i2c_i801 ahci igb usbcore i2c_algo_bit
usb_common libahci i2c_core ptp pps_core aacraid
CPU: 5 PID: 6589 Comm: btrfs-transacti Tainted: G W O 4.4.65+132-ph
#1
Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
0000000000000000 ffff880a7847bbb8 ffffffff813c23f3 ffff880a7847bc00
ffffffffc037114d ffff880a7847bbf0 ffffffff81083746 ffff880ff32f23c0
ffff880cac6c2000 ffff880fb4ee6060 ffffffffffffffff ffff880cac6c2000
Call Trace:
[<ffffffff81008b45>] try_stack_unwind+0x175/0x190
[<ffffffff81007d24>] dump_trace+0x64/0x380
[<ffffffff81008bad>] show_trace_log_lvl+0x4d/0x60
[<ffffffff81008148>] show_stack_log_lvl+0x108/0x170
[<ffffffff81008c05>] show_stack+0x25/0x70
[<ffffffff813c23f3>] dump_stack+0x63/0x90
[<ffffffff81083746>] warn_slowpath_common+0x86/0xc0
[<ffffffff810837cc>] warn_slowpath_fmt+0x4c/0x50
[<ffffffffc02d6322>] btrfs_run_delayed_refs+0x282/0x2e0 [btrfs]
[<ffffffffc02ebdd8>] create_pending_snapshot.isra.23+0x5b8/0x1010 [btrfs]
[<ffffffffc02ec8ba>] create_pending_snapshots+0x8a/0xb0 [btrfs]
[<ffffffffc02ecf11>] btrfs_commit_transaction.part.24+0x371/0xa30 [btrfs]
[<ffffffffc02ed60a>] btrfs_commit_transaction+0x3a/0x70 [btrfs]
[<ffffffffc02e78df>] transaction_kthread+0x1ff/0x260 [btrfs]
[<ffffffff810a0f5b>] kthread+0xeb/0x110
[<ffffffff816bdf0f>] ret_from_fork+0x3f/0x70
DWARF2 unwinder stuck at ret_from_fork+0x3f/0x70
Leftover inexact backtrace:
[<ffffffff810a0e70>] ? kthread_park+0x60/0x60
---[ end trace 457d5c0a7792677e ]---
BTRFS: error (device sdc1) in btrfs_run_delayed_refs:2956: errno=-28 No space
left
BTRFS info (device sdc1): forced readonly
BTRFS: error (device sdc1) in create_pending_snapshot:1603: errno=-28 No space
left
BTRFS warning (device sdc1): Skipping commit of aborted transaction.
BTRFS: error (device sdc1) in cleanup_transaction:1852: errno=-28 No space left
BTRFS error (device sdc1): pending csums is 383705088
I i umount the device i get a new trace:
# umount /backup
# dmesg -t
BTRFS error (device sdc1): cleaner transaction attach returned -30
------------[ cut here ]------------
WARNING: CPU: 2 PID: 12105 at fs/btrfs/extent-tree.c:133
btrfs_put_block_group+0x60/0x70 [btrfs]()
Modules linked in: netconsole xt_multiport ipt_REJECT nf_reject_ipv4 xt_set
iptable_filter ip_tables x_tables ip_set_hash_net ip_set nfnetlink 8021q garp
bonding ipmi_devintf x86_pkg_temp_thermal coretemp ast kvm_intel ttm kvm
drm_kms_helper irqbypass drm crc32_pclmul ghash_clmulni_intel sb_edac
fb_sys_fops syscopyarea sysfillrect sysimgblt edac_core i40e(O) vxlan
ip6_udp_tunnel udp_tunnel shpchp wmi ipmi_si ipmi_msghandler button loop btrfs
dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid0 multipath linear usbhid raid1 md_mod sg sd_mod
xhci_pci ehci_pci xhci_hcd ehci_hcd i2c_i801 ahci igb usbcore i2c_algo_bit
usb_common libahci i2c_core ptp pps_core aacraid
CPU: 2 PID: 12105 Comm: umount Tainted: G W O 4.4.65+132-ph #1
Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
0000000000000000 ffff880753a13ce0 ffffffff813c23f3 0000000000000000
ffffffffc037114d ffff880753a13d18 ffffffff81083746 ffff881030d25000
ffff881030d25000 ffff88103c3c4088 ffff88103c3c4000 ffff88103c3c4098
Call Trace:
[<ffffffff81008b45>] try_stack_unwind+0x175/0x190
[<ffffffff81007d24>] dump_trace+0x64/0x380
[<ffffffff81008bad>] show_trace_log_lvl+0x4d/0x60
[<ffffffff81008148>] show_stack_log_lvl+0x108/0x170
[<ffffffff81008c05>] show_stack+0x25/0x70
[<ffffffff813c23f3>] dump_stack+0x63/0x90
[<ffffffff81083746>] warn_slowpath_common+0x86/0xc0
[<ffffffff8108383a>] warn_slowpath_null+0x1a/0x20
[<ffffffffc02cd670>] btrfs_put_block_group+0x60/0x70 [btrfs]
[<ffffffffc02d5358>] btrfs_free_block_groups+0xf8/0x430 [btrfs]
[<ffffffffc02e7a8d>] close_ctree+0x14d/0x320 [btrfs]
[<ffffffffc02b6429>] btrfs_put_super+0x19/0x20 [btrfs]
[<ffffffff811d2aff>] generic_shutdown_super+0x6f/0x100
[<ffffffff811d2ba2>] kill_anon_super+0x12/0x20
[<ffffffffc02b71b6>] btrfs_kill_super+0x16/0x90 [btrfs]
[<ffffffff811d24d3>] deactivate_locked_super+0x43/0x70
[<ffffffff811d255c>] deactivate_super+0x5c/0x60
[<ffffffff811ee23f>] cleanup_mnt+0x3f/0x90
[<ffffffff811ee2d2>] __cleanup_mnt+0x12/0x20
[<ffffffff8109f51e>] task_work_run+0x7e/0xa0
[<ffffffff81002383>] exit_to_usermode_loop+0xf3/0x100
[<ffffffff81002bb4>] syscall_return_slowpath+0xd4/0x130
[<ffffffff816bdd0c>] int_ret_from_sys_call+0x25/0x8f
DWARF2 unwinder stuck at int_ret_from_sys_call+0x25/0x8f
Leftover inexact backtrace:
---[ end trace 457d5c0a7792677f ]---
------------[ cut here ]------------
WARNING: CPU: 2 PID: 12105 at fs/btrfs/extent-tree.c:9818
btrfs_free_block_groups+0x291/0x430 [btrfs]()
Modules linked in: netconsole xt_multiport ipt_REJECT nf_reject_ipv4 xt_set
iptable_filter ip_tables x_tables ip_set_hash_net ip_set nfnetlink 8021q garp
bonding ipmi_devintf x86_pkg_temp_thermal coretemp ast kvm_intel ttm kvm
drm_kms_helper irqbypass drm crc32_pclmul ghash_clmulni_intel sb_edac
fb_sys_fops syscopyarea sysfillrect sysimgblt edac_core i40e(O) vxlan
ip6_udp_tunnel udp_tunnel shpchp wmi ipmi_si ipmi_msghandler button loop btrfs
dm_mod raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid0 multipath linear usbhid raid1 md_mod sg sd_mod
xhci_pci ehci_pci xhci_hcd ehci_hcd i2c_i801 ahci igb usbcore i2c_algo_bit
usb_common libahci i2c_core ptp pps_core aacraid
CPU: 2 PID: 12105 Comm: umount Tainted: G W O 4.4.65+132-ph #1
Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
0000000000000000 ffff880753a13cf8 ffffffff813c23f3 0000000000000000
ffffffffc037114d ffff880753a13d30 ffffffff81083746 ffff880f0feb2200
ffff880f0feb2288 ffff88103c3c4088 ffff88103c3c4000 ffff88103c3c4098
Call Trace:
[<ffffffff81008b45>] try_stack_unwind+0x175/0x190
[<ffffffff81007d24>] dump_trace+0x64/0x380
[<ffffffff81008bad>] show_trace_log_lvl+0x4d/0x60
[<ffffffff81008148>] show_stack_log_lvl+0x108/0x170
[<ffffffff81008c05>] show_stack+0x25/0x70
[<ffffffff813c23f3>] dump_stack+0x63/0x90
[<ffffffff81083746>] warn_slowpath_common+0x86/0xc0
[<ffffffff8108383a>] warn_slowpath_null+0x1a/0x20
[<ffffffffc02d54f1>] btrfs_free_block_groups+0x291/0x430 [btrfs]
[<ffffffffc02e7a8d>] close_ctree+0x14d/0x320 [btrfs]
[<ffffffffc02b6429>] btrfs_put_super+0x19/0x20 [btrfs]
[<ffffffff811d2aff>] generic_shutdown_super+0x6f/0x100
[<ffffffff811d2ba2>] kill_anon_super+0x12/0x20
[<ffffffffc02b71b6>] btrfs_kill_super+0x16/0x90 [btrfs]
[<ffffffff811d24d3>] deactivate_locked_super+0x43/0x70
[<ffffffff811d255c>] deactivate_super+0x5c/0x60
[<ffffffff811ee23f>] cleanup_mnt+0x3f/0x90
[<ffffffff811ee2d2>] __cleanup_mnt+0x12/0x20
[<ffffffff8109f51e>] task_work_run+0x7e/0xa0
[<ffffffff81002383>] exit_to_usermode_loop+0xf3/0x100
[<ffffffff81002bb4>] syscall_return_slowpath+0xd4/0x130
[<ffffffff816bdd0c>] int_ret_from_sys_call+0x25/0x8f
DWARF2 unwinder stuck at int_ret_from_sys_call+0x25/0x8f
Leftover inexact backtrace:
---[ end trace 457d5c0a77926780 ]---
BTRFS info (device sdc1): space_info 1 has 144388956160 free, is not full
BTRFS info (device sdc1): space_info total=21839908700160, used=21695519477760,
pinned=0, reserved=4096, may_use=0, readonly=262144
# btrfs filesystem df /vmbackup/
Data, RAID0: total=19.86TiB, used=19.72TiB
System, RAID0: total=64.00MiB, used=1.23MiB
Metadata, RAID0: total=76.00GiB, used=72.62GiB
GlobalReserve, single: total=512.00MiB, used=6.16MiB
# btrfs filesystem usage /vmbackup/
Overall:
Device size: 58.20TiB
Device allocated: 19.94TiB
Device unallocated: 38.26TiB
Device missing: 0.00B
Used: 19.79TiB
Free (estimated): 38.41TiB (min: 38.41TiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 213.95MiB)
Data,RAID0: Size:19.86TiB, Used:19.72TiB
/dev/sdc1 4.97TiB
/dev/sdd1 4.97TiB
/dev/sde1 4.97TiB
/dev/sdf1 4.97TiB
Metadata,RAID0: Size:76.00GiB, Used:72.61GiB
/dev/sdc1 19.00GiB
/dev/sdd1 19.00GiB
/dev/sde1 19.00GiB
/dev/sdf1 19.00GiB
System,RAID0: Size:64.00MiB, Used:1.23MiB
/dev/sdc1 16.00MiB
/dev/sdd1 16.00MiB
/dev/sde1 16.00MiB
/dev/sdf1 16.00MiB
Unallocated:
/dev/sdc1 9.57TiB
/dev/sdd1 9.57TiB
/dev/sde1 9.57TiB
/dev/sdf1 9.57TiB
I already tried to clear the space cache but it happens again after some time.
Greets,
Stefan