[opensuse-kernel] 13.1 RC1: unable to rename any directory or file on btrfs
Is it expected? I obviously do have enough space and can create (and delete) files and directories but am not able to rename anything. Do I need to open bug report? linux-1a7f:~ # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB linux-1a7f:~ # cd /mnt/dir1/ linux-1a7f:/mnt/dir1 # ls -l total 0 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 linux-1a7f:/mnt/dir1 # mv dir2 dir3 mv: cannot move ‘dir2’ to ‘dir3’: No space left on device linux-1a7f:/mnt/dir1 # mv file1 file2 mv: cannot move ‘file1’ to ‘file2’: No space left on device linux-1a7f:/mnt/dir1 # echo ghdfgshf hdsghfj > file2 linux-1a7f:/mnt/dir1 # mkdir dir3 linux-1a7f:/mnt/dir1 # ls -l total 4 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 drwxr-xr-x 1 root root 0 Oct 14 10:58 dir3 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 -rw-r--r-- 1 root root 17 Oct 14 10:58 file2 linux-1a7f:/mnt/dir1 # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB linux-1a7f:~ # uname -a Linux linux-1a7f 3.11.3-1-pae #1 SMP Wed Oct 2 05:54:02 UTC 2013 (485f0d1) i686 i686 i386 GNU/Linux -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hello What does # df -h show --Glenn example: # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 12G 8.7G 2.4G 79% / devtmpfs 3.8G 16K 3.8G 1% /dev tmpfs 3.8G 76K 3.8G 1% /dev/shm tmpfs 3.8G 4.5M 3.8G 1% /run tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup tmpfs 3.8G 4.5M 3.8G 1% /var/run tmpfs 3.8G 4.5M 3.8G 1% /var/lock /dev/sda3 28G 24G 4.6G 84% /home -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
В Sun, 13 Oct 2013 01:51:29 +1100
Hello What does # df -h show --Glenn
example: # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda2 12G 8.7G 2.4G 79% / devtmpfs 3.8G 16K 3.8G 1% /dev tmpfs 3.8G 76K 3.8G 1% /dev/shm tmpfs 3.8G 4.5M 3.8G 1% /run tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup tmpfs 3.8G 4.5M 3.8G 1% /var/run tmpfs 3.8G 4.5M 3.8G 1% /var/lock /dev/sda3 28G 24G 4.6G 84% /home
Filesystem Size Used Avail Use% Mounted on /dev/sda2 8.7G 1.3G 7.0G 15% / devtmpfs 371M 32K 371M 1% /dev tmpfs 375M 0 375M 0% /dev/shm tmpfs 375M 1.4M 373M 1% /run tmpfs 375M 0 375M 0% /sys/fs/cgroup tmpfs 375M 1.4M 373M 1% /var/run tmpfs 375M 1.4M 373M 1% /var/lock /dev/sdb1 9.0M 64K 5.0M 2% /mnt -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hi, due to the btree and copy-on-write nature, such strange behaviour might appear. We are looking for such bugs and try to fix them urgently. Please open a bugreport. Ciao, Marcus On Mon, Oct 14, 2013 at 11:05:12AM +0400, Andrey Borzenkov wrote:
Is it expected? I obviously do have enough space and can create (and delete) files and directories but am not able to rename anything. Do I need to open bug report?
linux-1a7f:~ # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB linux-1a7f:~ # cd /mnt/dir1/ linux-1a7f:/mnt/dir1 # ls -l total 0 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 linux-1a7f:/mnt/dir1 # mv dir2 dir3 mv: cannot move ‘dir2’ to ‘dir3’: No space left on device linux-1a7f:/mnt/dir1 # mv file1 file2 mv: cannot move ‘file1’ to ‘file2’: No space left on device linux-1a7f:/mnt/dir1 # echo ghdfgshf hdsghfj > file2 linux-1a7f:/mnt/dir1 # mkdir dir3 linux-1a7f:/mnt/dir1 # ls -l total 4 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 drwxr-xr-x 1 root root 0 Oct 14 10:58 dir3 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 -rw-r--r-- 1 root root 17 Oct 14 10:58 file2 linux-1a7f:/mnt/dir1 # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB linux-1a7f:~ # uname -a Linux linux-1a7f 3.11.3-1-pae #1 SMP Wed Oct 2 05:54:02 UTC 2013 (485f0d1) i686 i686 i386 GNU/Linux -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
Hello What does # du -m /mnt/dir1/ | sort -n show ? I see in your report:5.0M Avail on /mnt Filesystem Size Used Avail Use% Mounted on /dev/sdb1 9.0M 64K 5.0M 2% /mnt If dir2 is > 5.0M ,expect the move not to work as there only 5.0M free on disk. --Glenn example: # du -m /tmp/YaST2-02248-3RjZxH | sort -n 1 /tmp/YaST2-02248-3RjZxH 1 /tmp/YaST2-02248-3RjZxH/etc 1 /tmp/YaST2-02248-3RjZxH/etc/default 1 /tmp/YaST2-02248-3RjZxH/etc/sysconfig 1 /tmp/YaST2-02248-3RjZxH/product-license 1 /tmp/YaST2-02248-3RjZxH/product-license/LiveCD -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
В Sun, 13 Oct 2013 07:39:06 +1100
Hello What does # du -m /mnt/dir1/ | sort -n show ? I see in your report:5.0M Avail on /mnt
Filesystem Size Used Avail Use% Mounted on /dev/sdb1 9.0M 64K 5.0M 2% /mnt
If dir2 is > 5.0M ,expect the move not to work as there only 5.0M free on disk.
linux-1a7f:~ # du -m /mnt/dir1 | sort -n 0 /mnt/dir1/dir2 0 /mnt/dir1/dir3 1 /mnt/dir1 linux-1a7f:~ # exit -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 10/14/13 4:59 AM, Marcus Meissner wrote:
Hi,
due to the btree and copy-on-write nature, such strange behaviour might appear.
We are looking for such bugs and try to fix them urgently. Please open a bugreport.
Indeed. Though for a 10 MB file system, I expect the answer is that the calculation is being a bit overzealous. 10 MB would be in the noise for most real use cases. -Jeff
Ciao, Marcus On Mon, Oct 14, 2013 at 11:05:12AM +0400, Andrey Borzenkov wrote:
Is it expected? I obviously do have enough space and can create (and delete) files and directories but am not able to rename anything. Do I need to open bug report?
linux-1a7f:~ # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB linux-1a7f:~ # cd /mnt/dir1/ linux-1a7f:/mnt/dir1 # ls -l total 0 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 linux-1a7f:/mnt/dir1 # mv dir2 dir3 mv: cannot move ‘dir2’ to ‘dir3’: No space left on device linux-1a7f:/mnt/dir1 # mv file1 file2 mv: cannot move ‘file1’ to ‘file2’: No space left on device linux-1a7f:/mnt/dir1 # echo ghdfgshf hdsghfj > file2 linux-1a7f:/mnt/dir1 # mkdir dir3 linux-1a7f:/mnt/dir1 # ls -l total 4 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 drwxr-xr-x 1 root root 0 Oct 14 10:58 dir3 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 -rw-r--r-- 1 root root 17 Oct 14 10:58 file2 linux-1a7f:/mnt/dir1 # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB linux-1a7f:~ # uname -a Linux linux-1a7f 3.11.3-1-pae #1 SMP Wed Oct 2 05:54:02 UTC 2013 (485f0d1) i686 i686 i386 GNU/Linux -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
-- Jeff Mahoney SUSE Labs
On Mon, Oct 14, 2013 at 11:05:12AM +0400, Andrey Borzenkov wrote:
Is it expected? I obviously do have enough space and can create (and delete) files and directories but am not able to rename anything. Do I need to open bug report?
As Jeff noted, 10MB is really small and the effects of the safety over-reservation calculations are noticeable. Following the steps
linux-1a7f:~ # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB
(~5mb of space to operate)
linux-1a7f:~ # cd /mnt/dir1/ linux-1a7f:/mnt/dir1 # ls -l total 0 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 linux-1a7f:/mnt/dir1 # mv dir2 dir3 mv: cannot move ‘dir2’ to ‘dir3’: No space left on device linux-1a7f:/mnt/dir1 # mv file1 file2 mv: cannot move ‘file1’ to ‘file2’: No space left on device
I can rename the files with 3.12-rc5, and I believe this is due to patch "Btrfs: fix worst case calculator for space usage" that should be safe to take to opensuse 13.1, but not at this time of release period. david -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On 10/15/13 7:55 AM, David Sterba wrote:
On Mon, Oct 14, 2013 at 11:05:12AM +0400, Andrey Borzenkov wrote:
Is it expected? I obviously do have enough space and can create (and delete) files and directories but am not able to rename anything. Do I need to open bug report?
As Jeff noted, 10MB is really small and the effects of the safety over-reservation calculations are noticeable.
Following the steps
linux-1a7f:~ # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB
(~5mb of space to operate)
linux-1a7f:~ # cd /mnt/dir1/ linux-1a7f:/mnt/dir1 # ls -l total 0 drwxr-xr-x 1 root root 0 Oct 14 09:43 dir2 -rw-r--r-- 1 root root 0 Oct 14 09:43 file1 linux-1a7f:/mnt/dir1 # mv dir2 dir3 mv: cannot move ‘dir2’ to ‘dir3’: No space left on device linux-1a7f:/mnt/dir1 # mv file1 file2 mv: cannot move ‘file1’ to ‘file2’: No space left on device
I can rename the files with 3.12-rc5, and I believe this is due to patch "Btrfs: fix worst case calculator for space usage" that should be safe to take to opensuse 13.1, but not at this time of release period.
I still can't rename a file using -rc5, but I have some data as to why. I hacked up two patches to diagnose this. The first is the following, which adds reservation failures to the btrfs_space_reservation trace. You can see it in the line with "failed: 28" (28 = ENOSPC). mv-4103 [001] .... 225.695967: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 mv-4103 [001] .... 225.696005: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 mv-4103 [001] .... 225.696024: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 mv-4103 [001] .... 225.696058: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 mv-4103 [001] .... 225.696163: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 mv-4103 [001] .... 225.696184: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 mv-4103 [001] .... 225.696262: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 reserve 516096 mv-4103 [001] .... 225.696263: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 491520 mv-4103 [001] .... 225.794161: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: failed: 28 reserve 720896 touch-4104 [001] .... 227.817348: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 reserve 327680 touch-4104 [001] .... 227.817354: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: transaction: 62 reserve 327680 touch-4104 [001] .... 227.817376: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 touch-4104 [001] .... 227.817469: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 touch-4104 [001] .... 227.817492: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 touch-4104 [001] .... 227.817510: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 touch-4104 [001] .... 227.817537: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: delayed_inode: 504 reserve 65536 touch-4104 [001] .... 227.817548: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 4096 touch-4104 [001] .... 227.817572: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: delayed_item: 257 reserve 65536 touch-4104 [001] .... 227.817575: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: delayed_inode: 257 reserve 65536 touch-4104 [001] .... 227.817600: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: transaction: 62 release 327680 touch-4104 [001] .... 227.817601: btrfs_space_reservation: 4e4ff907-1979-41da-a754-3f0adb285bd2: space_info: 5 release 110592 The second exports how much space is in the global reservation pool: # cat /sys/fs/btrfs/4e4ff907-1979-41da-a754-3f0adb285bd2/global_rsv 4194304 sled1:/home/jeffm # btrfs fi df /media/test System: total=4.00MiB, used=4.00KiB Data+Metadata: total=6.00MiB, used=1.51MiB So with ~ 4MB reserved, we end up returning ENOSPC for a ~700k reservation for the mv vs the ~300k required for the touch. 4 MB does seem a little high here, since that's about 60% of the size of the file system, but the reservation is rounded up to 4MB chunks. I've looked up the commit that added that, and it's been there since global reservations were introduced, so I'm not sure what the logic is there. Probably just to ballpark it. We already limit the size of the reservation to 512MB, so it might make sense to limit it further to something like 25% of the file system. -Jeff -- Jeff Mahoney SUSE Labs
On Tue, Oct 15, 2013 at 05:43:46PM -0400, Jeff Mahoney wrote:
On 10/15/13 7:55 AM, David Sterba wrote:
linux-1a7f:~ # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB
I can rename the files with 3.12-rc5, and I believe this is due to patch "Btrfs: fix worst case calculator for space usage" that should be safe to take to opensuse 13.1, but not at this time of release period.
I still can't rename a file using -rc5, but I have some data as to why.
So with ~ 4MB reserved, we end up returning ENOSPC for a ~700k reservation for the mv vs the ~300k required for the touch.
This explains why it worked for me, there was more than 4MB of free space. The actual fs size was 9MB so we have 4MB for system and 5MB for data+metadata, with some kilobytes consumed.
4 MB does seem a little high here, since that's about 60% of the size of the file system, but the reservation is rounded up to 4MB chunks. I've looked up the commit that added that, and it's been there since global reservations were introduced, so I'm not sure what the logic is there. Probably just to ballpark it.
The reservations still shoot too high: the calculations are done for a full tree height, ie. 8, but this does not make sense for a filesystem that's 10MB in size.
We already limit the size of the reservation to 512MB, so it might make sense to limit it further to something like 25% of the file system.
There is always a test that can break the limit, even 512 MB is not enough when there are many cpus and lots of unwritten data that rely on the global reserve. Josef sent a patch to lift it to 1G (but the test was still able to hit the limit). Also the percentage guesses work sensibly only for some range of fs size. I'm not sure we can't find a simple formula that would work for any filesystem size, so we can try to make separate logic based entierly on fs size. But this should rather be discussed upstream. david -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On Wed, Oct 16, 2013 at 8:27 AM, David Sterba
On Tue, Oct 15, 2013 at 05:43:46PM -0400, Jeff Mahoney wrote:
On 10/15/13 7:55 AM, David Sterba wrote:
linux-1a7f:~ # btrfs filesystem df /mnt System: total=4.00MiB, used=4.00KiB Data+Metadata: total=5.00MiB, used=60.00KiB
I can rename the files with 3.12-rc5, and I believe this is due to patch "Btrfs: fix worst case calculator for space usage" that should be safe to take to opensuse 13.1, but not at this time of release period.
I still can't rename a file using -rc5, but I have some data as to why.
So with ~ 4MB reserved, we end up returning ENOSPC for a ~700k reservation for the mv vs the ~300k required for the touch.
This explains why it worked for me, there was more than 4MB of free space. The actual fs size was 9MB so we have 4MB for system and 5MB for data+metadata, with some kilobytes consumed.
4 MB does seem a little high here, since that's about 60% of the size of the file system, but the reservation is rounded up to 4MB chunks. I've looked up the commit that added that, and it's been there since global reservations were introduced, so I'm not sure what the logic is there. Probably just to ballpark it.
The reservations still shoot too high: the calculations are done for a full tree height, ie. 8, but this does not make sense for a filesystem that's 10MB in size.
We already limit the size of the reservation to 512MB, so it might make sense to limit it further to something like 25% of the file system.
There is always a test that can break the limit, even 512 MB is not enough when there are many cpus and lots of unwritten data that rely on the global reserve. Josef sent a patch to lift it to 1G (but the test was still able to hit the limit). Also the percentage guesses work sensibly only for some range of fs size.
I'm not sure we can't find a simple formula that would work for any filesystem size, so we can try to make separate logic based entierly on fs size. But this should rather be discussed upstream.
david
Does continuing this discussion on this list really make any sense? Why not just put in a openSUSE specific patch to error out of mkfs.btrfs if the filesystem is less than X GB. I don't know what X should be, but at least 1 and maybe more. Then, if, and when, upstream adds usable support for smaller filesystems the patch can be adjusted or removed as appropriate. Greg -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
On Wed, Oct 16, 2013 at 09:29:32AM -0400, Greg Freemyer wrote:
Why not just put in a openSUSE specific patch to error out of mkfs.btrfs if the filesystem is less than X GB. I don't know what X should be, but at least 1 and maybe more.
Filesystems smaller than 1 GB automatically receive a special setting (so called mixed block groups), but the problem was reported on a 10 MB filesystem. Fixing that requires more than 1 line in mkfs. -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org
participants (6)
-
Andrey Borzenkov -
David Sterba -
doiggl＠velocitynet.com.au -
Greg Freemyer -
Jeff Mahoney -
Marcus Meissner