[opensuse] /var/log and btrfs on Leap 42.3
I have a Leap 42.3 system that was working fine. I started a zypper dup, and part way through, it started complaining that there was no space on /var/log. This is not true. There are a couple of GB free on the /var file system. Still, I deleted a couple of log files to see what happened. The files deleted fine, and the free space as reported by df increased ad expected. However, I cannot write anything to /var/log. If I just try to touch a new file it complains that there is no free space on the device. I have rebooted, but the problem persists. The setup is btrfs as Leap 42.3 set up. Until this, all has been working great. When you cannot write to /var/log or such places, lots of system stuff no longer starts. So even though I can log in, all is rather basic. If this was the old ext4, I would run fsck. I have to admit that the options with btrfs are less clear to me. I have tried the suggestions here: https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#I_get_.22No_space_left_o... When I run "btrfs fi balance start -dusage=5 /mount/point", no chunks are relocated. So I tried increasing the usage= value. When I hit 70, it complains that there is no space left on the device. So this seems not to be able to solve the problem. I cannot get it re reallocate a chunk. I have also checked here: https://www.suse.com/documentation/sles11/stor_admin/data/trbl_btrfs_volfull... It suggests pretty much the same thing. I have not tried removing a snapshot. I would only expect that to help when there is actually no free space. But that is not the case. I do not want to destroy this system It is my work system. I'm open to suggestions. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
might be useful to see how the free space is distributed (data vs meta) sudo btrfs filesystem usage -h / -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Feb 16, 2018 at 10:17 AM, nicholas cunliffe <ndcunliffe@gmail.com> wrote:
might be useful to see how the free space is distributed (data vs meta) sudo btrfs filesystem usage -h /
Overall: Device size: 64.00GiB Device allocated: 64.00GiB Device unallocated: 1.00MiB Device missing: 0.00B Used: 60.20GiB Free (estimated): 3.40GiB (min: 3.40GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 127.53MiB (used: 0.00B) Data,single: Size:60.94GiB, Used:57.53GiB /dev/sda2 60.94GiB Metadata,DUP: Size:1.50GiB, Used:1.33GiB /dev/sda2 3.00GiB System,DUP: Size:32.00MiB, Used:16.00KiB /dev/sda2 64.00MiB Unallocated: /dev/sda2 1.00MiB I should add that when I let the system sit for a while (an hour?), it eventually stopped complaining, and I can now start things as they should. But I really would like to know what happened so I am better prepared to deal with this next time. Or perhaps improve my layout if it seems I am on the edge of not having certain btrfs resources. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
im no expert, but.. are you sure snapper (or any other btrfs timer) didnt run in the hour time interval? can you balance now? you appear to be running with not a lot of reserve (required since e.g. you have zero unallocated and can in principle run out of meta data space before actual space goes to zero. also i note that an update takes approx 3* the zypper quoted 'space required' on btrfs) i have 29gb root with 5gb free, why is root so big - do you have home on btrfs? if not, snapshot size is proportional to age - perhaps check what snapper is hording. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Feb 16, 2018 at 11:08 AM, nicholas cunliffe <ndcunliffe@gmail.com> wrote:
im no expert, but.. are you sure snapper (or any other btrfs timer) didnt run in the hour time interval? can you balance now? you appear to be running with not a lot of reserve (required since e.g. you have zero unallocated and can in principle run out of meta data space before actual space goes to zero. also i note that an update takes approx 3* the zypper quoted 'space required' on btrfs) i have 29gb root with 5gb free, why is root so big - do you have home on btrfs? if not, snapshot size is proportional to age - perhaps check what snapper is hording.
I know about the space requirements for zypper. The download size was 400 MB. I think there was enough space. I have had issues with this on a laptop where I must be sure there is lots of space. In that case, df reports that there is too little space. I am guessing that something did run that sorted out the btrfs issue. But I might like to see if I could increase whatever storage is needed to make it less likely to happen again. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Of course I had to restart zypper dup. And of course I am once again getting the complaint about no space on the device. But I see that there are a couple of GB free. It seems to be a different file system. I have tried creating a file in the obvious places and it does not complain. Here is the message: : Installation of kernel-docs-html-4.4.114-42.1.noarch failed: Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed on file /usr/share/doc/kernel/html/kernel-api/API-vzalloc.html: cpio: rename failed - No space left on device error: kernel-docs-html-4.4.114-42.1.noarch: install failed error: kernel-docs-html-4.4.104-39.1.noarch: erase skipped Sigh. On Fri, Feb 16, 2018 at 11:19 AM, Roger Oberholtzer <roger.oberholtzer@gmail.com> wrote:
On Fri, Feb 16, 2018 at 11:08 AM, nicholas cunliffe <ndcunliffe@gmail.com> wrote:
im no expert, but.. are you sure snapper (or any other btrfs timer) didnt run in the hour time interval? can you balance now? you appear to be running with not a lot of reserve (required since e.g. you have zero unallocated and can in principle run out of meta data space before actual space goes to zero. also i note that an update takes approx 3* the zypper quoted 'space required' on btrfs) i have 29gb root with 5gb free, why is root so big - do you have home on btrfs? if not, snapshot size is proportional to age - perhaps check what snapper is hording.
I know about the space requirements for zypper. The download size was 400 MB. I think there was enough space. I have had issues with this on a laptop where I must be sure there is lots of space. In that case, df reports that there is too little space.
I am guessing that something did run that sorted out the btrfs issue. But I might like to see if I could increase whatever storage is needed to make it less likely to happen again.
-- Roger Oberholtzer
-- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
When the complaint is happening, I see this, which looks pretty much the same as when it was okay. Overall: Device size: 64.00GiB Device allocated: 64.00GiB Device unallocated: 1.00MiB Device missing: 0.00B Used: 61.29GiB Free (estimated): 2.38GiB (min: 2.38GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 129.14MiB (used: 0.00B) Data,single: Size:60.94GiB, Used:58.56GiB /dev/sda2 60.94GiB Metadata,DUP: Size:1.50GiB, Used:1.37GiB /dev/sda2 3.00GiB System,DUP: Size:32.00MiB, Used:16.00KiB /dev/sda2 64.00MiB Unallocated: /dev/sda2 1.00MiB On Fri, Feb 16, 2018 at 11:34 AM, Roger Oberholtzer <roger.oberholtzer@gmail.com> wrote:
Of course I had to restart zypper dup. And of course I am once again getting the complaint about no space on the device. But I see that there are a couple of GB free. It seems to be a different file system. I have tried creating a file in the obvious places and it does not complain. Here is the message:
:
Installation of kernel-docs-html-4.4.114-42.1.noarch failed: Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed on file /usr/share/doc/kernel/html/kernel-api/API-vzalloc.html: cpio: rename failed - No space left on device error: kernel-docs-html-4.4.114-42.1.noarch: install failed error: kernel-docs-html-4.4.104-39.1.noarch: erase skipped
Sigh.
On Fri, Feb 16, 2018 at 11:19 AM, Roger Oberholtzer <roger.oberholtzer@gmail.com> wrote:
On Fri, Feb 16, 2018 at 11:08 AM, nicholas cunliffe <ndcunliffe@gmail.com> wrote:
im no expert, but.. are you sure snapper (or any other btrfs timer) didnt run in the hour time interval? can you balance now? you appear to be running with not a lot of reserve (required since e.g. you have zero unallocated and can in principle run out of meta data space before actual space goes to zero. also i note that an update takes approx 3* the zypper quoted 'space required' on btrfs) i have 29gb root with 5gb free, why is root so big - do you have home on btrfs? if not, snapshot size is proportional to age - perhaps check what snapper is hording.
I know about the space requirements for zypper. The download size was 400 MB. I think there was enough space. I have had issues with this on a laptop where I must be sure there is lots of space. In that case, df reports that there is too little space.
I am guessing that something did run that sorted out the btrfs issue. But I might like to see if I could increase whatever storage is needed to make it less likely to happen again.
-- Roger Oberholtzer
-- Roger Oberholtzer
-- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Feb 16, 2018 at 12:44 PM, Roger Oberholtzer <roger.oberholtzer@gmail.com> wrote:
On Fri, Feb 16, 2018 at 10:17 AM, nicholas cunliffe <ndcunliffe@gmail.com> wrote:
might be useful to see how the free space is distributed (data vs meta) sudo btrfs filesystem usage -h /
Overall: Device size: 64.00GiB Device allocated: 64.00GiB
You likely cannot run balance in this state (not sure if this is improved in latest kernels). There is no space to allocate new chunks and balance requires new chunks where data is copied.
Device unallocated: 1.00MiB Device missing: 0.00B Used: 60.20GiB Free (estimated): 3.40GiB (min: 3.40GiB) Data ratio: 1.00 Metadata ratio: 2.00 Global reserve: 127.53MiB (used: 0.00B)
Data,single: Size:60.94GiB, Used:57.53GiB /dev/sda2 60.94GiB
Metadata,DUP: Size:1.50GiB, Used:1.33GiB /dev/sda2 3.00GiB
It is also close to being full. You may want to contact btrfs mailing list where you probably get better advice how to proceed, but in general running btrfs filled up to this level is challenging and probably requires ongoing monitoring and maintenance. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
perhaps its possible free space is allocated temporarily at least on a subvolume basis?. i.e. maybe its possible to run out of space on one subvolume whilst still having space overall. it would have been a nice test to see if you could create a file on another subvolume whilst out of space. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Feb 16, 2018 at 1:57 PM, nicholas cunliffe <ndcunliffe@gmail.com> wrote:
perhaps its possible free space is allocated temporarily at least on a subvolume basis?. i.e. maybe its possible to run out of space on one subvolume whilst still having space overall. it would have been a nice test to see if you could create a file on another subvolume whilst out of space.
That was possible. When it first happened, only /var/log would not accept a write.Other subvolumes seems fine. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (3)
-
Andrei Borzenkov -
nicholas cunliffe -
Roger Oberholtzer