[opensuse-factory] after big gcc6 update I keep getting "no space left on device"
Hi all, When the big update with the gcc6 arrived I did my usual dist-upgrade. Despite having at least 7Gb of free space on /, it took me 3 or 4 attempts to finish the installation because of "no space left on device". With a combination of removing some file in /tmp, logout-login and some waiting I managed to install all the packages and rebooted. But then I started getting "no space left on device" for no apparent reason (I've also rebooted a couple of times in the meanwhile). I randomly get it e.g. when trying to install something with pip, doing tab-completion on bash or using sudo. Often after a few minutes the problem disappears by itself for some time. Today I was trying to create a python virtual environment and got this: OSError: [Errno 28] No space left on device: '/usr/tmp/tmpzrux9b_p' I have no idea of what is going on, as I have plenty of space available. This is the situation now, but haven't changed much in the last week (I still haven't pull the last two updates): ~> df -h /dev/sdb2 41G 33G 7.2G 83% / [...] ~> btrfs filesystem df / Data, single: total=38.23GiB, used=31.09GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=1.75GiB, used=1.30GiB GlobalReserve, single: total=448.00MiB, used=0.00B ~> du -hs /tmp 47M /tmp ~> du -hs /var/tmp/ 174M /var/tmp/ Is this somehow related with Bug 931571? Where can I get more info about what is going on? Ciao, Francesco -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
You may have a look e.g. here: http://www.nrtm.org/index.php/2012/03/13/the-joys-of-btrfs-and-opensuse-or-n... I hope this explains whats going on. That's why I have a quite large root :-) On Mittwoch, 29. Juni 2016 21:52:01 CEST Francesco Montesano wrote:
Hi all,
When the big update with the gcc6 arrived I did my usual dist-upgrade. Despite having at least 7Gb of free space on /, it took me 3 or 4 attempts to finish the installation because of "no space left on device". With a combination of removing some file in /tmp, logout-login and some waiting I managed to install all the packages and rebooted.
But then I started getting "no space left on device" for no apparent reason (I've also rebooted a couple of times in the meanwhile). I randomly get it e.g. when trying to install something with pip, doing tab-completion on bash or using sudo. Often after a few minutes the problem disappears by itself for some time. Today I was trying to create a python virtual environment and got this: OSError: [Errno 28] No space left on device: '/usr/tmp/tmpzrux9b_p'
I have no idea of what is going on, as I have plenty of space available. This is the situation now, but haven't changed much in the last week (I still haven't pull the last two updates):
~> df -h /dev/sdb2 41G 33G 7.2G 83% / [...] ~> btrfs filesystem df / Data, single: total=38.23GiB, used=31.09GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=1.75GiB, used=1.30GiB GlobalReserve, single: total=448.00MiB, used=0.00B ~> du -hs /tmp 47M /tmp ~> du -hs /var/tmp/ 174M /var/tmp/
Is this somehow related with Bug 931571? Where can I get more info about what is going on?
Ciao,
Francesco
On Wednesday, June 29, 2016 10:07:19 PM CDT Robby Engelmann wrote:
You may have a look e.g. here: http://www.nrtm.org/index.php/2012/03/13/the-joys-of-btrfs-and-opensuse-or-> no-space-left-on-device/
I hope this explains whats going on. That's why I have a quite large root :-)
Most of us here have probably run through this gamut as some kind of school- of-hard-knocks introduction to btrfs in openSUSE. Perhaps it would be prudent for there to be some introductory documentation on first installation to warn of these and other potential problems?
On 06/30/2016 02:21 PM, Chan Ju Ping wrote:
On Wednesday, June 29, 2016 10:07:19 PM CDT Robby Engelmann wrote:
You may have a look e.g. here: http://www.nrtm.org/index.php/2012/03/13/the-joys-of-btrfs-and-opensuse-or-> no-space-left-on-device/
I hope this explains whats going on. That's why I have a quite large root :-)
Most of us here have probably run through this gamut as some kind of school- of-hard-knocks introduction to btrfs in openSUSE. Perhaps it would be prudent for there to be some introductory documentation on first installation to warn of these and other potential problems?
After this happened to a few people in the last big update I know people were atleast looking at modifying the default root directory size in the installer so people wouldn't keep running into this issue. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Thu, Jun 30, 2016 at 02:28:59PM +0930, Simon Lees wrote:
On 06/30/2016 02:21 PM, Chan Ju Ping wrote:
On Wednesday, June 29, 2016 10:07:19 PM CDT Robby Engelmann wrote:
You may have a look e.g. here: http://www.nrtm.org/index.php/2012/03/13/the-joys-of-btrfs-and-opensuse-or-> no-space-left-on-device/
I hope this explains whats going on. That's why I have a quite large root :-)
Most of us here have probably run through this gamut as some kind of school- of-hard-knocks introduction to btrfs in openSUSE. Perhaps it would be prudent for there to be some introductory documentation on first installation to warn of these and other potential problems?
After this happened to a few people in the last big update I know people were atleast looking at modifying the default root directory size in the installer so people wouldn't keep running into this issue.
Would modifying zypper to gather some stats on btrfs partitions, apply some magic numbers and flash a warning be helpful to avoid this situation? I guess the current "additional n XiB will be used" isn't able to second guess the sophistication of btrfs and interaction with snapper during updates, snagging people. Daniel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
you may want to remove old snapshots (snapper list and snapper remove 2-last number) and issue e.g. manually (is done by cron job as far as I remember) btrfs fi balance start -musage=55 / btrfs fi balance start -dusage=55 / This normally helped with me. But with 30 Gb root I regularily ran out of space with btrfs. Cheers. On Mittwoch, 29. Juni 2016 21:52:01 CEST Francesco Montesano wrote:
Hi all,
When the big update with the gcc6 arrived I did my usual dist-upgrade. Despite having at least 7Gb of free space on /, it took me 3 or 4 attempts to finish the installation because of "no space left on device". With a combination of removing some file in /tmp, logout-login and some waiting I managed to install all the packages and rebooted.
But then I started getting "no space left on device" for no apparent reason (I've also rebooted a couple of times in the meanwhile). I randomly get it e.g. when trying to install something with pip, doing tab-completion on bash or using sudo. Often after a few minutes the problem disappears by itself for some time. Today I was trying to create a python virtual environment and got this: OSError: [Errno 28] No space left on device: '/usr/tmp/tmpzrux9b_p'
I have no idea of what is going on, as I have plenty of space available. This is the situation now, but haven't changed much in the last week (I still haven't pull the last two updates):
~> df -h /dev/sdb2 41G 33G 7.2G 83% / [...] ~> btrfs filesystem df / Data, single: total=38.23GiB, used=31.09GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=1.75GiB, used=1.30GiB GlobalReserve, single: total=448.00MiB, used=0.00B ~> du -hs /tmp 47M /tmp ~> du -hs /var/tmp/ 174M /var/tmp/
Is this somehow related with Bug 931571? Where can I get more info about what is going on?
Ciao,
Francesco
On 06/29/2016 04:13 PM, Robby Engelmann wrote:
you may want to remove old snapshots (snapper list and snapper remove 2-last number)
and issue e.g. manually (is done by cron job as far as I remember)
btrfs fi balance start -musage=55 / btrfs fi balance start -dusage=55 /
This normally helped with me. But with 30 Gb root I regularily ran out of space with btrfs.
Cheers.
On Mittwoch, 29. Juni 2016 21:52:01 CEST Francesco Montesano wrote:
Hi all,
When the big update with the gcc6 arrived I did my usual dist-upgrade. Despite having at least 7Gb of free space on /, it took me 3 or 4 attempts to finish the installation because of "no space left on device". With a combination of removing some file in /tmp, logout-login and some waiting I managed to install all the packages and rebooted.
But then I started getting "no space left on device" for no apparent reason (I've also rebooted a couple of times in the meanwhile). I randomly get it e.g. when trying to install something with pip, doing tab-completion on bash or using sudo. Often after a few minutes the problem disappears by itself for some time. Today I was trying to create a python virtual environment and got this: OSError: [Errno 28] No space left on device: '/usr/tmp/tmpzrux9b_p'
I have no idea of what is going on, as I have plenty of space available. This is the situation now, but haven't changed much in the last week (I still haven't pull the last two updates):
~> df -h /dev/sdb2 41G 33G 7.2G 83% / [...] ~> btrfs filesystem df / Data, single: total=38.23GiB, used=31.09GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=1.75GiB, used=1.30GiB GlobalReserve, single: total=448.00MiB, used=0.00B ~> du -hs /tmp 47M /tmp ~> du -hs /var/tmp/ 174M /var/tmp/
Is this somehow related with Bug 931571? Where can I get more info about what is going on?
Ciao,
Francesco
You no longer need a separate /boot partition with btrfs. Back in 2012 we did. I'm using 90 GB on a 500 GB drive. This gives me ample space. Using Robby's commands are a good idea. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi,
2016-06-29 22:13 GMT+02:00 Robby Engelmann
you may want to remove old snapshots (snapper list and snapper remove 2-last number)
and issue e.g. manually (is done by cron job as far as I remember)
btrfs fi balance start -musage=55 / btrfs fi balance start -dusage=55 /
This normally helped with me. But with 30 Gb root I regularily ran out of space with btrfs.
Cheers.
Thank you for the advises and link. I did run into the problem of running out of space once some time ago and that made me learn a good deal of btrfs. When that happened, "btrfs fi" was properly reporting no free space. I think that this time is a different issue, as btrfs tools are reporting lots of free space. So I guess that it's not a snapshot issue, unless btrfs fi usage/df/ect ignore the snapshot (in which case the tool would not do its job) Ciao, Fra
On Mittwoch, 29. Juni 2016 21:52:01 CEST Francesco Montesano wrote:
Hi all,
When the big update with the gcc6 arrived I did my usual dist-upgrade. Despite having at least 7Gb of free space on /, it took me 3 or 4 attempts to finish the installation because of "no space left on device". With a combination of removing some file in /tmp, logout-login and some waiting I managed to install all the packages and rebooted.
But then I started getting "no space left on device" for no apparent reason (I've also rebooted a couple of times in the meanwhile). I randomly get it e.g. when trying to install something with pip, doing tab-completion on bash or using sudo. Often after a few minutes the problem disappears by itself for some time. Today I was trying to create a python virtual environment and got this: OSError: [Errno 28] No space left on device: '/usr/tmp/tmpzrux9b_p'
I have no idea of what is going on, as I have plenty of space available. This is the situation now, but haven't changed much in the last week (I still haven't pull the last two updates):
~> df -h /dev/sdb2 41G 33G 7.2G 83% / [...] ~> btrfs filesystem df / Data, single: total=38.23GiB, used=31.09GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=1.75GiB, used=1.30GiB GlobalReserve, single: total=448.00MiB, used=0.00B ~> du -hs /tmp 47M /tmp ~> du -hs /var/tmp/ 174M /var/tmp/
Is this somehow related with Bug 931571? Where can I get more info about what is going on?
Ciao,
Francesco
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano
Hi all,
When the big update with the gcc6 arrived I did my usual dist-upgrade. Despite having at least 7Gb of free space on /, it took me 3 or 4 attempts to finish the installation because of "no space left on device". With a combination of removing some file in /tmp, logout-login and some waiting I managed to install all the packages and rebooted.
But then I started getting "no space left on device" for no apparent reason (I've also rebooted a couple of times in the meanwhile). I randomly get it e.g. when trying to install something with pip, doing tab-completion on bash or using sudo. Often after a few minutes the problem disappears by itself for some time. Today I was trying to create a python virtual environment and got this: OSError: [Errno 28] No space left on device: '/usr/tmp/tmpzrux9b_p'
I have no idea of what is going on, as I have plenty of space available. This is the situation now, but haven't changed much in the last week (I still haven't pull the last two updates):
~> df -h /dev/sdb2 41G 33G 7.2G 83% / [...] ~> btrfs filesystem df / Data, single: total=38.23GiB, used=31.09GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=1.75GiB, used=1.30GiB GlobalReserve, single: total=448.00MiB, used=0.00B
It may not be true that you have 7.2G free. What do you get for 'btrfs fi us /' ? What you have is 7.14GiB unused in data block groups. And 450MiB unused in metadata block groups. If the installation is very metadata centric, or contains many small files (less than about 15KiB) then those files get written in the metadata block group as an inline extent. There is a known enospc bug that results in errno -28. But it hasn't been fixed. http://www.spinics.net/lists/linux-btrfs/msg50945.html However I'm not certain you are being hit by this bug so it's better to report back btrfs fi us /. You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed. But again it's hard to say if that's enough space for this task. What is the estimated installation size? There are some other ways to find out what's going on, what block group is probably affecting this, but it means doing some iteration. -- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Chris,
2016-06-30 0:44 GMT+02:00 Chris Murphy
On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano
wrote: [...]
~> df -h /dev/sdb2 41G 33G 7.2G 83% / [...] ~> btrfs filesystem df / Data, single: total=38.23GiB, used=31.09GiB System, single: total=32.00MiB, used=16.00KiB Metadata, single: total=1.75GiB, used=1.30GiB GlobalReserve, single: total=448.00MiB, used=0.00B
It may not be true that you have 7.2G free. What do you get for 'btrfs fi us /' ?
What you have is 7.14GiB unused in data block groups. And 450MiB unused in metadata block groups.
~> sudo btrfs filesystem usage / [sudo] password for root: Overall: Device size: 40.01GiB Device allocated: 40.01GiB Device unallocated: 1.00MiB Device missing: 0.00B Used: 32.39GiB Free (estimated): 7.14GiB (min: 7.14GiB) Data ratio: 1.00 Metadata ratio: 1.00 Global reserve: 448.00MiB (used: 0.00B) Data,single: Size:38.23GiB, Used:31.09GiB /dev/sdb2 38.23GiB Metadata,single: Size:1.75GiB, Used:1.30GiB /dev/sdb2 1.75GiB System,single: Size:32.00MiB, Used:16.00KiB /dev/sdb2 32.00MiB Unallocated: /dev/sdb2 1.00MiB
If the installation is very metadata centric, or contains many small files (less than about 15KiB) then those files get written in the metadata block group as an inline extent.
There is a known enospc bug that results in errno -28. But it hasn't been fixed. http://www.spinics.net/lists/linux-btrfs/msg50945.html
Thanks for the link
However I'm not certain you are being hit by this bug so it's better to report back btrfs fi us /.
You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed. But again it's hard to say if that's enough space for this task.
I can try this. A little background, if useful: * When there was the big update with the glibc update I run into the problem of the disk full. There I did some btrfs balance and managed to finish the installation. I also enabled the btrfsmaintenance-refresh service. * as said with the gcc6 updates I did run out of space (and I keep having this) despite everything showing that I have various GB of free space
What is the estimated installation size?
I did what the installer suggested: 40GB of space for / with btrf.
There are some other ways to find out what's going on, what block group is probably affecting this, but it means doing some iteration.
I like learning new stuff. Ciao, Fra
-- Chris Murphy
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jun 30, 2016 at 1:53 AM, Francesco Montesano
~> sudo btrfs filesystem usage / [sudo] password for root: Overall: Device size: 40.01GiB Device allocated: 40.01GiB Device unallocated: 1.00MiB
This part means Btrfs cannot create new block groups. Block groups are also called chunks in Btrfs.
Data,single: Size:38.23GiB, Used:31.09GiB /dev/sdb2 38.23GiB
This is the allocated and used space for the data block group. There's 7.14GiB unused. The data block group stores data extents only.
Metadata,single: Size:1.75GiB, Used:1.30GiB /dev/sdb2 1.75GiB
This is the allocated and used space for the metadata block group, which is the file system itself: The fs trees, chunk tree, dev tree, checksum tree, extent tree, leafs and nodes. By default Btrfs has 16KiB nodesizes, into which it'll stuff many item keys of fs metadata. But sometimes this includes data if it's small, and in that case the data is written inline the node, rather than being saved as an extent in a data chunk.
From this perspective, there's less than 500MiB of unused space.
What's needed is to free up some of the 7GiB of unused data chunk space, so it becomes unallocated, and now Btrfs will be able to make it a metadata chunk, and probably avoid the enospc you're having. It's expected the kernel code will get smarter than this, but that work isn't done yet, so in these cases we resort to manually having to fix the problem. (Typically it happens when the file system becomes nearly full and also fragmented, which many snapshots and snapshot deletions will exacerbate on a smaller file system. So really part of the problem is Btrfs rootfs size recommendation is too small for the aggressive snapshotting that Snapper does. I'm not the only one who has said it, but people who make these decisions seem to keep ignoring the problem for whatever reason. I never have this problem with 40-50GiB file systems because I'm not making more than a dozen snapshots, rather than hundreds or thousands that Snapper is doing by default - which I think is really unhelpful considering Btrfs is simply not smart enough yet to automatically deallocate partially empty block groups. It only does this for completely empty block groups.) -- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi,
2016-06-30 0:44 GMT+02:00 Chris Murphy
On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano [...]
You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed.
~> sudo btrfs balance start -dusage=25 / Done, had to relocate 0 out of 54 chunks ~> sudo btrfs balance status / No balance found on '/' I've tried to use 35 with the same result. BTW: now I'm trying to compile a big c++ library and I have to run make many times because I keep getting: fatal error: error writing to /tmp/ccZoakJY.s: No space left on device I run make, after few minutes I get the error; reruning make just afterwards runs for a few minutes and then crashes, and so on... Ciao, Fra
-- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
You may check whether /tmp is a subvolume (here on my TW installation it is.... btrfs subvolume list / ) and have an apropriate size. I cannot really advice you, because the concept and work with subvolumes is not absolutely clear to me. On Donnerstag, 30. Juni 2016 16:24:54 CEST Francesco Montesano wrote:
Hi,
2016-06-30 0:44 GMT+02:00 Chris Murphy
: On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano [...]
You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed.
~> sudo btrfs balance start -dusage=25 / Done, had to relocate 0 out of 54 chunks ~> sudo btrfs balance status / No balance found on '/'
I've tried to use 35 with the same result.
BTW: now I'm trying to compile a big c++ library and I have to run make many times because I keep getting:
fatal error: error writing to /tmp/ccZoakJY.s: No space left on device
I run make, after few minutes I get the error; reruning make just afterwards runs for a few minutes and then crashes, and so on...
Ciao,
Fra
-- Chris Murphy
Hi,
2016-06-30 16:35 GMT+02:00 Robby Engelmann
You may check whether /tmp is a subvolume (here on my TW installation it is....
btrfs subvolume list /
) and have an apropriate size. I cannot really advice you, because the concept and work with subvolumes is not absolutely clear to me.
ID 262 gen 264136 top level 5 path tmp [...] ID 271 gen 264623 top level 5 path var/tmp Does it make sense to you? Notice that I never had issues with this before and I'm 99% sure that I had times with my /tmp being way bigger than the 44M is now Fra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Maybe completely off-target, but what does "snapper list" say about snapshots? Libor On Thu 30-06-16 16:35:40, Robby Engelmann wrote:
You may check whether /tmp is a subvolume (here on my TW installation it is....
btrfs subvolume list /
) and have an apropriate size. I cannot really advice you, because the concept and work with subvolumes is not absolutely clear to me.
On Donnerstag, 30. Juni 2016 16:24:54 CEST Francesco Montesano wrote:
Hi,
2016-06-30 0:44 GMT+02:00 Chris Murphy
: On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano [...]
You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed.
~> sudo btrfs balance start -dusage=25 / Done, had to relocate 0 out of 54 chunks ~> sudo btrfs balance status / No balance found on '/'
I've tried to use 35 with the same result.
BTW: now I'm trying to compile a big c++ library and I have to run make many times because I keep getting:
fatal error: error writing to /tmp/ccZoakJY.s: No space left on device
I run make, after few minutes I get the error; reruning make just afterwards runs for a few minutes and then crashes, and so on...
Ciao,
Fra
-- Chris Murphy
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi,
2016-06-30 16:44 GMT+02:00 Libor Pechacek
Maybe completely off-target, but what does "snapper list" say about snapshots?
~> sudo snapper list Type | # | Pre # | Date | User | Cleanup | Description | Userdata -------+------+-------+-------------------------------+------+---------+--------------+-------------- single | 0 | | | root | | current | pre | 994 | | Fri 03 Jun 2016 10:55:11 CEST | root | number | zypp(zypper) | important=yes post | 995 | 994 | Fri 03 Jun 2016 11:00:44 CEST | root | number | | important=yes pre | 996 | | Sat 04 Jun 2016 08:29:03 CEST | root | number | zypp(zypper) | important=yes post | 997 | 996 | Sat 04 Jun 2016 08:33:47 CEST | root | number | | important=yes pre | 1006 | | Fri 10 Jun 2016 00:00:17 CEST | root | number | zypp(zypper) | important=yes post | 1007 | 1006 | Fri 10 Jun 2016 00:09:57 CEST | root | number | | important=yes pre | 1012 | | Tue 14 Jun 2016 08:49:01 CEST | root | number | zypp(zypper) | important=yes post | 1013 | 1012 | Tue 14 Jun 2016 08:54:04 CEST | root | number | | important=yes pre | 1014 | | Thu 16 Jun 2016 08:43:25 CEST | root | number | zypp(zypper) | important=yes post | 1015 | 1014 | Thu 16 Jun 2016 08:50:31 CEST | root | number | | important=yes pre | 1016 | | Wed 22 Jun 2016 21:43:45 CEST | root | number | zypp(zypper) | important=yes pre | 1017 | | Wed 22 Jun 2016 22:55:41 CEST | root | number | zypp(zypper) | important=no pre | 1018 | | Wed 22 Jun 2016 23:04:03 CEST | root | number | zypp(zypper) | important=no pre | 1019 | | Wed 22 Jun 2016 23:05:38 CEST | root | number | zypp(zypper) | important=no pre | 1020 | | Wed 22 Jun 2016 23:16:18 CEST | root | number | zypp(zypper) | important=no pre | 1021 | | Wed 22 Jun 2016 23:30:56 CEST | root | number | zypp(zypper) | important=no post | 1022 | 1021 | Wed 22 Jun 2016 23:33:12 CEST | root | number | | important=no pre | 1023 | | Thu 23 Jun 2016 20:39:52 CEST | root | number | zypp(zypper) | important=no pre | 1024 | | Fri 24 Jun 2016 21:11:12 CEST | root | number | zypp(zypper) | important=no post | 1025 | 1024 | Fri 24 Jun 2016 21:16:21 CEST | root | number | | important=no
Libor
Fra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu 30-06-16 16:47:56, Francesco Montesano wrote:
Hi,
2016-06-30 16:44 GMT+02:00 Libor Pechacek
: Maybe completely off-target, but what does "snapper list" say about snapshots?
~> sudo snapper list Type | # | Pre # | Date | User | Cleanup | Description | Userdata -------+------+-------+-------------------------------+------+---------+--------------+-------------- single | 0 | | | root | | current | pre | 994 | | Fri 03 Jun 2016 10:55:11 CEST | root | number | zypp(zypper) | important=yes post | 995 | 994 | Fri 03 Jun 2016 11:00:44 CEST | root | number | | important=yes pre | 996 | | Sat 04 Jun 2016 08:29:03 CEST | root | number | zypp(zypper) | important=yes post | 997 | 996 | Sat 04 Jun 2016 08:33:47 CEST | root | number | | important=yes pre | 1006 | | Fri 10 Jun 2016 00:00:17 CEST | root | number | zypp(zypper) | important=yes post | 1007 | 1006 | Fri 10 Jun 2016 00:09:57 CEST | root | number | | important=yes pre | 1012 | | Tue 14 Jun 2016 08:49:01 CEST | root | number | zypp(zypper) | important=yes post | 1013 | 1012 | Tue 14 Jun 2016 08:54:04 CEST | root | number | | important=yes pre | 1014 | | Thu 16 Jun 2016 08:43:25 CEST | root | number | zypp(zypper) | important=yes post | 1015 | 1014 | Thu 16 Jun 2016 08:50:31 CEST | root | number | | important=yes pre | 1016 | | Wed 22 Jun 2016 21:43:45 CEST | root | number | zypp(zypper) | important=yes pre | 1017 | | Wed 22 Jun 2016 22:55:41 CEST | root | number | zypp(zypper) | important=no pre | 1018 | | Wed 22 Jun 2016 23:04:03 CEST | root | number | zypp(zypper) | important=no pre | 1019 | | Wed 22 Jun 2016 23:05:38 CEST | root | number | zypp(zypper) | important=no pre | 1020 | | Wed 22 Jun 2016 23:16:18 CEST | root | number | zypp(zypper) | important=no pre | 1021 | | Wed 22 Jun 2016 23:30:56 CEST | root | number | zypp(zypper) | important=no post | 1022 | 1021 | Wed 22 Jun 2016 23:33:12 CEST | root | number | | important=no pre | 1023 | | Thu 23 Jun 2016 20:39:52 CEST | root | number | zypp(zypper) | important=no pre | 1024 | | Fri 24 Jun 2016 21:11:12 CEST | root | number | zypp(zypper) | important=no post | 1025 | 1024 | Fri 24 Jun 2016 21:16:21 CEST | root | number | | important=no
Seeing the snapshots, I would say deleting some snapshots (sudo snapper delete
994-
Отправлено с iPhone
30 июня 2016 г., в 17:24, Francesco Montesano
написал(а): Hi,
2016-06-30 0:44 GMT+02:00 Chris Murphy
: On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano [...]
You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed.
~> sudo btrfs balance start -dusage=25 / Done, had to relocate 0 out of 54 chunks ~> sudo btrfs balance status / No balance found on '/'
I've tried to use 35 with the same result.
BTW: now I'm trying to compile a big c++ library and I have to run make many times because I keep getting:
fatal error: error writing to /tmp/ccZoakJY.s: No space left on device
Let's start with obvious question - where is /tmp mounted?
I run make, after few minutes I get the error; reruning make just afterwards runs for a few minutes and then crashes, and so on...
Ciao,
Fra
-- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi,
2016-06-30 16:36 GMT+02:00 Andrei Borzenkov
Отправлено с iPhone
30 июня 2016 г., в 17:24, Francesco Montesano
написал(а): [...] Let's start with obvious question - where is /tmp mounted?
fstab tells me: UUID="very long id" /tmp btrfs subvol=tmp 0 0 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jun 30, 2016 at 8:24 AM, Francesco Montesano
Hi,
2016-06-30 0:44 GMT+02:00 Chris Murphy
: On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano [...]
You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed.
~> sudo btrfs balance start -dusage=25 / Done, had to relocate 0 out of 54 chunks ~> sudo btrfs balance status / No balance found on '/'
OK there are no empyt or nearly empty chunks. The problem with a full balance is that it is CoW. That's good and bad. Good because it should be safe. Bad because Btrfs must have unallocated space to create a new chunk to copy extents into before it can deallocate the origin chunk. But there is no unallocated space on this file system. There are two options: 1. You delete a bunch of snapshots and hopefully an entire data chunk becomes empty and therefore reverts to unallocated, and now Btrfs can make it into a metadata chunk, which seems not quite chrystal clear is what it wants to do. There's no guarantee this frees up an entire chunk, as snapshots can reference extents across chunks. 2. You need to make more unallocated space available by temporarily adding another device to the volume. Now you can do a balance, and then remove the extra device so that you're back to a single device volume. This device doesn't need to be big, in fact the smaller the better (up to a point). I'd say even a 2GiB USB stick is adequate. A 4GiB or 8GiB is better. If it's bigger than 8GiB I'd partition it, with an 8GiB partition and then point btrfs to that partition. Why? Btrfs will try to make the free space on each device the same, so the more free space on the USB stick, the more it'll use it. And the USB stick is slow. Slower than a hard drive, unless this is a fast USB 3.0 stick. So it looks like this: btrfs dev add /dev/sdXY / ###where XY is the block device and partition you intend to use. btrfs fi show ## will show this is now a two device Btrfs volume btrfs balance start --dusage=100 ### this will completely rewrite all data chunks, some of which will end up on the USB stick so you cannot physical disconnect it until it's removed from the volume The balance might take a while depending on the speed of the devices. Not much will be written to the USB stick. But for 38GiB it could take maybe 10 minutes if it's a hard drive. And maybe a couple minutes on SSD. If you want to watch this, you can use & after the balance command, and then use top or iotop to keep track of Btrfs activity. btrfs fi show ##will give you an idea of how much is used on the stick so you can estimate how long device removal should take Once the balance completes, the metadata and data on the USB stick is returned back to the first device like this: btrfs dev rem /dev/sdXY / ###where XY is the same as before, for the USB stick you're removing, could be slow and take a minute or 10 minutes depending on how much is used. Once that command returns to prompt, you can confirm: btrfs fi show And see that the volume is now a single device volume again. Only then is it safe to physically disconnect the USB stick. -- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 30.06.16 18:20 Chris Murphy wrote: [lots and lots of useful information] Dear Chris, although it is not my problem that you are solving here, let me thank you for your useful, detailed and very understandable mails in regard to btrfs. I am learning a lot from your mails, which could come in useful, as I have some BTRFS volumes myself. Johannes
Hi Chris,
1) thank you very much for the amazing explanation.
Il giorno gio 30 giu 2016 alle ore 18:20 Chris Murphy
On Thu, Jun 30, 2016 at 8:24 AM, Francesco Montesano
wrote: Hi,
2016-06-30 0:44 GMT+02:00 Chris Murphy
: On Wed, Jun 29, 2016 at 1:52 PM, Francesco Montesano [...]
You could try 'btrfs balance start -dusage=25 /' which should free up one or more data block groups into unallocated space so that Btrfs can reallocate it as a metadata block group if needed.
~> sudo btrfs balance start -dusage=25 / Done, had to relocate 0 out of 54 chunks ~> sudo btrfs balance status / No balance found on '/'
OK there are no empyt or nearly empty chunks.
The problem with a full balance is that it is CoW. That's good and bad. Good because it should be safe. Bad because Btrfs must have unallocated space to create a new chunk to copy extents into before it can deallocate the origin chunk. But there is no unallocated space on this file system.
There are two options:
1. You delete a bunch of snapshots and hopefully an entire data chunk becomes empty and therefore reverts to unallocated, and now Btrfs can make it into a metadata chunk, which seems not quite chrystal clear is what it wants to do. There's no guarantee this frees up an entire chunk, as snapshots can reference extents across chunks.
I've tried removing some of the oldest snapshots. I cleared some space: Data,single: Size:38.23GiB, Used:27.35GiB /dev/sdb2 38.23GiB Metadata,single: Size:1.75GiB, Used:1.09GiB /dev/sdb2 1.75GiB System,single: Size:32.00MiB, Used:16.00KiB /dev/sdb2 32.00MiB Then I've tried to run the balance command with -dusage 25 with no success. Then I tried: sudo btrfs balance start -dusage=50 / ERROR: error during balancing '/': No space left on device There may be more info in syslog - try dmesg | tail Given your explanation it makes sense, as I guess the balance needs disk space (metadata?) that apparently is not that abundant. In one of your messaged you talked about fragmentation; should I try to run "btrfs fi defragment"?
2. You need to make more unallocated space available by temporarily adding another device to the volume. Now you can do a balance, and then remove the extra device so that you're back to a single device volume. This device doesn't need to be big, in fact the smaller the better (up to a point). I'd say even a 2GiB USB stick is adequate. A 4GiB or 8GiB is better. If it's bigger than 8GiB I'd partition it, with an 8GiB partition and then point btrfs to that partition. Why? Btrfs will try to make the free space on each device the same, so the more free space on the USB stick, the more it'll use it. And the USB stick is slow. Slower than a hard drive, unless this is a fast USB 3.0 stick.
So it looks like this:
btrfs dev add /dev/sdXY / ###where XY is the block device and partition you intend to use. btrfs fi show ## will show this is now a two device Btrfs volume btrfs balance start --dusage=100 ### this will completely rewrite all data chunks, some of which will end up on the USB stick so you cannot physical disconnect it until it's removed from the volume
The balance might take a while depending on the speed of the devices. Not much will be written to the USB stick. But for 38GiB it could take maybe 10 minutes if it's a hard drive. And maybe a couple minutes on SSD. If you want to watch this, you can use & after the balance command, and then use top or iotop to keep track of Btrfs activity.
btrfs fi show ##will give you an idea of how much is used on the stick so you can estimate how long device removal should take Once the balance completes, the metadata and data on the USB stick is returned back to the first device like this:
btrfs dev rem /dev/sdXY / ###where XY is the same as before, for the USB stick you're removing, could be slow and take a minute or 10 minutes depending on how much is used.
Once that command returns to prompt, you can confirm:
btrfs fi show
And see that the volume is now a single device volume again. Only then is it safe to physically disconnect the USB stick.
I'll wait a day or two to see how btrfs behaves after removing the snapshot. If nothing happens, I'll try this. Ciao, Fra
-- Chris Murphy
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jun 30, 2016 at 3:38 PM, Francesco Montesano
I've tried removing some of the oldest snapshots. I cleared some space:
Data,single: Size:38.23GiB, Used:27.35GiB /dev/sdb2 38.23GiB
Metadata,single: Size:1.75GiB, Used:1.09GiB /dev/sdb2 1.75GiB
System,single: Size:32.00MiB, Used:16.00KiB /dev/sdb2 32.00MiB
Then I've tried to run the balance command with -dusage 25 with no success. Then I tried:
sudo btrfs balance start -dusage=50 / ERROR: error during balancing '/': No space left on device There may be more info in syslog - try dmesg | tail
Given your explanation it makes sense, as I guess the balance needs disk space (metadata?) that apparently is not that abundant.
Balance always needs some minimum amount of unallocated space that isn't already used for a block group. And this file system has no unallocated space. It has unused space in allocated block groups, however. But Btrfs will not use unused space for balance, it does a copy on write operation only to a new block group, once that's successful it'll delete the former source. As it does the balance each new block group is pretty much filled 100%, so it should always be true that eventually you end up with unused space = unallocated space. The last block group of course does have some unused space in it. So the problem here is there's still no completely empty block group for any kind of balance to be successful.
In one of your messaged you talked about fragmentation; should I try to run "btrfs fi defragment"?
No I wouldn't worry too much about that. You can check files with 'filefrag' if you want, but some fragmentation is normal and isn't a problem. It's a big problem for VM and database images though. Btrfs has in some sense two kinds of fragmentation: files that have more than one extent; and block groups with pockets of unused space in between extents. 'fi defrag' only does the former, and balance only does the latter. Snapshots indirectly can make block group fragmentation worse: more accurately when snapshots that contain few changes are deleted. An optimization to reduce this type of fragmentation on cow file systems might be to do frequent snapshots, which pins extents preventing their deletion. And instead of deleting them one at a time to delete large numbers of them at once which should free more contiguous extents, rather than piecemeal. But this is just an idea, someone will need to model it and figure out the patterns. But at some point it might be a good optimization for snapper.
I'll wait a day or two to see how btrfs behaves after removing the snapshot. If nothing happens, I'll try this.
It's a safe operation. But I'd still make sure you have backups of important things. -- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Francesco, can you try using kernel 4.5.7 by any chance? There's a patch in 4.5.7 and 4.7-rc's that might get you out of this problem. It's not yet in the 4.6 series. It's up to you if you go with the newer kernel approach, or the multiple device method I previously outlined. If you go with the kernel upgrade, try a filter value -dusage=25. It shouldn't enospc but it might say it relocated 0 chunks. If so increase the value, try -dusage=35, and increase until you get at least 1 chunk relocated. Next retry the activities that have caused enospc, and see if it's fixed. There is no wrong value for the filter. It just means to only apply the balance to chunks that are that % full or less. It's so you don't have to do a full balance which, especially on a HDD, can take a while. I think in this case there's no advantage to rewriting everything with a full balance, but it's OK to do that also by just omitting the filter. -- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Chris,
2016-07-02 21:13 GMT+02:00 Chris Murphy
Francesco, can you try using kernel 4.5.7 by any chance? There's a patch in 4.5.7 and 4.7-rc's that might get you out of this problem. It's not yet in the 4.6 series. It's up to you if you go with the newer kernel approach, or the multiple device method I previously outlined.
Thank you again. It's this already in the kernel:stable repo (http://kernel.opensuse.org/packages/stable)?
If you go with the kernel upgrade, try a filter value -dusage=25. It shouldn't enospc but it might say it relocated 0 chunks. If so increase the value, try -dusage=35, and increase until you get at least 1 chunk relocated. Next retry the activities that have caused enospc, and see if it's fixed.
a couple of days ago I removed a couple of snapshots and I still haven't had problems with space, except when trying to do -dusage=50. However right now I'm two snapshot (qt5 issues already reported in the list): I probably should try to resolve the situation before upgrading when the qt5 problems will be solved. I'll play with it during the week, I guess.
There is no wrong value for the filter. It just means to only apply the balance to chunks that are that % full or less. It's so you don't have to do a full balance which, especially on a HDD, can take a while. I think in this case there's no advantage to rewriting everything with a full balance, but it's OK to do that also by just omitting the filter.
root is on a SSD so that part should be fine. Ciao, Fra
-- Chris Murphy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (10)
-
Andrei Borzenkov
-
Chan Ju Ping
-
Chris Murphy
-
Daniel Morris
-
Francesco Montesano
-
Johannes Kastl
-
Libor Pechacek
-
Robby Engelmann
-
Roman Bysh
-
Simon Lees