Fwd: Fwd: snapshots filling up SSD
Users response. Regards Sid. -------- Forwarded Message -------- Subject: Re: Fwd: snapshots filling up SSD Date: Sat, 8 May 2021 15:29:09 +0000 From: Sav. Mellor <sav.mellor@outlook.com> To: Sid Boyce <sboyce@blueyonder.co.uk> Hi Sid, thanks for that. I think, though, that the point is that installing Opensuse Leap from the DVD download, configures the disk, regardless of technology, as a 40GB btrfs partition, a swap partition and the rest of the disk as an xfs partition. The reason the documentation gives is to protect your user data if you need to rollback a faulty update, which makes sense, I used to configure my Windows HDDs that way for the same reason, albeit with a much bigger system partition. The standard install for Opensuse then sets up the snapshots retention as 20 plus 20. I think the 40 GB and snapshot retention numbers are a mismatch. I think either a 64 GB btrfs and 20 plus 20 or 40 GB btrfs and 10 plus 10 would be better matched. As your correspondents noted, anyone encountering a btrfs configuration for the first time would expect that the "standard configuration" would be appropriate. Clearly, in the case of my SSD, it was not. The more I think about it, I'm convinced that a btrfs root "/" and xfs data "/home" configuration is fine for HDDs but not for SSDs, because of the wear considerations on SSDs. I think the "standard configuration" installer should allocate just a brtfs and a swap partition on a SSD, like your configuration, with the option to split root and home if you know what you are doing. Incidentally, on the two Windows systems I still have (applications not compatible with Linux) that have SSDs, I run them as a single partition, because of the wear considerations. Best regards, Sav On 09/05/2021 00:01, Sid Boyce wrote:
FYI. Sid. -------- Forwarded Message -------- Subject: Re: snapshots filling up SSD Date: Sat, 8 May 2021 09:18:21 +0200 From: Thorsten Kukuk <kukuk@suse.de> To: factory@lists.opensuse.org
On Sat, May 08, Michael Hamilton wrote:
The installer should restrict or warn users about the unusual space
requirements for OpenSUSE btrfs-root. Many people coming to OpenSUSE may not have encountered btrfs or snapshots before.
The installer is doing this. But YaST cannot look in the future what the user will do with the system afterwards.
So assume somebody is doing a standard installation. 40GB are clearly enough for this. Now he installs a lot of additional software
afterwards and does additional things like video recording or running
VMs, which are all very disk consuming -> the user will run into problems, which was not predictable by the installer.
Thorsten
-- Sid Boyce ... Hamradio License G3VBV, Licensed Private Pilot Emeritus IBM/Amdahl Mainframes and Sun/Fujitsu Servers Tech Support Senior Staff Specialist, Cricket Coach Microsoft Windows Free Zone - Linux used for all Computing Tasks
On 08/05/2021 17.37, Sid Boyce wrote:
Users response.
Regards
Sid.
-------- Forwarded Message -------- Subject: Re: Fwd: snapshots filling up SSD Date: Sat, 8 May 2021 15:29:09 +0000 From: Sav. Mellor <> To: Sid Boyce <>
Hi Sid, thanks for that. I think, though, that the point is that installing Opensuse Leap from the DVD download, configures the disk, regardless of technology, as a 40GB btrfs partition, a swap partition and the rest of the disk as an xfs partition. The reason the documentation gives is to protect your user data if you need to rollback a faulty update, which makes sense, I used to configure my Windows HDDs
that way for the same reason, albeit with a much bigger system partition. The standard install for Opensuse then sets up the snapshots
retention as 20 plus 20. I think the 40 GB and snapshot retention numbers are a mismatch. I think either a 64 GB btrfs and 20 plus 20 or 40 GB btrfs and 10 plus 10 would be better matched.
Sizing correctly a root partition for someone and get it right at the first go is an art. I'd probably select 80 or 120 GB for the root btrfs system (and the rest for /home and swap), but I'd probably end with a lot of space unused. I would also probably add a 10 GB spare partition for emergencies with a secondary system. It is normal to install a machine and sometime later redo it completely with different choices, after having some actual experience of the usage.
As your correspondents noted, anyone encountering a btrfs configuration
for the first time would expect that the "standard configuration" would
be appropriate. Clearly, in the case of my SSD, it was not.
That the SSD machine was impacted and another with "rotating rust" was not, can't be related to being SSD or not.
The more I think about it, I'm convinced that a btrfs root "/" and xfs data "/home" configuration is fine for HDDs but not for SSDs, because of the wear considerations on SSDs. I think the "standard configuration" installer should allocate just a brtfs and a swap partition on a SSD, like your configuration, with the option to split root and home if you know what you are doing. Incidentally, on the two Windows systems I still have (applications not compatible with Linux) that have SSDs, I run them as a single partition, because of the wear considerations.
SSD are partitioned the same as normal disks, there is no difference. You could leave some space outside of any partition for wear levelling. -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
On 08/05/2021 20.49, Larry Len Rainey wrote:
Why not have an option for novice that does choice of UEFI or Legacy install with ext4 full root and swap of ram size - now you don't have to worry about size and btrfs problems for novices users - that is better until they can comprehend btrfs and other file system types and partitions.
That would be my choice, but is not what the people that doit chose.
That is how I install every OpenSUSE for users - no btrfs - instead I provide an external 2gb drive and scripts to backup and restore the drives. I also provide a bootable usb drive with a saved blkid of the system and a script to make all the file systems needed and to restore the last full backup on the external drive so that the restore script can put everything back where it was before the "disaster".
Few users understand enough "linux or windows" to know what they are doing. You have to make it idiot proof. Like the sign over the testing room at Microsoft in 1982 said - "We try to make our Software Idiot Proof" someone with magic marker wrote below it "But the Idiots keep getting smarter".
Right now I have over 100 users doing this all over the world. All using english for the install but some have changed to their native keyboards
(france and turkey). All were ready to go back to Windows after trying Linux until I reinstalled an easy to support version.
Btrfs is not novice friendly and should be a choice for those that understand snapshots - I bet half the people with btrfs do not know how
to rollback a problem.
I have known novices rolling back the system when I did not know about it.
My 2 cents - Unix since 1973, Linux since 1995.
-- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
On Sunday 09 May 2021, Carlos E. R. wrote:
On 08/05/2021 20.49, Larry Len Rainey wrote:
Why not have an option for novice that does choice of UEFI or Legacy install with ext4 full root and swap of ram size - now you don't have to worry about size and btrfs problems for novices users - that is better until they can comprehend btrfs and other file system types and partitions.
That would be my choice, but is not what the people that doit chose.
...
Totally agree. To the uninformed the default btrfs install appears much like any non-btrfs installation. Many treat it as such until they strike trouble (or need to do some kind of maintenance such as moving to a new disk or setting up recoverable backups). Before running with btrfs a user needs to become familiar with the concepts and procedures. The simple things like df and du can no longer be trusted, that btrfs alternatives must be used (why doesn't someone amend the real df and real du - or make then error on btrfs). There are more complicated issues, such as how to you properly move or backup the root volume. Some understanding of the true/underlying structure of a btrfs root is necessary: this should include the subvolume layout, the special "@" subvolume the ID zero subvolume, and how snapshots fit into this structure. Perhaps there needs to be a daemon that checks the state of btrfs root volumes and takes action, or suggests action, well before space becomes an issue. Back in 2016 I used a VM to evaluate btrfs and concluded that ext4 was preferable for my relatively simple desktop needs. I wrote up those experiences and posted them: https://forums.opensuse.org/showthread.php/521277-LEAP-42-2-btrfs-root-files... (Note the root volume structure included in OpenSUSE has undergone some refinement and simplification since 2016.) It was only by performing that exercise that I came to appreciated how different an OpenSUSE btrfs root volume is from the ext4 equivalent and how much my current practices would have to change in making the switch. There now appears to some pretty good documentation for snapper/btrfs written for arch and SUSE, but I'm still surprised to see that issues such as those with df/du are still overlooked. I'm not knocking the advantage of btrfs/snapper, but people need to understand it a bit and be aware of how to use and maintain it properly. It may be that automated processes could remove the need for any good level of understanding, but I don't think we are at that point. Michael
The user (retired) is long in experience as a world authority on Mainframe hardware and Mainframe OS's development including UTS (Amdahl Mainframe Unix) and more than capable to spot the problem. If deleting snapshots didn't fix it his next move was btrfsck. Even years of experience don't shield you from the odd road block or embarrassment. The process certainly needs more thought than it did back in 2004 when I installed openSUSE for a 66 year old retired bus driver and a 77 year old retired welder who had never used a keyboard and had to be told what the backspace and space bar did. Those guys were able to do online updates and add cameras, printers, burn CD's, etc., email and all normal stuff completely unaided. Regards Sid. On 08/05/2021 19:49, Larry Len Rainey wrote:
Why not have an option for novice that does choice of UEFI or Legacy install with ext4 full root and swap of ram size - now you don't have to worry about size and btrfs problems for novices users - that is better until they can comprehend btrfs and other file system types and partitions.
That is how I install every OpenSUSE for users - no btrfs - instead I provide an external 2gb drive and scripts to backup and restore the drives. I also provide a bootable usb drive with a saved blkid of the system and a script to make all the file systems needed and to restore the last full backup on the external drive so that the restore script can put everything back where it was before the "disaster".
Few users understand enough "linux or windows" to know what they are doing. You have to make it idiot proof. Like the sign over the testing room at Microsoft in 1982 said - "We try to make our Software Idiot Proof" someone with magic marker wrote below it "But the Idiots keep getting smarter".
Right now I have over 100 users doing this all over the world. All using english for the install but some have changed to their native keyboards (france and turkey). All were ready to go back to Windows after trying Linux until I reinstalled an easy to support version.
Btrfs is not novice friendly and should be a choice for those that understand snapshots - I bet half the people with btrfs do not know how to rollback a problem.
My 2 cents - Unix since 1973, Linux since 1995.
On 5/8/21 1:24 PM, Carlos E. R. wrote:
On 08/05/2021 17.37, Sid Boyce wrote:
Users response.
Regards
Sid.
-------- Forwarded Message -------- Subject: Re: Fwd: snapshots filling up SSD Date: Sat, 8 May 2021 15:29:09 +0000 From: Sav. Mellor <> To: Sid Boyce <>
SSD are partitioned the same as normal disks, there is no difference. You could leave some space outside of any partition for wear levelling.
-- Sid Boyce ... Hamradio License G3VBV, Licensed Private Pilot Emeritus IBM/Amdahl Mainframes and Sun/Fujitsu Servers Tech Support Senior Staff Specialist, Cricket Coach Microsoft Windows Free Zone - Linux used for all Computing Tasks
Am Sonntag, 9. Mai 2021, 00:56:58 CEST schrieb Sid Boyce:
The user (retired) is long in experience as a world authority on Mainframe hardware and Mainframe OS's development including UTS (Amdahl Mainframe Unix) and more than capable to spot the problem. If deleting snapshots didn't fix it his next move was btrfsck.
Another thing that come to my mind is to check the content of /.snapshot I was hit (in the early btrfs days) by an orphaned snapshot which used an incredible amount of disk space. Deleting this /.snapshots/<number> which was not shown under 'snapper list' fixed the problem. No issues ever since then with btrfs Cheers Axel
On 09/05/2021 09:49, Axel Braun wrote:
Am Sonntag, 9. Mai 2021, 00:56:58 CEST schrieb Sid Boyce:
The user (retired) is long in experience as a world authority on Mainframe hardware and Mainframe OS's development including UTS (Amdahl Mainframe Unix) and more than capable to spot the problem. If deleting snapshots didn't fix it his next move was btrfsck. Another thing that come to my mind is to check the content of /.snapshot I was hit (in the early btrfs days) by an orphaned snapshot which used an incredible amount of disk space. Deleting this /.snapshots/<number> which was not shown under 'snapper list' fixed the problem. No issues ever since then with btrfs
Cheers Axel
Thanks Axel, That's exactly what I had to do last year when I saw the problem on one of my systems. Regards Sid. -- Sid Boyce ... Hamradio License G3VBV, Licensed Private Pilot Emeritus IBM/Amdahl Mainframes and Sun/Fujitsu Servers Tech Support Senior Staff Specialist, Cricket Coach Microsoft Windows Free Zone - Linux used for all Computing Tasks
The more I think about it, I'm convinced that a btrfs root "/" and xfs data "/home" configuration is fine for HDDs but not for SSDs, because of the wear considerations on SSDs.
I don't know why btrfs would be a problem, by default /home is not part of any snapshots so it doesn't incur additional CoW overhead. In any case, wear is not an issue on SSDs these days unless you fill them to the brim so they can't do proper wear leveling anymore. regards
Regarding btrfs-related defaults for SSD, I opened Bug 1184904 for /etc/sysconfig/btrfsmaintenance defaults. That particular bug is primarily targeting the filesystem corruption seen downstream - probably while live migrating VMs while a balance is running. Setting that value to "none" has stabilized our downstream VMs. Since making that setting, no additional systems have been lost to corrupted btrfs filesystems. Since btrfs balance rewrites a lot of data apparently without much if any benefit on a single disk non-RAID installation, I also like keeping it disabled to reduce wear on SSDs. For the use case of the pet Tumbleweeed VM that runs on my personal laptop, I have it set up an f2fs root filesystem. It's an old Macbook Pro with HDD, so I have the VM running on USB flash (at the under $20 price point, I'm assuming there is no wear leveling)
On 09/05/2021 18.43, peter.clark@ngic.com wrote:
Regarding btrfs-related defaults for SSD, I opened Bug 1184904 for /etc/sysconfig/btrfsmaintenance defaults.
That particular bug is primarily targeting the filesystem corruption seen downstream - probably while live migrating VMs while a balance is running. Setting that value to "none" has stabilized our downstream VMs. Since making that setting, no additional systems have been lost to corrupted btrfs filesystems.
Since btrfs balance rewrites a lot of data apparently without much if any benefit on a single disk non-RAID installation, I also like keeping it disabled to reduce wear on SSDs.
For the use case of the pet Tumbleweeed VM that runs on my personal laptop, I have it set up an f2fs root filesystem. It's an old Macbook Pro with HDD, so I have the VM running on USB flash (at the under $20 price point, I'm assuming there is no wear leveling)
I remember reading somewhere that btrfs was flash aware, but looking at "man mount" I see no references to this. There is some mention on "man mkfs.btrfs" -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
On Sun, May 9, 2021 at 2:08 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
On 09/05/2021 18.43, peter.clark@ngic.com wrote:
Regarding btrfs-related defaults for SSD, I opened Bug 1184904 for /etc/sysconfig/btrfsmaintenance defaults.
That particular bug is primarily targeting the filesystem corruption seen downstream - probably while live migrating VMs while a balance is running. Setting that value to "none" has stabilized our downstream VMs. Since making that setting, no additional systems have been lost to corrupted btrfs filesystems.
Since btrfs balance rewrites a lot of data apparently without much if any benefit on a single disk non-RAID installation, I also like keeping it disabled to reduce wear on SSDs.
For the use case of the pet Tumbleweeed VM that runs on my personal laptop, I have it set up an f2fs root filesystem. It's an old Macbook Pro with HDD, so I have the VM running on USB flash (at the under $20 price point, I'm assuming there is no wear leveling)
I remember reading somewhere that btrfs was flash aware, but looking at "man mount" I see no references to this. There is some mention on "man mkfs.btrfs"
It happens automatically, but the mount option "ssd" can be used to force it on. -- 真実はいつも一つ！/ Always, there's only one truth!
On 09/05/2021 20.17, Neal Gompa wrote:
On Sun, May 9, 2021 at 2:08 PM Carlos E. R. <> wrote:
I remember reading somewhere that btrfs was flash aware, but looking at "man mount" I see no references to this. There is some mention on "man mkfs.btrfs"
It happens automatically, but the mount option "ssd" can be used to force it on.
The manual on Leap doesn't mention this. Or it has eluded my search. :-? -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)
I took a look at a few VMs that are still on btrfs - Most of them have the 'ssd' flag set when looking at /etc/mtab, even going back to SLES 12, despite not having it specified in /etc/fstab anywhere. The ones that go read-only whenever a btrfs balance runs and need the root filesystem to be restored from backup each time are not showing the ssd flag in /etc/fstab. The backing storage is "all-flash arrays", some of which have NVRAM as the first stage before data written to them is deduplicated, compressed and written to SSD. A similar issue was also seen on a SLES 11 physical system with HDD using hardware RAID, so while lack of the ssd flag might be inducing extra wear on the all-flash arrays, the corruption induced by btrfs balance appears to be independent of that option. That SLES 11 system was aging, however, so it's possible that its hdds were beginning to have timeouts, though.
The ssd mount option appears to control the kernel's behavior, while /etc/sysconfig/btrfsmaintenance controls the frequency that filesystem maintenance jobs are run (or whether they are run automatically). By default on openSUSE/SLE, btrfs balance is a weekly scheduled job while scrub is monthly. This generates a lot of load which can trigger a live migration. When the live migrating happens on VMs with a large amount of RAM (e.g., 500GB), this often leads to filesystem corruption. Definitely would have appreciated a default that plays nicer in production while learning about the intricacies of btrfs behavior. The same goes for the pet VM on my personal laptop as well. ;)
participants (8)
-
Axel Braun -
Carlos E. R. -
Larry Len Rainey -
Maximilian Trummer -
Michael Hamilton -
Neal Gompa -
peter.clark＠ngic.com -
Sid Boyce