Re: [opensuse-factory] btrfs qgroup concerns
On 3 September 2016 at 18:24, Chris Murphy
On Fri, Sep 2, 2016 at 8:27 PM, Jeff Mahoney
wrote: No, it's not. We publish the code for every single one of our kernels both in patch series and expanded tree form. Every maintenance update is tagged and even if they weren't, the last commit that was included in the RPM is included in the RPM metadata. We also have a very longstanding, public policy regarding code going upstream and a outsize percentage of our kernel teams in upstream maintainer roles. So you can imagine that when someone suggests that we're not being good community members with our kernels, it ruffles a few feathers. You're the one leveling an accusation (using typically incorrect assumptions). You bring the proof.
1. On the Btrfs mailing list, most every time anyone has a quota problem, they're informed if they need stable quota support to use XFS or ext4. And not a single Btrfs developer ever contradicts that advice in the slightest way. And this most recent non-contradicted advice happened just a few weeks ago.
Well now you've seen at least one of our btrfs developers contradicting that advice. And it's not like Jeff is some novice who doesn't know what he's been talking about, he's been intimately involved with the kernel and explicitly btrfs and other filesystems for years now. He knows what he's talking about, he's very rarely wrong, and he is one of those boring "don't change anything, ship it only when it's absolutely ready" Enterprise developers. And even IF he is wrong, it's going to be the job of him and his team to fix it in both SLE and openSUSE, so you can take extra reassurance from that. And he works for SUSE, and SUSE has invested a lot of time, effort, engineering and testing on this. You've heard of the testing Jeff's team has done. In addition I can tell you that SUSE QA have been doing additional testing on btrfs qgroups and shaken out a corner case bug or two, but nothing that would classify as a 'Ship Stopper'. SUSE's Beta customers have all been using the feature also as part of the SLE 12 SP2 Open Beta programme. You've heard of the strategic importance of this feature from Matthias, SUSE's Senior Director of SLE Product Management. As a default option in our installation, almost all of SUSE's customers will be relying on this to work when they install SLE 12 SP2 at the end. SUSE isn't going to do that unless it's confident in the quality of the feature and its ability to support it for years. As an openSUSE contributor, you have an opportunity to contribute to this and to bring additional knowledge and data points to the table, but that comes with a responsibility to ensure that your data is accurate, correct, reproducible and actionable. By quoting random commits and mailinglist threads you do not have a full understanding of, while actively rejecting any suggestion to do your own testing, you have failed to contribute responsibly to this discussion.
The very first time I have ever read anywhere, by any Btrfs developer, that enabling quotas is considered stable, was on this list.
You heard it here first :)
2. https://github.com/kdave/btrfs-progs/commit/637c005ac881a7def923e3f56c9f9285...
Upstream has performance and stability concerns in the current code. That commit is two weeks old.
I will quote from that commit
"Signed-off-by: David Sterba
Matthias, Richard, and some of your comments, while circumspect, can be inferred to mean your quota code substantially differs from the current stable kernel, and that's why you consider it stable. But then just yesterday you say "the only [Btrfs] patches we have applied to the 4.4-based SLE12-SP2/Leap kernel are backports and patches we've submitted upstream".
4.4 is the latest upstream LTS Kernel 4.7 is the latest upstream kernel You have tested 4.7 in tumbleweed, and quoted from commits for 4.7 The current 4.4 Kernel contains many backports from 4.7. It also contains code which 4.7 has totally changed, so has specific fixes for 4.4 that are nothing to do with the current state in 4.7. These 4.4 backports+fixes number in the thousands. Many of them were written by SUSE engineers as part of their work for SLE 12 SP2/Leap 42.2 and then submitted upstream (it's our policy to submit all of our patches upstream). In the very unlikely event of upstream not accept a small amount of our patches, there is also the possibility that OUR 4.4 LTS Kernel might have some patches and configuration that differs from upstream vanilla 4.4 LTS. So, to put it bluntly, everything you tested in 4.7 is irrelevant unless you prove otherwise. everything you read about the current upstream development is not conclusive unless you prove otherwise. You were invited to help confirm the issues you feared were present in the 4.4 LTS Kernel used by Leap 42.2 or SLE 12 SP2. You were given advice from Jeff on how to do this. You have outright refused to do so, so frankly I feel you have no credibility to imply that there are deeper issues. To put it very directly, with all due respect, either contribute or shut up - this is a mailinglist for contributors to discuss the development of the openSUSE distributions, not a soap box for ill-informed rants. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 03.09.2016 23:40, Richard Brown wrote:
On 3 September 2016 at 18:24, Chris Murphy
wrote: On Fri, Sep 2, 2016 at 8:27 PM, Jeff Mahoney
wrote: No, it's not. We publish the code for every single one of our kernels both in patch series and expanded tree form. Every maintenance update is tagged and even if they weren't, the last commit that was included in the RPM is included in the RPM metadata. We also have a very longstanding, public policy regarding code going upstream and a outsize percentage of our kernel teams in upstream maintainer roles. So you can imagine that when someone suggests that we're not being good community members with our kernels, it ruffles a few feathers. You're the one leveling an accusation (using typically incorrect assumptions). You bring the proof.
1. On the Btrfs mailing list, most every time anyone has a quota problem, they're informed if they need stable quota support to use XFS or ext4. And not a single Btrfs developer ever contradicts that advice in the slightest way. And this most recent non-contradicted advice happened just a few weeks ago.
Well now you've seen at least one of our btrfs developers contradicting that advice. And it's not like Jeff is some novice who doesn't know what he's been talking about, he's been intimately involved with the kernel and explicitly btrfs and other filesystems for years now. He knows what he's talking about, he's very rarely wrong, and he is one of those boring "don't change anything, ship it only when it's absolutely ready" Enterprise developers. And even IF he is wrong, it's going to be the job of him and his team to fix it in both SLE and openSUSE, so you can take extra reassurance from that.
And he works for SUSE, and SUSE has invested a lot of time, effort, engineering and testing on this. You've heard of the testing Jeff's team has done. In addition I can tell you that SUSE QA have been doing additional testing on btrfs qgroups and shaken out a corner case bug or two, but nothing that would classify as a 'Ship Stopper'. SUSE's Beta customers have all been using the feature also as part of the SLE 12 SP2 Open Beta programme.
You've heard of the strategic importance of this feature from Matthias, SUSE's Senior Director of SLE Product Management. As a default option in our installation, almost all of SUSE's customers will be relying on this to work when they install SLE 12 SP2 at the end. SUSE isn't going to do that unless it's confident in the quality of the feature and its ability to support it for years.
As an openSUSE contributor, you have an opportunity to contribute to this and to bring additional knowledge and data points to the table, but that comes with a responsibility to ensure that your data is accurate, correct, reproducible and actionable. By quoting random commits and mailinglist threads you do not have a full understanding of, while actively rejecting any suggestion to do your own testing, you have failed to contribute responsibly to this discussion.
The very first time I have ever read anywhere, by any Btrfs developer, that enabling quotas is considered stable, was on this list.
You heard it here first :)
Matthias, Richard, and some of your comments, while circumspect, can be inferred to mean your quota code substantially differs from the current stable kernel, and that's why you consider it stable. But then just yesterday you say "the only [Btrfs] patches we have applied to the 4.4-based SLE12-SP2/Leap kernel are backports and patches we've submitted upstream".
You were invited to help confirm the issues you feared were present in the 4.4 LTS Kernel used by Leap 42.2 or SLE 12 SP2. You were given advice from Jeff on how to do this. You have outright refused to do so, so frankly I feel you have no credibility to imply that there are deeper issues.
To put it very directly, with all due respect, either contribute or shut up - this is a mailinglist for contributors to discuss the development of the openSUSE distributions, not a soap box for ill-informed rants.
I can't and will not say anything about particular bug discussed here BUT I think many people will agree that the reason why so many are getting those insane "not enough space" btrfs error varieties over the _years_ is that it's main purpose is to provide zfs-like enterprise features to corporation with big data storages while small-time enterprises and SOHO users of openSUSE and alike are readily treated as mere guinea pigs for that use case, meaning that openSUSE itself is an eternal "corner case". Recently, the distribution installer was removed from TW "due to bugs" while the only bug of that installer that I've ever seen is another "not enough space" right in the middle of installing 12Gb worth of packages on fresh ~20Gb-sized btrfs partition which I successfully have done multiple times before with different live builds and machines. Yeah, remove the installer of the system, not the thing that breaks it, that makes sense... Also a nice attempt to try to force more users on official desktop-dysfunctional TW builds as testers of "One True Way of SUSE". So, regarding "the development of the openSUSE distributions", since you're are the upstream and demanding from ordinary users having Ph.D. in computer science and endless free time is utterly insane, how about that: 1) Since the damn thing needs free space to even free its space, make it reserve that space for no other reason than to not shit itself when it's time to delete something. 2) Make it checkable and repairable on R/O mount or make your initrd able to check and repair it before mounting anything. Same goes for XFS. Right now they both are never checked at all. 3) Fix it's kernel code or all userspace space-querying software to make it able to figure out its real space utilization. 4) Make automatic snapshot deletion and balancing on potential "not enough space" before throwing it into userspace. Or at least enable your btrfsmaintenance crutches by default. 5) Make snapper defaults realistic and, better yet, make YaST partitioner generate them based on partition/volume parameters. 6) While you're at it, properly present all its features in YaST, especially compression. OR remove it from being default altogether because only those who can handle it need it anyway. PS: I will not even read the replies because I fear for my sanity at this point. I mean, I know I complained about the installer multiple times but only because I think that it's the most egregious example that SUSE leaders are totally out of their minds or cynical to the point of mind-boggling hypocrisy. Either way, feeding such well-written nonsense to the brain in big doses is simply dangerous, one may even start believing it.
Sergey, what did I say about this mailinglist not being a soapbox for
ill-informed rants?
On 4 September 2016 at 08:36, Sergey Kondakov
I can't and will not say anything about particular bug discussed here BUT I think many people will agree that the reason why so many are getting those insane "not enough space" btrfs error varieties over the _years_ is that it's main purpose is to provide zfs-like enterprise features to corporation with big data storages while small-time enterprises and SOHO users of openSUSE and alike are readily treated as mere guinea pigs for that use case, meaning that openSUSE itself is an eternal "corner case".
Not true. Bugs filed by the openSUSE community are treated with a very high level of respect and concern - heck, I can think of cases where me and my colleagues in SUSE QA have had to mention openSUSE bugs because our testing was considered a 'corner case' but as soon as we were able to show that at least one openSUSEr had hit the same issue 'in the real world' there was no longer any arguing and the bug got fixed, fast.
Recently, the distribution installer was removed from TW "due to bugs" while the only bug of that installer that I've ever seen is another "not enough space" right in the middle of installing 12Gb worth of packages on fresh ~20Gb-sized btrfs partition which I successfully have done multiple times before with different live builds and machines. Yeah, remove the installer of the system, not the thing that breaks it, that makes sense... Also a nice attempt to try to force more users on official desktop-dysfunctional TW builds as testers of "One True Way of SUSE".
Nonsense. The distribution installer was not removed. The LiveCD installer was removed because it had absolutely zero testing, zero maintainers, and ZERO SUPPORT for UEFI. Given 100% of modern laptops, desktops, and even servers these days ship with UEFI, most of the time as a default, having a LiveCD installer without any support for UEFI was an absolutely unacceptable situation. As no one was maintaining it, no one fixed it for many months. And therefore, it was removed. "Fix it, or it gets dropped" is a semi-regular occurrence in Tumbleweed. And since then, a contributor (thanks Fabian!) has done an amazing job of hacking together the Net installer onto our LiveCD's, so while the LiveCD installer is no longer present, the NET installer (which does support UEFI and is fully tested).
So, regarding "the development of the openSUSE distributions", since you're are the upstream and demanding from ordinary users having Ph.D. in computer science and endless free time is utterly insane, how about that: 1) Since the damn thing needs free space to even free its space, make it reserve that space for no other reason than to not shit itself when it's time to delete something.
That's exactly what the new snapper btrfs quota support does - literally, you're demanding the very thing which is the opposite of what Chris is demanding... http://snapper.io/2016/05/18/space-aware-cleanup.html
2) Make it checkable and repairable on R/O mount or make your initrd able to check and repair it before mounting anything. Same goes for XFS. Right now they both are never checked at all.
It is, that's what mount -o ro,recovery is for
3) Fix it's kernel code or all userspace space-querying software to make it able to figure out its real space utilization.
Why, when btrfs fi df / works? Read https://btrfs.wiki.kernel.org/index.php/FAQ#Understanding_free_space.2C_usin... to understand why this demand is unreasnoble (and also nothing to do with the topic at hand)
4) Make automatic snapshot deletion and balancing on potential "not enough space" before throwing it into userspace. Or at least enable your btrfsmaintenance crutches by default.
That's exactly what the new snapper btrfs quota support does - literally, you're demanding the very thing which is the opposite of what Chris is demanding... http://snapper.io/2016/05/18/space-aware-cleanup.html
5) Make snapper defaults realistic and, better yet, make YaST partitioner generate them based on partition/volume parameters.
That's exactly what the current parameters in YaST do - snapshots get disabled is the root filesystem is too small. If you disagree with those parameters, then file detailed bugs, not rants on this mailinglist asking for features we already have.
6) While you're at it, properly present all its features in YaST, especially compression.
no - we'll present the features we're happiest with people using, let's keep it simple at first, given emails like yours clearly show people can get confused by the subset of features we already expose.
PS: I will not even read the replies because I fear for my sanity at this point. I mean, I know I complained about the installer multiple times but only because I think that it's the most egregious example that SUSE leaders are totally out of their minds or cynical to the point of mind-boggling hypocrisy. Either way, feeding such well-written nonsense to the brain in big doses is simply dangerous, one may even start believing it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 4 September 2016 at 08:54, Richard Brown
On 4 September 2016 at 08:36, Sergey Kondakov
wrote:
2) Make it checkable and repairable on R/O mount or make your initrd able to check and repair it before mounting anything. Same goes for XFS. Right now they both are never checked at all.
It is, that's what mount -o ro,recovery is for
I just re-read this and realised I wasn't actually answering Sergeys question, I'd like to change my response to "No, that will just give us complaints of boot taking longer for no benefit" -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Richard Brown wrote:
Sergey, what did I say about this mailinglist not being a soapbox for ill-informed rants?
Richard, you're just another user here, right? You're obviously free to express your opinion, but to my knowledge, you're not the mailing list moderator. (that position might currently be unoccupied). -- Per Jessen, Zürich (21.4°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2016-09-04 08:36, Sergey Kondakov wrote:
Same goes for XFS. Right now they both are never checked at all.
I understand that XFS filesystems undergo a sanity check at mount time. There is no need to an explicit check. In fact, fsck.xfs is a script that says that it will not check the filesystem, and that's an upstream decision, not openSUSE's. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
Hi Richard, unfortunately, one of your bold statements is wrong IMNSHO. Am 03.09.2016 um 20:40 schrieb Richard Brown:
or two, but nothing that would classify as a 'Ship Stopper'. SUSE's Beta customers have all been using the feature also as part of the SLE 12 SP2 Open Beta programme.
This assumes all SLES customers participating in the Beta program are installing on btrfs. I am a SLES customer. I have 10s of thousands of servers and VMs installed on SLES12 The only ones that have btrfs are those that are accidentally installed the wrong way. Once we notice btrfs is in use, we immediately reinstall without btrfs. I know quite a few other SLES customers, and they all basically do the same. So just because "it's default on SLES" and "lots of people are doing tests on SLES12" does not prove anything wrt. stability of btrfs. :-) Best regards, and have a lot of fun... -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Stefan,
On 5 September 2016 at 20:39, Stefan Seyfried
I am a SLES customer. I have 10s of thousands of servers and VMs installed on SLES12
The only ones that have btrfs are those that are accidentally installed the wrong way. Once we notice btrfs is in use, we immediately reinstall without btrfs.
I know quite a few other SLES customers, and they all basically do the same.
So just because "it's default on SLES" and "lots of people are doing tests on SLES12" does not prove anything wrt. stability of btrfs.
:-)
I'm glad to hear you are a SLES customer, but if you are part of the SLES Beta programme, why have I not seen any posts from you on the SUSE Beta Mailinglists? SUSE have very large customers who willfully chose SLES 12 explictically for btrfs with snapshot-rollback. Those very large customers have very large environments, spanning not just the x86_64 architectures, but all other hardware architectures supported by SLES 12. We are well aware the number of customers taking the step to using btrfs are increasing, with an increasing number choosing to do so as part of their migrations from SLE 11 to SLE 12. Your experience may not match our observations, but I would say it is pretty hard to describe you as 'typical' example of either a SLES customer or an openSUSE contributor.
Best regards, and have a lot of fun...
You too :-) -- Richard Brown Technical Lead - openQA openSUSE Chairman SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (5)
-
Carlos E. R.
-
Per Jessen
-
Richard Brown
-
Sergey Kondakov
-
Stefan Seyfried