[opensuse-factory] btrfs qgroup / quotas concerns - revisited
Hi guys, Sometime ago, a BTRFS developer (Chris Murphy), which was helping me to fix a bug, warned that quotas must not be enabled by default in BTRFS because it is not stable. He actually sent a message to this mailing list: https://lists.opensuse.org/opensuse-factory/2016-09/msg00032.html However, some openSUSE developers contradicted Chris, specially Richard Brown: https://lists.opensuse.org/opensuse-factory/2016-09/msg00085.html Hence, nobody took the advice and quotas were enabled by default in Leap 42.2. I am sending this message because I think it is necessary to rethink this decision. A very annoying bug that I was having in all my openSUSE machines (Leap and Tumbleweed) is actually caused by quotas. Every week, when the maintenance script is started, my systems become unresponsive for almost 30 min. I mentioned it here but received just one answer: https://lists.opensuse.org/opensuse-factory/2016-09/msg00130.html Today, I saw a bug reported by Jan Ritzerfeld describing exactly the same behavior and mentioning that disabling quota fixes it: https://bugzilla.opensuse.org/show_bug.cgi?id=1017461 I tried, and I can confirm that the bug is indeed fixed by this workaround in Leap and Tumbleweed. My server, which was recently updated to Leap 42.2, started to show those freezes. Notice that quotas were not enabled in 42.1. Hence, I think we must revisit this subject as soon as possible. Best regards, Ronan Arraes -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
I am sending this message because I think it is necessary to rethink this decision. A very annoying bug that I was having in all my openSUSE machines (Leap and Tumbleweed) is actually caused by quotas. Every week, when the maintenance script is started, my systems become unresponsive for almost 30 min. I mentioned it here but received just one answer:
https://lists.opensuse.org/opensuse-factory/2016-09/msg00130.html
This also happens to me on TW, and is incredibly frustrating. So can we please also disable quotas on TW as well? -- Aleksa Sarai Software Engineer (Containers) SUSE Linux GmbH https://www.cyphar.com/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 01/03/2017 11:11 AM, Aleksa Sarai wrote:
I am sending this message because I think it is necessary to rethink this decision. A very annoying bug that I was having in all my openSUSE machines (Leap and Tumbleweed) is actually caused by quotas. Every week, when the maintenance script is started, my systems become unresponsive for almost 30 min. I mentioned it here but received just one answer:
https://lists.opensuse.org/opensuse-factory/2016-09/msg00130.html
This also happens to me on TW, and is incredibly frustrating. So can we please also disable quotas on TW as well?
Or we could give the kernel maintainers a couple of weeks to come back from there holidays and see if they can come up with a patch to fix the issue. This report is 1 week old and given the time of year thats not much time for anyone to deal with the issue. Once someone has time to look at it the bug (or if there is a long period of know one looking at it) then we can talk about changing defaults. "Feature X has a bug, Lets disable Feature X" doesn't seem like the best approach we should try and fix the bug first. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On 01/03/2017 11:11 AM, Aleksa Sarai wrote:
I am sending this message because I think it is necessary to rethink this decision. A very annoying bug that I was having in all my openSUSE machines (Leap and Tumbleweed) is actually caused by quotas. Every week, when the maintenance script is started, my systems become unresponsive for almost 30 min. I mentioned it here but received just one answer:
https://lists.opensuse.org/opensuse-factory/2016-09/msg00130.html
This also happens to me on TW, and is incredibly frustrating. So can we please also disable quotas on TW as well?
Or we could give the kernel maintainers ... kernel maintainers != brtfs developers; brtfs isn't in maintenance mode and likely won't be for some time. The problem seems caused by a user-mode script running. If that can lock up the kernel, wouldn't
Simon Lees wrote: that be good for a denial-of-service attack? But why give them any time if it is a bad decision? Why should quotas be on at all on a user system? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
im not sure i see any conclusive tests here. there are several factors 1 - enospc error vs performance hit 2 - maintainance script with or without quotas if it is claimed disbling quota system is a fix - how is this determined - wait one week, hope the load case is equal and see the results? a proper test would be a - balance (sudo /etc/cron.weekly/btrfs-balance) b - write say 10,000 blocks with dd c - run balance d - repeat without quotas enabled reading the wiki, quota performance can be impacted by too many snapshots -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Nicholas. Em ter, 2017-01-03 às 08:58 +0100, nicholas escreveu:
im not sure i see any conclusive tests here.
there are several factors 1 - enospc error vs performance hit 2 - maintainance script with or without quotas
It is not ENOSPC bug. I had a ENOSPC bug, which we thought might be related to quotas that were enabled by default in Tumbleweed. Because of this, Chris sent the message to this mailing list suggesting to disable it in Leap / Tumbleweed. The freezing problem was described by me in September, but since nobody beside Achim Gratz replied, I thought it was something "normal" in BTRFS. One week ago, a user reported the bug because he is experiencing the freeze and verified that disabling quotas is a workaround.
if it is claimed disbling quota system is a fix - how is this determined - wait one week, hope the load case is equal and see the results?
a proper test would be a - balance (sudo /etc/cron.weekly/btrfs-balance) b - write say 10,000 blocks with dd c - run balance d - repeat without quotas enabled
I did something very similar to that and I can confirm that disabling quotas fixed the freeze. However, you mentioned something important. Maybe the number of snapshots + quotas is what cause the freeze. I will try to debug that. Regards, Ronan Arraes -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
I did something very similar to that and I can confirm that disabling quotas fixed the freeze. However, you mentioned something important. Maybe the number of snapshots + quotas is what cause the freeze. I will try to debug that.
Regards, Ronan Arraes yes - my point is that there are many moving parts here, and it is very difficult to run an objective/repeatable test. It is hard to tell from anecdotes with certainty the root or extent of the problem. It is hard to get bugs fixed without a test case.
there might be many workarounds. changing frequency and target level of the balanace. or perhaps it is merely the creation/removal of many snapshots, i dont know. I have snapshot on updates only and have no problems. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Dienstag, 3. Januar 2017, 11:50:19 CET schrieb Simon Lees:
[...] Or we could give the kernel maintainers a couple of weeks to come back from there holidays and see if they can come up with a patch to fix the issue. This report is 1 week old and given the time of year thats not much time for anyone to deal with the issue. [...]
The message well I hear, my faith alone is weak. This is actually happening to bug reports about btrfs: https://lists.opensuse.org/opensuse-factory/2017-01/msg00002.html Gruß Jan -- When in doubt, take all the defaults. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (6)
-
Aleksa Sarai -
Jan Ritzerfeld -
Linda Walsh -
nicholas -
Ronan Arraes Jardim Chagas -
Simon Lees