[opensuse] BTFS issues
Stick OSS 13.2 machine running kernel 3.16.7-21-default. Machine has four SATA drives of 500 GB each. All drives are partitioned identically with 3 partitions each. First partition is 1G of type BIOS Boot. Second partition is 2 G of type linux raid. Third partition is of 463G of type linux raid. Second partitions of all drives are setup as a RAID5 md device md0 and used as swap. Third partitions of all drives are setup as a RAID5 md device md1 and mount as / (the root filesystem). Machine has been running fine for some time, doing much of nothing other than daily zypper updates etc. A few days ago, I copied massive amounts of data to /data/save1 (unsure of total amount, copied close to 1.1 TB, then deleted about 500 MB, then copied about 600 or so MG again). The copies were done with rsync and completed with no errors whatsoever. Once the copies completed, I decided to a btrfs balance (btrfs balance start /)- and that is where the fun began. btrfs kept failing with messages like below in the log (below is a grep 'btrfs' from journalctl -b so the messages are not necessarily contiguous): [44950.935025] BTRFS info (device md1): relocating block group 3706217037824 flags 1 [44954.665729] BTRFS info (device md1): relocating block group 3705143296000 flags 1 [44966.689827] BTRFS info (device md1): found 156 extents [44985.223754] BTRFS: bdev /dev/md1 errs: wr 9911, rd 0, flush 0, corrupt 0, gen 0 [44985.224033] BTRFS: bdev /dev/md1 errs: wr 9912, rd 0, flush 0, corrupt 0, gen 0 [44985.224285] BTRFS: bdev /dev/md1 errs: wr 9913, rd 0, flush 0, corrupt 0, gen 0 Running a btrfs scrub / showed no errors or issues at all. Then I started doing btrfs balance start -dusage=X and noticed that it would succeed up to X = 73 and fail after that. Then I started deleting subsets of data. I did btrfs balance start -dusage=X after each delete and noticed that as more and more data was deleted, X kept increasing before the balance would error out. Now, 4th day of the saga (slow system!) I have btrfs balance running successfully with -dusage=95 but still failing with error similar to above with -dusage=100. Do I need to keep on deleting data and doing the balance until balance succeeds with -dusage=100 ? Above, while more or less a test (the data exists in the source still so I am not too worried about losing from this btrfs file system), has scared me a bit about using btrfs for truly production data. Does anyone know what the btrfs errors in the log mean? I am assuming it has something to do with the requirement of doing a btrfs balance every so often. I know that btrfs has issues when at high usage capacity, and one has to balance etc. However, this is the first time I am seeing it, and knowing it happened without the data copy failing with a disk full error or anything of that nature has left me with the impression that btrfs is not suitable for production use. I would much rather have had my data copy fail with some disk full type error or what have you than run into this apparent time bomb. Other than the balance errors, everything else is fine on the filesystem, I can create files, delete files, system reboots fine (even though this is /). Thanks and I eagerly await comments from others who have seen this. -- --Moby They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. -- Benjamin Franklin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 11:54 AM, Moby wrote:
Stick OSS 13.2 machine running kernel 3.16.7-21-default. Machine has four SATA drives of 500 GB each. All drives are partitioned identically with 3 partitions each.
First partition is 1G of type BIOS Boot. Second partition is 2 G of type linux raid. Third partition is of 463G of type linux raid.
Second partitions of all drives are setup as a RAID5 md device md0 and used as swap. Third partitions of all drives are setup as a RAID5 md device md1 and mount as / (the root filesystem).
Machine has been running fine for some time, doing much of nothing other than daily zypper updates etc.
A few days ago, I copied massive amounts of data to /data/save1 (unsure of total amount, copied close to 1.1 TB, then deleted about 500 MB, then copied about 600 or so MG again). The copies were done with rsync and completed with no errors whatsoever.
Once the copies completed, I decided to a btrfs balance (btrfs balance start /)- and that is where the fun began.
btrfs kept failing with messages like below in the log (below is a grep 'btrfs' from journalctl -b so the messages are not necessarily contiguous):
[44950.935025] BTRFS info (device md1): relocating block group 3706217037824 flags 1 [44954.665729] BTRFS info (device md1): relocating block group 3705143296000 flags 1 [44966.689827] BTRFS info (device md1): found 156 extents [44985.223754] BTRFS: bdev /dev/md1 errs: wr 9911, rd 0, flush 0, corrupt 0, gen 0 [44985.224033] BTRFS: bdev /dev/md1 errs: wr 9912, rd 0, flush 0, corrupt 0, gen 0 [44985.224285] BTRFS: bdev /dev/md1 errs: wr 9913, rd 0, flush 0, corrupt 0, gen 0
Running a btrfs scrub / showed no errors or issues at all.
Then I started doing btrfs balance start -dusage=X and noticed that it would succeed up to X = 73 and fail after that.
Then I started deleting subsets of data. I did btrfs balance start -dusage=X after each delete and noticed that as more and more data was deleted, X kept increasing before the balance would error out.
Now, 4th day of the saga (slow system!) I have btrfs balance running successfully with -dusage=95 but still failing with error similar to above with -dusage=100. Do I need to keep on deleting data and doing the balance until balance succeeds with -dusage=100 ?
Above, while more or less a test (the data exists in the source still so I am not too worried about losing from this btrfs file system), has scared me a bit about using btrfs for truly production data.
Does anyone know what the btrfs errors in the log mean? I am assuming it has something to do with the requirement of doing a btrfs balance every so often. I know that btrfs has issues when at high usage capacity, and one has to balance etc. However, this is the first time I am seeing it, and knowing it happened without the data copy failing with a disk full error or anything of that nature has left me with the impression that btrfs is not suitable for production use. I would much rather have had my data copy fail with some disk full type error or what have you than run into this apparent time bomb.
Other than the balance errors, everything else is fine on the filesystem, I can create files, delete files, system reboots fine (even though this is /).
Thanks and I eagerly await comments from others who have seen this.
Also, at present, btrfs fi show gives: ]btrfs fi show Label: none uuid: 33b98d97-606b-4968-a266-24a48a9fe50d Total devices 1 FS bytes used 1.17TiB devid 1 size 1.36TiB used 1.18TiB path /dev/md1 -- --Moby -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 12:54 PM, Moby wrote:
Stick OSS 13.2 machine running kernel 3.16.7-21-default
I'm running 4.0.5-2, I have both default and desktop. I subscribe to the btrfs list X-Mailing-List: linux-btrfs@vger.kernel.org and see patches every day. I see I've zypper up a new kernel this morning! 4.0.5-4 Yes I run BtrFS but not on RAID and not for any data or /home. Back in the 3.11 days I had it corrupt /home. I've never had any problems with it just on / but regular readers will know that I use LVM and have almost everything I can migrated off /. I'm sure people here will express doubts about using RAID5 especially with just three spindles. But that's another matter. The only reason I'm running BtrFS at all is that I want to avoid the inode exhaustion problems I encounter with the ext[234] files systems and consider over-provisioning aesthetically displeasing. I started BtrFS on / and it hasn't given any problems in the very reduced / that I have configured. Which raises the point - how have you partitioned you system? Is this a "everything is one big FS under /" ? I've been happy with ReiserFS and would like to continue with it. Together with LVM it does everything I've needed so far. SSDs may change that, but it may be that the best solution for SSDs is not BtrFS. I suspect that if you could better tie down what your problem is from the logs then a perusal of the patches to a more update version of BtrFS might be of use. Which version of btrfsprogs/btrfsmaintenance are you running? Information for package btrfsprogs: ----------------------------------- Repository: openSUSE BuildService - filesystems Name: btrfsprogs Version: 4.0-186.1 Arch: x86_64 Vendor: obs://build.opensuse.org/filesystems Installed: Yes Status: up-to-date Installed Size: 3.0 MiB Summary: Utilities for the Btrfs filesystem Description: Utilities needed to create and maintain btrfs file systems under Linux. Information for package btrfsmaintenance: ----------------------------------------- Repository: openSUSE BuildService - filesystems Name: btrfsmaintenance Version: 0.1-10.1 Arch: noarch Vendor: obs://build.opensuse.org/filesystems Installed: Yes Status: up-to-date Installed Size: 29.6 KiB Summary: Scripts for btrfs periodic maintenance tasks Description: Scripts for btrfs maintenance tasks like periodic scrub, balance, trim or defrag on selected mountpoints or directories. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 12:52 PM, Anton Aylward wrote:
On 06/19/2015 12:54 PM, Moby wrote:
Stick OSS 13.2 machine running kernel 3.16.7-21-default I'm running 4.0.5-2, I have both default and desktop.
I subscribe to the btrfs list X-Mailing-List: linux-btrfs@vger.kernel.org and see patches every day.
I see I've zypper up a new kernel this morning! 4.0.5-4
Yes I run BtrFS but not on RAID and not for any data or /home.
Back in the 3.11 days I had it corrupt /home. I've never had any problems with it just on / but regular readers will know that I use LVM and have almost everything I can migrated off /.
I'm sure people here will express doubts about using RAID5 especially with just three spindles. But that's another matter.
The only reason I'm running BtrFS at all is that I want to avoid the inode exhaustion problems I encounter with the ext[234] files systems and consider over-provisioning aesthetically displeasing. I started BtrFS on / and it hasn't given any problems in the very reduced / that I have configured.
Which raises the point - how have you partitioned you system? Is this a "everything is one big FS under /" ?
I've been happy with ReiserFS and would like to continue with it. Together with LVM it does everything I've needed so far. SSDs may change that, but it may be that the best solution for SSDs is not BtrFS.
I suspect that if you could better tie down what your problem is from the logs then a perusal of the patches to a more update version of BtrFS might be of use.
Which version of btrfsprogs/btrfsmaintenance are you running?
Information for package btrfsprogs: ----------------------------------- Repository: openSUSE BuildService - filesystems Name: btrfsprogs Version: 4.0-186.1 Arch: x86_64 Vendor: obs://build.opensuse.org/filesystems Installed: Yes Status: up-to-date Installed Size: 3.0 MiB Summary: Utilities for the Btrfs filesystem Description: Utilities needed to create and maintain btrfs file systems under Linux.
Information for package btrfsmaintenance: ----------------------------------------- Repository: openSUSE BuildService - filesystems Name: btrfsmaintenance Version: 0.1-10.1 Arch: noarch Vendor: obs://build.opensuse.org/filesystems Installed: Yes Status: up-to-date Installed Size: 29.6 KiB Summary: Scripts for btrfs periodic maintenance tasks Description: Scripts for btrfs maintenance tasks like periodic scrub, balance, trim or defrag on selected mountpoints or directories.
Thanks Anton. Yes, the system is one big FS under / I used to use REISERFS and EXT3 on top of LVM before, now I mainly use EXT4 on top of LVM in prod. One headache, at least for me. is the journalctl business (I cannot just open up /var/log/messages). Going through the whole log for btrfs related messages, I see just the ones I posted in the original post. BTRFS package wise the system has: rpm -qa |grep btrf btrfsprogs-4.0-7.1.x86_64 libbtrfs0-4.0-7.1.x86_64 btrfsmaintenance-0.1-1.1.noarch I am fairly certain I can get balance to succeed after deleting "enough" stuff - key is what is "enough". I deleted yet another batch of data, and btrfs balance is now running, so far, with -dusage=100. -- --Moby -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 02:05 PM, Moby wrote:
Yes, the system is one big FS under /
I realise that the designers envisioned something like this, but even with aggressive sub-volumes to handle partial backups, I think this stinks. it sets all my mental alarm bells running. As you've seen, it anything goes wrong EVERYTHING goes wrong. There's a good reason we had partitioning! I understand snapshotting can be carefully configured but the reality is that its obscure and awkward. The policy for something like zypper updates/patches and the ability to roll them back in the manner reminiscent of the mainframe era is distinct from the snapshotting for user space changes and possible accidental deletes. Snapshotting databases is, again, quite a different strategy. I can see that a site that is, in effect, 'mainframe replacement' might warrant this and might warrant sending the admin off for relevant training and having a standby "scratch" system he could practice on (perhaps a virtual machine), but I think its overblown for a home user or just a workstation.
One headache, at least for me. is the journalctl business (I cannot just open up /var/log/messages).
IIR there was a discussion of this earlier this year and some advice on how to run syslog to /var/log/messages in parallel. Hmm. I seem to be doing that :-) I must have paid attention at the time it was posted :-) Do check the archives ...
BTRFS package wise the system has: rpm -qa |grep btrf btrfsprogs-4.0-7.1.x86_64 libbtrfs0-4.0-7.1.x86_64 btrfsmaintenance-0.1-1.1.noarch
I would strongly advise updating your kernel at the very least! There's the old adage about keeping patches up to date. Try one of these. I prefer the first. Kernel_Stable http://download.opensuse.org/repositories/Kernel:/stable/standard/ Kernel_standard http://download.opensuse.org/repositories/Kernel:/openSUSE-13.2/standard/
I am fairly certain I can get balance to succeed after deleting "enough" stuff - key is what is "enough". I deleted yet another batch of data, and btrfs balance is now running, so far, with -dusage=100.
Hmmm. This sounds rather like cutting off your arms and legs so as to reduce the amount of work the heart has to do pumping blood around the system. I would really really really suggest installing a patched and up to date version of BtrFS - the kernel - as well as the utilities (see my earlier email for the repositories) rather than amputating your data. -- "Now look," Forrester said patiently, "progress is an outmoded idea. We've got to be in step with the times. We've got to ask ourselves what progress ever did for us." -- Randall Garett, "Pagan Passions", 1959 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 01:45 PM, Anton Aylward wrote:
On 06/19/2015 02:05 PM, Moby wrote:
Yes, the system is one big FS under / I realise that the designers envisioned something like this, but even with aggressive sub-volumes to handle partial backups, I think this stinks. it sets all my mental alarm bells running.
As you've seen, it anything goes wrong EVERYTHING goes wrong. There's a good reason we had partitioning!
I understand snapshotting can be carefully configured but the reality is that its obscure and awkward. The policy for something like zypper updates/patches and the ability to roll them back in the manner reminiscent of the mainframe era is distinct from the snapshotting for user space changes and possible accidental deletes. Snapshotting databases is, again, quite a different strategy.
I can see that a site that is, in effect, 'mainframe replacement' might warrant this and might warrant sending the admin off for relevant training and having a standby "scratch" system he could practice on (perhaps a virtual machine), but I think its overblown for a home user or just a workstation.
One headache, at least for me. is the journalctl business (I cannot just open up /var/log/messages). IIR there was a discussion of this earlier this year and some advice on how to run syslog to /var/log/messages in parallel. Hmm. I seem to be doing that :-) I must have paid attention at the time it was posted :-) Do check the archives ...
BTRFS package wise the system has: rpm -qa |grep btrf btrfsprogs-4.0-7.1.x86_64 libbtrfs0-4.0-7.1.x86_64 btrfsmaintenance-0.1-1.1.noarch I would strongly advise updating your kernel at the very least! There's the old adage about keeping patches up to date.
Try one of these. I prefer the first.
Kernel_Stable http://download.opensuse.org/repositories/Kernel:/stable/standard/ Kernel_standard
http://download.opensuse.org/repositories/Kernel:/openSUSE-13.2/standard/
I am fairly certain I can get balance to succeed after deleting "enough" stuff - key is what is "enough". I deleted yet another batch of data, and btrfs balance is now running, so far, with -dusage=100. Hmmm. This sounds rather like cutting off your arms and legs so as to reduce the amount of work the heart has to do pumping blood around the system.
I would really really really suggest installing a patched and up to date version of BtrFS - the kernel - as well as the utilities (see my earlier email for the repositories) rather than amputating your data.
Well, the machine is patched as per "standard" oss 132 repos (http://download.opensuse.org/repositories/openSUSE:/13.2:/Update/standard/). My prod systems all run syslog, this one machine I was trying to somewhat stay true to oss defaults (other than making the filesystem one big /). I have no problem adding the kernel and filesystem repos you mention (I have 3 other machines running tumbleweed with kernel from http://download.opensuse.org/repositories/Kernel:/stable/standard (those are at 4.0.5-4 as of this morning). This particular machine where btrfs balance is failing is still fully useable, just that balance keeps failing. My main reason for playing on this machine is really to see what type of setup to go with my prod servers when I upgrade those from 13.1 to 13.2 ... I normally stick with "stock" oss repos for prod systems (mainly just http://download.opensuse.org/repositories/openSUSE:/13.2:/Update/standard/ type stuff). I can easily rebuild the machine in question, the data is available elsewhere and is not in jeopardy). I have been deleting stuff and re-running balance, and I am at a point where it succeeds with -dusage=98 but fails with higher numbers. I am almost at the point to rule out using btrfs in my prod systems, rebuild this particular machine with ext4 on top of lvm on top of md, and then keep on going with my testing. One odd thing I noticed is that btrfs balance succeeds with -dusage=98, and fails with 99. However, if it I run it again with -dusage=98, it still "does stuff" and succeeds. Which makes me wonder if perhaps multiple passes are required with -dusage=98, at least until it stop reporting having done stuff. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 02:56 PM, Moby wrote:
Well, the machine is patched as per "standard" oss 132 repos (http://download.opensuse.org/repositories/openSUSE:/13.2:/Update/standard/).
My prod systems all run syslog, this one machine I was trying to somewhat stay true to oss defaults (other than making the filesystem one big /).
Big :-) "You get what you pay for", or rather you get the results of the decision tree you followed.
I have no problem adding the kernel and filesystem repos you mention (I have 3 other machines running tumbleweed with kernel from http://download.opensuse.org/repositories/Kernel:/stable/standard (those are at 4.0.5-4 as of this morning).
I don't run tumbleweed. I might if I decide to stay with 13.1 I take it the 4.0.5-4 machines are handling btrFS better than this one? I'll reboot tomorrow to my own 4.0.5-4
This particular machine where btrfs balance is failing is still fully useable, just that balance keeps failing.
That is almost certainly indicative of a deeper problem. Its pretty obvious that you are running an 'early' (?earlier?) version of BtrFS & tools. There must be an avalanche of know problems that have been fixed between then and now. And more to come. As I've posted elsewhere, RAID5/6 seems to be particularly problematic.
.... I am almost at the point to rule out using btrfs in my prod systems, ....
All the indication I see watching the btrfs lists are that its not production-ready yet.
... rebuild this particular machine with ext4 on top of lvm on top of md, and then keep on going with my testing.
You might consider XFS. I'm sure someone will come in here to tell you how great XFS is. Or ZFS https://en.wikipedia.org/wiki/ZFS#Data_integrity Or NILFS https://en.wikipedia.org/wiki/NILFS whose snapshotting/versioning looks, to me, more attractive than that of BtrFS. Not least of all for SSDs http://www.linux-mag.com/id/7345/ Please note: there are many things that I think should NOT NOT NOT be on a file system like BtrFS, most especially certain types of database storage. This too will be critical in a production system.
One odd thing I noticed is that btrfs balance succeeds with -dusage=98, and fails with 99. However, if it I run it again with -dusage=98, it still "does stuff" and succeeds. Which makes me wonder if perhaps multiple passes are required with -dusage=98, at least until it stop reporting having done stuff.
-- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 02:25 PM, Anton Aylward wrote:
On 06/19/2015 02:56 PM, Moby wrote:
Well, the machine is patched as per "standard" oss 132 repos (http://download.opensuse.org/repositories/openSUSE:/13.2:/Update/standard/).
My prod systems all run syslog, this one machine I was trying to somewhat stay true to oss defaults (other than making the filesystem one big /). Big :-) "You get what you pay for", or rather you get the results of the decision tree you followed.
I have no problem adding the kernel and filesystem repos you mention (I have 3 other machines running tumbleweed with kernel from http://download.opensuse.org/repositories/Kernel:/stable/standard (those are at 4.0.5-4 as of this morning). I don't run tumbleweed. I might if I decide to stay with 13.1
I take it the 4.0.5-4 machines are handling btrFS better than this one? I'll reboot tomorrow to my own 4.0.5-4
This particular machine where btrfs balance is failing is still fully useable, just that balance keeps failing. That is almost certainly indicative of a deeper problem. Its pretty obvious that you are running an 'early' (?earlier?) version of BtrFS & tools. There must be an avalanche of know problems that have been fixed between then and now. And more to come.
As I've posted elsewhere, RAID5/6 seems to be particularly problematic.
.... I am almost at the point to rule out using btrfs in my prod systems, .... All the indication I see watching the btrfs lists are that its not production-ready yet.
... rebuild this particular machine with ext4 on top of lvm on top of md, and then keep on going with my testing. You might consider XFS. I'm sure someone will come in here to tell you how great XFS is.
Or ZFS https://en.wikipedia.org/wiki/ZFS#Data_integrity
Or NILFS https://en.wikipedia.org/wiki/NILFS whose snapshotting/versioning looks, to me, more attractive than that of BtrFS. Not least of all for SSDs http://www.linux-mag.com/id/7345/
Please note: there are many things that I think should NOT NOT NOT be on a file system like BtrFS, most especially certain types of database storage. This too will be critical in a production system.
One odd thing I noticed is that btrfs balance succeeds with -dusage=98, and fails with 99. However, if it I run it again with -dusage=98, it still "does stuff" and succeeds. Which makes me wonder if perhaps multiple passes are required with -dusage=98, at least until it stop reporting having done stuff.
I upgraded to kernel 4.0.5-4.g56152db-default and btrfsprogs-4.0-186.1.x86_64 and still have same results, namely everything works fine except that btrfs balance errors out when run without -dusage or with dusage=99 or higher. I think it is time for me to move back to ext4. -- --Moby They that can give up essential liberty to obtain a little temporary safety deserve neither libert -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-06-20 01:31, Moby wrote:
I upgraded to kernel 4.0.5-4.g56152db-default and btrfsprogs-4.0-186.1.x86_64 and still have same results, namely everything works fine except that btrfs balance errors out when run without -dusage or with dusage=99 or higher. I think it is time for me to move back to ext4.
I imagine that upgrading to a much newer kernel does not suffice. You would also have to upgrade the filesystem itself, ie, format it, to take full advantage of developments... just a wild guess. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlWEqkgACgkQja8UbcUWM1xygQD9GufswqwtDYlGmpXRDXBzYh3S 9I0F84P2eX232SQf/wMA/3iLz1rW0kJqfncsT/ULvejFqyDvdhCzBBndwTgsJb/d =6M/B -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-06-20 01:31, Moby wrote:
I upgraded to kernel 4.0.5-4.g56152db-default and btrfsprogs-4.0-186.1.x86_64 and still have same results, namely everything works fine except that btrfs balance errors out when run without -dusage or with dusage=99 or higher. I think it is time for me to move back to ext4. I imagine that upgrading to a much newer kernel does not suffice. You would also have to upgrade the filesystem itself, ie, format it, to take full advantage of developments... just a wild guess.
- -- Cheers / Saludos,
Carlos E. R.
(from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)
iF4EAREIAAYFAlWEqkgACgkQja8UbcUWM1xygQD9GufswqwtDYlGmpXRDXBzYh3S 9I0F84P2eX232SQf/wMA/3iLz1rW0kJqfncsT/ULvejFqyDvdhCzBBndwTgsJb/d =6M/B -----END PGP SIGNATURE----- Thanks Carlos. I can understand if I was changing filesystems or changing file system versions - hopefully I do not have to recreate the filesystem to get over this balance error issue. If so, then that in and of itself is enough for me to dump btrfs. I see btrfs balance continues to succeed with -dusage=98 and each time it logs moving 1 extent, and the extent id keeps on changing. So I think it will eventually succeed without the -dusage, except that I will
On 06/19/2015 06:48 PM, Carlos E. R. wrote: likely have to run it enough (not sure how many - hundreds? thousands?) times. As I said, this was mainly a test, and test is leaning towards indicating that I should stay away from btrfs for now. -- --Moby They that can give up essential liberty to obtain a little temporary safety deserve neither libert -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
Yes I run BtrFS but not on RAID and not for any data or /home.
Back in the 3.11 days I had it corrupt /home. I've never had any problems with it just on / but regular readers will know that I use LVM and have almost everything I can migrated off /.
---- Ditto on that. Cept my raid meltdown was back under the 2.x kernels. Decided to go w/hardware raid or BIOS-raid for 1+0 setups.
I'm sure people here will express doubts about using RAID5 especially with just three spindles. But that's another matter.
??? How many should you have? My current root (haven't quite gotten up the gumption to xfr root to a RAID-10 based on orange-housing blinking lights -- despite the fact that the controller claims it is fine) has 2 data spindles, 1 parity. But it is running on 15000RPM disks that are 'short stroked' by about 50% to lower seek times. I figured my root would need more spread-out and smaller I/O's.
The only reason I'm running BtrFS at all is that I want to avoid the inode exhaustion problems I encounter with the ext[234] files systems and consider over-provisioning aesthetically displeasing.
---- xfs doesn't have those problems, but the new CRC-check on the meta data has me very worried -- You can't even change a disks GUID w/o the disk becoming corrupt. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 07/07/2015 09:45 PM, Linda Walsh wrote:
Anton Aylward wrote:
Yes I run BtrFS but not on RAID and not for any data or /home.
Back in the 3.11 days I had it corrupt /home. I've never had any problems with it just on / but regular readers will know that I use LVM and have almost everything I can migrated off /.
---- Ditto on that. Cept my raid meltdown was back under the 2.x kernels. Decided to go w/hardware raid or BIOS-raid for 1+0 setups.
I can understand that. My AIX boxes in the racks reach have mirroring of the basic "root" and the 'data warehouse' in the "back room" (wall to wall) is whatever RAID that IBM deems it is. All this is is, as you say, BIOS or hardware and is quite transparent.
I'm sure people here will express doubts about using RAID5 especially with just three spindles. But that's another matter.
??? How many should you have?
My personal OPINION, based on my work with hamming coding in the 1970s and my experience with rebuilding RAID with a series of unfortunately unreliable drives in the 1990s is that I'd settle for 5. I might like more; the mathematics of hamming makes it easier when you start thinking in terms of 32 or even 64 bit wide data. But assembling that to write as 4K blocks is a but, well, excessive. You OPINION may be different. You ECONOMICS may be different. I'm NOT *REPEAT NOT* going to argue this point. It is my OPINION. I will note, however, that for the home user a RAID array of 5 or more 1T or 2T drives (since they come in at around US$50 http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=744346&csid=_61[1]) means that you can store more movies that your local video rental store has in stock. [2] Its getting awkward to find a reliable and consistent source of smaller SATA drives. On that point, I should mention, as I have before, some of those old <100G (some even less than 10G!) drives deep in the pile at the Closet of Anxieties are still working. There must be a lesson here about "Designing for a limited lifetime" for economic reasons[4].
My current root (haven't quite gotten up the gumption to xfr root to a RAID-10 based on orange-housing blinking lights -- despite the fact that the controller claims it is fine) has 2 data spindles, 1 parity. But it is running on 15000RPM disks that are 'short stroked' by about 50% to lower seek times. I figured my root would need more spread-out and smaller I/O's.
I've see many, MANY, !MANY! models and supporting mathematics and simulation over the performance of disk "striping", disk and cache turning. Some even take into account the nature of the application base and profile. The old acronym, EMACS - "Eight Megagbyes and Constantly Swapping" - was never that funny. many applications that users consider fundamental, Firefox most notably, so far exceed 8 meg *in code* that its not even remotely funny. But then again, long gone are the days of UNIX on a PDP-11 with one meg of 'core' and roll-in/roll-out. Even when we moved to the VAX and had twice or even four times that memory, and virtual memory support (that you Bill Joy) we still tried to write code that employed 'The Principle of Locality" so as to minimise paging. I don't know that modern compilers, linker, loaders and VM caching makes a better job of it. Yes we can tune the VM/paging/caching system. But is it worth it? Whenever any of the IT people I deal with ask me I tell them to get motherboards that they can load up with all the memory it can take and do so. Corporate budgets are more flexible on that point than home users, but sometimes contractual constraints are an impediment. They may _want_ a Toshiba or an AlienWare so it can be loaded up, but 'policy' says they have to have a (comparatively crippled) Leveno or a Dell, whose upgrade to 32G would cost more than three Toshibas[5]. Yes, I've seen that. The irony is that avoiding corporate procurement beats the economics. http://www.amazon.com/Corsair-Vengeance-Laptop-Memory-CMSX16GX3M2A1600C10/dp...
The only reason I'm running BtrFS at all is that I want to avoid the inode exhaustion problems I encounter with the ext[234] files systems and consider over-provisioning aesthetically displeasing.
xfs doesn't have those problems,
Yes, I've aware of that. Neither does ReiserFs. ZFS looks interesting but seeing what oracle is doing with Java makes me apprehensive about the longer term implications using ZFS. Even more so than the longer term implications for ReiserFS. Then there's NilFS. I can this being useful for the personal workstations or a shared workspace for developers of authors. As far as snapshots go, a pure journalling system beats out the way BtrFS seems to work. And for those of us who expect to work past 2038 its seems a good choice.
but the new CRC-check on the meta data has me very worried -- You can't even change a disks GUID w/o the disk becoming corrupt.
I'll stick with ReiserFS for my rotating rust for the foreseeable future. I plan to move my RootFS off BtrFS and abandon BtrFS. For SSD I can see Nil2 for low latency items such as VM loading of binaries and F2FS for any streaming applications (though its unlikely I'll keep movies & music on SSD) [1] See also http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=7739052&csid=_61 for another economic break point. [2] Of course if you live in a major city the library might have an even larger stock but the quality might be lacking[3]. [3] You could, of course, download from YouTube if you have enough bandwidth, and time, to spare. [4] I recall a story about the original Henry Ford finding out that the most common item in scrapyards was the rear axel of wrecked Model T cars. It was so well built! He decided it must have been 'over-engineered' and had it redesigned to be less so, less reliable, less robust. I don't know how true this anecdote is. [5] Such as http://www.laptopmag.com/reviews/laptops/toshiba-qosmio-x875-q7390 although not at that price :-) -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Jul 7, 2015 at 7:45 PM, Linda Walsh
Ditto on that. Cept my raid meltdown was back under the 2.x kernels. Decided to go w/hardware raid or BIOS-raid for 1+0 setups.
For what it's worth, firmware RAID were previously managed by the device mapper (dmraid), and now these days it's managed by md/mdadm. The firmware understands the ondisk metadata format and assembles the array so that things like bootloaders can be properly found and used in the pre-boot environment. But after all that, the kernel (and mdadm) are completely responsible for handling this kind of RAID, the firmware is not involved.
I'm sure people here will express doubts about using RAID5 especially with just three spindles. But that's another matter.
??? How many should you have?
I'm lost here too. Three is more reliable than four with respect to data to parity ratio. And two disk RAID 5 is the same thing as mirroring except also weird so why bother?
xfs doesn't have those problems, but the new CRC-check on the meta data has me very worried -- You can't even change a disks GUID w/o the disk becoming corrupt.
Yes. I suspect this is a short term issue because right now snapshotting is almost completely unworkable (LVM snapshots of either variety, as well as qcow2, etc) if that volume UUID appears to the kernel twice. -- Chris Murphy -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-07-08 16:46, Chris Murphy wrote:
And two disk RAID 5 is the same thing as mirroring except also weird so why bother?
A two disk raid 5 is "degraded", and it will complain till you complete it. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlWdmBcACgkQja8UbcUWM1z5kQD/WP0+cZ4fcDkxwIqzRlJi9DwI rh2DRjWoCwaoiH2jouQA/2ITTpKQCB0+BSPQu1AuBjfpsY9gO3yz1PI/zhy6+bgj =JTKV -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
В Wed, 08 Jul 2015 23:37:28 +0200
"Carlos E. R."
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-07-08 16:46, Chris Murphy wrote:
And two disk RAID 5 is the same thing as mirroring except also weird so why bother?
A two disk raid 5 is "degraded", and it will complain till you complete it.
Where have you go this idea? -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlWd57sACgkQR6LMutpd94ws2gCfY1Fp2oaUbQoMfBCSe1hymRES VA0AnRZ8ku13pb5pvkBTxiDpteGvtpnj =4ROu -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-07-09 05:17, Andrei Borzenkov wrote:
В Wed, 08 Jul 2015 23:37:28 +0200 "Carlos E. R." <> пишет:
A two disk raid 5 is "degraded", and it will complain till you complete it.
Where have you go this idea?
Several sources. For instance: https://en.wikipedia.org/wiki/RAID#Standard_levels «RAID 5 RAID 5 consists of block-level striping with distributed parity. Unlike in RAID 4, parity information is distributed among the drives. It requires that all drives but one be present to operate. Upon failure of a single drive, subsequent reads can be calculated from the distributed parity such that no data is lost. RAID 5 requires at least three disks.[11] RAID 5 is seriously affected by the general trends regarding array rebuild time and the chance of drive failure during rebuild.[22] Rebuilding an array requires reading all data from all disks, opening a chance for a second drive failure and the loss of entire array. In August 2012, Dell posted an advisory against the use of RAID 5 in any configuration on Dell EqualLogic arrays and RAID 50 with "Class 2 7200 RPM drives of 1 TB and higher capacity" for business-critical data.[23]» Notice the «RAID 5 requires at least three disks.» - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlWePTgACgkQja8UbcUWM1yT/gEAj6+hEfKCHgUpsHF/OhR1/1Ag yDpbQD0yFrtO792URgUA/j1+EYDitwu3BrZB2SjvcLP4Amk9iry65JhxvQERnSWO =Hp8x -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Jul 9, 2015 at 12:22 PM, Carlos E. R.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-07-09 05:17, Andrei Borzenkov wrote:
В Wed, 08 Jul 2015 23:37:28 +0200 "Carlos E. R." <> пишет:
A two disk raid 5 is "degraded", and it will complain till you complete it.
Where have you go this idea?
Several sources. For instance:
https://en.wikipedia.org/wiki/RAID#Standard_levels
«RAID 5 RAID 5 consists of block-level striping with distributed parity. Unlike in RAID 4, parity information is distributed among the drives. It requires that all drives but one be present to operate. Upon failure of a single drive, subsequent reads can be calculated from the distributed parity such that no data is lost. RAID 5 requires at least three disks.[11] RAID 5 is seriously affected by the general trends regarding array rebuild time and the chance of drive failure during rebuild.[22] Rebuilding an array requires reading all data from all disks, opening a chance for a second drive failure and the loss of entire array. In August 2012, Dell posted an advisory against the use of RAID 5 in any configuration on Dell EqualLogic arrays and RAID 50 with "Class 2 7200 RPM drives of 1 TB and higher capacity" for business-critical data.[23]»
Notice the «RAID 5 requires at least three disks.»
Personalities : [raid6] [raid5] [raid4] md0 : active raid5 loop1[2] loop0[0] 8704 blocks super 1.2 level 5, 512k chunk, algorithm 2 [2/2] [UU] I have also seen statements that RAID5 must have exactly 5 disks. In reality nothing in RAID5 *algorithm* prohibits or makes impossible having one data disk; it just makes no sense (it is more CPU intensive with zero advantages over RAID1 in this case). It is true that many implementations of hardware RAID won't allow you to create RAID5 with less than 3 disks, for the reasons I just mentioned. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-07-09 15:39, Andrei Borzenkov wrote:
On Thu, Jul 9, 2015 at 12:22 PM, Carlos E. R.
Notice the «RAID 5 requires at least three disks.»
Personalities : [raid6] [raid5] [raid4] md0 : active raid5 loop1[2] loop0[0] 8704 blocks super 1.2 level 5, 512k chunk, algorithm 2 [2/2] [UU]
I have also seen statements that RAID5 must have exactly 5 disks.
In reality nothing in RAID5 *algorithm* prohibits or makes impossible having one data disk; it just makes no sense (it is more CPU intensive with zero advantages over RAID1 in this case). It is true that many implementations of hardware RAID won't allow you to create RAID5 with less than 3 disks, for the reasons I just mentioned.
Well... https://raid.wiki.kernel.org/index.php/Overview#RAID-5 « RAID-5 This is perhaps the most useful RAID mode when one wishes to combine a larger number of physical disks, and still maintain some redundancy. RAID-5 can be (usefully) used on three or more disks, with zero or more spare-disks. The resulting RAID-5 device size will be (N-1)*S, just like RAID-4. The big difference between RAID-5 and -4 is, that the parity information is distributed evenly among the participating drives, avoiding the bottleneck problem in RAID-4, and also getting more performance out of the disk when reading, as all drives will then be used. » That's the official information. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlWefN0ACgkQja8UbcUWM1y3uAEAgPqCsL13XDSgux2NRbxZTqjX JTm3ylg9EQx02+G0UT0BAJaCRv86loDdBI5nGPv9G7S3iiYBtQHRY2PxINhBRxYM =EE+P -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Jul 8, 2015 at 3:37 PM, Carlos E. R.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-07-08 16:46, Chris Murphy wrote:
And two disk RAID 5 is the same thing as mirroring except also weird so why bother?
A two disk raid 5 is "degraded", and it will complain till you complete it.
mkfs.btrfs does not complain at mkfs time or mount time for a 2 device raid5. [1] mdadm -C does not complain at create time or mkfs time for a 2 device raid5. [2] Neither are considered degraded. However, lvcreate complains at create time for a 2 device raid5. [3] So clearly this depends on implementation. [1] [root@localhost ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sr0 11:0 1 1024M 0 rom vda 252:0 0 40G 0 disk └─vda1 252:1 0 40G 0 part / vdb 252:16 0 10G 0 disk vdc 252:32 0 10G 0 disk [root@localhost ~]# mkfs.btrfs -m raid5 -d raid5 /dev/vdb /dev/vdc btrfs-progs v4.1 See http://btrfs.wiki.kernel.org for more information. Label: (null) UUID: 16fdf954-ad92-4360-93ad-0041664bbd91 Node size: 16384 Sector size: 4096 Filesystem size: 20.00GiB Block group profiles: Data: RAID5 1.01GiB Metadata: RAID5 1.01GiB System: RAID5 12.00MiB SSD detected: no Incompat features: extref, raid56, skinny-metadata Number of devices: 2 Devices: ID SIZE PATH 1 10.00GiB /dev/vdb 2 10.00GiB /dev/vdc [root@localhost ~]# mount /dev/vdb /mnt [root@localhost ~]# btfs fi show /mnt bash: btfs: command not found... [root@localhost ~]# btrfs fi show /mnt Label: none uuid: 16fdf954-ad92-4360-93ad-0041664bbd91 Total devices 2 FS bytes used 640.00KiB devid 1 size 10.00GiB used 2.03GiB path /dev/vdb devid 2 size 10.00GiB used 2.01GiB path /dev/vdc btrfs-progs v4.1 [root@localhost ~]# btrfs fi df /mnt Data, single: total=8.00MiB, used=0.00B Data, RAID5: total=1.00GiB, used=512.00KiB System, single: total=4.00MiB, used=0.00B System, RAID5: total=8.00MiB, used=16.00KiB Metadata, single: total=8.00MiB, used=0.00B Metadata, RAID5: total=1.00GiB, used=112.00KiB GlobalReserve, single: total=16.00MiB, used=0.00B [2] [root@localhost ~]# mdadm -C /dev/md0 -n 2 -l raid5 --assume-clean /dev/vdb /dev/vdc mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started. [root@localhost ~]# mdadm -E /dev/vdb /dev/vdb: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 1a6cb32c:84aa0f29:27b255f9:0838034b Name : localhost.localdomain:0 (local to host localhost.localdomain) Creation Time : Thu Jul 9 09:16:22 2015 Raid Level : raid5 Raid Devices : 2 Avail Dev Size : 20955136 (9.99 GiB 10.73 GB) Array Size : 10477568 (9.99 GiB 10.73 GB) Data Offset : 16384 sectors Super Offset : 8 sectors Unused Space : before=16296 sectors, after=0 sectors State : clean Device UUID : 599dc079:b7afcc9e:869a74eb:0831cd84 Update Time : Thu Jul 9 09:16:22 2015 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 8c195d60 - correct Events : 0 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) [root@localhost ~]# mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Thu Jul 9 09:16:22 2015 Raid Level : raid5 Array Size : 10477568 (9.99 GiB 10.73 GB) Used Dev Size : 10477568 (9.99 GiB 10.73 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Thu Jul 9 09:16:22 2015 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : localhost.localdomain:0 (local to host localhost.localdomain) UUID : 1a6cb32c:84aa0f29:27b255f9:0838034b Events : 0 Number Major Minor RaidDevice State 0 252 16 0 active sync /dev/vdb 1 252 32 1 active sync /dev/vdc [root@localhost ~]# cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 vdc[1] vdb[0] 10477568 blocks super 1.2 level 5, 512k chunk, algorithm 2 [2/2] [UU] unused devices: <none> [3] [root@localhost ~]# lvcreate -L 15G -n lv5 --type raid5 rtest Number of stripes must be at least 2 for raid5 -- Chris Murphy -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
I note that on the btrfs list there have been recent (this month) patches to addres RAID5/6/ issues. one post says <quote src="Subject: Re: Does raid56 work on latest kernel? Date: Thu, 11 Jun 2015 04:14:25 +0000 (UTC) Message-ID: <pan$4058c$7888c445$459683cf$d6eac669@cox.net> "> ... Technically, all the raid56 support should be there now -- it's code- complete. Practically, however, I've been recommending people continue to stay off of it for anything but pure data-loss-ok testing, for at *LEAST* another couple kernels, to shake out some of the inevitable bugs and let the code settle at least a /bit/. And point-of-fact, we /did/ have some bad raid56 mode reports shortly after 4.0, from people who had /not/ let it shake out a bit and were trying to use it in normal-case situations. Whether they're actually fixed now (with the latest stable or with late 4.1-rcs for 4.1 release) I'm not sure, tho the number of bad reports has died down quite a bit, but I don't know whether that is people actually following the recommendation, or because it's fixed now. Either way, however, I'd not be using it myself, and couldn't recommend it for others except as I said purely for data-loss-doesn't-matter testing, until at LEAST 4.2, as there's still likely to be a few more critical bugs to shake out. And even at 4.2, I'd still recommend raid56 mode only for those willing to be bleeding edge testers, if that's what it takes to be leading edge testers, because point-of-fact, that code still won't be as stable as the raid0/1/10 modes, which are basically as stable as btrfs itself is by this point. It's going to take some time, as well as the reports of those leading/bleeding edge testers, to shake out further bugs and stabilize that still very new code. For more btrfs-mainstream users[1], I'd recommend waiting about a year, five kernel cycles, for btrfs raid56 mode to stabilize. </quote> -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 02:01 PM, Anton Aylward wrote:
I note that on the btrfs list there have been recent (this month) patches to addres RAID5/6/ issues. one post says
<quote src="Subject: Re: Does raid56 work on latest kernel? Date: Thu, 11 Jun 2015 04:14:25 +0000 (UTC) Message-ID: <pan$4058c$7888c445$459683cf$d6eac669@cox.net> "> ...
Technically, all the raid56 support should be there now -- it's code- complete. Practically, however, I've been recommending people continue to stay off of it for anything but pure data-loss-ok testing, for at *LEAST* another couple kernels, to shake out some of the inevitable bugs and let the code settle at least a /bit/.
And point-of-fact, we /did/ have some bad raid56 mode reports shortly after 4.0, from people who had /not/ let it shake out a bit and were trying to use it in normal-case situations. Whether they're actually fixed now (with the latest stable or with late 4.1-rcs for 4.1 release) I'm not sure, tho the number of bad reports has died down quite a bit, but I don't know whether that is people actually following the recommendation, or because it's fixed now. Either way, however, I'd not be using it myself, and couldn't recommend it for others except as I said purely for data-loss-doesn't-matter testing, until at LEAST 4.2, as there's still likely to be a few more critical bugs to shake out.
And even at 4.2, I'd still recommend raid56 mode only for those willing to be bleeding edge testers, if that's what it takes to be leading edge testers, because point-of-fact, that code still won't be as stable as the raid0/1/10 modes, which are basically as stable as btrfs itself is by this point. It's going to take some time, as well as the reports of those leading/bleeding edge testers, to shake out further bugs and stabilize that still very new code.
For more btrfs-mainstream users[1], I'd recommend waiting about a year, five kernel cycles, for btrfs raid56 mode to stabilize.
</quote> Thanks Anton - sorry if I did not mention earlier, the RAID5 setup is using md raid, not using btrfs ...
-- --Moby -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/19/2015 02:03 PM, Moby wrote:
On 06/19/2015 02:01 PM, Anton Aylward wrote:
I note that on the btrfs list there have been recent (this month) patches to addres RAID5/6/ issues. one post says
<quote src="Subject: Re: Does raid56 work on latest kernel? Date: Thu, 11 Jun 2015 04:14:25 +0000 (UTC) Message-ID: <pan$4058c$7888c445$459683cf$d6eac669@cox.net> "> ...
Technically, all the raid56 support should be there now -- it's code- complete. Practically, however, I've been recommending people continue to stay off of it for anything but pure data-loss-ok testing, for at *LEAST* another couple kernels, to shake out some of the inevitable bugs and let the code settle at least a /bit/.
And point-of-fact, we /did/ have some bad raid56 mode reports shortly after 4.0, from people who had /not/ let it shake out a bit and were trying to use it in normal-case situations. Whether they're actually fixed now (with the latest stable or with late 4.1-rcs for 4.1 release) I'm not sure, tho the number of bad reports has died down quite a bit, but I don't know whether that is people actually following the recommendation, or because it's fixed now. Either way, however, I'd not be using it myself, and couldn't recommend it for others except as I said purely for data-loss-doesn't-matter testing, until at LEAST 4.2, as there's still likely to be a few more critical bugs to shake out.
And even at 4.2, I'd still recommend raid56 mode only for those willing to be bleeding edge testers, if that's what it takes to be leading edge testers, because point-of-fact, that code still won't be as stable as the raid0/1/10 modes, which are basically as stable as btrfs itself is by this point. It's going to take some time, as well as the reports of those leading/bleeding edge testers, to shake out further bugs and stabilize that still very new code.
For more btrfs-mainstream users[1], I'd recommend waiting about a year, five kernel cycles, for btrfs raid56 mode to stabilize.
</quote> Thanks Anton - sorry if I did not mention earlier, the RAID5 setup is using md raid, not using btrfs ...
Same machine (upgraded kernel and btrfsprogs as per Anton's suggestion). Doing a btrfs balance with -dusage=95 now shows negative values in percentage left! Every 15.0s: sh -c date;btrfs balance status -v / Wed Jun 24 12:29:12 2015 Wed Jun 24 12:29:12 CDT 2015 Balance on '/' is running 306 out of about 145 chunks balanced (312 considered), -111% left Dumping filters: flags 0x1, state 0x1, force is off DATA (flags 0x2): balancing, usage=95 -- --Moby They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. -- Benjamin Franklin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/24/2015 12:33 PM, Moby wrote:
On 06/19/2015 02:03 PM, Moby wrote:
On 06/19/2015 02:01 PM, Anton Aylward wrote:
I note that on the btrfs list there have been recent (this month) patches to addres RAID5/6/ issues. one post says
<quote src="Subject: Re: Does raid56 work on latest kernel? Date: Thu, 11 Jun 2015 04:14:25 +0000 (UTC) Message-ID: <pan$4058c$7888c445$459683cf$d6eac669@cox.net> "> ...
Technically, all the raid56 support should be there now -- it's code- complete. Practically, however, I've been recommending people continue to stay off of it for anything but pure data-loss-ok testing, for at *LEAST* another couple kernels, to shake out some of the inevitable bugs and let the code settle at least a /bit/.
And point-of-fact, we /did/ have some bad raid56 mode reports shortly after 4.0, from people who had /not/ let it shake out a bit and were trying to use it in normal-case situations. Whether they're actually fixed now (with the latest stable or with late 4.1-rcs for 4.1 release) I'm not sure, tho the number of bad reports has died down quite a bit, but I don't know whether that is people actually following the recommendation, or because it's fixed now. Either way, however, I'd not be using it myself, and couldn't recommend it for others except as I said purely for data-loss-doesn't-matter testing, until at LEAST 4.2, as there's still likely to be a few more critical bugs to shake out.
And even at 4.2, I'd still recommend raid56 mode only for those willing to be bleeding edge testers, if that's what it takes to be leading edge testers, because point-of-fact, that code still won't be as stable as the raid0/1/10 modes, which are basically as stable as btrfs itself is by this point. It's going to take some time, as well as the reports of those leading/bleeding edge testers, to shake out further bugs and stabilize that still very new code.
For more btrfs-mainstream users[1], I'd recommend waiting about a year, five kernel cycles, for btrfs raid56 mode to stabilize.
</quote> Thanks Anton - sorry if I did not mention earlier, the RAID5 setup is using md raid, not using btrfs ...
Same machine (upgraded kernel and btrfsprogs as per Anton's suggestion). Doing a btrfs balance with -dusage=95 now shows negative values in percentage left!
Every 15.0s: sh -c date;btrfs balance status -v / Wed Jun 24 12:29:12 2015
Wed Jun 24 12:29:12 CDT 2015 Balance on '/' is running 306 out of about 145 chunks balanced (312 considered), -111% left Dumping filters: flags 0x1, state 0x1, force is off DATA (flags 0x2): balancing, usage=95
I upgraded today to kernel 4.1.0-1.gfcf8349-default and btrfs-progs v4.1+20150622. Then I kicked off a btrfs balance with no -duage. So far it is running fine and is at 28% left. Keeping my fingers crossed that this updates fixes the issue. -- --Moby They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. -- Benjamin Franklin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/24/2015 09:01 PM, Moby wrote:
kernel 4.1.0-1.gfcf8349-default and btrfs-progs v4.1+20150622
Upgrading to kernel 4.1.0-1.gfcf8349-default and btrfs-progs v4.1+20150622 seems to have fixed the problem. btrfs balance now completes without any errors. -- --Moby -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/25/2015 10:40 AM, Moby wrote:
On 06/24/2015 09:01 PM, Moby wrote:
kernel 4.1.0-1.gfcf8349-default and btrfs-progs v4.1+20150622
Upgrading to kernel 4.1.0-1.gfcf8349-default and btrfs-progs v4.1+20150622 seems to have fixed the problem. btrfs balance now completes without any errors.
What repositories are those from? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/25/2015 09:44 AM, Anton Aylward wrote:
On 06/25/2015 10:40 AM, Moby wrote:
On 06/24/2015 09:01 PM, Moby wrote:
kernel 4.1.0-1.gfcf8349-default and btrfs-progs v4.1+20150622 Upgrading to kernel 4.1.0-1.gfcf8349-default and btrfs-progs v4.1+20150622 seems to have fixed the problem. btrfs balance now completes without any errors.
What repositories are those from?
The kernel is from http://download.opensuse.org/repositories/Kernel:/stable/standard/ and btrfsprogs is from http://download.opensuse.org/repositories/filesystems/openSUSE_13.2/ -- --Moby They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. -- Benjamin Franklin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (6)
-
Andrei Borzenkov -
Anton Aylward -
Carlos E. R. -
Chris Murphy -
Linda Walsh -
Moby