On Thursday 04 Jul 2013 11:54:05 Jeff Mahoney wrote:

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

On a several desktops and two small hosts and several VMs with limited size (20GB) root volumes, I had to disable/remove snapper and manually delete all the snapper created sub volumes. Without warning, the root volumes became un- writeable with write operations from various services and applications reporting no available space. I understand why df/du and monitoring tools are unable to correctly report free space in the same way that we are used to with more traditional non COW filesystems; but unless we educate users/admins about "btrfs filesystem df" or improve the various traditional reporting tools I bet the problem is going to bite more people. I didn't investigate the problem further to determine whether it was a lack of space for meta information/extents or whether the sheer number of subvolumes (even with COW) was taking up the space. The problem generally manifested itself after a medium period of about 3-4 months use, but on one desktop and one VM which were heavily used for testing packages, the problem appeared within a few weeks. Not good at all... I'm concerned that smallish sized root volumes may be quite a common choice for VM's and desktops and that this problem might appear more often as adoption increases. The second issue I ran into was a while ago and I apologise for no bug report on this one (I had no clue if btrfs was even officially supported in yast at the time). I can't remember the specifics but I was manually creating a btrfs filesystem with either multiple physical disks or md devices. The system partitioner (yast) was unable to recognise the filesystem; but what's worse is that I was able to add a btrfs filesystem via yast to the same physical or md devices that I had already manually created an fs on. In general, btrfs support in yast appears fairly limited and I would imagine I'd get really pissed off in future if the only supported brtfs use was via yast and that didn't allow for what I imagine will be fairly popular btrfs usage (multiple physical devices for large storage system). So I guess what I'm saying is I'd love to see some more love for btrfs in YAST with better support for btrfs options ;) Cheers the noo, Graham -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, 4 Jul 2013 17:54, Jeff Mahoney wrote: [snip]

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

Using OSS 12.3 (out of the box) it was not possible to use btrfs for /boot, most boot-loaders disliked that something firce. Has that changed? What's the prognosis? (For lilo, grub-legacy, grub2, gummi-boot, syslinux, ...) Otherwise, I'd like to say fine so far, but I did not have any disaster so far, too. The snapper tools could use some love, esp. the filters: excluding files / dirs would be very nice, expanding the man-page with some more examples (or adding a man snapper-config) Thanks so far, for a desktop using btrfs for /home with use of snapper has helped to reduce some panic-attacks. On performance, we could not 'feel' any difference between ext4 and btrfs, the difference between ssd and hdd is much easier to feel. The missing implemention of a reserved space is much more critical. (create a small btrfs partition, fill it, then try to delete some, that is NOT working, you have to expand the partition and the fs first) - Yamaban. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thursday 2013-07-04 17:54, Jeff Mahoney wrote:

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

Using `rsync --inplace` to update files that have shared extents (like, in snapshots) leads to what seems to be terrible fragmentation — so much that said rsync command progresses at considerably reduced speed, noticable when updating "big" files where there is a chance to look at the rsync -P progress. I know it is a tradeoff, and I am not sure there is actually any fits-it-all solution. I know I could drop the --inplace argument, at the cost of spending more space. The bigger the file and the lesser the amount of data changed per snapshot, the costlier this will be. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thursday 04 Jul 2013 11:54:05 Jeff Mahoney wrote:

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

When I tested a variety of KVM image formats and settings to see what worked well with regards to disk read/write speed; I found that any disk image stored on a btrfs volume made writing to the the VM filesystem (ext4) too slow to be usable for any kind of real work load. I didn't test using btrfs volumes inside the VM's themselves. Cheers the noo, Graham -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Jul 4, 2013 at 10:54 AM, Jeff Mahoney wrote:

Hi all -

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

I'm using openSUSE 12.3 with and without more recent kernels. On one 4-disk btrfs filesystem, I ran into filesystem corruption that - even though the upstream btrfs folks have been working on it - is not yet resolved. I was (and am) able to mount the filesystem with "-o ro,recovery" but that's it. I use btrfs on 3 out of 5 computers running openSUSE. I've had some really amazing positive experiences with it. In one case, I had a 4-disk btrfs filesystem where I had accidentally bumped the sata cable, just enough that it would kinda-sorta work but threw errors left and right. It accrued over 90 thousand errors before I noticed. After reseating the cable and running a "scrub", all 90 thousand errors were repaired. On my (only) SSD, it's been pretty solid. Unfortunately, all 3.8, 3.9 kernels lock up almost instantly when a file is "defrag"d. A combination of two patches from upstream appears to resolve that for me, but those patches are not in the stock openSUSE or vanilla kernels. Overall, it's been positive with some performance (fragmentation) issues. I filed a number of bugs in the openSUSE bug tracker, some of which have gone unacknowledged and all of which remain open, unfortunately. Please do not take that as an indictment of the huge quantity of work, effort, and expertise that a great many persons put into openSUSE - I recommend no other distribution!

Lastly, I'd like to ask that you take the opportunity to test with the latest openSUSE Factory kernel running 3.10-final. 12.3 used the 3.7 kernel since it was the most recent release going into the beta cycle and we've seen a bunch of btrfs fixes since that release.

I'll give it a try as soon as it's available. Are you talking about the OBS project with title: "Kernel builds for branch stable (standard)" ?

I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases.

13.1 is looking very nice! -- Jon -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On 07/04/2013 08:54 AM, Jeff Mahoney wrote:

I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases.

Hi Jeff, I experimented with btrfs just a few days ago and bumped into a fatal problem. Our application calls for writing data as fast as possible to a hardware RAID-6 array. The data comes from as many as four Gig-E Ethernet tuned to run at 950-Mb/sec steady-state. So we need to record as much as 475-MBytes/sec. We've been doing this for years with two Ethernet channels and we wanted to test to see if the boxes would handle four. We've been using XFS without issue in the past. So I built a RAID-6 array with eleven 2-TB Seagate Constellation disks. An additional disk is used as a hot-spare. The array grosses out at 17162760M with XFS as reported by "df -BM". I then write 4000 4-GB files and use the wall-clock time to calculate MB/sec. The files are all identical, but contain random binary data. This all on a fresh install of openSuSE 12.3 with a subsequent zypper -dup. The hardware has two Xeon E5-2643 cpus running at 3.3GHz. The box has 64-GB of ram with a LSI MegaRAID SAS 9271-8i raid controller in a 24-bay chassis. System disks are two Intel SSD's configured as RAID-1. The XFS array writes at 973-MB/sec and df -h shows: /dev/sdc1 17T 16T 761G 96% /export/data1 I tried reiserfs but it wouldn't build on the 17-TB array, it supports only up to 16-TB I guess. I then formatted a second identical array using yast2 with btrfs. But the write test failed about 55% of the way through. The system was still operable, but the writing just stopped. /var/log/messages had kernel problems with btrfs and generated a trace. I've included a sample below. The hostname is "beef". A second attempt stopped in approximately the same place. The write rates for btrfs seemed to be about twice as fast as XFS, but was irregular. I then repeated the process with ext4 and got 992-MB/sec, but the filesystem filled up. It looks like ext4 uses about 800-GB more space for its overhead than XFS, for a 17-TB array. So for now, we'll stick with XFS since we have enough bandwidth, but can use the additional storage space. Regards, Lew beef's /var/log/message output showing btrfs crash: 2013-07-02T16:28:20.899462+00:00 beef kernel: [75249.184041] use_block_rsv: 10505 callbacks suppressed 2013-07-02T16:28:20.899478+00:00 beef kernel: [75249.184056] btrfs: block rsv returned -28 2013-07-02T16:28:20.899479+00:00 beef kernel: [75249.184057] ------------[ cut here ]------------ 2013-07-02T16:28:20.899480+00:00 beef kernel: [75249.184073] WARNING: at /home/abuild/rpmbuild/BUILD/kernel-desktop-3.7.10/linux-3.7/fs/btrfs/extent-tree.c:6297 btrfs_alloc_free_block+0x3b1/0x3c0 [btrfs]() 2013-07-02T16:28:20.899481+00:00 beef kernel: [75249.184074] Hardware name: X9DRH-7TF/7F/iTF/iF 2013-07-02T16:28:20.899482+00:00 beef kernel: [75249.184075] Modules linked in: mpt2sas raid_class mptctl mptbase btrfs zlib_deflate libcrc32c st sr_mod cdrom dm_mod minix hfs vfat fat af_packet xfs mperf joydev coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul iTCO_wdt gpio_ich iTCO_vendor_support microcode pcspkr sb_edac edac_core i2c_i801 lpc_ich mfd_core mei igb ses enclosure ptp pps_core sg ioatdma dca container button autofs4 reiserfs mgag200 ttm drm_kms_helper drm isci i2c_algo_bit sysimgblt sysfillrect syscopyarea libsas scsi_transport_sas processor thermal_sys scsi_dh_alua scsi_dh_hp_sw scsi_dh_emc scsi_dh_rdac scsi_dh megaraid_sas 2013-07-02T16:28:20.899483+00:00 beef kernel: [75249.184113] Pid: 28610, comm: rm Tainted: G W 3.7.10-1.16-desktop #1 2013-07-02T16:28:20.899483+00:00 beef kernel: [75249.184114] Call Trace: 2013-07-02T16:28:20.899484+00:00 beef kernel: [75249.184124] [<ffffffff81004818>] dump_trace+0x88/0x300 2013-07-02T16:28:20.899485+00:00 beef kernel: [75249.184129] [<ffffffff8158af33>] dump_stack+0x69/0x6f 2013-07-02T16:28:20.899486+00:00 beef kernel: [75249.184134] [<ffffffff81045249>] warn_slowpath_common+0x79/0xc0 2013-07-02T16:28:20.899486+00:00 beef kernel: [75249.184141] [<ffffffffa05d13b1>] btrfs_alloc_free_block+0x3b1/0x3c0 [btrfs] 2013-07-02T16:28:20.899487+00:00 beef kernel: [75249.184172] [<ffffffffa05bd3d7>] __btrfs_cow_block+0x137/0x570 [btrfs] 2013-07-02T16:28:20.899488+00:00 beef kernel: [75249.184183] [<ffffffffa05bd98f>] btrfs_cow_block+0xff/0x250 [btrfs] 2013-07-02T16:28:20.899488+00:00 beef kernel: [75249.184195] [<ffffffffa05c1f99>] btrfs_search_slot+0x429/0x990 [btrfs] 2013-07-02T16:28:20.899489+00:00 beef kernel: [75249.184209] [<ffffffffa05d7ab5>] btrfs_del_csums+0x125/0x300 [btrfs] 2013-07-02T16:28:20.899490+00:00 beef kernel: [75249.184230] [<ffffffffa05caedf>] __btrfs_free_extent+0x60f/0x890 [btrfs] 2013-07-02T16:28:20.899490+00:00 beef kernel: [75249.184246] [<ffffffffa05cf852>] run_clustered_refs+0x322/0xac0 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184264] [<ffffffffa05d2d5a>] btrfs_run_delayed_refs+0xca/0x320 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184293] [<ffffffffa05e3343>] __btrfs_end_transaction+0x123/0x420 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184321] [<ffffffffa05edd14>] btrfs_evict_inode+0x334/0x380 [btrfs] 2013-07-02T16:28:20.899492+00:00 beef kernel: [75249.184361] [<ffffffff811887c7>] evict+0xa7/0x1a0 2013-07-02T16:28:20.899492+00:00 beef kernel: [75249.184373] [<ffffffff8117e37d>] do_unlinkat+0x12d/0x1d0 2013-07-02T16:28:20.899498+00:00 beef kernel: [75249.184385] [<ffffffff8159eaad>] system_call_fastpath+0x1a/0x1f 2013-07-02T16:28:20.899499+00:00 beef kernel: [75249.184390] [<00007f21a89400ed>] 0x7f21a89400ec 2013-07-02T16:28:20.899500+00:00 beef kernel: [75249.184391] ---[ end trace c2679e952b898a26 ]--- 2013-07-02T16:28:20.900367+00:00 beef kernel: [75249.184578] btrfs: block rsv returned -28 2013-07-02T16:28:20.900373+00:00 beef kernel: [75249.184579] ------------[ cut here ]------------ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

On Thu, Jul 4, 2013 at 5:54 PM, Jeff Mahoney wrote:

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

I ran it exclusively for about 6 months with the openSUSE 12.1 kernel (3.1) and then a self compiled one going through 3.2 rcs, 3.3 rcs and 3.4 rcs. I had performance issues with it, reported them upstream but didn't get far (after a reply from Josef Bacik the thread died). You can see the upstream thread which includes lots of details at http://thread.gmane.org/gmane.comp.file-systems.btrfs/17638 Note that the thread above talks about slow mounts but it just generally got slow as time passed. I reported the slow mount particularly because it seemed like a discrete operation that was easy to time and diagnose separately from everything else, the plan was to get that fixed and then look into the runtime performance issues. At the time I had the feeling that what was different between me and other people that were running btrfs day to day was that I was using snapper (therefore taking lots of snapshots automatically, hourly I think) and lzo compression (which was quite new at the time). The runtime performance issues seemed like huge lantency for IO operations which ended up being unbearable. I remember a git rebase -i (granted on the big Libreoffice repo) taking something on the order of a minute just to come up with the list of my patches that needed to be rebased on top of origin/master, while the disk was spinning all the time. In general, there were lots of times when windows seemed unresponsive, Firefox would freeze etc. all while the disk was continuously spinning. Some seconds later (not uncommonly 10, 20, 30 seconds later) everything would respond again after disk activity stopped. I'm quite sure that it got progressively worse (which made me suspect snapper), it definitely wasn't unbearable the first day after installation, and it definitely got unbearable the day I decided to format and switch to ext4. I say I'm sure it got progressively slower because after switching to ext4 it literally seemed like I got a new computer which was orders of magnitude faster. I'm sure I bared the slowness only because the water the frog was in got gradually warmer and so it took a while to realize how bad it had gotten. So I'm a bit worried that after months of use performance degrades significantly. Do you know of somebody that ran with lots of snapshots and lzo for some months with no issues? Hope this helps -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi all, My laptop is running on a btrfs root filesystem every since oS 12.2. I have had two issues in the beginning, but now it is running perfectly for more than 6 months already. I use the latest kernel though (i.e. now 3.10). Since oS 12.3 I do not use a separate boot partition anymore, without any issues. Kind regards, Erwin -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, in my company laptop (Lenovo X220) I installed openSUSE 12.3 RC2 with btrfs then I updated to 12.3. I had an encrypted partition. The performance was very good at the beginning. But it became worse and worse over time due to my hardisk being busy for no reason. At some point I began to experience time slots up to one minute in which I my laptop got frozen and came back to life again like nothing happened. I thought it was due to Akonadi. So I deactivated it but problems remained. Ludwig Nussel checked it and determined that the source of the problem most likely was btrFS. I reinstalled my machine some days ago with ext4 and full hard disk encrypted and so far the laptop goes fine. Nothing different from BtrFS behavior at this point in time for me though. Please remember that I am not an engineer and my knowledge about the system is very limited. On Thursday 04 July 2013 11:54:05 Jeff Mahoney wrote:

Hi all -

Over the past year, btrfs has received substantial attention in the areas of both stability and performance. I've been running it, personally, on a 4 TB data volume as well as my root partitions for my openSUSE 12.2/12.3 systems. I've been able to do stress tests like a long fsmark run simultaneously with creating and removing snapshots (with syncs to force them to disk) and haven't run into any issues. David Sterba, who's been leading the charge at SUSE for btrfs stability has likewise regularly run it through an array of torture tests and has found that problems are occurring a whole lot less frequently than they have in the past. The upstream btrfs community in which we participate has also started focusing less on feature development and more on stability and performance.

But these are only anecdotal data points from two file system guys and if my experiences working for a major operating systems vendor have taught me anything, it's that our users can be a lot more creative at finding ways to break things than we are. ;)

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

Lastly, I'd like to ask that you take the opportunity to test with the latest openSUSE Factory kernel running 3.10-final. 12.3 used the 3.7 kernel since it was the most recent release going into the beta cycle and we've seen a bunch of btrfs fixes since that release.

I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases.

Thanks!

-Jeff -- Agustin Benito Bethencourt openSUSE Team Lead at SUSE abebe@suse.com

Le jeudi 04 juillet 2013 à 11:54 -0400, Jeff Mahoney a écrit :

Hi all -

Over the past year, btrfs has received substantial attention in the areas of both stability and performance. I've been running it, personally, on a 4 TB data volume as well as my root partitions for my openSUSE 12.2/12.3 systems. I've been able to do stress tests like a long fsmark run simultaneously with creating and removing snapshots (with syncs to force them to disk) and haven't run into any issues. David Sterba, who's been leading the charge at SUSE for btrfs stability has likewise regularly run it through an array of torture tests and has found that problems are occurring a whole lot less frequently than they have in the past. The upstream btrfs community in which we participate has also started focusing less on feature development and more on stability and performance.

But these are only anecdotal data points from two file system guys and if my experiences working for a major operating systems vendor have taught me anything, it's that our users can be a lot more creative at finding ways to break things than we are. ;)

So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?

Lastly, I'd like to ask that you take the opportunity to test with the latest openSUSE Factory kernel running 3.10-final. 12.3 used the 3.7 kernel since it was the most recent release going into the beta cycle and we've seen a bunch of btrfs fixes since that release.

I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases.

Some feedback on my btrfs usage : - as I mentioned in another reply, running KVM hosting VM on btrfs is consuming a lot of IO cycle and we should probably disable CoW for such images, by default (I already opened a FATE for that) - I have btrfs running on my main laptop, using two separate partitions : * / is using btrfs since last February (first 12.2, then 12.3 and then Factory since this week). It has been working nicely, except a big crash (on 12.3 caused by Nouveau) which stopped my system to boot at all (btrfs couldn't replay its log). I had to zero btrfs log to be able to recover my system (of course, it was just as I was starting a talk in front of an audience, so I didn't had time to test mount with recovery and nor to take a copy of the FS). We need to improve btrfs reliance in this sort of event (maybe automatically mount with recover when mounting / doesn't work, etc..) * /home over LUKS, since several weeks. No issue found so far. - I'm also using btrfs for / (not /home) at my home, no issue found. - I agree with Matthias on our default snapper configuration which can easily fill a system without user knowledge and causing strange error like: file system full, removing file refused with "no space on device" because snapshot should be removed, not files. Other than that, I'm quite happy with btrfs. -- Frederic Crozat SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org