[opensuse-factory] BtrFS Usage/Complaints
Hi all - Over the past year, btrfs has received substantial attention in the areas of both stability and performance. I've been running it, personally, on a 4 TB data volume as well as my root partitions for my openSUSE 12.2/12.3 systems. I've been able to do stress tests like a long fsmark run simultaneously with creating and removing snapshots (with syncs to force them to disk) and haven't run into any issues. David Sterba, who's been leading the charge at SUSE for btrfs stability has likewise regularly run it through an array of torture tests and has found that problems are occurring a whole lot less frequently than they have in the past. The upstream btrfs community in which we participate has also started focusing less on feature development and more on stability and performance. But these are only anecdotal data points from two file system guys and if my experiences working for a major operating systems vendor have taught me anything, it's that our users can be a lot more creative at finding ways to break things than we are. ;) So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs? Lastly, I'd like to ask that you take the opportunity to test with the latest openSUSE Factory kernel running 3.10-final. 12.3 used the 3.7 kernel since it was the most recent release going into the beta cycle and we've seen a bunch of btrfs fixes since that release. I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases. Thanks! -Jeff -- Jeff Mahoney -- Jeff Mahoney SUSE Labs
On Thu, Jul 4, 2013 at 12:54 PM, Jeff Mahoney <jeffm@suse.com> wrote:
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
Last I heard, btrfs had no provision for reserved space. This means, once a btrfs partition becomes full, it may become rather difficult to fix[0]: deleting can't succeed if there really isn't any space left. That really needs fixing. [0] https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_o... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 01:56:38PM -0300, Claudio Freire wrote:
There's some space left for deletion purposes while the filesystem does not accept any new data. Bugs that prevented deletion on a full filesytem have been fixed.
That really needs fixing.
[0] https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_o...
This lists all the workarounds that accumulated over time, wiki is unfortunatelly not up to date everywhere. david -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 04 Jul 2013 11:54:05 Jeff Mahoney wrote:
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
On a several desktops and two small hosts and several VMs with limited size (20GB) root volumes, I had to disable/remove snapper and manually delete all the snapper created sub volumes. Without warning, the root volumes became un- writeable with write operations from various services and applications reporting no available space. I understand why df/du and monitoring tools are unable to correctly report free space in the same way that we are used to with more traditional non COW filesystems; but unless we educate users/admins about "btrfs filesystem df" or improve the various traditional reporting tools I bet the problem is going to bite more people. I didn't investigate the problem further to determine whether it was a lack of space for meta information/extents or whether the sheer number of subvolumes (even with COW) was taking up the space. The problem generally manifested itself after a medium period of about 3-4 months use, but on one desktop and one VM which were heavily used for testing packages, the problem appeared within a few weeks. Not good at all... I'm concerned that smallish sized root volumes may be quite a common choice for VM's and desktops and that this problem might appear more often as adoption increases. The second issue I ran into was a while ago and I apologise for no bug report on this one (I had no clue if btrfs was even officially supported in yast at the time). I can't remember the specifics but I was manually creating a btrfs filesystem with either multiple physical disks or md devices. The system partitioner (yast) was unable to recognise the filesystem; but what's worse is that I was able to add a btrfs filesystem via yast to the same physical or md devices that I had already manually created an fs on. In general, btrfs support in yast appears fairly limited and I would imagine I'd get really pissed off in future if the only supported brtfs use was via yast and that didn't allow for what I imagine will be fairly popular btrfs usage (multiple physical devices for large storage system). So I guess what I'm saying is I'd love to see some more love for btrfs in YAST with better support for btrfs options ;) Cheers the noo, Graham -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hello Graham and all, On 2013-07-04 T 18:03 +0100 Graham Anderson wrote:
Admittedly, this is the concern I hear most often, and it is (at least to a certain degree) based on too "snapshot friendly" settings in the Snapper configuration for "/" (root). Or to put it differently: Snapper should delete older snapshots much more frequently and aggressivly. I would propose a Snapper update to enforce this also on 12.2 and 12.3. Something like this (pseudo code): if ( Snapper-config-for-"root" has default values ) { Change to more aggressive values } else { Do not touch the configuration, but issue a warning that the administrator should change this manually } What do you think? [...]
That sounds weird. If you run into it again, please create a bugreport.
Everything needs a start -- so did btrfs support in the YaST partitioner. I see a request for RAID 1 support in openFATE: https://features.opensuse.org/313130 and one about improved subvolume handling: https://features.opensuse.org/314606 Going forward: feel free to add your requests to the openFATE database, ... So long - MgE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 06:03:51PM +0100, Graham Anderson wrote:
Although I've got used to reading the current terse 'btrfs fi df/show' output, this is insufficient and buggers everyone. I'm going to attempt to merge the pending 'df' patches again. david -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 4 Jul 2013 17:54, Jeff Mahoney <jeffm@...> wrote: [snip]
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
Using OSS 12.3 (out of the box) it was not possible to use btrfs for /boot, most boot-loaders disliked that something firce. Has that changed? What's the prognosis? (For lilo, grub-legacy, grub2, gummi-boot, syslinux, ...) Otherwise, I'd like to say fine so far, but I did not have any disaster so far, too. The snapper tools could use some love, esp. the filters: excluding files / dirs would be very nice, expanding the man-page with some more examples (or adding a man snapper-config) Thanks so far, for a desktop using btrfs for /home with use of snapper has helped to reduce some panic-attacks. On performance, we could not 'feel' any difference between ext4 and btrfs, the difference between ssd and hdd is much easier to feel. The missing implemention of a reserved space is much more critical. (create a small btrfs partition, fill it, then try to delete some, that is NOT working, you have to expand the partition and the fs first) - Yamaban. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2013-07-04 19:21, Yamaban wrote:
You need a boot loader that understands the btrfs layout. IIRC, that's the behemoth called grub2. But then again, I have arranged myself with having /boot on some "simple" filesystem, also because the delayed writeout, or internal rebalancing of some filesystems (and maybe even manual defragmentation!) means that the location of stage2 might change, which is totally uncool for bootloaders that grab stage2 solely by LBA numbers. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
В Thu, 4 Jul 2013 19:31:29 +0200 (CEST) Jan Engelhardt <jengelh@inai.de> пишет:
btrfs provides space for bootloader and grub2 will by embed itself in this space. So it is affected by file layout change. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 07:21:28PM +0200, Yamaban wrote:
The snapper tools could use some love, esp. the filters: excluding files / dirs would be very nice, expanding the
If you or someone else provides input for further filters I can add them to snapper. It would of course be ideally if each package provides its own filter file. Regards, Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, 6 Jul 2013 15:31, Arvin Schnell <aschnell@...> wrote:
Thanks for responding. What I'm missing most is the docu for the filter files. No word or example in the manpage on what and how they work. - Will the matched files / dirs included or excluded? - How do I exclude a dir and it's contends from snapshoted (e.g. $HOME/.cache, /var/cache)? - How do I include a file(s) / dir(s) into being snapshoted for sure? That is the difficulty atm. NO docu for filters. E.G. a simple sentence in the manpage that the matched files/dirs from the filters will be excluded form the snapshots would be very helpfull. Otherwise thanks so far. - Yamaban. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, Jul 06, 2013 at 03:53:38PM +0200, Yamaban wrote:
Yes, I'll add it to the manpage. It simply uses fnmatch (glibc function) and all files that do match a line in any filter file are ignored. One sidenote: The filters don't affect what files are included in the snapshots on the filesystem level. The filters only make snapper ignore those files, e.g. when listing files with the status command. If you don't want certain directories to be snapshotted (e.g. to save disk space) you have to create btrfs subvolumes. Regards, Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2013-07-04 17:54, Jeff Mahoney wrote:
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
Using `rsync --inplace` to update files that have shared extents (like, in snapshots) leads to what seems to be terrible fragmentation — so much that said rsync command progresses at considerably reduced speed, noticable when updating "big" files where there is a chance to look at the rsync -P progress. I know it is a tradeoff, and I am not sure there is actually any fits-it-all solution. I know I could drop the --inplace argument, at the cost of spending more space. The bigger the file and the lesser the amount of data changed per snapshot, the costlier this will be. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 07:24:33PM +0200, Jan Engelhardt wrote:
I don't think that autodefrag mount option could improve that significantly, though it could have a slightly positive effect in some cases. If a block is being rewritten, the surrounding block up to limit 64k are read and rewritten as well moved to a new location. david -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 04 Jul 2013 11:54:05 Jeff Mahoney wrote:
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
When I tested a variety of KVM image formats and settings to see what worked well with regards to disk read/write speed; I found that any disk image stored on a btrfs volume made writing to the the VM filesystem (ext4) too slow to be usable for any kind of real work load. I didn't test using btrfs volumes inside the VM's themselves. Cheers the noo, Graham -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le jeudi 04 juillet 2013 à 18:26 +0100, Graham Anderson a écrit :
I'm running btrfs on such setup and the way to get decent performance (either when using only btrfs on host on in both host and guest) is to disable CoW on the VM images, using "chattr -C". I'm wondering if we shouldn't do that by default in qemu-img and libvirt.. -- Frederic Crozat <fcrozat@suse.com> SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le samedi 13 juillet 2013 à 14:37 +0400, Andrey Borzenkov a écrit :
I haven't tried.. But from Jeff and David replies to other mails in this thread, it appears we might loose this feature.. -- Frederic Crozat <fcrozat@suse.com> SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 06:26:32PM +0100, Graham Anderson wrote:
VM and database loads do not play well with the COW mechanism. The NOCOW file attribute (chattr +C) fixed the performance problems with VM images for me and reportedly for others. This comes at a price, the checksums are turned off for the file. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 7/8/13 8:14 AM, David Sterba wrote:
For database workloads, that's typically no price. That's removing unnecessary redundancy. The bigger DB workloads just want to use the FS as a way to have names map to blocks on disk and would otherwise be happy to have the FS keep out of the way. They usually don't even want the page cache involved. For VM images, I'd argue it may be more appropriate to have the integrity checking inside the VM itself anyway. ext4 can do this now, at least in the journal, as can btrfs. I can't comment on Windows VMs, of course. -Jeff -- Jeff Mahoney SUSE Labs
On Mon, 8 Jul 2013 14:14:19 +0200 David Sterba <dsterba@suse.cz> wrote:
IIUC, that also removes the ability to instantaneously clone VM images using cp --reflink. I'd consider that a pretty severe cost for a workload where Btrfs normally excels. Cheers, David -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Jul 08, 2013 at 04:17:36PM +0200, David Disseldorp wrote:
That's right and more serious than the csums. As a compromise, the vm images may be a mixed set of cow and nocow files, let's say the rootfs is cow-based and data partitions are nocow. david -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 4, 2013 at 10:54 AM, Jeff Mahoney <jeffm@suse.com> wrote:
I'm using openSUSE 12.3 with and without more recent kernels. On one 4-disk btrfs filesystem, I ran into filesystem corruption that - even though the upstream btrfs folks have been working on it - is not yet resolved. I was (and am) able to mount the filesystem with "-o ro,recovery" but that's it. I use btrfs on 3 out of 5 computers running openSUSE. I've had some really amazing positive experiences with it. In one case, I had a 4-disk btrfs filesystem where I had accidentally bumped the sata cable, just enough that it would kinda-sorta work but threw errors left and right. It accrued over 90 thousand errors before I noticed. After reseating the cable and running a "scrub", all 90 thousand errors were repaired. On my (only) SSD, it's been pretty solid. Unfortunately, all 3.8, 3.9 kernels lock up almost instantly when a file is "defrag"d. A combination of two patches from upstream appears to resolve that for me, but those patches are not in the stock openSUSE or vanilla kernels. Overall, it's been positive with some performance (fragmentation) issues. I filed a number of bugs in the openSUSE bug tracker, some of which have gone unacknowledged and all of which remain open, unfortunately. Please do not take that as an indictment of the huge quantity of work, effort, and expertise that a great many persons put into openSUSE - I recommend no other distribution!
I'll give it a try as soon as it's available. Are you talking about the OBS project with title: "Kernel builds for branch stable (standard)" ?
I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases.
13.1 is looking very nice! -- Jon -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 4, 2013 at 1:40 PM, Jon Nelson <jnelson-suse@jamponi.net> wrote:
Unfortunately (and this appears to have been reported elsewhere) 3.10 runs super hot for me on my Thinkpad T520. Over 152F (normally it's 115F to 120F). -- Jon -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 07/04/2013 08:54 AM, Jeff Mahoney wrote:
I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases.
Hi Jeff, I experimented with btrfs just a few days ago and bumped into a fatal problem. Our application calls for writing data as fast as possible to a hardware RAID-6 array. The data comes from as many as four Gig-E Ethernet tuned to run at 950-Mb/sec steady-state. So we need to record as much as 475-MBytes/sec. We've been doing this for years with two Ethernet channels and we wanted to test to see if the boxes would handle four. We've been using XFS without issue in the past. So I built a RAID-6 array with eleven 2-TB Seagate Constellation disks. An additional disk is used as a hot-spare. The array grosses out at 17162760M with XFS as reported by "df -BM". I then write 4000 4-GB files and use the wall-clock time to calculate MB/sec. The files are all identical, but contain random binary data. This all on a fresh install of openSuSE 12.3 with a subsequent zypper -dup. The hardware has two Xeon E5-2643 cpus running at 3.3GHz. The box has 64-GB of ram with a LSI MegaRAID SAS 9271-8i raid controller in a 24-bay chassis. System disks are two Intel SSD's configured as RAID-1. The XFS array writes at 973-MB/sec and df -h shows: /dev/sdc1 17T 16T 761G 96% /export/data1 I tried reiserfs but it wouldn't build on the 17-TB array, it supports only up to 16-TB I guess. I then formatted a second identical array using yast2 with btrfs. But the write test failed about 55% of the way through. The system was still operable, but the writing just stopped. /var/log/messages had kernel problems with btrfs and generated a trace. I've included a sample below. The hostname is "beef". A second attempt stopped in approximately the same place. The write rates for btrfs seemed to be about twice as fast as XFS, but was irregular. I then repeated the process with ext4 and got 992-MB/sec, but the filesystem filled up. It looks like ext4 uses about 800-GB more space for its overhead than XFS, for a 17-TB array. So for now, we'll stick with XFS since we have enough bandwidth, but can use the additional storage space. Regards, Lew beef's /var/log/message output showing btrfs crash: 2013-07-02T16:28:20.899462+00:00 beef kernel: [75249.184041] use_block_rsv: 10505 callbacks suppressed 2013-07-02T16:28:20.899478+00:00 beef kernel: [75249.184056] btrfs: block rsv returned -28 2013-07-02T16:28:20.899479+00:00 beef kernel: [75249.184057] ------------[ cut here ]------------ 2013-07-02T16:28:20.899480+00:00 beef kernel: [75249.184073] WARNING: at /home/abuild/rpmbuild/BUILD/kernel-desktop-3.7.10/linux-3.7/fs/btrfs/extent-tree.c:6297 btrfs_alloc_free_block+0x3b1/0x3c0 [btrfs]() 2013-07-02T16:28:20.899481+00:00 beef kernel: [75249.184074] Hardware name: X9DRH-7TF/7F/iTF/iF 2013-07-02T16:28:20.899482+00:00 beef kernel: [75249.184075] Modules linked in: mpt2sas raid_class mptctl mptbase btrfs zlib_deflate libcrc32c st sr_mod cdrom dm_mod minix hfs vfat fat af_packet xfs mperf joydev coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul iTCO_wdt gpio_ich iTCO_vendor_support microcode pcspkr sb_edac edac_core i2c_i801 lpc_ich mfd_core mei igb ses enclosure ptp pps_core sg ioatdma dca container button autofs4 reiserfs mgag200 ttm drm_kms_helper drm isci i2c_algo_bit sysimgblt sysfillrect syscopyarea libsas scsi_transport_sas processor thermal_sys scsi_dh_alua scsi_dh_hp_sw scsi_dh_emc scsi_dh_rdac scsi_dh megaraid_sas 2013-07-02T16:28:20.899483+00:00 beef kernel: [75249.184113] Pid: 28610, comm: rm Tainted: G W 3.7.10-1.16-desktop #1 2013-07-02T16:28:20.899483+00:00 beef kernel: [75249.184114] Call Trace: 2013-07-02T16:28:20.899484+00:00 beef kernel: [75249.184124] [<ffffffff81004818>] dump_trace+0x88/0x300 2013-07-02T16:28:20.899485+00:00 beef kernel: [75249.184129] [<ffffffff8158af33>] dump_stack+0x69/0x6f 2013-07-02T16:28:20.899486+00:00 beef kernel: [75249.184134] [<ffffffff81045249>] warn_slowpath_common+0x79/0xc0 2013-07-02T16:28:20.899486+00:00 beef kernel: [75249.184141] [<ffffffffa05d13b1>] btrfs_alloc_free_block+0x3b1/0x3c0 [btrfs] 2013-07-02T16:28:20.899487+00:00 beef kernel: [75249.184172] [<ffffffffa05bd3d7>] __btrfs_cow_block+0x137/0x570 [btrfs] 2013-07-02T16:28:20.899488+00:00 beef kernel: [75249.184183] [<ffffffffa05bd98f>] btrfs_cow_block+0xff/0x250 [btrfs] 2013-07-02T16:28:20.899488+00:00 beef kernel: [75249.184195] [<ffffffffa05c1f99>] btrfs_search_slot+0x429/0x990 [btrfs] 2013-07-02T16:28:20.899489+00:00 beef kernel: [75249.184209] [<ffffffffa05d7ab5>] btrfs_del_csums+0x125/0x300 [btrfs] 2013-07-02T16:28:20.899490+00:00 beef kernel: [75249.184230] [<ffffffffa05caedf>] __btrfs_free_extent+0x60f/0x890 [btrfs] 2013-07-02T16:28:20.899490+00:00 beef kernel: [75249.184246] [<ffffffffa05cf852>] run_clustered_refs+0x322/0xac0 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184264] [<ffffffffa05d2d5a>] btrfs_run_delayed_refs+0xca/0x320 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184293] [<ffffffffa05e3343>] __btrfs_end_transaction+0x123/0x420 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184321] [<ffffffffa05edd14>] btrfs_evict_inode+0x334/0x380 [btrfs] 2013-07-02T16:28:20.899492+00:00 beef kernel: [75249.184361] [<ffffffff811887c7>] evict+0xa7/0x1a0 2013-07-02T16:28:20.899492+00:00 beef kernel: [75249.184373] [<ffffffff8117e37d>] do_unlinkat+0x12d/0x1d0 2013-07-02T16:28:20.899498+00:00 beef kernel: [75249.184385] [<ffffffff8159eaad>] system_call_fastpath+0x1a/0x1f 2013-07-02T16:28:20.899499+00:00 beef kernel: [75249.184390] [<00007f21a89400ed>] 0x7f21a89400ec 2013-07-02T16:28:20.899500+00:00 beef kernel: [75249.184391] ---[ end trace c2679e952b898a26 ]--- 2013-07-02T16:28:20.900367+00:00 beef kernel: [75249.184578] btrfs: block rsv returned -28 2013-07-02T16:28:20.900373+00:00 beef kernel: [75249.184579] ------------[ cut here ]------------ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
El 04/07/13 15:22, Lew Wolfgang escribió:
beef's /var/log/message output showing btrfs crash:
that's a bug, please file it in bugzilla so it does not get lost in the sea of -factory mail. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 03:36:31PM -0400, Cristian Rodríguez wrote:
It's an annoying warning, a known one and harmless. Starting with kernel 3.8 it's bee moved under enospc_debug mount option and I haven't seen it with 3.10 kernels, so hopefully it's gone for good. david -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-----Original Message----- From: Jeff Mahoney <jeffm@suse.com> To: opensuse-factory@opensuse.org Cc: David Sterba <dsterba@suse.cz>, Vojtech Pavlik <vojtech@suse.cz>, Mark Fasheh <mfasheh@suse.de>, Matthias Eckermann <mge@suse.com> Subject: [opensuse-factory] BtrFS Usage/Complaints Date: Thu, 04 Jul 2013 11:54:05 -0400 Hi all - Over the past year, btrfs has received substantial attention in the areas of both stability and performance. I've been running it, personally, on a 4 TB data volume as well as my root partitions for my openSUSE 12.2/12.3 systems. I've been able to do stress tests like a long fsmark run simultaneously with creating and removing snapshots (with syncs to force them to disk) and haven't run into any issues. David Sterba, who's been leading the charge at SUSE for btrfs stability has likewise regularly run it through an array of torture tests and has found that problems are occurring a whole lot less frequently than they have in the past. The upstream btrfs community in which we participate has also started focusing less on feature development and more on stability and performance. But these are only anecdotal data points from two file system guys and if my experiences working for a major operating systems vendor have taught me anything, it's that our users can be a lot more creative at finding ways to break things than we are. ;) So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs? -----Original Message----- Hi Jeff, Been using btrfs for quite some time now (release 12.2) for my data storage. I've noticed that (for a given amount of storage) it has less overhead compared with others, like ext4. Also i noticed, that if you fills it up completely (at least as root), you get a trace back in syslog. Seems informational, cause it just keeps working. What surprised me initially, that when you use an logical volume with ext4, you enlarge the LV and do resizing the filesystem, you specify again the LV. When doing this for a LV filled with btrfs, you start resizing the LV, but for btrfs, you specify the mountpoint. Finally, it scared the hell out of me seeing that for each btrfs-mountpoint, several processes are created: ps wwwaux |grep -v grep|grep btrfs |wc -l 554 for: cat /proc/mounts |grep btrfs -c 36 So, as far as i can see, btrfs works right out of the box. Hans -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 4, 2013 at 5:54 PM, Jeff Mahoney <jeffm@suse.com> wrote:
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
I ran it exclusively for about 6 months with the openSUSE 12.1 kernel (3.1) and then a self compiled one going through 3.2 rcs, 3.3 rcs and 3.4 rcs. I had performance issues with it, reported them upstream but didn't get far (after a reply from Josef Bacik the thread died). You can see the upstream thread which includes lots of details at http://thread.gmane.org/gmane.comp.file-systems.btrfs/17638 Note that the thread above talks about slow mounts but it just generally got slow as time passed. I reported the slow mount particularly because it seemed like a discrete operation that was easy to time and diagnose separately from everything else, the plan was to get that fixed and then look into the runtime performance issues. At the time I had the feeling that what was different between me and other people that were running btrfs day to day was that I was using snapper (therefore taking lots of snapshots automatically, hourly I think) and lzo compression (which was quite new at the time). The runtime performance issues seemed like huge lantency for IO operations which ended up being unbearable. I remember a git rebase -i (granted on the big Libreoffice repo) taking something on the order of a minute just to come up with the list of my patches that needed to be rebased on top of origin/master, while the disk was spinning all the time. In general, there were lots of times when windows seemed unresponsive, Firefox would freeze etc. all while the disk was continuously spinning. Some seconds later (not uncommonly 10, 20, 30 seconds later) everything would respond again after disk activity stopped. I'm quite sure that it got progressively worse (which made me suspect snapper), it definitely wasn't unbearable the first day after installation, and it definitely got unbearable the day I decided to format and switch to ext4. I say I'm sure it got progressively slower because after switching to ext4 it literally seemed like I got a new computer which was orders of magnitude faster. I'm sure I bared the slowness only because the water the frog was in got gradually warmer and so it took a while to realize how bad it had gotten. So I'm a bit worried that after months of use performance degrades significantly. Do you know of somebody that ran with lots of snapshots and lzo for some months with no issues? Hope this helps -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Catalin Iacob wrote:
I got exactly the same experience, both on a Laptop with encrypted root volume as well as in a VM I only used for testing. Those delays and slowness made me really angry about how bad Linux on the Desktop has gotten until I realized it was just btrfs. I'll stay with ext4 now. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi all, My laptop is running on a btrfs root filesystem every since oS 12.2. I have had two issues in the beginning, but now it is running perfectly for more than 6 months already. I use the latest kernel though (i.e. now 3.10). Since oS 12.3 I do not use a separate boot partition anymore, without any issues. Kind regards, Erwin -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hello Erwin and all, On 2013-07-05 T 10:14 +0200 Erwin Van de Velde wrote:
just for my curiousity, two questions: 1. How big is your "/" partition? 2. Are you using Snapper, and if yes, do you use the default settings in "/etc/snapper/configs/root" or did you configure a more aggressive deletion of snapshots? Thanks in advance! so long - MgE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi, in my company laptop (Lenovo X220) I installed openSUSE 12.3 RC2 with btrfs then I updated to 12.3. I had an encrypted partition. The performance was very good at the beginning. But it became worse and worse over time due to my hardisk being busy for no reason. At some point I began to experience time slots up to one minute in which I my laptop got frozen and came back to life again like nothing happened. I thought it was due to Akonadi. So I deactivated it but problems remained. Ludwig Nussel checked it and determined that the source of the problem most likely was btrFS. I reinstalled my machine some days ago with ext4 and full hard disk encrypted and so far the laptop goes fine. Nothing different from BtrFS behavior at this point in time for me though. Please remember that I am not an engineer and my knowledge about the system is very limited. On Thursday 04 July 2013 11:54:05 Jeff Mahoney wrote:
On Fri, Jul 05, 2013 at 10:59:11AM +0200, agustin benito bethencourt wrote:
in my company laptop (Lenovo X220) I installed openSUSE 12.3 RC2 with btrfs then I updated to 12.3. I had an encrypted partition.
SSD or HDD?
Have you tried the autodefrag mount option? Regards, Arvin -- Arvin Schnell, <aschnell@suse.de> Senior Software Engineer, Research & Development SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le samedi 06 juillet 2013 à 15:33 +0200, Arvin Schnell a écrit :
Based on discussion with one of our BtrFS hackers (David Sterba), using autodefrag isn't a great idea, because it causes BtrFS to block write when defragmenting (so, it would make the problem worse). He told me running "btrfs filesystem defragment" is better because it doesn't block IO. -- Frederic Crozat <fcrozat@suse.com> SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Jul 08, 2013 at 10:02:10AM +0200, Frederic Crozat wrote:
Le samedi 06 juillet 2013 à 15:33 +0200, Arvin Schnell a écrit :
On Fri, Jul 05, 2013 at 10:59:11AM +0200, agustin benito bethencourt wrote:
In that case, did they try btrfs filesystem defragment? Regards, Arvin -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le jeudi 04 juillet 2013 à 11:54 -0400, Jeff Mahoney a écrit :
Some feedback on my btrfs usage : - as I mentioned in another reply, running KVM hosting VM on btrfs is consuming a lot of IO cycle and we should probably disable CoW for such images, by default (I already opened a FATE for that) - I have btrfs running on my main laptop, using two separate partitions : * / is using btrfs since last February (first 12.2, then 12.3 and then Factory since this week). It has been working nicely, except a big crash (on 12.3 caused by Nouveau) which stopped my system to boot at all (btrfs couldn't replay its log). I had to zero btrfs log to be able to recover my system (of course, it was just as I was starting a talk in front of an audience, so I didn't had time to test mount with recovery and nor to take a copy of the FS). We need to improve btrfs reliance in this sort of event (maybe automatically mount with recover when mounting / doesn't work, etc..) * /home over LUKS, since several weeks. No issue found so far. - I'm also using btrfs for / (not /home) at my home, no issue found. - I agree with Matthias on our default snapper configuration which can easily fill a system without user knowledge and causing strange error like: file system full, removing file refused with "no space on device" because snapshot should be removed, not files. Other than that, I'm quite happy with btrfs. -- Frederic Crozat <fcrozat@suse.com> SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jul 5, 2013 at 11:37 AM, Frederic Crozat <fcrozat@suse.com> wrote:
That's only a stop-gap measure, this issue should be passed upstream for fixing. You probably have, but just in case, I'm reminding. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le vendredi 05 juillet 2013 à 12:33 -0300, Claudio Freire a écrit :
Of course. I've already opened a FATE to ensure such "manual" rescue would become automated in some way.. (and of course, btrfs kernel module needs to become smarter in handling this, but without a kernel stacktrace, such bugreport is useless..). -- Frederic Crozat <fcrozat@suse.com> SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
I'm using btrfs on data storage with it raid10 native option. Great no partition just bunch of disk. It's stable even with stock 3.7 under openSUSE 12.3 And using it with bunch of disk is really "kind of magic" compare to mdadm+lvm on top. On the annoying chapter : Hard to find my way on which disk are there, etc. How to know if a subvolume is full or not. Sparse documentation to explain the different parts. The second use case is still not available : Have full luks support for / with ssd And it look like it will still take a long time to get it. -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch openSUSE Member GPG KEY : D5C9B751C4653227 irc: tigerfoot
On Thu, Jul 4, 2013 at 12:54 PM, Jeff Mahoney <jeffm@suse.com> wrote:
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
Last I heard, btrfs had no provision for reserved space. This means, once a btrfs partition becomes full, it may become rather difficult to fix[0]: deleting can't succeed if there really isn't any space left. That really needs fixing. [0] https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_o... -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 01:56:38PM -0300, Claudio Freire wrote:
There's some space left for deletion purposes while the filesystem does not accept any new data. Bugs that prevented deletion on a full filesytem have been fixed.
That really needs fixing.
[0] https://btrfs.wiki.kernel.org/index.php/FAQ#Help.21_Btrfs_claims_I.27m_out_o...
This lists all the workarounds that accumulated over time, wiki is unfortunatelly not up to date everywhere. david -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 04 Jul 2013 11:54:05 Jeff Mahoney wrote:
So, I'd like to hear your stories. What's worked for you? What hasn't worked? What would you consider the pain points with using btrfs?
On a several desktops and two small hosts and several VMs with limited size (20GB) root volumes, I had to disable/remove snapper and manually delete all the snapper created sub volumes. Without warning, the root volumes became un- writeable with write operations from various services and applications reporting no available space. I understand why df/du and monitoring tools are unable to correctly report free space in the same way that we are used to with more traditional non COW filesystems; but unless we educate users/admins about "btrfs filesystem df" or improve the various traditional reporting tools I bet the problem is going to bite more people. I didn't investigate the problem further to determine whether it was a lack of space for meta information/extents or whether the sheer number of subvolumes (even with COW) was taking up the space. The problem generally manifested itself after a medium period of about 3-4 months use, but on one desktop and one VM which were heavily used for testing packages, the problem appeared within a few weeks. Not good at all... I'm concerned that smallish sized root volumes may be quite a common choice for VM's and desktops and that this problem might appear more often as adoption increases. The second issue I ran into was a while ago and I apologise for no bug report on this one (I had no clue if btrfs was even officially supported in yast at the time). I can't remember the specifics but I was manually creating a btrfs filesystem with either multiple physical disks or md devices. The system partitioner (yast) was unable to recognise the filesystem; but what's worse is that I was able to add a btrfs filesystem via yast to the same physical or md devices that I had already manually created an fs on. In general, btrfs support in yast appears fairly limited and I would imagine I'd get really pissed off in future if the only supported brtfs use was via yast and that didn't allow for what I imagine will be fairly popular btrfs usage (multiple physical devices for large storage system). So I guess what I'm saying is I'd love to see some more love for btrfs in YAST with better support for btrfs options ;) Cheers the noo, Graham -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hello Graham and all, On 2013-07-04 T 18:03 +0100 Graham Anderson wrote:
Admittedly, this is the concern I hear most often, and it is (at least to a certain degree) based on too "snapshot friendly" settings in the Snapper configuration for "/" (root). Or to put it differently: Snapper should delete older snapshots much more frequently and aggressivly. I would propose a Snapper update to enforce this also on 12.2 and 12.3. Something like this (pseudo code): if ( Snapper-config-for-"root" has default values ) { Change to more aggressive values } else { Do not touch the configuration, but issue a warning that the administrator should change this manually } What do you think? [...]
That sounds weird. If you run into it again, please create a bugreport.
Everything needs a start -- so did btrfs support in the YaST partitioner. I see a request for RAID 1 support in openFATE: https://features.opensuse.org/313130 and one about improved subvolume handling: https://features.opensuse.org/314606 Going forward: feel free to add your requests to the openFATE database, ... So long - MgE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Jul 04, 2013 at 06:03:51PM +0100, Graham Anderson wrote:
Although I've got used to reading the current terse 'btrfs fi df/show' output, this is insufficient and buggers everyone. I'm going to attempt to merge the pending 'df' patches again. david -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (20)
-
agustin benito bethencourt
-
Andrey Borzenkov
-
Arvin Schnell
-
Bruno Friedmann
-
Catalin Iacob
-
Claudio Freire
-
Cristian Rodríguez
-
David Disseldorp
-
David Sterba
-
Erwin Van de Velde
-
Frederic Crozat
-
Graham Anderson
-
Hans Witvliet
-
Jan Engelhardt
-
Jeff Mahoney
-
Jon Nelson
-
Lew Wolfgang
-
Ludwig Nussel
-
Matthias G. Eckermann
-
Yamaban