On 07/04/2013 08:54 AM, Jeff Mahoney wrote:
I look forward to your feedback and the opportunity to improve btrfs for the 13.1 release and future releases.
Hi Jeff, I experimented with btrfs just a few days ago and bumped into a fatal problem. Our application calls for writing data as fast as possible to a hardware RAID-6 array. The data comes from as many as four Gig-E Ethernet tuned to run at 950-Mb/sec steady-state. So we need to record as much as 475-MBytes/sec. We've been doing this for years with two Ethernet channels and we wanted to test to see if the boxes would handle four. We've been using XFS without issue in the past. So I built a RAID-6 array with eleven 2-TB Seagate Constellation disks. An additional disk is used as a hot-spare. The array grosses out at 17162760M with XFS as reported by "df -BM". I then write 4000 4-GB files and use the wall-clock time to calculate MB/sec. The files are all identical, but contain random binary data. This all on a fresh install of openSuSE 12.3 with a subsequent zypper -dup. The hardware has two Xeon E5-2643 cpus running at 3.3GHz. The box has 64-GB of ram with a LSI MegaRAID SAS 9271-8i raid controller in a 24-bay chassis. System disks are two Intel SSD's configured as RAID-1. The XFS array writes at 973-MB/sec and df -h shows: /dev/sdc1 17T 16T 761G 96% /export/data1 I tried reiserfs but it wouldn't build on the 17-TB array, it supports only up to 16-TB I guess. I then formatted a second identical array using yast2 with btrfs. But the write test failed about 55% of the way through. The system was still operable, but the writing just stopped. /var/log/messages had kernel problems with btrfs and generated a trace. I've included a sample below. The hostname is "beef". A second attempt stopped in approximately the same place. The write rates for btrfs seemed to be about twice as fast as XFS, but was irregular. I then repeated the process with ext4 and got 992-MB/sec, but the filesystem filled up. It looks like ext4 uses about 800-GB more space for its overhead than XFS, for a 17-TB array. So for now, we'll stick with XFS since we have enough bandwidth, but can use the additional storage space. Regards, Lew beef's /var/log/message output showing btrfs crash: 2013-07-02T16:28:20.899462+00:00 beef kernel: [75249.184041] use_block_rsv: 10505 callbacks suppressed 2013-07-02T16:28:20.899478+00:00 beef kernel: [75249.184056] btrfs: block rsv returned -28 2013-07-02T16:28:20.899479+00:00 beef kernel: [75249.184057] ------------[ cut here ]------------ 2013-07-02T16:28:20.899480+00:00 beef kernel: [75249.184073] WARNING: at /home/abuild/rpmbuild/BUILD/kernel-desktop-3.7.10/linux-3.7/fs/btrfs/extent-tree.c:6297 btrfs_alloc_free_block+0x3b1/0x3c0 [btrfs]() 2013-07-02T16:28:20.899481+00:00 beef kernel: [75249.184074] Hardware name: X9DRH-7TF/7F/iTF/iF 2013-07-02T16:28:20.899482+00:00 beef kernel: [75249.184075] Modules linked in: mpt2sas raid_class mptctl mptbase btrfs zlib_deflate libcrc32c st sr_mod cdrom dm_mod minix hfs vfat fat af_packet xfs mperf joydev coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul iTCO_wdt gpio_ich iTCO_vendor_support microcode pcspkr sb_edac edac_core i2c_i801 lpc_ich mfd_core mei igb ses enclosure ptp pps_core sg ioatdma dca container button autofs4 reiserfs mgag200 ttm drm_kms_helper drm isci i2c_algo_bit sysimgblt sysfillrect syscopyarea libsas scsi_transport_sas processor thermal_sys scsi_dh_alua scsi_dh_hp_sw scsi_dh_emc scsi_dh_rdac scsi_dh megaraid_sas 2013-07-02T16:28:20.899483+00:00 beef kernel: [75249.184113] Pid: 28610, comm: rm Tainted: G W 3.7.10-1.16-desktop #1 2013-07-02T16:28:20.899483+00:00 beef kernel: [75249.184114] Call Trace: 2013-07-02T16:28:20.899484+00:00 beef kernel: [75249.184124] [<ffffffff81004818>] dump_trace+0x88/0x300 2013-07-02T16:28:20.899485+00:00 beef kernel: [75249.184129] [<ffffffff8158af33>] dump_stack+0x69/0x6f 2013-07-02T16:28:20.899486+00:00 beef kernel: [75249.184134] [<ffffffff81045249>] warn_slowpath_common+0x79/0xc0 2013-07-02T16:28:20.899486+00:00 beef kernel: [75249.184141] [<ffffffffa05d13b1>] btrfs_alloc_free_block+0x3b1/0x3c0 [btrfs] 2013-07-02T16:28:20.899487+00:00 beef kernel: [75249.184172] [<ffffffffa05bd3d7>] __btrfs_cow_block+0x137/0x570 [btrfs] 2013-07-02T16:28:20.899488+00:00 beef kernel: [75249.184183] [<ffffffffa05bd98f>] btrfs_cow_block+0xff/0x250 [btrfs] 2013-07-02T16:28:20.899488+00:00 beef kernel: [75249.184195] [<ffffffffa05c1f99>] btrfs_search_slot+0x429/0x990 [btrfs] 2013-07-02T16:28:20.899489+00:00 beef kernel: [75249.184209] [<ffffffffa05d7ab5>] btrfs_del_csums+0x125/0x300 [btrfs] 2013-07-02T16:28:20.899490+00:00 beef kernel: [75249.184230] [<ffffffffa05caedf>] __btrfs_free_extent+0x60f/0x890 [btrfs] 2013-07-02T16:28:20.899490+00:00 beef kernel: [75249.184246] [<ffffffffa05cf852>] run_clustered_refs+0x322/0xac0 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184264] [<ffffffffa05d2d5a>] btrfs_run_delayed_refs+0xca/0x320 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184293] [<ffffffffa05e3343>] __btrfs_end_transaction+0x123/0x420 [btrfs] 2013-07-02T16:28:20.899491+00:00 beef kernel: [75249.184321] [<ffffffffa05edd14>] btrfs_evict_inode+0x334/0x380 [btrfs] 2013-07-02T16:28:20.899492+00:00 beef kernel: [75249.184361] [<ffffffff811887c7>] evict+0xa7/0x1a0 2013-07-02T16:28:20.899492+00:00 beef kernel: [75249.184373] [<ffffffff8117e37d>] do_unlinkat+0x12d/0x1d0 2013-07-02T16:28:20.899498+00:00 beef kernel: [75249.184385] [<ffffffff8159eaad>] system_call_fastpath+0x1a/0x1f 2013-07-02T16:28:20.899499+00:00 beef kernel: [75249.184390] [<00007f21a89400ed>] 0x7f21a89400ec 2013-07-02T16:28:20.899500+00:00 beef kernel: [75249.184391] ---[ end trace c2679e952b898a26 ]--- 2013-07-02T16:28:20.900367+00:00 beef kernel: [75249.184578] btrfs: block rsv returned -28 2013-07-02T16:28:20.900373+00:00 beef kernel: [75249.184579] ------------[ cut here ]------------ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org