[opensuse] Reiserfs unstable under load using large LV
Hi, I've a server with 24 500GB SATA 2 hard drives with RAID5 running 10.2. It also has a pair of small disks for the OS with ext3 FS and the large 11TB LV was using ReiserFS. The box was OK, but I found that when it was under heavy load the LV would lock up, where I was unable to read or write to it, the only way to get it back was a hard reset as the partition refused to unmount. My understanding is that the limit for ReiserFS is 17.6TB's so I'm well within the limit. I've switched the filesystem to XFS which has been fine although I generally prefer the performance of ReiserFS. I don't think this is a bug with SUSE as I tried Fedora on the box with exactly the same results. Using XFS isn't causing a problem, but I thought I post this to see what folks have to say on this subject. Cheers Matthew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Wednesday 2007-06-06 at 13:03 +0100, Matthew Stringer wrote:
I've a server with 24 500GB SATA 2 hard drives with RAID5 running 10.2.
It also has a pair of small disks for the OS with ext3 FS and the large 11TB LV was using ReiserFS.
The box was OK, but I found that when it was under heavy load the LV would lock up, where I was unable to read or write to it, the only way to get it back was a hard reset as the partition refused to unmount.
My understanding is that the limit for ReiserFS is 17.6TB's so I'm well within the limit.
I think you may be hitting what people say that reiserfs doesn't scale well. Let me see if I can find a reference... ah, here: http://lists.opensuse.org/opensuse-factory/2006-09/msg00542.html [Jeff Mahoney] ] ReiserFS has serious scalability problems. David Chinner's talk at OLS ] really underscored the problem well for a single, large, high bandwidth ] file system. While I realize that XFS-style scalability isn't a real ] goal for most users, ans isn't a target workload for reiserfs, the ] scalability problems are real. ReiserFS uses the BKL for synchronization ] everywhere, and since it's system-global lock, the problem doesn't go ] away when you split the file system into smaller ones. Lock contention ] alone is one problem, but it's made worse by cache bouncing between ] processors on larger systems. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFGZxS4tTMYHG2NR9URAsxcAJ9LsLjBnMh7RKlb0loxR4zYeWbSwACfVRzZ gbDr6wQ+btuoeYHRuaZHs+Q= =nJfk -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Matthew Stringer wrote:
Hi,
I've a server with 24 500GB SATA 2 hard drives with RAID5 running 10.2.
It also has a pair of small disks for the OS with ext3 FS and the large 11TB LV was using ReiserFS.
The box was OK, but I found that when it was under heavy load the LV would lock up, where I was unable to read or write to it, the only way to get it back was a hard reset as the partition refused to unmount.
My understanding is that the limit for ReiserFS is 17.6TB's so I'm well within the limit.
I've switched the filesystem to XFS which has been fine although I generally prefer the performance of ReiserFS.
I don't think this is a bug with SUSE as I tried Fedora on the box with exactly the same results.
Using XFS isn't causing a problem, but I thought I post this to see what folks have to say on this subject.
Cheers
Matthew
I understand you have to be careful with xfs. When there are large number of files being used, xfs caches a lot of the data. I read somewhere, that xfs under heavy load will lock up sometimes and use all your memory up. -- Joseph Loo jloo@acm.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
|-----Original Message----- |I understand you have to be careful with xfs. When there are |large number of files being used, xfs caches a lot of the |data. I read somewhere, that xfs under heavy load will lock up |sometimes and use all your memory up. | Maybe that was a fact years ago with immature releases pre 2.4.19 versions. That is _NOT_ true now. XFS is probably the most stressed filesystem on the planet. Most Hollywood studio backbones and linux workstations runs xfs. It's only drawback is delete which is slow by design, because it has to traverse the inodes to figure out which blocks to delete and not a direct hashed pointer-table. If you turn off internal logging and set the buffers straight it performs very well. (I've commented on this earlier) The extensive caching done by xfs makes it excellent running under vmware, because it minimize io-ops by writing large chunks. So it is a goodie for low end systems also. I've used the xfs_repair, xfsdump, xfsrestore and xfs_db tools extensively. I've had lots of bad blocks, zeroed inodes and other disk failures. I've almost everytime managed to restore most of the data on the disk. Once I had to use an old indy irix box to mount up the disk (luckily it was a scsi) but that was very early 2.4.x versions. -- MortenB -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Thursday 2007-06-07 at 10:12 +0200, Morten Bjørnsvik wrote:
Maybe that was a fact years ago with immature releases pre 2.4.19 versions. That is _NOT_ true now. XFS is probably the most stressed filesystem on the planet. Most Hollywood studio backbones and linux workstations runs xfs. It's only drawback is delete which is slow by design, because it has to traverse the inodes to figure out which blocks to delete and not a direct hashed pointer-table.
Slow? It is the faster deleting large files, almost instantaneous. So much so that it is recomended for use with mythtv for that very reason.
I've used the xfs_repair, xfsdump, xfsrestore and xfs_db tools extensively. I've had lots of bad blocks, zeroed inodes and other disk failures. I've almost everytime managed to restore most of the data on the disk.
I have an unrepairable xfs partition that crashes xfs_repair every time, with a bug report. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD4DBQFGZ837tTMYHG2NR9URAtgeAJ9PgDZdBsE8FLPyIOZmoiBruosRIQCUCOCS uIH7/+gbRjKcRmB/Vtmk0g== =Djvi -----END PGP SIGNATURE-----
Carlos E. R. wrote:
The Thursday 2007-06-07 at 10:12 +0200, Morten Bjørnsvik wrote:
Maybe that was a fact years ago with immature releases pre 2.4.19 versions. That is _NOT_ true now. XFS is probably the most stressed filesystem on the planet. Most Hollywood studio backbones and linux workstations runs xfs. It's only drawback is delete which is slow by design, because it has to traverse the inodes to figure out which blocks to delete and not a direct hashed pointer-table.
Slow? It is the faster deleting large files, almost instantaneous. So much so that it is recomended for use with mythtv for that very reason.
I've used the xfs_repair, xfsdump, xfsrestore and xfs_db tools extensively. I've had lots of bad blocks, zeroed inodes and other disk failures. I've almost everytime managed to restore most of the data on the disk.
I have an unrepairable xfs partition that crashes xfs_repair every time, with a bug report.
FWIW, I recently tried installing Ubuntu and thought I'd try XFS for a change. I got a warning message that GRUB has problems with XFS, so I went with JFS on the boot partition instead and XFS on /home. -- Use OpenOffice.org http://www.openoffice.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 07 June 2007 13:29, James Knott wrote:
Carlos E. R. wrote:
The Thursday 2007-06-07 at 10:12 +0200, Morten Bjørnsvik wrote:
Maybe that was a fact years ago with immature releases pre 2.4.19
versions.
That is _NOT_ true now. XFS is probably the most stressed filesystem on the planet. Most Hollywood studio backbones and linux
workstations runs xfs.
It's only drawback is delete which is slow by design, because it has to traverse the inodes to figure out which blocks to delete and not a
direct hashed
pointer-table.
Slow? It is the faster deleting large files, almost instantaneous. So much so that it is recomended for use with mythtv for that very reason.
I've used the xfs_repair, xfsdump, xfsrestore and xfs_db tools
extensively.
I've had lots of bad blocks, zeroed inodes and other disk failures.
I've almost
everytime managed to restore most of the data on the disk.
I have an unrepairable xfs partition that crashes xfs_repair every time, with a bug report.
FWIW, I recently tried installing Ubuntu and thought I'd try XFS for a change. I got a warning message that GRUB has problems with XFS, so I went with JFS on the boot partition instead and XFS on /home.
-- Use OpenOffice.org http://www.openoffice.org
Never had any problems with XFS. As people have stated; it cashes alot, so if you are unlucky and get at power out at the wrong moment, the filesystem MAY corrupt. But so far i have been lucky i guess. Even with power outs i havent had any trashed drives (yet) XFS is fast, reliable and scales well, or so i have read. And besides; SGI wouldnt use it if it werent stable... -- /Rikard ----------------------------------------------------------------------------- email : rikard.j@rikjoh.com web : http://www.rikjoh.com mob: : +46 (0)763 19 76 25 ------------------------ Public PGP fingerprint ---------------------------- < 15 28 DF 78 67 98 B2 16 1F D3 FD C5 59 D4 B6 78 46 1C EE 56 >
On Thursday 07 June 2007 12:47, John Andersen wrote:
On Thursday 07 June 2007, Rikard Johnels wrote:
Never had any problems with XFS. As people have stated; it cashes alot,
XFS is fast, reliable and scales well,
How the heck can you put those two statements in the same email???
I think in the first paragraph, second sentence, he meant "caches," not a typo of "crashes." I use XFS on several file systems and have since I installed 9.0 or 9.1 and have had no trouble I can recall (other than a little hitch the first time out when installing, 'cause the XFS file system module was missing from the installer kernel and required manual loading via a diskette, or something like that—it was a long time ago). Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, 2007-06-07 at 13:07 -0700, Randall R Schulz wrote:
On Thursday 07 June 2007 12:47, John Andersen wrote:
On Thursday 07 June 2007, Rikard Johnels wrote:
Never had any problems with XFS. As people have stated; it cashes alot,
XFS is fast, reliable and scales well,
How the heck can you put those two statements in the same email???
I think in the first paragraph, second sentence, he meant "caches," not a typo of "crashes."
No... He meant caches instead of cashes (as in cash a check) Re-check the spelling, I had to look three times. -- Ken Schneider UNIX since 1989, linux since 1994, SuSE since 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 07 June 2007 15:11, Kenneth Schneider wrote:
On Thu, 2007-06-07 at 13:07 -0700, Randall R Schulz wrote:
On Thursday 07 June 2007 12:47, John Andersen wrote:
On Thursday 07 June 2007, Rikard Johnels wrote:
Never had any problems with XFS. As people have stated; it cashes alot,
XFS is fast, reliable and scales well,
How the heck can you put those two statements in the same email???
I think in the first paragraph, second sentence, he meant "caches," not a typo of "crashes."
No... He meant caches instead of cashes (as in cash a check) Re-check the spelling, I had to look three times.
That's what I said. Clearly "cashes" makes no sense, so the two plausible hypotheses are "caches" and "crashes." "Crashes" makes the statements inconsistent, as it would combine "never had any problems" with "crashes a lot." So we conclude, "cashes" is a typo for "caches."
-- Ken Schneider
RRS -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Friday 08 June 2007 00:25, Randall R Schulz wrote:
On Thursday 07 June 2007 15:11, Kenneth Schneider wrote:
On Thu, 2007-06-07 at 13:07 -0700, Randall R Schulz wrote:
On Thursday 07 June 2007 12:47, John Andersen wrote:
On Thursday 07 June 2007, Rikard Johnels wrote:
Never had any problems with XFS. As people have stated; it cashes alot,
XFS is fast, reliable and scales well,
How the heck can you put those two statements in the same email???
I think in the first paragraph, second sentence, he meant "caches," not a typo of "crashes."
No... He meant caches instead of cashes (as in cash a check) Re-check the spelling, I had to look three times.
That's what I said.
Clearly "cashes" makes no sense, so the two plausible hypotheses are "caches" and "crashes." "Crashes" makes the statements inconsistent, as it would combine "never had any problems" with "crashes a lot."
So we conclude, "cashes" is a typo for "caches."
-- Ken Schneider
RRS
Yes. I was referring to "caches." English is not my native language. (Swedish is) And its WAY above my "thinking straight level" temperature limit... So i am sorry if i confused anyone with my typo... -- /Rikard ----------------------------------------------------------------------------- email : rikard.j@rikjoh.com web : http://www.rikjoh.com mob: : +46 (0)763 19 76 25 ------------------------ Public PGP fingerprint ---------------------------- < 15 28 DF 78 67 98 B2 16 1F D3 FD C5 59 D4 B6 78 46 1C EE 56 >
On Friday 08 June 2007 19:49, Rikard Johnels wrote:
Yes. I was referring to "caches." English is not my native language. (Swedish is)
Och vad menar du att det heter på svenska? :) Anyway, I'm using xfs for all my data storage, and I've never seen a problem with it -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Friday 08 June 2007 20:29, Anders Johansson wrote:
On Friday 08 June 2007 19:49, Rikard Johnels wrote:
Yes. I was referring to "caches." English is not my native language. (Swedish is)
Och vad menar du att det heter på svenska? :)
Anyway, I'm using xfs for all my data storage, and I've never seen a problem with it
Vad? "caches." Mellanlagrar? Låter iofs som man talade om ost... :P -- /Rikard ----------------------------------------------------------------------------- email : rikard.j@rikjoh.com web : http://www.rikjoh.com mob: : +46 (0)763 19 76 25 ------------------------ Public PGP fingerprint ---------------------------- < 15 28 DF 78 67 98 B2 16 1F D3 FD C5 59 D4 B6 78 46 1C EE 56 >
On Friday 08 June 2007 21:06, Rikard Johnels wrote:
On Friday 08 June 2007 20:29, Anders Johansson wrote:
On Friday 08 June 2007 19:49, Rikard Johnels wrote:
Yes. I was referring to "caches." English is not my native language. (Swedish is)
Och vad menar du att det heter på svenska? :)
Anyway, I'm using xfs for all my data storage, and I've never seen a problem with it
Vad? "caches." Mellanlagrar? Låter iofs som man talade om ost...
:P
Poängen var att man normalt inte översätter det. Alla jag känner säger bara "cache" även på "svenska" (med väldigt stora citationstecken) Mellanlagring går väl an, antar jag, men jag har aldrig hört det förr när det gäller datorer. Jag har bara hört den termen i fråga om kärnkraftverksbränsle -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Friday 08 June 2007, Anders Johansson wrote:
On Friday 08 June 2007 21:06, Rikard Johnels wrote:
On Friday 08 June 2007 20:29, Anders Johansson wrote:
On Friday 08 June 2007 19:49, Rikard Johnels wrote:
Yes. I was referring to "caches." English is not my native language. (Swedish is)
Och vad menar du att det heter på svenska? :)
Anyway, I'm using xfs for all my data storage, and I've never seen a problem with it
Vad? "caches." Mellanlagrar? Låter iofs som man talade om ost...
:P
Poängen var att man normalt inte översätter det. Alla jag känner säger bara "cache" även på "svenska" (med väldigt stora citationstecken)
Mellanlagring går väl an, antar jag, men jag har aldrig hört det förr när det gäller datorer. Jag har bara hört den termen i fråga om kärnkraftverksbränsle
Le tsallwri tein ot herlan guag esreg ardles softh eeng lishdes ignat ionoft hisl is tjus ttos howt hatwecan. -- _____________________________________ John Andersen -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Saturday 09 June 2007 01:39, John Andersen wrote:
On Friday 08 June 2007, Anders Johansson wrote:
On Friday 08 June 2007 21:06, Rikard Johnels wrote:
On Friday 08 June 2007 20:29, Anders Johansson wrote:
On Friday 08 June 2007 19:49, Rikard Johnels wrote:
Yes. I was referring to "caches." English is not my native language. (Swedish is)
Och vad menar du att det heter på svenska? :)
Anyway, I'm using xfs for all my data storage, and I've never seen a problem with it
Vad? "caches." Mellanlagrar? Låter iofs som man talade om ost...
:P
Poängen var att man normalt inte översätter det. Alla jag känner säger bara "cache" även på "svenska" (med väldigt stora citationstecken)
Mellanlagring går väl an, antar jag, men jag har aldrig hört det förr när det gäller datorer. Jag har bara hört den termen i fråga om kärnkraftverksbränsle
Le tsallwri tein ot herlan guag esreg ardles softh eeng lishdes ignat ionoft hisl is tjus ttos howt hatwecan.
-- _____________________________________ John Andersen
Sure! Why not. I write typo all the time, so for my part its not a big problem... /Thread closed... -- /Rikard ----------------------------------------------------------------------------- email : rikard.j@rikjoh.com web : http://www.rikjoh.com mob: : +46 (0)763 19 76 25 ------------------------ Public PGP fingerprint ---------------------------- < 15 28 DF 78 67 98 B2 16 1F D3 FD C5 59 D4 B6 78 46 1C EE 56 >
Rikard Johnels wrote:
Never had any problems with XFS. As people have stated; it cashes alot, so if you are unlucky and get at power out at the wrong moment, the filesystem MAY corrupt. But so far i have been lucky i guess. Even with power outs i havent had any trashed drives (yet)
XFS is fast, reliable and scales well, or so i have read. And besides; SGI wouldnt use it if it werent stable...
One thing Reiser supports is resizing with the LVM. Does XFS do that too? -- Use OpenOffice.org http://www.openoffice.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 07 June 2007 22:20, James Knott wrote:
One thing Reiser supports is resizing with the LVM. Does XFS do that too?
What does "resizing with the LVM" mean? If you mean growing the size of the file system, yes, xfs supports that -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thursday 07 June 2007, Morten Bjørnsvik wrote:
I've used the xfs_repair, xfsdump, xfsrestore and xfs_db tools extensively.
Why? I can count on the fingers of one hand ho many times I've had to use the equivalent utilities on the 10 or 12 servers I manage over the years. They all run Reiser or ext3. -- _____________________________________ John Andersen
participants (10)
-
Anders Johansson -
Carlos E. R. -
James Knott -
John Andersen -
Joseph Loo -
Kenneth Schneider -
Matthew Stringer -
Morten Bjørnsvik -
Randall R Schulz -
Rikard Johnels