RE: [SLE] Network File Systems
Many thanks for all the responses.
a. rsync. we use this tool as well, but experienced some discomfort
with it
in that, if we run a couple of rsync tasks on large transfers in
parallel,
they can grind our servers to a halt without too much difficulty.
Any further access to the same file system involved in the rsync will
certainly
be very slow, and the overall system performance is severely degraded
too.
By the way, we use gigabit switches in general, and the servers are
using reasonably
decent processors like dual opterons 246s and dual xeons 3GHz with
2GB ram and 3ware 8505
controllers.
A slight deficiency with rsync is that we cannot easily access files
as if they were local
for addition or modification as we can do so with NFS.
b. fish. I have not used it before. Can someone tell me where I can
find more details about it.
The on-line man page says fish is a card game, though sounds lovely
it is.
Many thanks.
Peter
-----Original Message-----
From: Jeffrey L. Taylor [mailto:suse@austinblues.dyndns.org]
Sent: 15 March 2005 20:13
To: suse-linux-e@suse.com
Subject: Re: [SLE] Network File Systems
Quoting Chiu, PCM (Peter)
Is there a more efficient networked file system other than NFS and samba?
The two both work, but can be rather slow for large transfers.
To paraphase the Mutt MUA logo, "All network file systems suck, some suck more than others." Having tried NFS, Samba, and AFS over the years, all of them are slow and unreliable. Rsync is fast, reliable, and great IF you can remember not to work on the same file on both ends. One thing to check, make sure your network is solid, and all links are running 100 or 1000Gbps, full duplex. And any switches or hubs have plenty of backplane bandwidth. Just my $0.02USD, Jeffrey -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
Peter, On Wednesday 16 March 2005 03:34, Chiu, PCM (Peter) wrote:
Many thanks for all the responses.
a. rsync. we use this tool as well, but experienced some discomfort with it in that, if we run a couple of rsync tasks on large transfers inparallel, they can grind our servers to a halt without too much difficulty.
Any further access to the same file system involved in the rsync will certainly be very slow, and the overall system performance is severely degraded too.
You can (probably) use a traffic shaping tool to throttle rsync traffic. I have no experience with it, but in general when the question of traffic control and shaping comes up, someone will recommend wondershaper. If you Google the SuSE lists for wondershaper, you'll get lots of hits (Google "Wondershaper site:lists.suse.com"). If you Google simply for "Wondershaper", the third hit you get is a tech note on the Novell / SuSE site: http://www.novell.com/products/linuxpackages/enterpriseserver/i386/wondersha.... Randall Schulz
On Wednesday 16 March 2005 15:59, Randall R Schulz wrote:
Peter,
On Wednesday 16 March 2005 03:34, Chiu, PCM (Peter) wrote:
Many thanks for all the responses.
a. rsync. we use this tool as well, but experienced some discomfort with it in that, if we run a couple of rsync tasks on large transfers inparallel, they can grind our servers to a halt without too much difficulty.
Any further access to the same file system involved in the rsync will certainly be very slow, and the overall system performance is severely degraded too.
You can (probably) use a traffic shaping tool to throttle rsync traffic.
That reminds me vaguely of something. man rsync: yes indeed: --bwlimit=KBPS limit I/O bandwidth, KBytes per second Cheers, Leen
Leen, On Wednesday 16 March 2005 07:41, Leendert Meyer wrote:
On Wednesday 16 March 2005 15:59, Randall R Schulz wrote:
Peter,
On Wednesday 16 March 2005 03:34, Chiu, PCM (Peter) wrote:
Many thanks for all the responses.
a. rsync. we use this tool as well, but experienced some discomfort with it in that, if we run a couple of rsync tasks on large transfers inparallel, they can grind our servers to a halt without too much difficulty.
Any further access to the same file system involved in the rsync will certainly be very slow, and the overall system performance is severely degraded too.
You can (probably) use a traffic shaping tool to throttle rsync traffic.
That reminds me vaguely of something.
man rsync: yes indeed:
--bwlimit=KBPS limit I/O bandwidth, KBytes per second
Dammit! I looked in the rsyncd.conf manual page for something like this before replying, but I didn't think of looking in the rsync manual page itself. Of course, this requires system administrators to rely on their users to limit their bandwidth consumption, which probably isn't the ideal solution to Peter's problem.
Cheers,
Leen
Randall Schulz
On Wednesday 16 March 2005 16:51, Randall R Schulz wrote:
Leen,
On Wednesday 16 March 2005 07:41, Leendert Meyer wrote:
On Wednesday 16 March 2005 15:59, Randall R Schulz wrote:
Peter,
On Wednesday 16 March 2005 03:34, Chiu, PCM (Peter) wrote:
Many thanks for all the responses.
a. rsync. we use this tool as well, but experienced some discomfort with it in that, if we run a couple of rsync tasks on large transfers inparallel, they can grind our servers to a halt without too much difficulty.
Any further access to the same file system involved in the rsync will certainly be very slow, and the overall system performance is severely degraded too.
You can (probably) use a traffic shaping tool to throttle rsync traffic.
That reminds me vaguely of something.
man rsync: yes indeed:
--bwlimit=KBPS limit I/O bandwidth, KBytes per second
Dammit! I looked in the rsyncd.conf manual page for something like this before replying, but I didn't think of looking in the rsync manual page itself.
Well, if it weren't there, I wouldn't be surprised. I would just wonder where I got that vague impression from... ;)
Of course, this requires system administrators to rely on their users to limit their bandwidth consumption, which probably isn't the ideal solution to Peter's problem.
I know, I realized that, but I just wanted to point out the possibility. Of course if users are in question, throttling via wondershaper or something similar would be better. OTOH, there is a --config=FILE option... But I suppose every option in there would be overridden by their commandline equivalent. Cheers, Leen
On Wednesday 16 Mar 2005 15:51 pm, Randall R Schulz wrote:
Leen,
On Wednesday 16 March 2005 07:41, Leendert Meyer wrote: <SNIP>
That reminds me vaguely of something.
man rsync: yes indeed:
--bwlimit=KBPS limit I/O bandwidth, KBytes per second
Dammit! I looked in the rsyncd.conf manual page for something like this before replying, but I didn't think of looking in the rsync manual page itself.
Of course, this requires system administrators to rely on their users to limit their bandwidth consumption, which probably isn't the ideal solution to Peter's problem.
The sysadmin could add it to the alias list somewhere suitable. Dylan
Cheers,
Leen
Randall Schulz
-- "I see your Schwartz is as big as mine" -Dark Helmet
Dylan, On Wednesday 16 March 2005 09:51, Dylan wrote:
On Wednesday 16 Mar 2005 15:51 pm, Randall R Schulz wrote:
Leen,
On Wednesday 16 March 2005 07:41, Leendert Meyer wrote:
<SNIP>
That reminds me vaguely of something.
man rsync: yes indeed:
--bwlimit=KBPS limit I/O bandwidth, KBytes per second
Dammit! I looked in the rsyncd.conf manual page for something like this before replying, but I didn't think of looking in the rsync manual page itself.
Of course, this requires system administrators to rely on their users to limit their bandwidth consumption, which probably isn't the ideal solution to Peter's problem.
The sysadmin could add it to the alias list somewhere suitable.
It's not trivial to enforce a policy in this manner. You'd have to make it impossible to access the rsync executable directly. Then you've have to supply a cover script that processed all the arguments supplied to strip out those that control the bandwidth limit and put the administratively mandated one in its place. Then you'd have to pass the invocation on to the real rsync binary. The hard part is making it possible for the cover script to invoke the real rsync binary while not allowing end users to do so directly. Setuid shell scripts are not an option on Linux, which then forces the administrator to write the cover program in C or C++ or some other language the produces binary executables. Of course, if all you're trying to do is relieve users from the burden of imposing the limits themselves, but not not to strictly enforce it, then a cover script is adequate and allows in-the-know but self-disciplined users to override the limits when there's a legitimate justification. This is why I looked to the system-wide rsync configuration for a bandwidth-limiting option. I clearly belongs there.
Dylan
Randall Schulz
On Wednesday 16 March 2005 18:51, Dylan wrote:
On Wednesday 16 Mar 2005 15:51 pm, Randall R Schulz wrote:
Leen,
On Wednesday 16 March 2005 07:41, Leendert Meyer wrote:
<SNIP>
That reminds me vaguely of something.
man rsync: yes indeed:
--bwlimit=KBPS limit I/O bandwidth, KBytes per second
Dammit! I looked in the rsyncd.conf manual page for something like this before replying, but I didn't think of looking in the rsync manual page itself.
Of course, this requires system administrators to rely on their users to limit their bandwidth consumption, which probably isn't the ideal solution to Peter's problem.
The sysadmin could add it to the alias list somewhere suitable.
Yes. But couldn't a user easily override that, e.g. by creating his own alias or by calling rsync directly (e.g. /usr/bin/rsync)? Cheers, Leen
On Wednesday 16 Mar 2005 18:17 pm, Leendert Meyer wrote:
On Wednesday 16 March 2005 18:51, Dylan wrote: <SNIP>
The sysadmin could add it to the alias list somewhere suitable.
Yes. But couldn't a user easily override that, e.g. by creating his own alias or by calling rsync directly (e.g. /usr/bin/rsync)?
Indeed, but I wasn't intending to suggest it was a perfect solution. Dylan
Cheers,
Leen
-- "I see your Schwartz is as big as mine" -Dark Helmet
On Wednesday 16 March 2005 12:34, Chiu, PCM (Peter) wrote:
b. fish. I have not used it before. Can someone tell me where I can find more details about it. The on-line man page says fish is a card game, though sounds lovely it is.
Fire up KInfoCenter (KDE-Menu -> System -> Monitor -> KInfoCenter on my KDE-3.4 from SuSE-9.2). Then select protocols -> fish In short: works with Konqueror. On the commandline, you would use scp (man scp), or, indeed, rsync. Cheers, Leen
participants (4)
-
Chiu, PCM (Peter) -
Dylan -
Leendert Meyer -
Randall R Schulz