[opensuse] TCP: Possible SYN flooding on port 873. Sending cookies.
I have a new rsync server that is part of a mirror network. Every hour it is hit by an apparently slight surge in the number of clients, but most of the time there is nothing to download. When this happens I see $SUBJ in the log. I've googled it quite a bit to understand the meaning, but I'm not sure I'm really getting anywhere :-) I guess the kernel starts sending cookies when certain conditions are met, possibly the length of tcp_max_syn_backlog ? I have increased it from the default 512 to 16384 (step-wise), but it doesn't seem to have had much effect. I've also measured the number of connections on port 873 in state SYN_RECV (netstat -ant | grep 873.*SYN_RECV), but checking that every second only showed up to 20 clients. The box isn't overly potent, but should easily handle e.g. 100 concurrent rsync clients, especially when there's no data to sync. What I'd like to understand - a) what does it mean when the kernel say $SUBJ, b) what are the criteria for (a), c) am I loosing clients when it does, and d) what can I do to prevent it? -- Per Jessen, Zürich (19.9°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-06-21 18:04, Per Jessen wrote:
I have a new rsync server that is part of a mirror network. Every hour it is hit by an apparently slight surge in the number of clients, but most of the time there is nothing to download. When this happens I see $SUBJ in the log. I've googled it quite a bit to understand the meaning, but I'm not sure I'm really getting anywhere :-)
I guess the kernel starts sending cookies when certain conditions are met, possibly the length of tcp_max_syn_backlog ? I have increased it from the default 512 to 16384 (step-wise), but it doesn't seem to have had much effect.
I've also measured the number of connections on port 873 in state SYN_RECV (netstat -ant | grep 873.*SYN_RECV), but checking that every second only showed up to 20 clients.
The box isn't overly potent, but should easily handle e.g. 100 concurrent rsync clients, especially when there's no data to sync.
What I'd like to understand -
a) what does it mean when the kernel say $SUBJ, b) what are the criteria for (a), c) am I loosing clients when it does, and d) what can I do to prevent it?
I don't know much about the subject either, but you've had no other replies yet ... Do you mean an rsync daemon? And I presume it's on the public internet? Does the rsyncd log show anything interesting? How many clients are there and how often do they check whether there is anything to sync? Just once an hour? All at the same time? Did your googling turn up these pages? https://www.frozentux.net/ipsysctl-tutorial/ipsysctl-tutorial.html http://blog.dubbelboer.com/2012/04/09/syn-cookies.html They seem the most useful I've found so far. Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Dave Howorth wrote:
I don't know much about the subject either, but you've had no other replies yet ...
Do you mean an rsync daemon? And I presume it's on the public internet?
Yes and yes.
Does the rsyncd log show anything interesting?
No, only the transfer log entries.
How many clients are there and how often do they check whether there is anything to sync? Just once an hour? All at the same time?
The mirror works on round-robin DNS, and I see about 30'000 unique clients a day. Many check once an hour, some more frequently, others less. $SUBJ appears only around the hour, e.g. from 1450 to 1455.
Did your googling turn up these pages?
https://www.frozentux.net/ipsysctl-tutorial/ipsysctl-tutorial.html http://blog.dubbelboer.com/2012/04/09/syn-cookies.html
They seem the most useful I've found so far.
I don't think I've seen net.core.somaxconn mentioned in any of the stuff I've read, I'll try tweaking that too. -- Per Jessen, Zürich (24.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Jun 21, 2016 at 8:04 PM, Per Jessen
I have a new rsync server that is part of a mirror network. Every hour it is hit by an apparently slight surge in the number of clients, but most of the time there is nothing to download. When this happens I see $SUBJ in the log. I've googled it quite a bit to understand the meaning, but I'm not sure I'm really getting anywhere :-)
I guess the kernel starts sending cookies when certain conditions are met, possibly the length of tcp_max_syn_backlog ? I have increased it from the default 512 to 16384 (step-wise), but it doesn't seem to have had much effect.
I've also measured the number of connections on port 873 in state SYN_RECV (netstat -ant | grep 873.*SYN_RECV), but checking that every second only showed up to 20 clients.
The box isn't overly potent, but should easily handle e.g. 100 concurrent rsync clients, especially when there's no data to sync.
What I'd like to understand -
a) what does it mean when the kernel say $SUBJ,
The number of pending connection requests exceeds socket backlog
b) what are the criteria for (a),
backlog is set by listen() call. Max value is clamped by both net.core.somaxconn and net.ipv4.tcp_max_syn_backlog, whatever is smaller.
c) am I loosing clients when it does, and
If you get these messages you are probably not losing them.
d) what can I do to prevent it?
Increase rsycnd backlog? listen backlog You can override the default backlog value when the daemon listens for connections. It defaults to 5. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
What I'd like to understand -
a) what does it mean when the kernel say $SUBJ,
The number of pending connection requests exceeds socket backlog
b) what are the criteria for (a),
backlog is set by listen() call. Max value is clamped by both net.core.somaxconn and net.ipv4.tcp_max_syn_backlog, whatever is smaller.
c) am I loosing clients when it does, and
If you get these messages you are probably not losing them.
That was my thought too, but I thought I was dealing with a system limit, not a per-application.
d) what can I do to prevent it?
Increase rsycnd backlog?
listen backlog You can override the default backlog value when the daemon listens for connections. It defaults to 5.
Ah, looks like I need a newer rsyncd. :-) thanks Andrei. -- Per Jessen, Zürich (24.8°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
Andrei Borzenkov wrote:
What I'd like to understand -
a) what does it mean when the kernel say $SUBJ,
The number of pending connection requests exceeds socket backlog
b) what are the criteria for (a),
backlog is set by listen() call. Max value is clamped by both net.core.somaxconn and net.ipv4.tcp_max_syn_backlog, whatever is smaller.
c) am I loosing clients when it does, and
If you get these messages you are probably not losing them.
That was my thought too, but I thought I was dealing with a system limit, not a per-application.
Did I get this right - rsyncd uses e.g. listen(x,5) and when the number of pending clients exceeds 5, the kernel will say $SUBJ ? -- Per Jessen, Zürich (27.9°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Jun 22, 2016 at 5:34 PM, Per Jessen
Per Jessen wrote:
Andrei Borzenkov wrote:
What I'd like to understand -
a) what does it mean when the kernel say $SUBJ,
The number of pending connection requests exceeds socket backlog
b) what are the criteria for (a),
backlog is set by listen() call. Max value is clamped by both net.core.somaxconn and net.ipv4.tcp_max_syn_backlog, whatever is smaller.
c) am I loosing clients when it does, and
If you get these messages you are probably not losing them.
That was my thought too, but I thought I was dealing with a system limit, not a per-application.
Did I get this right - rsyncd uses e.g. listen(x,5) and when the number of pending clients exceeds 5, the kernel will say $SUBJ ?
At least, that is my understanding based on information I have. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
On Wed, Jun 22, 2016 at 5:34 PM, Per Jessen
wrote: Per Jessen wrote:
Andrei Borzenkov wrote:
What I'd like to understand -
a) what does it mean when the kernel say $SUBJ,
The number of pending connection requests exceeds socket backlog
b) what are the criteria for (a),
backlog is set by listen() call. Max value is clamped by both net.core.somaxconn and net.ipv4.tcp_max_syn_backlog, whatever is smaller.
c) am I loosing clients when it does, and
If you get these messages you are probably not losing them.
That was my thought too, but I thought I was dealing with a system limit, not a per-application.
Did I get this right - rsyncd uses e.g. listen(x,5) and when the number of pending clients exceeds 5, the kernel will say $SUBJ ?
At least, that is my understanding based on information I have.
Good enough for me - can't help thinking it sounds odd, but I've been experimenting a bit this morning. My rsyncd on this system is slightly back level and doesn't support the "listen backlog" directive. Instead I switched to xinetd for kicking off "rsync --daemon". I have had three hourly peaks so far, and no $SUBJ in the logs. Looking at xinetd source, it uses listen(x,64), which seems to confirm your understanding. I've also looked at number of clients connecting - I was halfway expecting to see more with xinetd, but sofar I can't see any major difference. -- Per Jessen, Zürich (21.1°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
I've been experimenting a bit this morning. My rsyncd on this system is slightly back level and doesn't support the "listen backlog" directive. Instead I switched to xinetd for kicking off "rsync --daemon". I have had three hourly peaks so far, and no $SUBJ in the logs. Looking at xinetd source, it uses listen(x,64),
A quick update - no $SUBJ since I changed to xinetd yesterday. Doesn't mean using xinetd is _the_ solution, I am certain increasing the backlog in rsyncd with "listen backlog" will do the same. I am still puzzled as to why the kernel writes this message based on an application setting, but that's not so important anymore. -- Per Jessen, Zürich (25.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (3)
-
Andrei Borzenkov
-
Dave Howorth
-
Per Jessen