[opensuse] who is eating (my) packets?
I think I need a second pair of eyes. We are migrating a somewhat complex cluster setup from openSUSE 11.0 32bit to Leap 15.1 64bit. Just for completeness, for the topic at hand I don't think it matters, except I sort of suspect a change in behaviour due to the big jump in kernel versions. Maybe some default setting I need to tweak. I won't try to describe the complete setup unless I have to - it's not secret, just slightly complex. On a cluster node (tattoo16) I try to initiate an outbound connection over 10.0.1.144/30 on an ipip device. I use telnet - when the first response comes back, it seems to "disappear". The response comes from the external host (public IP) to my ipip network 10.0.11.146. (on outgoing, it has been appropriately snat'ed). So far so good. I see the response using tcpdump - but it never gets back to telnet. I started adding some iptables rules to log it: -t filter INPUT - nothing. -t filter FORWARD - nothing. I did the same with the nat and mangle tables, never caught anything. Then I tried the "raw" table and voila! I see my response (as I see it with tcpdump). In summary - I see the response with tcpdump and in the "raw" table, where else can I look for this response packet? I expected it to turn up in INPUT, the rest I tried in pure desperation. I know someone is going to say "firewall" at some point - the firewall has plain ACCEPT rules for everything on the ipip devices. -- Per Jessen, Zürich (19.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
I think I need a second pair of eyes.
[snip]
I started adding some iptables rules to log it:
-t filter INPUT - nothing. -t filter FORWARD - nothing.
I did the same with the nat and mangle tables, never caught anything.
Correction - in mangle::PREROUTING, I do see the response, but then in nat::PREROUTING, there is nothing. Nor in mangle::INPUT or ::FORWARD. So I see it in mangle::PREROUTING, I see the counters increment, then it disappears. -- Per Jessen, Zürich (21.5°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
Per Jessen wrote:
I think I need a second pair of eyes.
[snip]
I started adding some iptables rules to log it:
-t filter INPUT - nothing. -t filter FORWARD - nothing.
I did the same with the nat and mangle tables, never caught anything.
Correction - in mangle::PREROUTING, I do see the response, but then in nat::PREROUTING, there is nothing. Nor in mangle::INPUT or ::FORWARD.
So I see it in mangle::PREROUTING, I see the counters increment, then it disappears.
Weak guess I'm afraid, but when I had similar problems here it was a (bad) port forwarding definition -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Peter Suetterlin wrote:
Per Jessen wrote:
Per Jessen wrote:
I think I need a second pair of eyes.
[snip]
I started adding some iptables rules to log it:
-t filter INPUT - nothing. -t filter FORWARD - nothing.
I did the same with the nat and mangle tables, never caught anything.
Correction - in mangle::PREROUTING, I do see the response, but then in nat::PREROUTING, there is nothing. Nor in mangle::INPUT or ::FORWARD.
So I see it in mangle::PREROUTING, I see the counters increment, then it disappears.
Weak guess I'm afraid, but when I had similar problems here it was a (bad) port forwarding definition
Thanks for chipping in, but no port forwarding or much nat'ing here - except uplink where the 10.0.1.x address is snat'ed (masqueraded). What I really don't get is that this works fine, just on much older systems. I keep thinking there must be a setting or switch that has changed. I'm about to go run a diff on /proc/sys/net :-) -- Per Jessen, Zürich (26.4°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 7/2/19 9:08 AM, Per Jessen wrote:
Weak guess I'm afraid, but when I had similar problems here it was a (bad) port forwarding definition Thanks for chipping in, but no port forwarding or much nat'ing here - except uplink where the 10.0.1.x address is snat'ed (masqueraded).
What I really don't get is that this works fine, just on much older systems. I keep thinking there must be a setting or switch that has changed. I'm about to go run a diff on /proc/sys/net:-)
Just out of curiosity, Per, what firewall are you using? Firewalld? I tried to use firewalld in 15.0, but I didn't have the time to figure out the gui, so I went back to SuSEfirewall2. Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Lew Wolfgang wrote:
On 7/2/19 9:08 AM, Per Jessen wrote:
Weak guess I'm afraid, but when I had similar problems here it was a (bad) port forwarding definition Thanks for chipping in, but no port forwarding or much nat'ing here - except uplink where the 10.0.1.x address is snat'ed (masqueraded).
What I really don't get is that this works fine, just on much older systems. I keep thinking there must be a setting or switch that has changed. I'm about to go run a diff on /proc/sys/net:-)
Just out of curiosity, Per, what firewall are you using? Firewalld?
It's our own script, with iptables of course.
I tried to use firewalld in 15.0, but I didn't have the time to figure out the gui, so I went back to SuSEfirewall2.
Some of this stuff is older than that - I custom-wrote the firewall script a loooong time ago :-) In a complex setup, I find it is much easier to work directly with iptables than to try to bend any wrapper the right way. -- Per Jessen, Zürich (25.5°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
I think I need a second pair of eyes.
Let me try again. Start by looking at an iptables flowchart, e.g. the one shown here: https://stuffphilwrites.com/2014/09/iptables-processing-flowchart/ An incoming packet hits raw::prerouting Next it hits mangle::prerouting. (log1) Next it hits either nat::prerouting (log2) or mangle::input (log3). I have put logX entries at those places, first entry in the table, protocol tcp, source port 30077. log1 - check. log2 and log3 - nothing. Instead I then tried TRACE - only works in "raw", but afaict a packet will now be traced at all points, very useful. With this, I see exactly the same - the syn ack packet only reaches mangle::prerouting, then it disappears. [90946.009268] TRACE: raw:PREROUTING:policy:2 IN=ipip1 OUT= MAC= SRC=88.198.37.20 DST=10.0.1.146 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=30077 DPT=57649 SEQ=2614710311 ACK=2195080463 WINDOW=5792 RES=0x00 ACK SYN URGP=0 OPT (020405B40101080AD970158EA5C9C25E01030306) [90946.009289] TRACE: mangle:PREROUTING:policy:5 IN=ipip1 OUT= MAC= SRC=88.198.37.20 DST=10.0.1.146 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=30077 DPT=57649 SEQ=2614710311 ACK=2195080463 WINDOW=5792 RES=0x00 ACK SYN URGP=0 OPT (020405B40101080AD970158EA5C9C25E01030306) On other nodes, with kernel 2.6.25 or 3.16.7, the same trace shows packets neatly passing through (on 3.6.17): TRACE: raw:PREROUTING:policy:2 TRACE: mangle:PREROUTING:policy:5 TRACE: mangle:INPUT:policy:1 TRACE: filter:INPUT:rule:2 How can my packet(s) just be dropped after mangle::prerouting ??? I am missing something or totally misinterpreting something? I wonder if I might install an older kernel and see if it works with e.g. 3.16.xx. -- Per Jessen, Zürich (22.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, 03 Jul 2019 12:25:36 +0200 Per Jessen <per@computer.org> wrote:
Per Jessen wrote:
I think I need a second pair of eyes.
Let me try again. Start by looking at an iptables flowchart, e.g. the one shown here:
https://stuffphilwrites.com/2014/09/iptables-processing-flowchart/
An incoming packet hits raw::prerouting Next it hits mangle::prerouting. (log1) Next it hits either nat::prerouting (log2) or mangle::input (log3).
I have put logX entries at those places, first entry in the table, protocol tcp, source port 30077.
log1 - check. log2 and log3 - nothing.
Instead I then tried TRACE - only works in "raw", but afaict a packet will now be traced at all points, very useful.
With this, I see exactly the same - the syn ack packet only reaches mangle::prerouting, then it disappears.
[90946.009268] TRACE: raw:PREROUTING:policy:2 IN=ipip1 OUT= MAC= SRC=88.198.37.20 DST=10.0.1.146 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=30077 DPT=57649 SEQ=2614710311 ACK=2195080463 WINDOW=5792 RES=0x00 ACK SYN URGP=0 OPT (020405B40101080AD970158EA5C9C25E01030306) [90946.009289] TRACE: mangle:PREROUTING:policy:5 IN=ipip1 OUT= MAC= SRC=88.198.37.20 DST=10.0.1.146 LEN=60 TOS=0x00 PREC=0x00 TTL=59 ID=0 DF PROTO=TCP SPT=30077 DPT=57649 SEQ=2614710311 ACK=2195080463 WINDOW=5792 RES=0x00 ACK SYN URGP=0 OPT (020405B40101080AD970158EA5C9C25E01030306)
On other nodes, with kernel 2.6.25 or 3.16.7, the same trace shows packets neatly passing through (on 3.6.17):
TRACE: raw:PREROUTING:policy:2 TRACE: mangle:PREROUTING:policy:5 TRACE: mangle:INPUT:policy:1 TRACE: filter:INPUT:rule:2
How can my packet(s) just be dropped after mangle::prerouting ??? I am missing something or totally misinterpreting something?
I wonder if I might install an older kernel and see if it works with e.g. 3.16.xx.
Maybe try asking on the netfilter users list? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Dave Howorth wrote:
On Wed, 03 Jul 2019 12:25:36 +0200 Per Jessen <per@computer.org> wrote:
Per Jessen wrote:
I think I need a second pair of eyes.
Let me try again. Start by looking at an iptables flowchart, e.g. the one shown here: [snip] Maybe try asking on the netfilter users list?
Thanks! - why didn't I think of that ..... -- Per Jessen, Zürich (22.2°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
I wonder if I might install an older kernel and see if it works with e.g. 3.16.xx.
I tried this, installed a 5.1 from Tumbleweed, then went back to 3.16.7, also trying out a few 4.x.x on the way. Nothing worked. However - once I was on 3.16.7, it was easy to compare the contents of /proc/sys/net betweem a working and a non-working system. I only needed to set rp_filter=2 for my ipip devices. I really don't know how I had overlooked that. :-( -- Per Jessen, Zürich (24.3°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Jul 3, 2019 at 3:35 PM Per Jessen <per@computer.org> wrote:
Per Jessen wrote:
I wonder if I might install an older kernel and see if it works with e.g. 3.16.xx.
I tried this, installed a 5.1 from Tumbleweed, then went back to 3.16.7, also trying out a few 4.x.x on the way. Nothing worked. However - once I was on 3.16.7, it was easy to compare the contents of /proc/sys/net betweem a working and a non-working system. I only needed to set rp_filter=2 for my ipip devices. I really don't know how I had overlooked that. :-(
And the previous value was ... ? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
On Wed, Jul 3, 2019 at 3:35 PM Per Jessen <per@computer.org> wrote:
Per Jessen wrote:
I wonder if I might install an older kernel and see if it works with e.g. 3.16.xx.
I tried this, installed a 5.1 from Tumbleweed, then went back to 3.16.7, also trying out a few 4.x.x on the way. Nothing worked. However - once I was on 3.16.7, it was easy to compare the contents of /proc/sys/net betweem a working and a non-working system. I only needed to set rp_filter=2 for my ipip devices. I really don't know how I had overlooked that. :-(
And the previous value was ... ?
I am not sure - except for the two '2's, this is the default: /proc/sys/net/ipv4/conf/all/rp_filter = 1 /proc/sys/net/ipv4/conf/default/rp_filter = 0 /proc/sys/net/ipv4/conf/eth0/rp_filter = 0 /proc/sys/net/ipv4/conf/ipip0/rp_filter = 2 /proc/sys/net/ipv4/conf/ipip1/rp_filter = 2 /proc/sys/net/ipv4/conf/lo/rp_filter = 0 /proc/sys/net/ipv4/conf/tun0/rp_filter = 0 /proc/sys/net/ipv4/conf/tunl0/rp_filter = 0 -- Per Jessen, Zürich (24.9°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Andrei Borzenkov -
Dave Howorth -
Lew Wolfgang -
Per Jessen -
Peter Suetterlin