W dniu 07.09.2018 o 11:07, Andrei Borzenkov pisze:

On Thu, Sep 6, 2018 at 11:01 PM Adam Mizerski wrote:

...

Hi,

I have the following situation: I have a VPS with openSUSE 15 Leap on it. It has a public IPv6 address and connectivity with outside world works correctly. I also have a whole /64 subnet to use (public address is not in this subnet).

I want to configure WireGuard so that devices connected to it would have addresses from my subnet (for now set by hand, I'll play with autoconfiguration later) and connectivity to outside world.

My VSP operator requires that all IPv6 traffic should come out of my public address.

Here's what I've got:

# ip -6 a [...] 2: eth0: mtu 1500 state UP qlen 1000 inet6 {PUBLIC_IP}/64 scope global valid_lft forever preferred_lft forever [...] 3: wg0: mtu 1420 state UNKNOWN qlen 1000 inet6 {SUBNET}::1/64 scope global deprecated valid_lft forever preferred_lft 0sec

# ip -6 r local ::1 dev lo proto kernel metric 256 pref medium {SUBNET}::/64 dev wg0 proto kernel metric 256 pref medium {PUBLIC_IP_SUBNET}::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via {GATEWAY} dev eth0 src {PUBLIC_IP} metric 1024 pref medium

# sysctl -a | grep ipv6 | grep forwarding | grep -v mc_forwarding net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.default.forwarding = 1 net.ipv6.conf.eth0.forwarding = 1 net.ipv6.conf.lo.forwarding = 0 net.ipv6.conf.wg0.forwarding = 1

# ping6 -c1 sixxs.net -I eth0 PING sixxs.net(nlams01.paphosting.net (2a02:898:146::2)) from {PUBLIC_IP} eth0: 56 data bytes 64 bytes from nlams01.paphosting.net (2a02:898:146::2): icmp_seq=1 ttl=58 time=28.1 ms

--- sixxs.net ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 28.109/28.109/28.109/0.000 ms

# ping6 -c1 sixxs.net -I wg0 ping6: Warning: source address might be selected on device other than wg0. PING sixxs.net(nlams01.paphosting.net (2a02:898:146::2)) from {PUBLIC_IP} wg0: 56 data bytes From {PUBLIC_IP} icmp_seq=1 Destination unreachable: Address unreachable ping: sendmsg: Destination address required

--- sixxs.net ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

# ping6 -c1 {PUBLIC_IP} -I wg0 ping6: Warning: source address might be selected on device other than wg0. PING {PUBLIC_IP}({PUBLIC_IP}) from {PUBLIC_IP} wg0: 56 data bytes From {PUBLIC_IP} icmp_seq=1 Destination unreachable: Address unreachable ping: sendmsg: Destination address required

--- {PUBLIC_IP} ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

Address at wg0 has status "deprecated", because from what I've read, command # ip a change {SUBNET}::1 dev wg0 preferred_lft 0 should make connections not come out of this address. But it seems it's not working,

According to commands output you provided it *is* working - deprecated address is not used. All your examples show PUBLIC_IP as source address.

...

because if I run ping6 without "-I" option, it works the same if I use "-I wg0".

You forgot to show command you used and its output and explain what is wrong with it from your point of view.

So it looks like this: # ping6 -c1 sixxs.net PING sixxs.net(nlams01.paphosting.net (2a02:898:146::2)) 56 data bytes From {PUBLIC_IP} ({PUBLIC_IP}) icmp_seq=1 Destination unreachable: Address unreachable ping: sendmsg: Destination address required --- sixxs.net ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms As you can see, ping failed. But if I use `ping6 -c1 sixxs.net -I eth0` it works, as I wrote before. And I don't see the reason why it fails. Because address on wg0 is deprecated, I though both command should be equivalent.

...

iptables is managed by firewalld and wg0 interface is in "trusted" zone. As I understand output of "ip6tables -L -nv" command, from this interface all traffic (including FORWARD) is allowed.

Am I doing something wrong? Or is there something I'm not aware of? Because my understanding is that packets from wg0 interface should be forwarded to eth0 and from it further to the world. But that's not happening and I don't even know how to find out what's really going on.

Any help is appreciated, no question is dumb, and don't assume anything to be obvious.

-- Adam Mizerski

-- Adam Mizerski

W dniu 08.09.2018 o 07:40, Andrei Borzenkov pisze:

07.09.2018 23:47, Adam Mizerski пишет:

...

W dniu 07.09.2018 o 11:07, Andrei Borzenkov pisze:

...

On Thu, Sep 6, 2018 at 11:01 PM Adam Mizerski wrote:

...

Hi,

I have the following situation: I have a VPS with openSUSE 15 Leap on it. It has a public IPv6 address and connectivity with outside world works correctly. I also have a whole /64 subnet to use (public address is not in this subnet).

...

...

...

I want to configure WireGuard so that devices connected to it would have addresses from my subnet (for now set by hand, I'll play with autoconfiguration later) and connectivity to outside world.

My VSP operator requires that all IPv6 traffic should come out of my public address.

Here's what I've got:

# ip -6 a [...] 2: eth0: mtu 1500 state UP qlen 1000 inet6 {PUBLIC_IP}/64 scope global valid_lft forever preferred_lft forever [...] 3: wg0: mtu 1420 state UNKNOWN qlen 1000 inet6 {SUBNET}::1/64 scope global deprecated valid_lft forever preferred_lft 0sec

# ip -6 r local ::1 dev lo proto kernel metric 256 pref medium {SUBNET}::/64 dev wg0 proto kernel metric 256 pref medium {PUBLIC_IP_SUBNET}::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via {GATEWAY} dev eth0 src {PUBLIC_IP} metric 1024 pref medium

# sysctl -a | grep ipv6 | grep forwarding | grep -v mc_forwarding net.ipv6.conf.all.forwarding = 1 net.ipv6.conf.default.forwarding = 1 net.ipv6.conf.eth0.forwarding = 1 net.ipv6.conf.lo.forwarding = 0 net.ipv6.conf.wg0.forwarding = 1

# ping6 -c1 sixxs.net -I eth0 PING sixxs.net(nlams01.paphosting.net (2a02:898:146::2)) from {PUBLIC_IP} eth0: 56 data bytes 64 bytes from nlams01.paphosting.net (2a02:898:146::2): icmp_seq=1 ttl=58 time=28.1 ms

--- sixxs.net ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 28.109/28.109/28.109/0.000 ms

...

So it looks like this:

# ping6 -c1 sixxs.net PING sixxs.net(nlams01.paphosting.net (2a02:898:146::2)) 56 data bytes From {PUBLIC_IP} ({PUBLIC_IP}) icmp_seq=1 Destination unreachable: Address unreachable ping: sendmsg: Destination address required

--- sixxs.net ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

As you can see, ping failed. But if I use `ping6 -c1 sixxs.net -I eth0` it works, as I wrote before. And I don't see the reason why it fails. Because address on wg0 is deprecated, I though both command should be equivalent.

Earlier you said that connectivity to outside world works correctly. Does it mean only ping fails? Can you connect to any IPv6 using TCP/UDP?

strace of both successful and failed ping invocations would be useful.

Pings and everything works correctly before I start (and after I stop) WireGuard interface. here are strace outputs: https://etam.homelinux.com/index.php/s/wmJxaDps6Y3KrA9 It's not only ping failing. "curl -6 sixxs.net" gives error: curl: (7) Failed to connect to sixxs.net port 80: No route to host -- Adam Mizerski

W dniu 09.09.2018 o 08:49, Andrei Borzenkov pisze:

09.09.2018 00:13, Adam Mizerski пишет:

...

...

Earlier you said that connectivity to outside world works correctly. Does it mean only ping fails? Can you connect to any IPv6 using TCP/UDP?

strace of both successful and failed ping invocations would be useful.

Pings and everything works correctly before I start (and after I stop) WireGuard interface.

here are strace outputs: https://etam.homelinux.com/index.php/s/wmJxaDps6Y3KrA9

sendto() fails immediately with EDESTADDRREQ which makes no sense because destination address *is* provided. I cannot reproduce it at all. If I create the same configuration (to the extent rather scarce details you provided allow) I get "Network is unreachable" when explicitly selecting the second interface without default route. And for the first interface I either get timeout or address unreachable depending on which gateway I set (I get address unreachable if I set dummy gateway with real public address).

Something must be different on your system. You say VPS - is it some sort of container? What platform does it run on?

It's a KVM virtual machine, with openSUSE Leap 15, which I installed by myself from ISO. "to the extent rather scarce details you provided allow" - I tried to provide enough info to describe the situation, without dumping half of my system in one email. If you need more info, I can provide it, just tell me what you need.

...

It's not only ping failing. "curl -6 sixxs.net" gives error: curl: (7) Failed to connect to sixxs.net port 80: No route to host

-- Adam Mizerski

W dniu 15.09.2018 o 09:18, Andrei Borzenkov pisze:

11.09.2018 01:21, Adam Mizerski пишет:

...

W dniu 09.09.2018 o 08:49, Andrei Borzenkov pisze:

...

09.09.2018 00:13, Adam Mizerski пишет:

...

...

Earlier you said that connectivity to outside world works correctly. Does it mean only ping fails? Can you connect to any IPv6 using TCP/UDP?

strace of both successful and failed ping invocations would be useful.

Pings and everything works correctly before I start (and after I stop) WireGuard interface.

here are strace outputs: https://etam.homelinux.com/index.php/s/wmJxaDps6Y3KrA9

sendto() fails immediately with EDESTADDRREQ which makes no sense because destination address *is* provided. I cannot reproduce it at all. If I create the same configuration (to the extent rather scarce details you provided allow) I get "Network is unreachable" when explicitly selecting the second interface without default route. And for the first interface I either get timeout or address unreachable depending on which gateway I set (I get address unreachable if I set dummy gateway with real public address).

Something must be different on your system. You say VPS - is it some sort of container? What platform does it run on?

It's a KVM virtual machine, with openSUSE Leap 15, which I installed by myself from ISO.

"to the extent rather scarce details you provided allow" - I tried to provide enough info to describe the situation, without dumping half of my system in one email. If you need more info, I can provide it, just tell me what you need.

Full "ip a" and "ip r" output would be helpful. Also full output of "journalctl -b" in case there is something in logs.

Here you go: https://etam.homelinux.com/index.php/s/EWRQ8XP2rxcASX6

...

...

...

It's not only ping failing. "curl -6 sixxs.net" gives error: curl: (7) Failed to connect to sixxs.net port 80: No route to host

-- Adam Mizerski

W dniu 15.09.2018 o 15:48, Andrei Borzenkov pisze:

15.09.2018 14:09, Adam Mizerski пишет:

...

W dniu 15.09.2018 o 09:18, Andrei Borzenkov pisze:

...

11.09.2018 01:21, Adam Mizerski пишет:

...

W dniu 09.09.2018 o 08:49, Andrei Borzenkov pisze:

...

09.09.2018 00:13, Adam Mizerski пишет:

...

> > Earlier you said that connectivity to outside world works correctly. > Does it mean only ping fails? Can you connect to any IPv6 using TCP/UDP? > > strace of both successful and failed ping invocations would be useful. >

Pings and everything works correctly before I start (and after I stop) WireGuard interface.

here are strace outputs: https://etam.homelinux.com/index.php/s/wmJxaDps6Y3KrA9

sendto() fails immediately with EDESTADDRREQ which makes no sense because destination address *is* provided. I cannot reproduce it at all. If I create the same configuration (to the extent rather scarce details you provided allow) I get "Network is unreachable" when explicitly selecting the second interface without default route. And for the first interface I either get timeout or address unreachable depending on which gateway I set (I get address unreachable if I set dummy gateway with real public address).

Something must be different on your system. You say VPS - is it some sort of container? What platform does it run on?

It's a KVM virtual machine, with openSUSE Leap 15, which I installed by myself from ISO.

"to the extent rather scarce details you provided allow" - I tried to provide enough info to describe the situation, without dumping half of my system in one email. If you need more info, I can provide it, just tell me what you need.

Full "ip a" and "ip r" output would be helpful. Also full output of "journalctl -b" in case there is something in logs.

Here you go: https://etam.homelinux.com/index.php/s/EWRQ8XP2rxcASX6

Sep 15 12:57:41 etam-hitme wg-quick[2210]: [#] ip link set mtu 1420 dev wg0 Sep 15 12:57:41 etam-hitme wg-quick[2210]: [#] ip link set wg0 up Sep 15 12:57:41 etam-hitme wg-quick[2210]: [#] wg set wg0 fwmark 51820 Sep 15 12:57:41 etam-hitme wg-quick[2210]: [#] ip -6 route add ::/0 dev wg0 table 51820 Sep 15 12:57:41 etam-hitme wg-quick[2210]: [#] ip -6 rule add not fwmark 51820 table 51820 Sep 15 12:57:41 etam-hitme wg-quick[2210]: [#] ip -6 rule add table main suppress_prefixlength 0

This makes wg0 default interface while allowing only packets to directly connected LAN on other interfaces. My understanding is that you need exactly opposite. You probably need to change your WireGuard configuration to not declare wg0 as default route.

Thanks! I just learned about marking packets and having more than one routing table. https://i.kym-cdn.com/photos/images/original/000/288/648/776.gif Anyway, to keep things simple, I've added "Table = off" line to wireguard config, which disables messing with routing tables in wg-quick script. Now, with routing set manually, as I described before, outgoing connections are *not* broken. https://i.kym-cdn.com/photos/images/original/000/409/799/83b.jpg But I'm still struggling to get routing working. So far no success with having connection between my subnet and the rest of the world. Everywhere I read, it should be as easy as setting "sysctl -w net.ipv6.conf.all.forwarding=1", but for some unknown to me reason, it's not. Anyway, I still have some ideas to explore, so I'll write back when I have more details. -- Adam Mizerski

W dniu 19.09.2018 o 07:01, Andrei Borzenkov pisze:

19.09.2018 00:09, Adam Mizerski пишет:

...

But I'm still struggling to get routing working. So far no success with having connection between my subnet and the rest of the world. Everywhere I read, it should be as easy as setting "sysctl -w net.ipv6.conf.all.forwarding=1", but for some unknown to me reason, it's not.

You cannot just take arbitrary subnet and expect it to work. Your public IP must be registered as gateway into this subnet, otherwise outside world simply does not know where to forward packets addressed to this subnet.

I know. My VPS operator claims that everything is set up correctly on their side, so I'm trying to verify this. -- Adam Mizerski