On some servers we rent at Hetzner, we have recently (in the last month) begun to run Xen - and the Xen guests obviously have "made up" MAC addresses, e.g. 00:16:3e:bb:ac:82. For IPv4, we have enabled proxy_arp on the bridge interface, works well as always. We have not yet configuredany IPv6 in any of the guests, only on the xen host. The guests only have a link-local address. In this config, the only MAC addresses that should be seen on the network that of the only real interface, eth0/br0. However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses. I figured out we probably needed to enable ipv6 ndp - neighbour discovery protocol proxy : setting /proc/sys/net/ipv6/conf/*/proxy_ndp = 1 This did not have the desired effect, so I am clearly missing something? -- Per Jessen, Zürich (17.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland.
On 2021-04-27 8:04 a.m., Per Jessen wrote:
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
What do they mean by "leaking out"? How is that happening. MAC and link local addresses are never seen off the local network, unless something is relaying them.
James Knott wrote:
On 2021-04-27 8:04 a.m., Per Jessen wrote:
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
What do they mean by "leaking out"? How is that happening.
Well, because we at first had forgotten to enable ND proxying. Without that, the external network will see whatever happens on the bridge device.
MAC and link local addresses are never seen off the local network, unless something is relaying them.
The bridge device. -- Per Jessen, Zürich (17.3°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland.
James Knott wrote:
On 2021-04-27 8:47 a.m., Per Jessen wrote:
The bridge device.
What is the bridge device?
'br0' is the bridge device: 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000 link/ether 6c:62:6d:a0:75:aa brd ff:ff:ff:ff:ff:ff inet6 fe80::6e62:6dff:fea0:75aa/64 scope link valid_lft forever preferred_lft forever 3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 6c:62:6d:a0:75:aa brd ff:ff:ff:ff:ff:ff inet 88.198.24.136/27 brd 88.198.24.159 scope global br0 valid_lft forever preferred_lft forever inet 88.198.172.121/29 scope global br0 valid_lft forever preferred_lft forever inet 192.168.77.1/24 brd 192.168.77.255 scope global br0 valid_lft forever preferred_lft forever inet 88.198.172.125/29 scope global secondary br0 valid_lft forever preferred_lft forever inet 88.198.172.126/29 scope global secondary br0 valid_lft forever preferred_lft forever inet6 2a01:4f8:130:232d::99/64 scope global valid_lft forever preferred_lft forever inet6 2a01:4f8:130:232d::24/64 scope global valid_lft forever preferred_lft forever inet6 2a01:4f8:130:232d::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::6e62:6dff:fea0:75aa/64 scope link valid_lft forever preferred_lft forever
Bridges (essentially a 2 port switch) haven't been common for many years. However, it sounds like it's the cause of your problem.
It's needed in virtual hosts. -- Per Jessen, Zürich (19.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland.
On 2021-04-27 10:04 a.m., Per Jessen wrote:
It's needed in virtual hosts.
Where, in relation to the host, are the complaints coming from? Virtual machine MACs should appear on the local LAN. I run Virtualbox here, with the NIC in bridge mode. I see the MAC addresses of the VMs on the wire as I would with any real device.
James Knott wrote:
On 2021-04-27 10:04 a.m., Per Jessen wrote:
It's needed in virtual hosts.
Where, in relation to the host, are the complaints coming from?
From the external network, where a firewall or filter only accepts the real MAC address.
Virtual machine MACs should appear on the local LAN. I run Virtualbox here, with the NIC in bridge mode. I see the MAC addresses of the VMs on the wire as I would with any real device.
Yes, but that is not acceptable here, hence I have enabled proxy_arp in ipv4, which just works. With proxy_arp, the external network only 'sees' the real MAC address, not the virtual. I have not had to do this with IPv6 before, and it seems to work a bit differently. -- Per Jessen, Zürich (20.5°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.
On 2021-04-27 10:19 a.m., Per Jessen wrote:
From the external network, where a firewall or filter only accepts the real MAC address.
Is that connected to the same LAN as the VMs? If so, that's entirely normal and the problem is with the configuration of it. It should behave exactly the same way as though those VMs were real devices.
James Knott wrote:
On 2021-04-27 10:19 a.m., Per Jessen wrote:
From the external network, where a firewall or filter only accepts the real MAC address.
Is that connected to the same LAN as the VMs? If so, that's entirely normal and the problem is with the configuration of it.
Well, Hetzner disagrees with your view of "entirely normal" - they only accept real MAC addresses on their network, and that works fine with IPv4. James, it is not a problem, except for how to configure it with IPv6. -- Per Jessen, Zürich (20.6°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.
On Tue, Apr 27, 2021 at 3:22 PM Per Jessen <per@computer.org> wrote:
On some servers we rent at Hetzner, we have recently (in the last month) begun to run Xen - and the Xen guests obviously have "made up" MAC addresses, e.g. 00:16:3e:bb:ac:82.
For IPv4, we have enabled proxy_arp on the bridge interface, works well as always. We have not yet configuredany IPv6 in any of the guests, only on the xen host. The guests only have a link-local address.
In this config, the only MAC addresses that should be seen on the network that of the only real interface, eth0/br0.
What does it mean? eth0 is slave interface in br0? Where and how guests are connected? You really need to explain network topology better.
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
How exactly can you tell it? Do you have any packet capture?
I figured out we probably needed to enable ipv6 ndp - neighbour discovery protocol proxy :
setting /proc/sys/net/ipv6/conf/*/proxy_ndp = 1
This did not have the desired effect, so I am clearly missing something?
NDP proxy works between two interfaces (and you need to explicitly define which addresses are proxied). If guests are connected to the same br0 as eth0, then guests are on the same physical link and there is nothing to proxy. If guests are connected to something else, how comes MAC leaks? You really need to explain your topology.
Andrei Borzenkov wrote:
On Tue, Apr 27, 2021 at 3:22 PM Per Jessen <per@computer.org> wrote:
On some servers we rent at Hetzner, we have recently (in the last month) begun to run Xen - and the Xen guests obviously have "made up" MAC addresses, e.g. 00:16:3e:bb:ac:82.
For IPv4, we have enabled proxy_arp on the bridge interface, works well as always. We have not yet configuredany IPv6 in any of the guests, only on the xen host. The guests only have a link-local address.
In this config, the only MAC addresses that should be seen on the network that of the only real interface, eth0/br0.
What does it mean? eth0 is slave interface in br0? Where and how guests are connected? You really need to explain network topology better.
Yes, eth0 is bridged with the virtual interfaces into br0.
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
How exactly can you tell it? Do you have any packet capture?
Yes, I ran a tcpdump this morning to confirm what Hetzner told me. a tcpdump on "br0", looking for the two local link addresses from my two DomUs. tcpdump -n -i br0 host fe80::216:3eff:febb:ac7c or \ host fe80::216:3eff:febb:ac82
NDP proxy works between two interfaces (and you need to explicitly define which addresses are proxied).
Aha. I'll have to study that.
If guests are connected to the same br0 as eth0, then guests are on the same physical link and there is nothing to proxy.
Well, except for the "virtual" MAC addresses.
If guests are connected to something else, how comes MAC leaks?
You really need to explain your topology.
It is really simple - a single machine, a xen host. Single ethernet interface in the Dom0, bridged with the virtual interfaces into br0. -- Per Jessen, Zürich (19.6°C)
Per Jessen wrote:
Andrei Borzenkov wrote:
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
How exactly can you tell it? Do you have any packet capture?
Yes, I ran a tcpdump this morning to confirm what Hetzner told me. a tcpdump on "br0", looking for the two local link addresses from my two DomUs.
see attached. The DomU with 'fe80::216:3eff:febb:ac82' is currently down. It was running 15.3, but failed to boot after the latest 'zypper dup'. -- Per Jessen, Zürich (19.8°C)
On 27.04.2021 17:30, Per Jessen wrote:
Per Jessen wrote:
Andrei Borzenkov wrote:
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
How exactly can you tell it? Do you have any packet capture?
Yes, I ran a tcpdump this morning to confirm what Hetzner told me. a tcpdump on "br0", looking for the two local link addresses from my two DomUs.
see attached.
The DomU with 'fe80::216:3eff:febb:ac82' is currently down. It was running 15.3, but failed to boot after the latest 'zypper dup'.
Well, this does not show any MAC address, so is rather uninteresting. I suspect there is some misunderstanding and proxy_arp is actually red herring. You cannot use proxy ARP/NDP to hide MAC addresses on bridge. Even assuming proxy works, it will only handle ARP/NDP - it will *not* replace MAC address in any other packet. When proxy ARP/NDP is used, actual packet delivery after LL address has been determined is handled by L3 switching which explains why MAC addresses do not leak (because packet delivery remains local to each LAN segment). But you are on bridge so no L3. Proxied host will attempt to send packet directly using its own MAC address. So to actually hide guests MAC you would need separate external interface and separate bridge for guests and proxy each guest on external interface via bridge. You will need (host) route to each guest also. And I do not even want to think about DHCP or any other similar protocol :)
Andrei Borzenkov wrote:
On 27.04.2021 17:30, Per Jessen wrote:
Per Jessen wrote:
Andrei Borzenkov wrote:
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
How exactly can you tell it? Do you have any packet capture?
Yes, I ran a tcpdump this morning to confirm what Hetzner told me. a tcpdump on "br0", looking for the two local link addresses from my two DomUs.
see attached.
The DomU with 'fe80::216:3eff:febb:ac82' is currently down. It was running 15.3, but failed to boot after the latest 'zypper dup'.
Well, this does not show any MAC address, so is rather uninteresting.
I can post what Hetzner sent me, sorry about the folding: -----------------%<----------------- 2021-04-27 08:18:33 2a01:4f8::a:22:2:639 info local3 fpc0 PFE_FW_SYSLOG_ETH_IP6_ICMP: FW: ge-0/0/29.0 A 86dd 00:16:3e:bb:ac:82 -> 4c:16:fc:c8:e2:c2 icmpv6 SA fe80:0:0:0:216:3eff:febb:ac82 -> DA fe80:0:0:0:0:0:0:1 type 136 code 0 (1 packets) 2021-04-27 08:18:34 2a01:4f8::a:22:2:639 info local3 fpc0 PFE_FW_SYSLOG_ETH_IP6_ICMP: FW: ge-0/0/29.0 A 86dd 00:16:3e:bb:ac:82 -> 4c:16:fc:c8:e2:c2 icmpv6 SA fe80:0:0:0:216:3eff:febb:ac82 -> DA fe80:0:0:0:0:0:0:1 type 136 code 0 (1 packets) -----------------%<----------------- Just now trying to see what I get with 'tcpdump ether'.
I suspect there is some misunderstanding and proxy_arp is actually red herring.
You cannot use proxy ARP/NDP to hide MAC addresses on bridge. Even assuming proxy works, it will only handle ARP/NDP - it will *not* replace MAC address in any other packet.
Hmmm. Okay.
When proxy ARP/NDP is used, actual packet delivery after LL address has been determined is handled by L3 switching which explains why MAC addresses do not leak (because packet delivery remains local to each LAN segment). But you are on bridge so no L3. Proxied host will attempt to send packet directly using its own MAC address.
hmm, yes, that is what it looks like with 'tcpdump ether' too. Okay, I'll have to use NAT'ing instead I guess. -- Per Jessen, Zürich (20.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland.
On Tue, 27 Apr 2021 16:02:31 +0200 Per Jessen <per@computer.org> wrote:
Andrei Borzenkov wrote:
On Tue, Apr 27, 2021 at 3:22 PM Per Jessen <per@computer.org> wrote:
On some servers we rent at Hetzner, we have recently (in the last month) begun to run Xen - and the Xen guests obviously have "made up" MAC addresses, e.g. 00:16:3e:bb:ac:82.
For IPv4, we have enabled proxy_arp on the bridge interface, works well as always. We have not yet configuredany IPv6 in any of the guests, only on the xen host. The guests only have a link-local address.
In this config, the only MAC addresses that should be seen on the network that of the only real interface, eth0/br0.
What does it mean? eth0 is slave interface in br0? Where and how guests are connected? You really need to explain network topology better.
Yes, eth0 is bridged with the virtual interfaces into br0.
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
How exactly can you tell it? Do you have any packet capture?
Yes, I ran a tcpdump this morning to confirm what Hetzner told me. a tcpdump on "br0", looking for the two local link addresses from my two DomUs.
tcpdump -n -i br0 host fe80::216:3eff:febb:ac7c or \ host fe80::216:3eff:febb:ac82
NDP proxy works between two interfaces (and you need to explicitly define which addresses are proxied).
Aha. I'll have to study that.
If guests are connected to the same br0 as eth0, then guests are on the same physical link and there is nothing to proxy.
Well, except for the "virtual" MAC addresses.
If guests are connected to something else, how comes MAC leaks?
You really need to explain your topology.
It is really simple - a single machine, a xen host. Single ethernet interface in the Dom0, bridged with the virtual interfaces into br0.
Are these of any relevance? http://wiki.stocksy.co.uk/wiki/IPv6%2BXen_on_a_Hetzner_server_with_routing_t... https://gist.github.com/pulsar256/5f901a7fa903fae6332e
On Tue, 27 Apr 2021 16:14:31 +0100 Dave Howorth <dave@howorth.org.uk> wrote:
On Tue, 27 Apr 2021 16:02:31 +0200 Per Jessen <per@computer.org> wrote:
Andrei Borzenkov wrote:
On Tue, Apr 27, 2021 at 3:22 PM Per Jessen <per@computer.org> wrote:
On some servers we rent at Hetzner, we have recently (in the last month) begun to run Xen - and the Xen guests obviously have "made up" MAC addresses, e.g. 00:16:3e:bb:ac:82.
For IPv4, we have enabled proxy_arp on the bridge interface, works well as always. We have not yet configuredany IPv6 in any of the guests, only on the xen host. The guests only have a link-local address.
In this config, the only MAC addresses that should be seen on the network that of the only real interface, eth0/br0.
What does it mean? eth0 is slave interface in br0? Where and how guests are connected? You really need to explain network topology better.
Yes, eth0 is bridged with the virtual interfaces into br0.
However, Hetzner is complaining that some of our guest MAC addresses are "leaking out". AFAICT, this is happening in neighbour solitications and advertisements, with the link-local addresses.
How exactly can you tell it? Do you have any packet capture?
Yes, I ran a tcpdump this morning to confirm what Hetzner told me. a tcpdump on "br0", looking for the two local link addresses from my two DomUs.
tcpdump -n -i br0 host fe80::216:3eff:febb:ac7c or \ host fe80::216:3eff:febb:ac82
NDP proxy works between two interfaces (and you need to explicitly define which addresses are proxied).
Aha. I'll have to study that.
If guests are connected to the same br0 as eth0, then guests are on the same physical link and there is nothing to proxy.
Well, except for the "virtual" MAC addresses.
If guests are connected to something else, how comes MAC leaks?
You really need to explain your topology.
It is really simple - a single machine, a xen host. Single ethernet interface in the Dom0, bridged with the virtual interfaces into br0.
Are these of any relevance?
http://wiki.stocksy.co.uk/wiki/IPv6%2BXen_on_a_Hetzner_server_with_routing_t...
Or this? https://blog.kumina.nl/2011/06/proxying-neighbor-discovery-messages-ndproxy/
Dave Howorth wrote:
Are these of any relevance?
http://wiki.stocksy.co.uk/wiki/IPv6%2BXen_on_a_Hetzner_server_with_routing_t...
That one looks useful, yes! Thanks! -- Per Jessen, Zürich (20.2°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland.
On Tue, 27 Apr 2021 17:51:36 +0200 Per Jessen <per@computer.org> wrote:
Dave Howorth wrote:
Are these of any relevance?
http://wiki.stocksy.co.uk/wiki/IPv6%2BXen_on_a_Hetzner_server_with_routing_t...
That one looks useful, yes! Thanks!
That's good/lucky :) I didn't even use hetzner in the search terms but most of the hits were about their systems! I just used ipv6 xen ndp proxy
participants (4)
-
Andrei Borzenkov -
Dave Howorth -
James Knott -
Per Jessen