On 4/18/23 21:03, Andrei Borzenkov wrote:

On 19.04.2023 02:14, Lew Wolfgang wrote:

...

Hi Folks,

We've been collecting data through four 1-GbE Ethernet interfaces configured as independent connections on different subnets. This has been working well enough for years.

But now someone has suggested bonding those four interfaces into one virtual interface.  I've never had any experience with this sort of thing, it looks like there's "bonding" and a newer thing called "teaming".  We're already getting 950-Mb/channel so we'd be looking more for load-leveling and fall-over reliability.

I wonder if any of you might have some suggestions or experience you might share?

Do not touch the running system.

If you describe what problems you have with the current implementation and what you are trying to achieve it may be possible to estimate whether bonding is suitable for it.

Thanks Andrei, my thoughts exactly.  We'll set up a test rig to try it out.  The existing system has been working well, I'm not sure what the engineer is thinking about.  I'll speak with him soon about it.  I don't think we can improve total bandwidth with bonding, I'm guessing that he's concerned about fall-back and load leveling.  I'll keep you informed. Regards, Lew

On 4/19/23 00:06, Roger Oberholtzer wrote:

On Wed, Apr 19, 2023 at 6:37 AM Lew Wolfgang wrote:

...

On 4/18/23 21:03, Andrei Borzenkov wrote:

...

On 19.04.2023 02:14, Lew Wolfgang wrote:

...

Hi Folks,

We've been collecting data through four 1-GbE Ethernet interfaces configured as independent connections on different subnets. This has been working well enough for years.

But now someone has suggested bonding those four interfaces into one virtual interface. I've never had any experience with this sort of thing, it looks like there's "bonding" and a newer thing called "teaming". We're already getting 950-Mb/channel so we'd be looking more for load-leveling and fall-over reliability.

I wonder if any of you might have some suggestions or experience you might share? We have been using bonding in a system that provides disk storage to a cluster of computers. It is a very data intensive analysis system. The clients are other openSUSE systems, as well as Windows systems. It has been working great.

After setting it up (in YaST) some devices will be identified as MASTERS and some as SLAVES. And a new interface that is the bound device is made. My setup looks like this;

1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 2: eth2: mtu 1500 qdisc mq master bond1 state UP group default qlen 1000 link/ether ac:1f:6b:a4:e5:88 brd ff:ff:ff:ff:ff:ff altname enp101s0f0 3: eth0: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:db:c4:66 brd ff:ff:ff:ff:ff:ff altname eno1 altname enp4s0 4: eth3: mtu 1500 qdisc mq master bond1 state UP group default qlen 1000 link/ether ac:1f:6b:a4:e5:88 brd ff:ff:ff:ff:ff:ff altname enp101s0f1 5: eth4: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether ac:1f:6b:a4:e5:8a brd ff:ff:ff:ff:ff:ff altname enp101s0f2 inet 172.22.1.1/16 brd 172.22.255.255 scope global eth4 valid_lft forever preferred_lft forever 6: eth1: mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether ac:1f:6b:db:c4:67 brd ff:ff:ff:ff:ff:ff altname eno2 altname enp5s0 7: eth5: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether ac:1f:6b:a4:e5:8b brd ff:ff:ff:ff:ff:ff altname enp101s0f3 8: eth6: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether ac:1f:6b:f5:dd:3c brd ff:ff:ff:ff:ff:ff altname enp102s0f0 altname ens7f0 inet 10.2.184.3/26 brd 10.2.184.63 scope global eth6 valid_lft forever preferred_lft forever 9: eth7: mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether ac:1f:6b:f5:dd:3d brd ff:ff:ff:ff:ff:ff altname enp102s0f1 altname ens7f1 10: eth8: mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether ac:1f:6b:f5:dd:3e brd ff:ff:ff:ff:ff:ff altname enp102s0f2 altname ens7f2 11: eth9: mtu 1500 qdisc mq master bond0 state UP group default qlen 1000 link/ether ac:1f:6b:f5:dd:3e brd ff:ff:ff:ff:ff:ff altname enp102s0f3 altname ens7f3 12: bond0: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ac:1f:6b:f5:dd:3e brd ff:ff:ff:ff:ff:ff inet 10.2.184.2/26 brd 10.2.184.63 scope global bond0 valid_lft forever preferred_lft forever 13: bond1: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether ac:1f:6b:a4:e5:88 brd ff:ff:ff:ff:ff:ff inet 10.2.184.50/26 brd 10.2.184.63 scope global bond1 valid_lft forever preferred_lft forever

Note that the SLAVE devices do not have their own IP address.

As to performance, all seems snappy. We have not exactly bench-marked it. But we feel the speed is what we would expect from the devices involved.

Be sure to verify that your switch works with bounding. I think that these days most do. But do check that.

Thanks Roger.  Do you have two "bond" interfaces?  Did yast set that up? Also, would you benefit from a larger MTU size?  We've been using 8192 on our data intensive interfaces and it does make a difference.  We can juggle some buffer sizes with sysctl.conf and with the larger MTU's we can get 950-Mb/sec per interface on a continuous basis. We experimented with larger MTU, but the switches we had wouldn't take it. Regards, Lew

On 2023/04/19 08:34, Per Jessen wrote:

for iSCSI over ethernet, we experimented with 9000, which is the max both nics and switches would take, but it caused to much latency. Not enough IO.

---- iSCSI didn't look especially efficient for high speed file transfers -- it looked pretty bad because instead of being able to send a large file with a large TCP window, all files seemed to be broken down into iSCSI's idea of a block size. I found 9kb packet sizes gave about 50% more throughput over CIFS/SMB. The limiting factor was CPU and number of send/rcv buffers on client. I had to reduce number of send buffers on a win7 client considerably to get reasonable latency as the send buffers filled up. More send buffers on client resulted in lower cpu load on client, but higher cpu load on linux server. As for latency -- you're probably right, 1.5kb would likely help w/latency, though other param, like coalescing and how long to wait to coalesce might also have a strong effect. Also -- strong effect -- speed of link. Takes alot less time to send a packet (of any size) over a 10Gb link vs. a 1Gb link. Also of note was that combining 2x10Gb links resulted in no throughput improvement *in my setup* due to larger CPU load in recombining/re-ordering traffic. Eventually, I've had to settle for 1.5kb packets for compat with IoT (like media equipment in my living room).

On 4/19/23 10:50, Andrei Borzenkov wrote:

On 19.04.2023 18:24, Lew Wolfgang wrote:

...

Thanks Roger.  Do you have two "bond" interfaces?  Did yast set that up?

YaST sets up what you tell it to set up.

Knowing nothing about bonding I was asking why two bonded interfaces show up.  Maybe they're going to two physically different storage arrays? Or is there some other reason? Regards, Lew

On Thu, Apr 20, 2023 at 8:38 AM Andrei Borzenkov wrote:

On Wed, Apr 19, 2023 at 9:31 PM Lew Wolfgang wrote:

...

On 4/19/23 10:50, Andrei Borzenkov wrote:

...

On 19.04.2023 18:24, Lew Wolfgang wrote:

...

Thanks Roger. Do you have two "bond" interfaces? Did yast set that up?

YaST sets up what you tell it to set up.

Knowing nothing about bonding I was asking why two bonded interfaces show up.

You need multiple interfaces if your host is connected to multiple separate L2 domains. Whether these interfaces are physical or logical is an entirely different question.

The primary use case for bonding is redundancy. Whether this is actually useful depends on your infrastructure and applications. While bonding may increase bandwidth, whether this will result in better performance again depends on your infrastructure, applications, traffic patterns etc.

Our setup is a data analysis system that uses a dozen or so computers to do various types of processing. Some paths are data. That is, disk I/O. We have this on one network. Another is user activity, which includes remote desktop access from India (we are in Sweden). That is on another network. Perhaps there are better ways to set this up. But it works for us. -- Roger Oberholtzer

On Thu, Apr 20, 2023 at 10:01 AM Lew Wolfgang wrote:

On 4/19/23 23:37, Andrei Borzenkov wrote:

...

On Wed, Apr 19, 2023 at 9:31 PM Lew Wolfgang wrote:

...

On 4/19/23 10:50, Andrei Borzenkov wrote:

...

On 19.04.2023 18:24, Lew Wolfgang wrote:

...

Thanks Roger. Do you have two "bond" interfaces? Did yast set that up?

YaST sets up what you tell it to set up. Knowing nothing about bonding I was asking why two bonded interfaces show up. You need multiple interfaces if your host is connected to multiple separate L2 domains. Whether these interfaces are physical or logical is an entirely different question.

The primary use case for bonding is redundancy. Whether this is actually useful depends on your infrastructure and applications. While bonding may increase bandwidth, whether this will result in better performance again depends on your infrastructure, applications, traffic patterns etc.

I heard from the engineer who's requesting this test and he's hoping to bond 8 interfaces together for bandwidth and easier software design.

Easier software design could alone be the valid reason.

We plan to connect two servers directly together to measure bandwidth, then run the eight interfaces through a switch and measure again.

Single flow (let's say TCP connection) will use one pair of physical interfaces on both sides and the result will be exactly the same as with physical interfaces. You need multiple flows that are distributed across multiple interface pairs to actually see *total* increased bandwidth.

I've got a feeling that the switch will slow things down and that the high network bandwidth will reveal other bottlenecks. It should be interesting in any case.

Regards, Lew

On 5/10/23 11:15, L A Walsh wrote:

On 2023/04/18 16:14, Lew Wolfgang wrote:

...

Hi Folks,

We've been collecting data through four 1-GbE Ethernet interfaces configured as independent connections on different subnets. This has been working well enough for years.

But now someone has suggested bonding those four interfaces into one virtual interface.  I've never had any experience with this sort of thing, it looks like there's "bonding" and a newer thing called "teaming".  We're already getting 950-Mb/channel so we'd be looking more for load-leveling and fall-over reliability.

I wonder if any of you might have some suggestions or experience you might share?

Regards, Lew honestly, i forget, but I thought teaming was more suitable for failover, at the expense of aggregate throughput, which you don't really need, it sounds like.

I don't remember(?) bridging having the same feature.

Could very well be true about teaming.  We were testing this without too much success when we had to move the hardware elsewhere for a more important test.  We'll return to this issue when we can.  Our test results so far have been rather disappointing. Regards, Lew