On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang wrote:

On 05/30/2017 11:22 AM, Greg Freemyer wrote:

...

I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.

I'm on the client that is uploading the data to a Windows fileserver.

They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:

...

netstat -in; sleep 120; netstat -in

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.

It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.

The machines are just a few feet apart and connected to the same 1Gbit switch.

Hi Greg,

That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?

Regards, Lew

There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long. I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem. Thanks Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Tue, May 30, 2017 at 7:27 PM, Lew Wolfgang wrote:

On 05/30/2017 03:30 PM, Greg Freemyer wrote:

...

On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang wrote:

...

On 05/30/2017 11:22 AM, Greg Freemyer wrote:

...

I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.

I'm on the client that is uploading the data to a Windows fileserver.

They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:

...

netstat -in; sleep 120; netstat -in

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.

It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.

The machines are just a few feet apart and connected to the same 1Gbit switch.

Hi Greg,

That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?

Regards, Lew

There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long.

I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem.

I think something's wrong. You seem to be dropping almost 6% of your packets.

For example, here's one of my boxes with an uptime of 52-days

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg enp5s0f0 1500 0 11758029247 0 0 0 52174553831 0 0 0 BMRU eth0 1500 0 4188629239 0 31 0 12046855552 0 0 0 BMRU lo 65536 0 376527440 0 0 0 376527440 0 0 0 LRU

The enp5s0f0 interface connects to an RFC-1918 private subnet. So the outside eth0 interface has a drop rate of .0000000074, or so.

Agreed. But what does that actually mean. Network gear is store and forward, so I assume a packet checksum is verified at every hop, including the switch that is mounted to my workstation. And then when a TCP packet gets to my openSUSE 42.2 client the sequence tracker for the TCP packets is checked. Does a 6% drop rate mean 6% of my packets are dropped due to a bad packet checksum? Or does it mean 6% of my packets are TCP packets coming in out of order and resends have to be requested? Thanks Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/30/2017 08:29 PM, Lew Wolfgang wrote:

Dropped packets could be caused by the buffers overflowing in the switch or receiving NIC. Frames lost at the switch wouldn't be counted because the computer would have no way to know about them.

Dropped packets aren't out-of-order packets, but they do require retransmits to get the order right. Out-of-order packets wouldn't necessarily cause retransmits, I'd think.

They might, if so delayed they're assumed lost. However, normally with TCP, the acks tell the sender what's expected next. If, after a period of time, the ack doesn't advance, the sender assumes a loss and resends. On receipt, the receiver can then advance the ack through all received sequential packets. Again, that;s a couple of levels up the stack. As for error checking, it can happen at the Ethernet frame level, the IP packet level in IPv4 only and TCP/UDP level in IPv4 and IPv6. With IPv6 UDP checking is mandatory, whereas it was optional in IPv4. There is no IP error checking in IPv6. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Hello, On Tue, 30 May 2017, Lew Wolfgang wrote:

On 05/30/2017 03:30 PM, Greg Freemyer wrote:

...

On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang wrote:

...

...

They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:

...

netstat -in; sleep 120; netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets. [..] That seems a bit high to me too. Do you have another switch to drop into

On 05/30/2017 11:22 AM, Greg Freemyer wrote: [..] place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?

Gigabit Ethernet _mandates_ Auto-MDIx. So Greg, you can use any cable long enough and CAT 5 or higher.

I think something's wrong. You seem to be dropping almost 6% of your packets. [..] The enp5s0f0 interface connects to an RFC-1918 private subnet. So the outside eth0 interface has a drop rate of .0000000074, or so.

Nope. $ calc '(105198-105154)/(1759965-1446805)' ~0.00014050325712096053 So, it's just 0.014% being dropped. HTH, -dnh -- I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone. -- Bjarne Stroustrup -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2017-05-31 13:12, Anton Aylward wrote:

On 30/05/17 10:07 PM, David Haller wrote:

...

So, it's just 0.014% being dropped.

I keep saying "Context is Everything".

Noise on the wire?

Sure. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" (Minas Tirith))

On 05/30/2017 06:30 PM, Greg Freemyer wrote:

...

That seems a bit high to me too. Do you have another switch to drop into

...

place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?

Regards, Lew There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long.

I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem.

Dropped packets generally mean corrupt. That could be caused by a defective interface, at either end, or a poor connection. I agree some swapping things might reveal the issue. BTW, with gigabit, there's no need for a cross over, as all 4 pairs are used in both directions. A properly functioning switch shouldn't pass corrupt frames. I don't think modern switches use cut through anymore. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 05/31/2017 01:44 AM, David Haller wrote:

Hello,

On Wed, 31 May 2017, Per Jessen wrote:

...

...

I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets. I looked at some of my interface stats, just a random selection, and

Greg Freemyer wrote: they all show dropped or overrun packets. On a plain desktop machine, no one working on it, 1 packet overrun in 2 minutes, no drops. On the opensuse mail list server, 12 drops in 2 minutes. On a busy mirror machine - 58 overrun. Good point. Looking at my WAN interface:

RX packets:11064938 errors:0 dropped:1015 overruns:0 frame:0 TX packets:5446401 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:16478062049 (15714.7 Mb) TX bytes:301289242 (287.3 Mb)

That's ~0.009% packets dropped, so about 2/3 droprate of what Greg gets. Same ballpark I'd say.

I was looking at the total number of drops over the total number of received packets, not just the two-minute delta. Here's Greg's second reading: Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU That comes out to about 5.9% since the stack was started, if I'm not mistaken. The old rule-of-thumb was to start worrying when rates exceed 1%. I remember that collision rates on a non-switched bus network could get up to about 2%, but you shouldn't see too many problems when using a switch, especially when just connecting between two ports on the same switch. If I had Greg's numbers I'd assume something was wrong. BTW, here are the numbers from my home machine that I'm using now. It's been up for 12-days. Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg br0 1500 0 0 0 0 0 87221 0 0 0 BMRU enp2s0 1500 0 12010582 0 0 0 5266048 0 0 0 BMRU lo 65536 0 78 0 0 0 78 0 0 0 LRU Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Wed, May 31, 2017 at 12:02 PM, Per Jessen wrote:

Lew Wolfgang wrote:

...

I was looking at the total number of drops over the total number of received packets, not just the two-minute delta. Here's Greg's second reading:

Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

That comes out to about 5.9% since the stack was started, if I'm not mistaken. The old rule-of-thumb was to start worrying when rates exceed 1%.

Isn't that numnber of dropped messages though? I see dropped messages on virtually all interfaces (virtually all = just checked a few of them).

From the above:

RX-OK 1759965 RX-DRP 105198 5.9% dropped I saw number like that first which is what triggered me to do the 2-min check before and after. I had a lot of local traffic during that 2 minutes. I don't really know what all the traffic was prior to that 2 minutes. Might have been web browsing, or zypper up, etc. If all 105198 packet drops occurred in either my local switch, or my NIC, then it seems excessive. By chance, I've rebooted that PC since this discussion started. It now reports:

netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 88364752 0 21978 0 90246677 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

So, 22,000 dropped packets out of 88 million. Far less than 1%. I did a lot of SMB uploading since the reboot, thus the 88 million received packets. Most would have been simple ACKs for the transmitted packets that had a data payload. Not sure why I had a 6% cumulative drop rate earlier. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

John Andersen wrote:

On 05/31/2017 09:48 AM, Greg Freemyer wrote:

...

By chance, I've rebooted that PC since this discussion started. It now reports:

...

netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 88364752 0 21978 0 90246677 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU

So, 22,000 dropped packets out of 88 million.

I Still think something is amiss.

I just checked three machines on my network, all connected by switches, plus one machine connected directly to my cable modem (not my internal network) and I have zero drops on any machine (other than the same two drops I reported yesterday).

These are fairly heavily used machines. (Except the Raspberry pi connected directly to the cable modem (acting as my backup wifi access-point - it only sees significant traffic when I stream video via it).

Maybe your numbers don't yet rise to the level of being an issue, but in a modern network where you can see both ends with the naked eye, you shouldn't see any.

My home server, which I rebooted 30 days ago for mains power failure, has 52 RX-DRPs (out of 6,458,705) zero TX-Drps (out of 14,862,261).

Some more examples - on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy. On a mirror+squid, uptime 45days, I see 7% drop on one interface. On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ... two webservers, uptime 1600 days and 92 days, 0 rx drops. The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts. -- Per Jessen, Zürich (18.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, Jun 1, 2017 at 6:25 AM, Carlos E. R. wrote:

On 2017-06-01 08:31, Per Jessen wrote:

...

Some more examples -

on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.

On a mirror+squid, uptime 45days, I see 7% drop on one interface.

On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...

two webservers, uptime 1600 days and 92 days, 0 rx drops.

The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.

Question.

The network "hardware" in the xen guest, is real or emulated?

Ignoring Per's issue. My PC with the 6% dropped packets over some period of time is a physical machine with a X99 MB and 64GB of RAM. No VM involved at all. The 1Gbit NIC is built into the MB. The CPU is 6 cores, 12 threads (via hyper-threading).

A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.

I wonder if my X99 MB truly has all that functionality? Also, when I'm beating my NIC to death, I'm also hitting USB-3 hard. I likely pulled down a couple TB of data via SMB and placed it on a 5TB USB-3 drive during the time I lost all those packets. ie. rsync <SMB host> <USB-3 drive> (or equivalent drag & drop via dolphin) It could well be that heavy USB-3 usage is interfering with my NIC? Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, 1 Jun 2017 14:19, Greg Freemyer wrote:

On Thu, Jun 1, 2017 at 6:25 AM, Carlos E. R. wrote:

...

On 2017-06-01 08:31, Per Jessen wrote:

...

Some more examples -

on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.

On a mirror+squid, uptime 45days, I see 7% drop on one interface.

On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...

two webservers, uptime 1600 days and 92 days, 0 rx drops.

The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.

Question.

The network "hardware" in the xen guest, is real or emulated?

Ignoring Per's issue.

My PC with the 6% dropped packets over some period of time is a physical machine with a X99 MB and 64GB of RAM. No VM involved at all. The 1Gbit NIC is built into the MB.

The CPU is 6 cores, 12 threads (via hyper-threading).

...

A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.

I wonder if my X99 MB truly has all that functionality?

Also, when I'm beating my NIC to death, I'm also hitting USB-3 hard. I likely pulled down a couple TB of data via SMB and placed it on a 5TB USB-3 drive during the time I lost all those packets.

ie. rsync <SMB host> <USB-3 drive> (or equivalent drag & drop via dolphin)

It could well be that heavy USB-3 usage is interfering with my NIC?

Hmm, looking at Intels docu for the X99 chipset, it could well be. The X99 PCH has a dedicated Gbit MAC integrated, and all the USBs (3.0 and 2.0) are also handled by the PCH. Also on the PCH are all the Sata, PCIe 2.0, M.2, and the Audio. All that has to pass through the DMI 2.0 x4 (16 or 20Gbit) connect to the CPU. The PCIe 3.0 and the RAM are connected directly to the CPU. It could well be that the PCH has hiccups in the flow-control, and faces the decission of mishandling the USB3 timing or dropping eth frames. https://en.wikipedia.org/wiki/Intel_X99 (for overview) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2017-06-01 18:53, Yamaban wrote:

Hmm, looking at Intels docu for the X99 chipset, it could well be. The X99 PCH has a dedicated Gbit MAC integrated, and all the USBs (3.0 and 2.0) are also handled by the PCH. Also on the PCH are all the Sata, PCIe 2.0, M.2, and the Audio. All that has to pass through the DMI 2.0 x4 (16 or 20Gbit) connect to the CPU.

The PCIe 3.0 and the RAM are connected directly to the CPU.

It could well be that the PCH has hiccups in the flow-control, and faces the decission of mishandling the USB3 timing or dropping eth frames.

https://en.wikipedia.org/wiki/Intel_X99 (for overview)

Can the network hardware read/write directly to a RAM block, without using the main CPU, through that DMI thing? That is, the network hardware gets a frame, writes it to the memory block that it was told earlier to use, then ping the CPU to tell it op completed. Can it do that? Or does it simply tell the CPU that it got a frame, please pick it up, word by word? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)

On 2017-06-01 14:22, Per Jessen wrote:

Carlos E. R. wrote:

...

On 2017-06-01 08:31, Per Jessen wrote:

...

Some more examples -

on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.

On a mirror+squid, uptime 45days, I see 7% drop on one interface.

On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...

two webservers, uptime 1600 days and 92 days, 0 rx drops.

The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.

Question.

The network "hardware" in the xen guest, is real or emulated?

They're virtual network devices, bridged with the physical device.

...

Because if it is emulated or virtual, not real hardware, it depends on the host CPU having time to attend it at the precise instant it is needed. A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.

The physical card still works the way it always has, but there is obvioulsy more code to be executed to get a network interrupt serviced in a guest. Something like that.

I have to wonder at how/if the card can write to ram that belongs to the guest. :-? :-o -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)