[opensuse] Is 40 dropped packets out of 300,000 high? This is wired cat5 LAN.
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets. I'm on the client that is uploading the data to a Windows fileserver. They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets. It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets. The machines are just a few feet apart and connected to the same 1Gbit switch. Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 11:22 AM, Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I'm on the client that is uploading the data to a Windows fileserver.
They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.
It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.
The machines are just a few feet apart and connected to the same 1Gbit switch.
Greg -- Greg Freemyer
Does seem like a lot. I was backing up some Virtual Machines to my NAS today so I thought I would watch that process.
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 183040 0 2 0 92579 0 0 0 BMRU lo 65536 0 1640 0 0 0 1640 0 0 0 LRU
...> Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 3017854 0 2 0 5671711 0 0 0 BMRU lo 65536 0 2004 0 0 0 2004 0 0 0 LRU
I was measuring from the sending end, haven't found any way to pull those numbers from the NAS box. Laptop 100Meg ---> NAS 1000Meg interface. Windows share protocol running on the NAS. But for a similar traffic load I had no indication of any additional drops. It has 1000Meg switch between them in an otherwise unbusy network. -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 11:22 AM, Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I'm on the client that is uploading the data to a Windows fileserver.
They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.
It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.
The machines are just a few feet apart and connected to the same 1Gbit switch.
Hi Greg, That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable? Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
On 05/30/2017 11:22 AM, Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I'm on the client that is uploading the data to a Windows fileserver.
They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.
It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.
The machines are just a few feet apart and connected to the same 1Gbit switch.
Hi Greg,
That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?
Regards, Lew
There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long. I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem. Thanks Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 03:30 PM, Greg Freemyer wrote:
On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
On 05/30/2017 11:22 AM, Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I'm on the client that is uploading the data to a Windows fileserver.
They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.
It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.
The machines are just a few feet apart and connected to the same 1Gbit switch.
Hi Greg,
That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?
Regards, Lew
There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long.
I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem.
I think something's wrong. You seem to be dropping almost 6% of your packets. For example, here's one of my boxes with an uptime of 52-days Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg enp5s0f0 1500 0 11758029247 0 0 0 52174553831 0 0 0 BMRU eth0 1500 0 4188629239 0 31 0 12046855552 0 0 0 BMRU lo 65536 0 376527440 0 0 0 376527440 0 0 0 LRU The enp5s0f0 interface connects to an RFC-1918 private subnet. So the outside eth0 interface has a drop rate of .0000000074, or so. Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, May 30, 2017 at 7:27 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
On 05/30/2017 03:30 PM, Greg Freemyer wrote:
On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
On 05/30/2017 11:22 AM, Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I'm on the client that is uploading the data to a Windows fileserver.
They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.
It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.
The machines are just a few feet apart and connected to the same 1Gbit switch.
Hi Greg,
That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?
Regards, Lew
There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long.
I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem.
I think something's wrong. You seem to be dropping almost 6% of your packets.
For example, here's one of my boxes with an uptime of 52-days
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg enp5s0f0 1500 0 11758029247 0 0 0 52174553831 0 0 0 BMRU eth0 1500 0 4188629239 0 31 0 12046855552 0 0 0 BMRU lo 65536 0 376527440 0 0 0 376527440 0 0 0 LRU
The enp5s0f0 interface connects to an RFC-1918 private subnet. So the outside eth0 interface has a drop rate of .0000000074, or so.
Agreed. But what does that actually mean. Network gear is store and forward, so I assume a packet checksum is verified at every hop, including the switch that is mounted to my workstation. And then when a TCP packet gets to my openSUSE 42.2 client the sequence tracker for the TCP packets is checked. Does a 6% drop rate mean 6% of my packets are dropped due to a bad packet checksum? Or does it mean 6% of my packets are TCP packets coming in out of order and resends have to be requested? Thanks Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 05:10 PM, Greg Freemyer wrote:
On Tue, May 30, 2017 at 7:27 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
On 05/30/2017 03:30 PM, Greg Freemyer wrote:
On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
On 05/30/2017 11:22 AM, Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I'm on the client that is uploading the data to a Windows fileserver.
They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets.
It's not outrageous, but it caught me off-guard. And I don't often look at the number of dropped packets.
The machines are just a few feet apart and connected to the same 1Gbit switch.
Hi Greg,
That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?
Regards, Lew
There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long.
I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem.
I think something's wrong. You seem to be dropping almost 6% of your packets.
For example, here's one of my boxes with an uptime of 52-days
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg enp5s0f0 1500 0 11758029247 0 0 0 52174553831 0 0 0 BMRU eth0 1500 0 4188629239 0 31 0 12046855552 0 0 0 BMRU lo 65536 0 376527440 0 0 0 376527440 0 0 0 LRU
The enp5s0f0 interface connects to an RFC-1918 private subnet. So the outside eth0 interface has a drop rate of .0000000074, or so. Agreed.
But what does that actually mean. Network gear is store and forward, so I assume a packet checksum is verified at every hop, including the switch that is mounted to my workstation.
And then when a TCP packet gets to my openSUSE 42.2 client the sequence tracker for the TCP packets is checked.
Does a 6% drop rate mean 6% of my packets are dropped due to a bad packet checksum?
Or does it mean 6% of my packets are TCP packets coming in out of order and resends have to be requested?
I think that a failed checksum will give you an RX-ERR. Dropped packets could be caused by the buffers overflowing in the switch or receiving NIC. I think that a rule-of-thumb for drops is they aren't really too significant until they reach 1% of your total packet count. At 6% I'd think that you'd have noticeable performance issues cause by the requirement to re-transmit those dropped packets. Dropped packets aren't out-of-order packets, but they do require retransmits to get the order right. Out-of-order packets wouldn't necessarily cause retransmits, I'd think. Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 08:29 PM, Lew Wolfgang wrote:
Dropped packets could be caused by the buffers overflowing in the switch or receiving NIC. Frames lost at the switch wouldn't be counted because the computer would have no way to know about them.
Dropped packets aren't out-of-order packets, but they do require retransmits to get the order right. Out-of-order packets wouldn't necessarily cause retransmits, I'd think.
They might, if so delayed they're assumed lost. However, normally with TCP, the acks tell the sender what's expected next. If, after a period of time, the ack doesn't advance, the sender assumes a loss and resends. On receipt, the receiver can then advance the ack through all received sequential packets. Again, that;s a couple of levels up the stack. As for error checking, it can happen at the Ethernet frame level, the IP packet level in IPv4 only and TCP/UDP level in IPv4 and IPv6. With IPv6 UDP checking is mandatory, whereas it was optional in IPv4. There is no IP error checking in IPv6. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 08:10 PM, Greg Freemyer wrote:
But what does that actually mean. Network gear is store and forward, so I assume a packet checksum is verified at every hop, including the switch that is mounted to my workstation.
Older switches could be configured for cut through, where the switch would start forwarding before it finished receiving a frame. This would allow corrupt frames through.
Or does it mean 6% of my packets are TCP packets coming in out of order and resends have to be requested? At that level it doesn't even understand IP. Those are Ethernet frame errors. TCP errors are 2 levels up the stack.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Tue, 30 May 2017, Lew Wolfgang wrote:
On 05/30/2017 03:30 PM, Greg Freemyer wrote:
On Tue, May 30, 2017 at 6:10 PM, Lew Wolfgang <wolfgang@sweet-haven.com> wrote:
They are on receive side which should not have much data in the packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1446805 0 105154 0 10527829 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
If you look at the delta over those 2 minutes, it's 44 dropped RX packets out of a little over 300,000 overall received packets. [..] That seems a bit high to me too. Do you have another switch to drop into
On 05/30/2017 11:22 AM, Greg Freemyer wrote: [..] place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?
Gigabit Ethernet _mandates_ Auto-MDIx. So Greg, you can use any cable long enough and CAT 5 or higher.
I think something's wrong. You seem to be dropping almost 6% of your packets. [..] The enp5s0f0 interface connects to an RFC-1918 private subnet. So the outside eth0 interface has a drop rate of .0000000074, or so.
Nope. $ calc '(105198-105154)/(1759965-1446805)' ~0.00014050325712096053 So, it's just 0.014% being dropped. HTH, -dnh -- I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone. -- Bjarne Stroustrup -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 30/05/17 10:07 PM, David Haller wrote:
So, it's just 0.014% being dropped.
I keep saying "Context is Everything". Noise on the wire? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/31/2017 07:12 AM, Anton Aylward wrote:
On 30/05/17 10:07 PM, David Haller wrote:
So, it's just 0.014% being dropped. I keep saying "Context is Everything".
Noise on the wire?
Sorry, didn't hear what you said. Too much noise on the wire. ;-) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 06:30 PM, Greg Freemyer wrote:
That seems a bit high to me too. Do you have another switch to drop into
place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?
Regards, Lew There are 4 computers on the switch and an uplink headed to the Internet connected router in the closet. So bypassing the switch isn't going to fly. Or at least not for long.
I likely have a spare switch I could grab, and plenty of cables. I just didn't want to start troubleshooting a non-existent problem.
Dropped packets generally mean corrupt. That could be caused by a defective interface, at either end, or a poor connection. I agree some swapping things might reveal the issue. BTW, with gigabit, there's no need for a cross over, as all 4 pairs are used in both directions. A properly functioning switch shouldn't pass corrupt frames. I don't think modern switches use cut through anymore. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/30/2017 05:10 PM, Lew Wolfgang wrote:
Hi Greg,
That seems a bit high to me too. Do you have another switch to drop into place as a test? Maybe try some different cables too? You might be able to eliminate the switch entirely if your NIC's auto sense. That, or try a cross-over cable?
Regards, Lew
In addition to the switch/cables check, depending on hardware age, there is always the possibility if a capacitor on the nic or motherboard going bad. A quick check to make sure you don't have any frozen Dr. Pepper can looking caps may be in order. If the hardware is critical irreplaceable, the $70 and a week at badcaps.net is well worth the repair... -- David C. Rankin, J.D.,P.E. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I looked at some of my interface stats, just a random selection, and they all show dropped or overrun packets. On a plain desktop machine, no one working on it, 1 packet overrun in 2 minutes, no drops. On the opensuse mail list server, 12 drops in 2 minutes. On a busy mirror machine - 58 overrun. -- Per Jessen, Zürich (19.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hello, On Wed, 31 May 2017, Per Jessen wrote:
Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I looked at some of my interface stats, just a random selection, and they all show dropped or overrun packets. On a plain desktop machine, no one working on it, 1 packet overrun in 2 minutes, no drops. On the opensuse mail list server, 12 drops in 2 minutes. On a busy mirror machine - 58 overrun.
Good point. Looking at my WAN interface: RX packets:11064938 errors:0 dropped:1015 overruns:0 frame:0 TX packets:5446401 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:16478062049 (15714.7 Mb) TX bytes:301289242 (287.3 Mb) That's ~0.009% packets dropped, so about 2/3 droprate of what Greg gets. Same ballpark I'd say. -dnh -- The true sysadmin does not adjust his behaviour to fit the machine. He adjusts the machine until it behaves properly. With a hammer, if necessary. -- Brian Kantor -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/31/2017 01:44 AM, David Haller wrote:
Hello,
On Wed, 31 May 2017, Per Jessen wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets. I looked at some of my interface stats, just a random selection, and
Greg Freemyer wrote: they all show dropped or overrun packets. On a plain desktop machine, no one working on it, 1 packet overrun in 2 minutes, no drops. On the opensuse mail list server, 12 drops in 2 minutes. On a busy mirror machine - 58 overrun. Good point. Looking at my WAN interface:
RX packets:11064938 errors:0 dropped:1015 overruns:0 frame:0 TX packets:5446401 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:16478062049 (15714.7 Mb) TX bytes:301289242 (287.3 Mb)
That's ~0.009% packets dropped, so about 2/3 droprate of what Greg gets. Same ballpark I'd say.
I was looking at the total number of drops over the total number of received packets, not just the two-minute delta. Here's Greg's second reading: Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU That comes out to about 5.9% since the stack was started, if I'm not mistaken. The old rule-of-thumb was to start worrying when rates exceed 1%. I remember that collision rates on a non-switched bus network could get up to about 2%, but you shouldn't see too many problems when using a switch, especially when just connecting between two ports on the same switch. If I had Greg's numbers I'd assume something was wrong. BTW, here are the numbers from my home machine that I'm using now. It's been up for 12-days. Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg br0 1500 0 0 0 0 0 87221 0 0 0 BMRU enp2s0 1500 0 12010582 0 0 0 5266048 0 0 0 BMRU lo 65536 0 78 0 0 0 78 0 0 0 LRU Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Lew Wolfgang wrote:
I was looking at the total number of drops over the total number of received packets, not just the two-minute delta. Here's Greg's second reading:
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
That comes out to about 5.9% since the stack was started, if I'm not mistaken. The old rule-of-thumb was to start worrying when rates exceed 1%.
Isn't that numnber of dropped messages though? I see dropped messages on virtually all interfaces (virtually all = just checked a few of them). -- Per Jessen, Zürich (26.9°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, May 31, 2017 at 12:02 PM, Per Jessen <per@computer.org> wrote:
Lew Wolfgang wrote:
I was looking at the total number of drops over the total number of received packets, not just the two-minute delta. Here's Greg's second reading:
Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 1759965 0 105198 0 13887199 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
That comes out to about 5.9% since the stack was started, if I'm not mistaken. The old rule-of-thumb was to start worrying when rates exceed 1%.
Isn't that numnber of dropped messages though? I see dropped messages on virtually all interfaces (virtually all = just checked a few of them).
From the above:
RX-OK 1759965 RX-DRP 105198 5.9% dropped I saw number like that first which is what triggered me to do the 2-min check before and after. I had a lot of local traffic during that 2 minutes. I don't really know what all the traffic was prior to that 2 minutes. Might have been web browsing, or zypper up, etc. If all 105198 packet drops occurred in either my local switch, or my NIC, then it seems excessive. By chance, I've rebooted that PC since this discussion started. It now reports:
netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 88364752 0 21978 0 90246677 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
So, 22,000 dropped packets out of 88 million. Far less than 1%. I did a lot of SMB uploading since the reboot, thus the 88 million received packets. Most would have been simple ACKs for the transmitted packets that had a data payload. Not sure why I had a 6% cumulative drop rate earlier. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/31/2017 09:48 AM, Greg Freemyer wrote:
By chance, I've rebooted that PC since this discussion started. It now reports:
netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 88364752 0 21978 0 90246677 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
So, 22,000 dropped packets out of 88 million.
I Still think something is amiss. I just checked three machines on my network, all connected by switches, plus one machine connected directly to my cable modem (not my internal network) and I have zero drops on any machine (other than the same two drops I reported yesterday). These are fairly heavily used machines. (Except the Raspberry pi connected directly to the cable modem (acting as my backup wifi access-point - it only sees significant traffic when I stream video via it). Maybe your numbers don't yet rise to the level of being an issue, but in a modern network where you can see both ends with the naked eye, you shouldn't see any. My home server, which I rebooted 30 days ago for mains power failure, has 52 RX-DRPs (out of 6,458,705) zero TX-Drps (out of 14,862,261). -- After all is said and done, more is said than done. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
John Andersen wrote:
On 05/31/2017 09:48 AM, Greg Freemyer wrote:
By chance, I've rebooted that PC since this discussion started. It now reports:
netstat -in Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg eth0 1500 0 88364752 0 21978 0 90246677 0 0 0 BMRU lo 65536 0 69 0 0 0 69 0 0 0 LRU
So, 22,000 dropped packets out of 88 million.
I Still think something is amiss.
I just checked three machines on my network, all connected by switches, plus one machine connected directly to my cable modem (not my internal network) and I have zero drops on any machine (other than the same two drops I reported yesterday).
These are fairly heavily used machines. (Except the Raspberry pi connected directly to the cable modem (acting as my backup wifi access-point - it only sees significant traffic when I stream video via it).
Maybe your numbers don't yet rise to the level of being an issue, but in a modern network where you can see both ends with the naked eye, you shouldn't see any.
My home server, which I rebooted 30 days ago for mains power failure, has 52 RX-DRPs (out of 6,458,705) zero TX-Drps (out of 14,862,261).
Some more examples - on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy. On a mirror+squid, uptime 45days, I see 7% drop on one interface. On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ... two webservers, uptime 1600 days and 92 days, 0 rx drops. The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts. -- Per Jessen, Zürich (18.8°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-06-01 08:31, Per Jessen wrote:
Some more examples -
on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.
On a mirror+squid, uptime 45days, I see 7% drop on one interface.
On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...
two webservers, uptime 1600 days and 92 days, 0 rx drops.
The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.
Question. The network "hardware" in the xen guest, is real or emulated? Because if it is emulated or virtual, not real hardware, it depends on the host CPU having time to attend it at the precise instant it is needed. A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On Thu, Jun 1, 2017 at 6:25 AM, Carlos E. R. <robin.listas@telefonica.net> wrote:
On 2017-06-01 08:31, Per Jessen wrote:
Some more examples -
on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.
On a mirror+squid, uptime 45days, I see 7% drop on one interface.
On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...
two webservers, uptime 1600 days and 92 days, 0 rx drops.
The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.
Question.
The network "hardware" in the xen guest, is real or emulated?
Ignoring Per's issue. My PC with the 6% dropped packets over some period of time is a physical machine with a X99 MB and 64GB of RAM. No VM involved at all. The 1Gbit NIC is built into the MB. The CPU is 6 cores, 12 threads (via hyper-threading).
A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.
I wonder if my X99 MB truly has all that functionality? Also, when I'm beating my NIC to death, I'm also hitting USB-3 hard. I likely pulled down a couple TB of data via SMB and placed it on a 5TB USB-3 drive during the time I lost all those packets. ie. rsync <SMB host> <USB-3 drive> (or equivalent drag & drop via dolphin) It could well be that heavy USB-3 usage is interfering with my NIC? Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Greg Freemyer wrote:
Also, when I'm beating my NIC to death, I'm also hitting USB-3 hard. I likely pulled down a couple TB of data via SMB and placed it on a 5TB USB-3 drive during the time I lost all those packets.
ie. rsync <SMB host> <USB-3 drive> (or equivalent drag & drop via dolphin)
It could well be that heavy USB-3 usage is interfering with my NIC?
That sounds like a plausible explanation. You can easily test it by reducing the max bandwidth on the rsync. -- Per Jessen, Zürich (23.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, 1 Jun 2017 14:19, Greg Freemyer wrote:
On Thu, Jun 1, 2017 at 6:25 AM, Carlos E. R. wrote:
On 2017-06-01 08:31, Per Jessen wrote:
Some more examples -
on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.
On a mirror+squid, uptime 45days, I see 7% drop on one interface.
On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...
two webservers, uptime 1600 days and 92 days, 0 rx drops.
The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.
Question.
The network "hardware" in the xen guest, is real or emulated?
Ignoring Per's issue.
My PC with the 6% dropped packets over some period of time is a physical machine with a X99 MB and 64GB of RAM. No VM involved at all. The 1Gbit NIC is built into the MB.
The CPU is 6 cores, 12 threads (via hyper-threading).
A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.
I wonder if my X99 MB truly has all that functionality?
Also, when I'm beating my NIC to death, I'm also hitting USB-3 hard. I likely pulled down a couple TB of data via SMB and placed it on a 5TB USB-3 drive during the time I lost all those packets.
ie. rsync <SMB host> <USB-3 drive> (or equivalent drag & drop via dolphin)
It could well be that heavy USB-3 usage is interfering with my NIC?
Hmm, looking at Intels docu for the X99 chipset, it could well be. The X99 PCH has a dedicated Gbit MAC integrated, and all the USBs (3.0 and 2.0) are also handled by the PCH. Also on the PCH are all the Sata, PCIe 2.0, M.2, and the Audio. All that has to pass through the DMI 2.0 x4 (16 or 20Gbit) connect to the CPU. The PCIe 3.0 and the RAM are connected directly to the CPU. It could well be that the PCH has hiccups in the flow-control, and faces the decission of mishandling the USB3 timing or dropping eth frames. https://en.wikipedia.org/wiki/Intel_X99 (for overview) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Jun 1, 2017 at 12:53 PM, Yamaban <foerster@lisas.de> wrote:
On Thu, 1 Jun 2017 14:19, Greg Freemyer wrote:
On Thu, Jun 1, 2017 at 6:25 AM, Carlos E. R. wrote:
Question.
The network "hardware" in the xen guest, is real or emulated?
Ignoring Per's issue.
My PC with the 6% dropped packets over some period of time is a physical machine with a X99 MB and 64GB of RAM. No VM involved at all. The 1Gbit NIC is built into the MB.
The CPU is 6 cores, 12 threads (via hyper-threading).
A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.
I wonder if my X99 MB truly has all that functionality?
Also, when I'm beating my NIC to death, I'm also hitting USB-3 hard. I likely pulled down a couple TB of data via SMB and placed it on a 5TB USB-3 drive during the time I lost all those packets.
ie. rsync <SMB host> <USB-3 drive> (or equivalent drag & drop via dolphin)
It could well be that heavy USB-3 usage is interfering with my NIC?
Hmm, looking at Intels docu for the X99 chipset, it could well be. The X99 PCH has a dedicated Gbit MAC integrated, and all the USBs (3.0 and 2.0) are also handled by the PCH. Also on the PCH are all the Sata, PCIe 2.0, M.2, and the Audio. All that has to pass through the DMI 2.0 x4 (16 or 20Gbit) connect to the CPU.
The PCIe 3.0 and the RAM are connected directly to the CPU.
It could well be that the PCH has hiccups in the flow-control, and faces the decission of mishandling the USB3 timing or dropping eth frames.
https://en.wikipedia.org/wiki/Intel_X99 (for overview)
If a $30 NIC would improve my rsync like performance on that machine, its a no-brainer decision for me: https://www.amazon.com/Intel-Gigabit-Network-Adapter-EXPI9301CTBLK/dp/B001CY... I have a free slot. But, would that NIC use the PCIe 3.0 interface or the PCIe 2.0 that comes in via the x99 chip. Here's a link to the diagram of my MB: https://www.asus.com/Motherboards/X99S/ According to that, all 5 of the PCIe slots I have are PCIe 3.0 capable, but I don't know if there is a legacy mode that might get invoked by plugging in to old/cheap of a NIC. Thanks Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-06-01 18:53, Yamaban wrote:
Hmm, looking at Intels docu for the X99 chipset, it could well be. The X99 PCH has a dedicated Gbit MAC integrated, and all the USBs (3.0 and 2.0) are also handled by the PCH. Also on the PCH are all the Sata, PCIe 2.0, M.2, and the Audio. All that has to pass through the DMI 2.0 x4 (16 or 20Gbit) connect to the CPU.
The PCIe 3.0 and the RAM are connected directly to the CPU.
It could well be that the PCH has hiccups in the flow-control, and faces the decission of mishandling the USB3 timing or dropping eth frames.
https://en.wikipedia.org/wiki/Intel_X99 (for overview)
Can the network hardware read/write directly to a RAM block, without using the main CPU, through that DMI thing? That is, the network hardware gets a frame, writes it to the memory block that it was told earlier to use, then ping the CPU to tell it op completed. Can it do that? Or does it simply tell the CPU that it got a frame, please pick it up, word by word? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
On 2017-06-01 08:31, Per Jessen wrote:
Some more examples -
on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.
On a mirror+squid, uptime 45days, I see 7% drop on one interface.
On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...
two webservers, uptime 1600 days and 92 days, 0 rx drops.
The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.
Question.
The network "hardware" in the xen guest, is real or emulated?
They're virtual network devices, bridged with the physical device.
Because if it is emulated or virtual, not real hardware, it depends on the host CPU having time to attend it at the precise instant it is needed. A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.
The physical card still works the way it always has, but there is obvioulsy more code to be executed to get a network interrupt serviced in a guest. Something like that. -- Per Jessen, Zürich (23.6°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2017-06-01 14:22, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-06-01 08:31, Per Jessen wrote:
Some more examples -
on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.
On a mirror+squid, uptime 45days, I see 7% drop on one interface.
On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...
two webservers, uptime 1600 days and 92 days, 0 rx drops.
The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.
Question.
The network "hardware" in the xen guest, is real or emulated?
They're virtual network devices, bridged with the physical device.
Because if it is emulated or virtual, not real hardware, it depends on the host CPU having time to attend it at the precise instant it is needed. A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.
The physical card still works the way it always has, but there is obvioulsy more code to be executed to get a network interrupt serviced in a guest. Something like that.
I have to wonder at how/if the card can write to ram that belongs to the guest. :-? :-o -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Carlos E. R. wrote:
On 2017-06-01 14:22, Per Jessen wrote:
Carlos E. R. wrote:
On 2017-06-01 08:31, Per Jessen wrote:
Some more examples -
on machines with uptimes between 30 and 60 days, depending on their load, I see drop rates of up to 1%. 0.99% on a xen guest, fairly busy.
On a mirror+squid, uptime 45days, I see 7% drop on one interface.
On four xen guests, not the most potent host machine, each has 10-11% dropped rx packages ...
two webservers, uptime 1600 days and 92 days, 0 rx drops.
The only possible pattern here might be the xen guests which exhibit somewhat higher rates than physical machines. My guess is that a machine dropping packages is simply busy - too much incoming traffic or too busy to process the interrupts.
Question.
The network "hardware" in the xen guest, is real or emulated?
They're virtual network devices, bridged with the physical device.
Because if it is emulated or virtual, not real hardware, it depends on the host CPU having time to attend it at the precise instant it is needed. A real network card will have some internal memory and processing power, and probably interrupts the machine only when the data chunk is ready. Perhaps even moves the chunk to main ram via dma, thus no cpu load.
The physical card still works the way it always has, but there is obvioulsy more code to be executed to get a network interrupt serviced in a guest. Something like that.
I have to wonder at how/if the card can write to ram that belongs to the guest. :-? :-o
It can't and it doesn't. At that level, the Dom0 (xen host) is in charge. In this case, the dom0 has been up for 140days, and shows a drop rate of 0.004%. The packages are dropped in the guests. I am sure I could reduce the drop rate by running just one guest for instance, but I need four to emulate a production setup. -- Per Jessen, Zürich (18.9°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wednesday, 31 May 2017 3:52:12 ACST Greg Freemyer wrote:
I happened to look at netstat -in with a large SMB upload ongoing and was surprised to see dropped packets.
I'm on the client that is uploading the data to a Windows fileserver.
They are on receive side which should not have much data in the
packets at all. Just ACKs for the data being uploaded:
netstat -in; sleep 120; netstat -in [...]
Apart from all the other responses, packets can be dropped for a number of different reasons; corruption, collisions (not a problem with a switch where every segment is its own broadcast domain, unless you have a duplexing issue), hardware or cabling faults (but you’d likely see a much higher percentage and notice a performance hit) or multicast packets being received for a multicast group that the interface is not registered for. This can happen if you are running a dynamic routing protocol (RIP, OSPF, EIGRP just to name a few) that use multicast packets for routing management (and the routing protocol is configured so that it runs on the relevant port), or even broadcast packets (e.g. ARP requests) that don’t need to be responded to by the local machine. At that low percentage, I wouldn’t be concerned. If you see it increasing over time, or start to notice performance issues, look further. HTH. Rodney. -- ============================================================== Rodney Baker VK5ZTV rodney.baker@iinet.net.au ============================================================== -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/31/2017 09:10 AM, Rodney Baker wrote:
or multicast packets being received for a multicast group that the interface is not registered for.
Shouldn't those just be ignored? Unless a switch supports IGMP snooping, it will treat multicasts as broadcasts and send those packets to all ports. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (12)
-
Anton Aylward -
Carlos E. R. -
Carlos E. R. -
David C. Rankin -
David Haller -
Greg Freemyer -
James Knott -
John Andersen -
Lew Wolfgang -
Per Jessen -
Rodney Baker -
Yamaban