User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36
Build Identifier:
Hi folks,
We have observed that Leap 15.2 kernels 5.3.18-lp152.72.1 onwards frequently
deliver UDP packets out of order.
The issue is 100% reproducable on Haswell hardware, for example an Intel(R)
Core(TM) i3-4130 CPU @ 3.40GHz machine.
I know UDP makes no guarantee of packet delivery order, but previous kernels
always seemed to deliver packets in order.
Did something change in 72.1 that would affect this?
We noticed this affecting video streaming, but it is easily reproducable using
iperf.
Please see steps below.
Thanks,
Simon
Reproducible: Always
Steps to Reproduce:
- Run an iperf server on the Haswell box.
iperf3 -s
- Now run a set of 10 tests, asking a remote client to receive 300Mbit blocks
of UDP data.
for iter in 0 1 2 3 4 5 6 7 8 9; do echo "iter: $iter"; iperf3 -c 192.168.1.10
-u -b 300M -R | grep "datagrams received out-of-order"; done
Using kernel 5.3.18-lp152.60 I see no out-of-order datagrams.
Using kernel 5.3.18-lp152.87 I see > 10k out-of-order datagrams on each run.
Actual Results:
Using the 5.3.18-lp152.87 kernel:
iter: 0
[SUM] 0.0-10.0 sec 22283 datagrams received out-of-order
iter: 1
[SUM] 0.0-10.0 sec 23910 datagrams received out-of-order
iter: 2
[SUM] 0.0-10.0 sec 20661 datagrams received out-of-order
iter: 3
[SUM] 0.0-10.0 sec 20182 datagrams received out-of-order
iter: 4
[SUM] 0.0-10.0 sec 18224 datagrams received out-of-order
iter: 5
[SUM] 0.0-10.0 sec 19448 datagrams received out-of-order
iter: 6
[SUM] 0.0-10.0 sec 27404 datagrams received out-of-order
iter: 7
[SUM] 0.0-10.0 sec 17781 datagrams received out-of-order
iter: 8
[SUM] 0.0-10.0 sec 11182 datagrams received out-of-order
iter: 9
[SUM] 0.0-10.0 sec 25676 datagrams received out-of-order
Here's the full output from one run:
[ 5] local 192.168.1.8 port 53594 connected to 192.168.1.10 port 5201
[ ID] Interval Transfer Bitrate Jitter Lost/Total
Datagrams
[ 5] 0.00-1.00 sec 35.8 MBytes 300 Mbits/sec 0.010 ms 0/25904 (0%)
[ 5] 1.00-2.00 sec 35.7 MBytes 300 Mbits/sec 0.024 ms 0/25888 (0%)
[ 5] 2.00-3.00 sec 35.8 MBytes 300 Mbits/sec 0.027 ms 0/25897 (0%)
[ 5] 3.00-4.00 sec 35.8 MBytes 300 Mbits/sec 0.012 ms 1/25908
(0.0039%)
[ 5] 4.00-5.00 sec 35.8 MBytes 300 Mbits/sec 0.021 ms -1/25897
(-0.0039%)
[ 5] 5.00-6.00 sec 35.8 MBytes 300 Mbits/sec 0.017 ms 0/25897 (0%)
[ 5] 6.00-7.00 sec 35.7 MBytes 300 Mbits/sec 0.032 ms 0/25887 (0%)
[ 5] 7.00-8.00 sec 35.8 MBytes 300 Mbits/sec 0.012 ms 0/25909 (0%)
[ 5] 8.00-9.00 sec 35.8 MBytes 300 Mbits/sec 0.009 ms 0/25899 (0%)
[ 5] 9.00-10.00 sec 35.8 MBytes 300 Mbits/sec 0.015 ms 0/25897 (0%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total
Datagrams
[ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.000 ms 0/259008 (0%)
sender
[SUM] 0.0-10.0 sec 22745 datagrams received out-of-order
[ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.015 ms 0/258983 (0%)
receiver
Going the other way
simon@localhost:~> iperf3 -c 192.168.1.10 -u -b 300M
Connecting to host 192.168.1.10, port 5201
[ 5] local 192.168.1.8 port 56004 connected to 192.168.1.10 port 5201
[ ID] Interval Transfer Bitrate Total Datagrams
[ 5] 0.00-1.00 sec 35.7 MBytes 300 Mbits/sec 25878
[ 5] 1.00-2.00 sec 35.8 MBytes 300 Mbits/sec 25898
[ 5] 2.00-3.00 sec 35.8 MBytes 300 Mbits/sec 25900
[ 5] 3.00-4.00 sec 35.8 MBytes 300 Mbits/sec 25910
[ 5] 4.00-5.00 sec 35.7 MBytes 300 Mbits/sec 25878
[ 5] 5.00-6.00 sec 35.8 MBytes 300 Mbits/sec 25919
[ 5] 6.00-7.00 sec 35.7 MBytes 300 Mbits/sec 25882
[ 5] 7.00-8.00 sec 35.8 MBytes 300 Mbits/sec 25897
[ 5] 8.00-9.00 sec 35.8 MBytes 300 Mbits/sec 25898
[ 5] 9.00-10.00 sec 35.8 MBytes 300 Mbits/sec 25898
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Jitter Lost/Total
Datagrams
[ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.000 ms 0/258958 (0%)
sender
[ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.013 ms 6/258958
(0.0023%) receiver
Expected Results:
Based on kernel 5.3.18-lp152.60, I expect no out-of-order datagrams.