http://bugzilla.opensuse.org/show_bug.cgi?id=1189808 Bug ID: 1189808 Summary: udp datagrams received out-of-order with kernel 5.3.18-lp152.72.1 onwards Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.2 Hardware: x86-64 OS: openSUSE Leap 15.2 Status: NEW Severity: Normal Priority: P5 - None Component: Kernel Assignee: kernel-bugs@opensuse.org Reporter: simonalogan@gmail.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36 Build Identifier: Hi folks, We have observed that Leap 15.2 kernels 5.3.18-lp152.72.1 onwards frequently deliver UDP packets out of order. The issue is 100% reproducable on Haswell hardware, for example an Intel(R) Core(TM) i3-4130 CPU @ 3.40GHz machine. I know UDP makes no guarantee of packet delivery order, but previous kernels always seemed to deliver packets in order. Did something change in 72.1 that would affect this? We noticed this affecting video streaming, but it is easily reproducable using iperf. Please see steps below. Thanks, Simon Reproducible: Always Steps to Reproduce: - Run an iperf server on the Haswell box. iperf3 -s - Now run a set of 10 tests, asking a remote client to receive 300Mbit blocks of UDP data. for iter in 0 1 2 3 4 5 6 7 8 9; do echo "iter: $iter"; iperf3 -c 192.168.1.10 -u -b 300M -R | grep "datagrams received out-of-order"; done Using kernel 5.3.18-lp152.60 I see no out-of-order datagrams. Using kernel 5.3.18-lp152.87 I see > 10k out-of-order datagrams on each run. Actual Results: Using the 5.3.18-lp152.87 kernel: iter: 0 [SUM] 0.0-10.0 sec 22283 datagrams received out-of-order iter: 1 [SUM] 0.0-10.0 sec 23910 datagrams received out-of-order iter: 2 [SUM] 0.0-10.0 sec 20661 datagrams received out-of-order iter: 3 [SUM] 0.0-10.0 sec 20182 datagrams received out-of-order iter: 4 [SUM] 0.0-10.0 sec 18224 datagrams received out-of-order iter: 5 [SUM] 0.0-10.0 sec 19448 datagrams received out-of-order iter: 6 [SUM] 0.0-10.0 sec 27404 datagrams received out-of-order iter: 7 [SUM] 0.0-10.0 sec 17781 datagrams received out-of-order iter: 8 [SUM] 0.0-10.0 sec 11182 datagrams received out-of-order iter: 9 [SUM] 0.0-10.0 sec 25676 datagrams received out-of-order Here's the full output from one run: [ 5] local 192.168.1.8 port 53594 connected to 192.168.1.10 port 5201 [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 35.8 MBytes 300 Mbits/sec 0.010 ms 0/25904 (0%) [ 5] 1.00-2.00 sec 35.7 MBytes 300 Mbits/sec 0.024 ms 0/25888 (0%) [ 5] 2.00-3.00 sec 35.8 MBytes 300 Mbits/sec 0.027 ms 0/25897 (0%) [ 5] 3.00-4.00 sec 35.8 MBytes 300 Mbits/sec 0.012 ms 1/25908 (0.0039%) [ 5] 4.00-5.00 sec 35.8 MBytes 300 Mbits/sec 0.021 ms -1/25897 (-0.0039%) [ 5] 5.00-6.00 sec 35.8 MBytes 300 Mbits/sec 0.017 ms 0/25897 (0%) [ 5] 6.00-7.00 sec 35.7 MBytes 300 Mbits/sec 0.032 ms 0/25887 (0%) [ 5] 7.00-8.00 sec 35.8 MBytes 300 Mbits/sec 0.012 ms 0/25909 (0%) [ 5] 8.00-9.00 sec 35.8 MBytes 300 Mbits/sec 0.009 ms 0/25899 (0%) [ 5] 9.00-10.00 sec 35.8 MBytes 300 Mbits/sec 0.015 ms 0/25897 (0%) - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.000 ms 0/259008 (0%) sender [SUM] 0.0-10.0 sec 22745 datagrams received out-of-order [ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.015 ms 0/258983 (0%) receiver Going the other way simon@localhost:~> iperf3 -c 192.168.1.10 -u -b 300M Connecting to host 192.168.1.10, port 5201 [ 5] local 192.168.1.8 port 56004 connected to 192.168.1.10 port 5201 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 35.7 MBytes 300 Mbits/sec 25878 [ 5] 1.00-2.00 sec 35.8 MBytes 300 Mbits/sec 25898 [ 5] 2.00-3.00 sec 35.8 MBytes 300 Mbits/sec 25900 [ 5] 3.00-4.00 sec 35.8 MBytes 300 Mbits/sec 25910 [ 5] 4.00-5.00 sec 35.7 MBytes 300 Mbits/sec 25878 [ 5] 5.00-6.00 sec 35.8 MBytes 300 Mbits/sec 25919 [ 5] 6.00-7.00 sec 35.7 MBytes 300 Mbits/sec 25882 [ 5] 7.00-8.00 sec 35.8 MBytes 300 Mbits/sec 25897 [ 5] 8.00-9.00 sec 35.8 MBytes 300 Mbits/sec 25898 [ 5] 9.00-10.00 sec 35.8 MBytes 300 Mbits/sec 25898 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.000 ms 0/258958 (0%) sender [ 5] 0.00-10.00 sec 358 MBytes 300 Mbits/sec 0.013 ms 6/258958 (0.0023%) receiver Expected Results: Based on kernel 5.3.18-lp152.60, I expect no out-of-order datagrams. -- You are receiving this mail because: You are the assignee for the bug.