http://bugzilla.opensuse.org/show_bug.cgi?id=1172541 http://bugzilla.opensuse.org/show_bug.cgi?id=1172541#c4 --- Comment #4 from James Carter --- I installed kernel-default-5.7.1-1.2.x86_64 (except aarch64 on one machine). The three machines that froze up on kernel-default-5.6.14-1.1.x86_64, also froze up on 5.7.1. I kept the other six on 5.7.1 for about 48 hours witn no freezeups. Again there is no trace of the problem in the logs, when it happens. I have two machines (Intel NUC6CAYH, Celeron J3455, Intel HD Graphics 500, 2x4Gb RAM, Realtek RTL8111/8168/8411 NIC) with identical hardware, for a quick replacement if the more critical one fails: this is the main router, music library, directory server, etc. The other is a leaf node that does audio performance. The router is always the first to freeze up, whereas the leaf node has never frozen yet, despite substantial net traffic for the music. I suspect that hardware is irrelevant, but it's way too soon to blow off this possibility. I suspect that the roles exercise a vulnerable data path on the router, but there is no evidence of this either. However, when I put 5.7.1 on my "non-failing" hosts I got a rash of connection failures. This is like chasing ghosts: here's the issue that I worked on first, since it was the most mission-critical. I don't seriously expect anyone to figure it out, but I'm getting it on record in case it provides a clue. My publicly exposed webserver is on a VM (running 5.7.1 on the Celeron). IPv4+6 connections from the wild side come into the main router (running 5.6.12) and get DNAT to the webserver. HTTP and HTTPS on IPv4 work. IPv6 from the wild side used to work, not now, but with the "comorbid" network issues IPv6 is too complicated to give any clues. A tester on the webserver VM attempts https://www.jfcarter.net/ and it times out, both IPv4 and IPv6. I did lots of troubleshooting including tcpdumps and reboots (into 5.7.1). The client would initiate a TLS connection with SNI, the server would send its cert which the client would accept as valid, but the server then just sat there, not sending a TLS session ticket. Finally I tried a scorched earth solution: rebooted all the leaf nodes and VMs into 5.6.12. The net issues miraculously vanished (not seen for 2 days), and specifically, web service was back to normal. It looks like kernel 5.7.2 is now out. I'm going to install it on one vulnerable machine and see if it freezes. But in the meantime I'm going to learn to use git and I'll try to use the bisect feature to find the commit that messed it up. I wish there were a symptom that would appear immediately and left the machine alive enough that you could log in remotely and boot back to 5.6.12. But the world isn't arranged for our convenience. If I ever accidentally uninstall 5.6.12 I'll have a real problem. Several times I've wanted to revert a bad update, but my only recourse was either to wait for SuSE to release a fixed version or to fix it myself, e.g. https://bugzilla.opensuse.org/show_bug.cgi?id=1172256 . The present issue (bug 1172541) finally got me moving to set up an archive of all RPMs installed on any of my machines, keeping no longer installed versions for a month or two, in case of reversion. Did I reimplement the wheel, that is, does SuSE have such an archive server? If not, maybe you should. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1172541 http://bugzilla.opensuse.org/show_bug.cgi?id=1172541#c6 --- Comment #6 from James Carter --- "Good" news from a test: I have these two identical machines, one with a VM and one without; the VM host freezes reliably and the leaf node has never been seen to freeze. OK, on the leaf note (Iris) I installed qemu-kvm-5.0.0-3.1.x86_64 libvirt-daemon-6.4.0-1.1.x86_64 and numerous friends (the latest) and created a new VM called Orion, basically copying the XML and software selection from my other VMs. For testing I ran a console session on the guest (VNC through virt-viewer) and set the guest's screensaver to show random eye candy, giving the guest and host consistent but not excessive net traffic and CPU load. (I have the impression that the eye candy is not necessary; just being booted is enough; but that's not assured and it's more prudent to load up the guest, to maybe avoid a false negative.) With kernel 5.6.12 on the host and 5.7.1 on the guest, it ran for 14 hours without freezing. With 5.7.1 on both the host and the guest, it ran for 5 hours and the host froze up, being rebooted by its watchdog timer. I can't tell if the guest had any issues just before the host froze; I was in bed at the time. As usual, the log files on both the host and guest look perfectly normal, then end with a block of \0, and continue on with normal-looking output from the rebooted session. So this does support the hypothesis that machines running kernel 5.6.14+ (here, 5.7.1) will freeze if and only if they are host to a VM. It does not support the alternative fact that the recent update to libvirt and qemu may have made the host stop freezing. In the upgrades to 5.6.14 and kernel-default-5.7.0-3.1.gad96a07.x86_64 two of the VM hosts would freeze within 5 minutes of booting (tried several times each), but hosts that are less mission critical and more convenient for testing also take a long time to freeze, if they're going to. It's going to be hard to get developers to pay attention to this problem if they can't reproduce it in-house, and I'm going to try to put together an easily describable software and VM configuration that will freeze up. @tiwai, I retrieved kernel-default-5.6.12-1.3.x86_64 from http://download.opensuse.org/history/ , thanks for the link! -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1172541 http://bugzilla.opensuse.org/show_bug.cgi?id=1172541#c9 --- Comment #9 from James Carter --- @miroslav: During the summer I tried to investigate but the problem is, the thing won't fail reliably, which makes bisection very hard. And then USA politics went into underdrive, which took a lot of time and focus away from "important" work like the kernel bug. I did a test last night (2020-10-21 AM). First with kernel-default-5.8.14-1.2, but unbound wouldn't start, see bug 1177715. I reverted to 5.7.9-1-default (on the VM host), started the guest, gave it some minimal work (other host logs in with X-Windows propagation, guest pings various hosts and shows a graph of the results with X-Windows), and let it run. Every 3 hours both hosts do some monitoring and auditing tasks which fork off up to 10 tests in parallel, 50-60 tests overall. I've noticed that the host is most likely to crash during these tests, but my attempts to point the finger of blame at a specific test were fruitless. The tests at midnight went OK (no complaints, no crash), but 03:04 was the last message in syslog on both machines, and the host crashed (no watchdog). So kernel 5.7.9 doesn't have a fix. It's interesting that neither the host nor the guest's syslog has a block of \0's after the last message, while back in June they did. That means that then and now the log file's inode is getting written showing how much stuff had been sent to the kernel, but formerly the data was not yet synced to disc, while now it does get synced more promptly. rsyslogd has an option to sync after every write, which I didn't turn on, and perhaps I should. [Done.] @joerg: None of my machines has a serial port. I could try buying a pair of USB serial dongles, with a wire between. If I get this working I'll post the result. USB is useless while booting, but if the machine is fully functional until the crash, the USB dongle should send at least something. I upgraded the guest's packages, specifically libvirt, qemu-x86 and numerous friends, to get recent important-seeming bugfixes that had not been installed on the mothballed guest due to political distractions. I'll try the test again. Wish me luck, I'm going to need it. -- You are receiving this mail because: You are the assignee for the bug.