[Bug 1175642] New: libvirtd crashes when stopped
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 Bug ID: 1175642 Summary: libvirtd crashes when stopped Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.2 Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Virtualization:Other Assignee: virt-bugs@suse.de Reporter: ralf.koelmel@kit.edu QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Created attachment 840947 --> http://bugzilla.opensuse.org/attachment.cgi?id=840947&action=edit stacktrace of the crash the libvirtd daemon from libvirt-daemomon (version 6.0.0-lp152.8.1) is replicably crashing during stopping. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c1 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jfehlig@suse.com Assignee|virt-bugs@suse.de |jfehlig@suse.com --- Comment #1 from James Fehlig <jfehlig@suse.com> --- It's not obvious from the stacktrace what the problem is. Perhaps a double free. I will try to reproduce the bug. Do you see the crash if running libvirtd in the foreground? E.g. # systemctl stop libvirtd # /usr/sbin/libvirtd then ctr-c to terminate it. Does it crash in that scenario? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c2 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ralf.koelmel@kit.edu Flags| |needinfo?(ralf.koelmel@kit. | |edu) --- Comment #2 from James Fehlig <jfehlig@suse.com> --- I'm not able to reproduce the crash. Along with my question in #1, does the crash happen if simply starting libvirtd, then stopping it? Or do you need to run a guest, perhaps one that uses nwfilter rules? If we are encountering a double free, valgrind may help find it. Can you run libvirtd under valgrind and attach the outputs when reproducing the crash? E.g. something like valgrind --trace-children=yes --log-file="/tmp/libvirt-%p-log.out" --leak-check=full /usr/sbin/libvirtd -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c3 Ralf Kölmel <ralf.koelmel@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ralf.koelmel@kit. | |edu) | --- Comment #3 from Ralf Kölmel <ralf.koelmel@kit.edu> --- Created attachment 841016 --> http://bugzilla.opensuse.org/attachment.cgi?id=841016&action=edit libvirtd (without nwfilter configuration) stacktrace with crash during stopping it doesn't happen if i start the libvirtd on a commandline, but i believe the stopping with ctrl-c is different from a normal stop. the libvirtd, which is started and stopped via libvirtd, has these crashes without running guests, but i have a nwfilter configuration. It happens with both configurations (--listen and --timeout 120, which is now obviously default with Leap 15.2). Because of the new --timeout starting these crashes are happening every 2 minutes on a idle libvirtd configuration. If i remove my nwfilter configuration, i have again a coredump, but with a different stack (s. attachment) So the stopping with and without a nwfilter configuration is not clean. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c4 --- Comment #4 from Ralf Kölmel <ralf.koelmel@kit.edu> --- until now i had used a VM storage mounted via NFS and bind mounted to /var/lib/libvirt/images. For test i removed this and the referenced storage places are empty. Now i have no crash during stopping. So the crash is depending on my VM storage. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c5 --- Comment #5 from Ralf Kölmel <ralf.koelmel@kit.edu> --- but with configured nwfilter and without my storage the first crash is present again. the 2. crash without nwfilter is probably depending of my storage place. The 1. crash is depending of a configured nwfilter. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c6 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(ralf.koelmel@kit. | |edu) --- Comment #6 from James Fehlig <jfehlig@suse.com> --- (In reply to Ralf Kölmel from comment #4)
until now i had used a VM storage mounted via NFS and bind mounted to /var/lib/libvirt/images. For test i removed this and the referenced storage places are empty. Now i have no crash during stopping. So the crash is depending on my VM storage.
Thanks, I can give another shot at reproducing. Please provide your mount command to ensure I use the same options. (In reply to Ralf Kölmel from comment #3)
If i remove my nwfilter configuration
What do you mean by this? Does adding or removing your nwfilter configuration simply mean installing or removing libvirt-daemon-config-nwfilter? Please describe the configuration so I can replicate it here. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c7 --- Comment #7 from James Fehlig <jfehlig@suse.com> --- I'm not sure why it is difficult for me to reproduce either of the crashes. For the storage related one I have an NFS mount on the host which I then bind mount to /var/lib/libvirt/images. libvirtd terminates after the timeout with no crash. I'll need more details of your configuration. Some other items that would be interesting to try if you are able: Does the crash occur if you start libvirtd on the command line *with* the timeout value? E.g. '/usr/sbin/libvirtd --timeout 60'. That would make it a bit easier to use valgrind. Can you see if the crash still appears using the latest libvirt release? Packages for 15.2 are here https://download.opensuse.org/repositories/Virtualization/openSUSE_Leap_15.2... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c8 --- Comment #8 from James Fehlig <jfehlig@suse.com> --- BTW, Virtuozzo has also observed crashes on shutdown. It sure smells like you are encountering some of these now. They have been working on a patch series to address the crashes, some of which have been ACKed but none pushed yet https://www.redhat.com/archives/libvir-list/2020-July/msg01606.html If you are willing to test the latest libvirt packages from the Virtualization repo we may be able to help review/test the series. E.g. it would be interesting to see if you can reproduce with libvirt 6.6.0, then add that series on top and see if it helps. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c9 Ralf Kölmel <ralf.koelmel@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ralf.koelmel@kit. | |edu) | --- Comment #9 from Ralf Kölmel <ralf.koelmel@kit.edu> --- Created attachment 841039 --> http://bugzilla.opensuse.org/attachment.cgi?id=841039&action=edit libvirtd v. 6.6.0 (with nwfilter configuration) stacktrace with crash during stopping Dear James, with removing i mean deleting the nwfilter configuration file under /etc/libvirt/nwfilter/. I couldn't reproduce the nwfilter crash with valgrind. It seems that also some race condition is necessary to reproduce, which is changing when running under valgrind. I tried several times (with libvirtd version 6.0 and 6.6.0), but no chance to reproduce the crash. But on a commandline started libvirtd is also reproducibly crashing. With libvirt-daemon version 6.6 package( 6.6.0-lp152.905.1) from Virtualization repo the nwfilter crash is also happening with a little changed stacktrace (s. new attachment). The crash without nwfilter and with my NFS mounted VM storage is not occuring with the new libvirtd version. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c10 --- Comment #10 from Ralf Kölmel <ralf.koelmel@kit.edu> --- Created attachment 841040 --> http://bugzilla.opensuse.org/attachment.cgi?id=841040&action=edit the nwfilter config for reproducing the crash for the nwfilter crash it is enough to have a nwfilter configuration, which is not referenced in a VM domain definition. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 Ralf Kölmel <ralf.koelmel@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Summary|libvirtd crashes when |libvirtd with nwfilter |stopped |config crashes when stopped -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c11 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(ralf.koelmel@kit. | |edu) --- Comment #11 from James Fehlig <jfehlig@suse.com> --- (In reply to Ralf Kölmel from comment #9)
With libvirt-daemon version 6.6 package( 6.6.0-lp152.905.1) from Virtualization repo the nwfilter crash is also happening with a little changed stacktrace (s. new attachment).
Thanks a lot for the courage to test various packages on your system :-). Can you please test the libvirt packages containing Virtuozzo's patches that address shutdown hangs/crashes? You can find them here https://download.opensuse.org/repositories/home:/jfehlig:/branches:/Virtuali... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c12 Ralf Kölmel <ralf.koelmel@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ralf.koelmel@kit. | |edu) | --- Comment #12 from Ralf Kölmel <ralf.koelmel@kit.edu> --- Dear James, it's a minimum to test opensource software especially if you and other people do the harder part to develop this software ! The coredump at the stop with a nwfilter configuration is still existing with your libvirtd v6.6.0-lp15.2-907-1 and with the same stacktrace as https://bugzilla.opensuse.org/attachment.cgi?id=841039. I could also reproduce the coredump when i started libvirtd on a commandline and hit Ctrl-C. Again i had no change to reproduce the coredump under valgrind :-( Do you have tried with a nwfilter config ? Thank you for the support ! -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c14 James Fehlig <jfehlig@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |NORESPONSE --- Comment #14 from James Fehlig <jfehlig@suse.com> --- It has been a long time with no response. In the meantime libvirt in TW has been updated to 7.7.0 and 7.8.0 will be landing soon https://build.opensuse.org/request/show/922672 Feel free to reopen if you still see the issue with libvirt 7.8.0. Thanks for your patience! -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1175642 http://bugzilla.opensuse.org/show_bug.cgi?id=1175642#c15 Ralf K�lmel <ralf.koelmel@kit.edu> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(ralf.koelmel@kit. | |edu) | --- Comment #15 from Ralf K�lmel <ralf.koelmel@kit.edu> --- i couldn't replicate the crash at stopping with active nwfilters on openSUSE Leap 15.3 with libvird-daemon version 7.1.0-6.5.1. Thank you ! -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com