[Bug 780966] New: systemctl restart libvirtd.service kills all running VMs
https://bugzilla.novell.com/show_bug.cgi?id=780966 https://bugzilla.novell.com/show_bug.cgi?id=780966#c0 Summary: systemctl restart libvirtd.service kills all running VMs Classification: openSUSE Product: openSUSE 12.2 Version: Final Platform: Other OS/Version: Other Status: NEW Severity: Major Priority: P5 - None Component: Other AssignedTo: jfehlig@suse.com ReportedBy: rhafer@suse.com QAContact: qa-bugs@suse.de Found By: Development Blocker: --- I just noticed that running "systemctl restart libvirtd.service" on 12.2 will killall all qemu-kvm instances started by libvirtd. This seems to be a issue with libvirtd's service file or system as it does not happen when manually calling /./etc/init.d/libvirtd restart. systemctl stop libvirtd.service also keeps the VMs alive, but they get destroyed with a subsequent systemctl start ... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c1
James Fehlig
I just noticed that running "systemctl restart libvirtd.service" on 12.2 will killall all qemu-kvm instances started by libvirtd.
I'm seeing this too.
This seems to be a issue with libvirtd's service file or system as it does not happen when manually calling /./etc/init.d/libvirtd restart.
'/etc/init.d/libvirtd restart' kills my running instances.
systemctl stop libvirtd.service also keeps the VMs alive, but they get destroyed with a subsequent systemctl start ...
Yep, I see the same behavior. But I can start/stop libvirtd directly (e.g. startproc /usr/sbin/libvirtd -l -d; killproc /usr/sbin/libvirtd; startproc /usr/sbin/libvirtd -l -d) and the instances continue to run. I'm not sure why doing the same thing via init scripts cause the instances to be killed off! Any crazy ideas? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c2
Christian Boltz
Any crazy ideas?
systemd. Is that crazy enough? ;-) Seriously: systemd kills _all processes in the cgroup_ on stop or restart, which probably includes all VMs started by libvirtd. The init script is also redirected to systemd, so you'll get the same behaviour. sshd had a similar problem (closing all connections on restart), and the sshd.service file now contains [Service] [...] KillMode=process Be warned that I never used libvirtd and the above is just a guess. Ask Frederic if you want to be sure if I'm right ;-) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c3
James Fehlig
Seriously: systemd kills _all processes in the cgroup_ on stop or restart, which probably includes all VMs started by libvirtd.
I think you nailed it :). libvirt creates a directory for each vm, e.g. /sys/fs/cgroup/cpu/libvirt/qemu/<vm-name>. So if everything under libvirt gets killed, bye-bye vms.
sshd had a similar problem (closing all connections on restart), and the sshd.service file now contains
[Service] [...] KillMode=process
libvirt already has that: # cat /lib/systemd/system/libvirtd.service # NB we don't use socket activation. When libvirtd starts it will # spawn any virtual machines registered for autostart. We want this # to occur on every boot, regardless of whether any client connects # to a socket. Thus socket activation doesn't have any benefit [Unit] Description=Virtualization daemon Before=libvirt-guests.service After=network.target [Service] EnvironmentFile=-/etc/sysconfig/libvirtd ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS ExecReload=/bin/kill -HUP $MAINPID KillMode=process # Override the maximum number of opened files #LimitNOFILE=2048 [Install] WantedBy=multi-user.target
Be warned that I never used libvirtd and the above is just a guess. Ask Frederic if you want to be sure if I'm right ;-)
Maybe he has a suggestion... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c4
James Fehlig
(In reply to comment #2)
Seriously: systemd kills _all processes in the cgroup_ on stop or restart, which probably includes all VMs started by libvirtd.
I think you nailed it :). libvirt creates a directory for each vm, e.g. /sys/fs/cgroup/cpu/libvirt/qemu/<vm-name>. So if everything under libvirt gets killed, bye-bye vms.
Thinking about this some more, the vms are killed off when doing 'systemctl start libvirtd.service', they are still running when doing stop.
sshd had a similar problem (closing all connections on restart), and the sshd.service file now contains
[Service] [...] KillMode=process
So I'm not sure any of this applies since the vms are running after stop. Perhaps this is an issue in cgroups, or libvirt's use of cgroups. After starting the service, I noticed that all of the /sys/fs/cgroup/*/libvirt/qemu/<vm-name> subdirectories of the hierarchy are removed. Nothing telling found in any of the logs. Frederic, any comments or suggestions? Thanks! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c5
Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c6
--- Comment #6 from James Fehlig
looks like duplicate of RH bug https://bugzilla.redhat.com/show_bug.cgi?id=805942
Blah, I should have found that one. Thanks for the pointer.
Please test package from http://download.opensuse.org/repositories/home:/fcrozat:/branches:/openSUSE:...
on my test system, it looks like it fixes the issue.
Yep, fixes the issue for me too.
You might also want to backport some upstream fixes in the .service shipped in libvirt (http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=79ca7e4e5763795b041e957... and http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=c18dc28b1ff29282ca1effc... )
Those are for libvirt-guests service file, so unrelated to this bug. But agreed that they should be backported for 12.2. Thanks for the help Frederic! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c7
Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c8
--- Comment #8 from James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c9
--- Comment #9 from Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c10
Patrick Schaaf
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c11
Patrick Schaaf
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c12
James Fehlig
Installed and tested both libvirt-0.9.11.6-1.5.1, current from the 12.2 update repository, and libvirt-1.0.0-... from the Virtualization repository. Also tried, with the libvirt-0.9.11.6-1.5.1 package installed, a complete reboot of the server, and behaviour afterwards.
Well, the real fix is in systemd, not libvirt. Did you try the packages Frederic referenced in #5? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c13
--- Comment #13 from Patrick Schaaf
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c14
Patrick Schaaf
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c15
--- Comment #15 from Frederic Crozat
Frederic - any idea when that next systemd maintainance update will hit the normal update repo, so I can remove your home: repo again?
I'd like to finish investigating another bug which might be in systemd before issuing the update. I'll try to work on it next week. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c16
James Fehlig
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c17
Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c18
Frederic Crozat
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c19
Benjamin Brunner
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c20
Benjamin Brunner
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c21
--- Comment #21 from Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=780966
https://bugzilla.novell.com/show_bug.cgi?id=780966#c22
--- Comment #22 from Patrick Schaaf
participants (1)
-
bugzilla_noreply＠novell.com