[opensuse-packaging] not restarting the docker service on running zypper update?
Hi packagers, I am packaging docker and we have this problem that if a user runs an unattended update, docker gets updated and so restarted. However, this is causing downtime issues on their services which are inside containers. I know there is the "--skip-interactive" flag for zypper ... So my question is. Should we mark docker as "interactive" and if so, how do we do that? Should we not restart docker when installing? I think you should, but am I wrong? Are there other packages with similar problems? mariadb? thanks in advance jordi -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Fri, 2017-01-20 at 12:02 +0100, Jordi Massaguer Pla wrote:
Hi packagers,
I am packaging docker and we have this problem that if a user runs an unattended update, docker gets updated and so restarted. However, this is causing downtime issues on their services which are inside containers.
I know there is the "--skip-interactive" flag for zypper ...
So my question is. Should we mark docker as "interactive" and if so, how do we do that?
That's not the right approach
Should we not restart docker when installing? I think you should, but am I wrong?
There is a config flag in /etc/susconfig that gives users control over if they normally want their services restarted on update (which is the default and we should keep doing for all services that do not 'break the update sequence') See /etc/sysconfig/services: DISABLE_RESTART_ON_UPDATE= and DISABLE_STOP_ON_REMOVAL= Packages that KNOW that they case very negative effects on restart (e.g. dbus, the X server, PackageKit) can export this variable in their post script just before executing the service handling macros. But for docker, where the issue is 'only' downtime, I'd argue it is up to the admin to decide if he wants to skip the restart (by setting the corresponding variables in his system) - and thus take care of actually enabling the new services, or not (follow the default, restart the service up update)
Are there other packages with similar problems? mariadb?
The 'rule' is basically: if the service can be restarted without impacting the update session, it should be done. Known exceptions so far are: * PackageKit: users updating with any PK-related tool would result in an interrupted system update - hence PK must not be restarted * xdm: a restart of the display-manager terminates running X-sessions of a user. This clearly impacts the running update process if not done in a screen/tmux session (or on a text console) * dbus-1: just seen last week: restarting dbus makes pretty much all running dbus-sessions disapear. with X running on systemd-logind nowadays, this is a bad idea. Cheers, Dominique
The 'rule' is basically: if the service can be restarted without impacting the update session, it should be done. Known exceptions so far are: * PackageKit: users updating with any PK-related tool would result in an interrupted system update - hence PK must not be restarted * xdm: a restart of the display-manager terminates running X-sessions of a user. This clearly impacts the running update process if not done in a screen/tmux session (or on a text console) * dbus-1: just seen last week: restarting dbus makes pretty much all running dbus-sessions disapear. with X running on systemd-logind nowadays, this is a bad idea.
Running live updates on virtualisation hosts (xen, kvm, docker) without clever cherrypicking can potentially give you a lot more issues, especially if you cannot simply reboot the host on short notice. That shouldn't be an issue with docker on laptops - what could possibly run there? - but with servers acting as docker hosts, you should carefully select your live updates and postpone everything else until you can failover your services to another host (or switch them off for a while) -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Mon, 2017-01-30 at 12:09 +0100, Ralf Lang wrote:
The 'rule' is basically: if the service can be restarted without impacting the update session, it should be done. Known exceptions so far are: * PackageKit: users updating with any PK-related tool would result in an interrupted system update - hence PK must not be restarted * xdm: a restart of the display-manager terminates running X- sessions of a user. This clearly impacts the running update process if not done in a screen/tmux session (or on a text console) * dbus-1: just seen last week: restarting dbus makes pretty much all running dbus-sessions disapear. with X running on systemd-logind nowadays, this is a bad idea.
Running live updates on virtualisation hosts (xen, kvm, docker) without clever cherrypicking can potentially give you a lot more issues, especially if you cannot simply reboot the host on short notice.
That shouldn't be an issue with docker on laptops - what could possibly run there? - but with servers acting as docker hosts, you should carefully select your live updates and postpone everything else until you can failover your services to another host (or switch them off for a while)
Sorry, but on such a business critical system I expect the admin to have disabled automatic restarting of all services in /etc/sysconfig anyway. Cheers, Dominiqe
On 01/31/2017 06:56 PM, Dominique Leuenberger / DimStar wrote:
On Mon, 2017-01-30 at 12:09 +0100, Ralf Lang wrote:
The 'rule' is basically: if the service can be restarted without impacting the update session, it should be done. Known exceptions so far are: * PackageKit: users updating with any PK-related tool would result in an interrupted system update - hence PK must not be restarted * xdm: a restart of the display-manager terminates running X- sessions of a user. This clearly impacts the running update process if not done in a screen/tmux session (or on a text console) * dbus-1: just seen last week: restarting dbus makes pretty much all running dbus-sessions disapear. with X running on systemd-logind nowadays, this is a bad idea.
Running live updates on virtualisation hosts (xen, kvm, docker) without clever cherrypicking can potentially give you a lot more issues, especially if you cannot simply reboot the host on short notice.
That shouldn't be an issue with docker on laptops - what could possibly run there? - but with servers acting as docker hosts, you should carefully select your live updates and postpone everything else until you can failover your services to another host (or switch them off for a while)
Sorry, but on such a business critical system I expect the admin to have disabled automatic restarting of all services in /etc/sysconfig anyway.
Cheers, Dominiqe
I'm just a messenger, I don't disagree here, but what we expect doesn't seem to always happen. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On 31.01.2017 09:26, Dominique Leuenberger / DimStar wrote:
On Mon, 2017-01-30 at 12:09 +0100, Ralf Lang wrote:
The 'rule' is basically: if the service can be restarted without impacting the update session, it should be done. Known exceptions so far are: * PackageKit: users updating with any PK-related tool would result in an interrupted system update - hence PK must not be restarted * xdm: a restart of the display-manager terminates running X- sessions of a user. This clearly impacts the running update process if not done in a screen/tmux session (or on a text console) * dbus-1: just seen last week: restarting dbus makes pretty much all running dbus-sessions disapear. with X running on systemd-logind nowadays, this is a bad idea.
Running live updates on virtualisation hosts (xen, kvm, docker) without clever cherrypicking can potentially give you a lot more issues, especially if you cannot simply reboot the host on short notice.
That shouldn't be an issue with docker on laptops - what could possibly run there? - but with servers acting as docker hosts, you should carefully select your live updates and postpone everything else until you can failover your services to another host (or switch them off for a while)
Sorry, but on such a business critical system I expect the admin to have disabled automatic restarting of all services in /etc/sysconfig anyway.
I just wanted to advise against admins running random live updates on virtualisation hosts. Packaging cannot anticipate use cases and admins should not assume it does. I don't disagree with you. -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Am 2. Februar 2017 19:48:37 MEZ, schrieb Ralf Lang <lang@b1-systems.de>:
Sorry, but on such a business critical system I expect the admin to have disabled automatic restarting of all services in /etc/sysconfig anyway.
I just wanted to advise against admins running random live updates on virtualisation hosts. Packaging cannot anticipate use cases and admins should not assume it does. I don't disagree with you.
From my personal point as sysadmin maintaining hundreds of machines over years: I trust the packagers that they know best about their services. I never touched the variable you mentioned on any of my systems since I use Linux.
If a services has to be up for the magic 99,999%, you need a HA setup anyway. In all other cases, I expect: * security updates might require a restart to have the patched service running instead of the vulnerable one * a restart might take some time, but I expect the service to be up as before without loss of configured features * as even productive machines might have a hardware problem, see a power outage or simply need a reboot after kernel/ systemd update, the system as such has to come up with all needed services up and running after a reboot So from packagers I would except: * never ever change (runtime) configurations * if you restart a service, do it right (I saw untested conditions in init scripts that resulted in a failure during restart) * if needed (and only then) allow a sysadmin to disable an automated restart - but this requires a lot of documentation as this is nothing an admin like me expects as a default setting => and add a big warning sign that disabling an automated restart might have big security implications! Lars -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 02/03/2017 07:59 AM, Lars Vogdt wrote:
Am 2. Februar 2017 19:48:37 MEZ, schrieb Ralf Lang <lang@b1-systems.de>:
Sorry, but on such a business critical system I expect the admin to have disabled automatic restarting of all services in /etc/sysconfig anyway. I just wanted to advise against admins running random live updates on virtualisation hosts. Packaging cannot anticipate use cases and admins should not assume it does. I don't disagree with you. From my personal point as sysadmin maintaining hundreds of machines over years: I trust the packagers that they know best about their services. I never touched the variable you mentioned on any of my systems since I use Linux.
If a services has to be up for the magic 99,999%, you need a HA setup anyway.
In all other cases, I expect: * security updates might require a restart to have the patched service running instead of the vulnerable one * a restart might take some time, but I expect the service to be up as before without loss of configured features * as even productive machines might have a hardware problem, see a power outage or simply need a reboot after kernel/ systemd update, the system as such has to come up with all needed services up and running after a reboot
So from packagers I would except: * never ever change (runtime) configurations * if you restart a service, do it right (I saw untested conditions in init scripts that resulted in a failure during restart)
The problem with docker is that restarting docker stops your containers and they will only come back if, and only if, you had started them with the option "restart". https://docs.docker.com/engine/reference/run/#/restart-policies---restart However, I have the feeling most users do not expect this behavior, they expect to have the same containers running after an update.
* if needed (and only then) allow a sysadmin to disable an automated restart - but this requires a lot of documentation as this is nothing an admin like me expects as a default setting => and add a big warning sign that disabling an automated restart might have big security implications!
Lars
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 01/23/2017 08:47 PM, Dominique Leuenberger / DimStar wrote:
On Fri, 2017-01-20 at 12:02 +0100, Jordi Massaguer Pla wrote:
Hi packagers,
I am packaging docker and we have this problem that if a user runs an unattended update, docker gets updated and so restarted. However, this is causing downtime issues on their services which are inside containers.
Should we not restart docker when installing? I think you should, but am I wrong?
Packages that KNOW that they case very negative effects on restart (e.g. dbus, the X server, PackageKit) can export this variable in their post script just before executing the service handling macros. But for docker, where the issue is 'only' downtime, I'd argue it is up to the admin to decide if he wants to skip the restart (by setting the corresponding variables in his system) - and thus take care of actually enabling the new services, or not (follow the default, restart the service up update)
In http://bugzilla.suse.com/show_bug.cgi?id=980555 it was decided that as migrations can take many hours in this case the service shouldn't be automatically restarted. (When docker isn't doing a migration it should be). -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On 01/20/2017 09:32 PM, Jordi Massaguer Pla wrote:
Hi packagers,
I am packaging docker and we have this problem that if a user runs an unattended update, docker gets updated and so restarted. However, this is causing downtime issues on their services which are inside containers.
I know there is the "--skip-interactive" flag for zypper ...
So my question is. Should we mark docker as "interactive" and if so, how do we do that?
Should we not restart docker when installing? I think you should, but am I wrong?
Are there other packages with similar problems? mariadb?
thanks in advance
jordi
Sorry, I only just noticed this today, we already added handling for this in the docker spec file [1] late last year. Changing the "last_migration_version" will cause "DISABLE_RESTART_ON_UPDATE=yes" to be exported in the postun script which will cause the service not to be restarted see boo#980555. This means the docker service will only be restarted if a migration won't be caused. For dbus we have done it slightly differently [2] by modifying the file in /etc/sysconfig it allowed us to block the update even though DISABLE_RESTART_ON_UPDATE is not exported in the older version's postun section, we also now always export DISABLE_RESTART_ON_UPDATE=yes for dbus as we never want it to restart during an update so in a few months time we will probably remove the code messing with config files as hopefully most of tumbleweed will have migrated away. Appologies again that it took this long for me to miss and then see the email, I was off at a conference. 1. https://build.opensuse.org/package/view_file/Virtualization:containers/docke... 2. https://build.opensuse.org/package/view_file/Base:System/dbus-1/dbus-1.spec?... -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
participants (5)
-
Dominique Leuenberger / DimStar -
Jordi Massaguer Pla -
Lars Vogdt -
Ralf Lang -
Simon Lees