[opensuse-factory] Why systemd shutdown send SIGHUP to user processes?

newer
Re: [opensuse-factory] roll back...

older
[opensuse-factory] Gpg key expiring

Ruediger Meier

9 Aug 2016 9 Aug '16

19:33

Hi, I'm debugging why my processes do not terminate gracefully at shutdown and found that openSUSE's systemd setup seems to send SIGHUP to user's processes. This is the output of my test program while "shutdown -h" 2016-08-09 21:05:00.556999 catched SIGNAL 1 (SIGHUP) 2016-08-09 21:05:00.557086 catched SIGNAL 15 (SIGTERM) 2016-08-09 21:05:00.557572 catched SIGNAL 1 (SIGHUP) On other sysvinit systems I get only SIGTERM _one_ time before it would send the final SIGKILL. Problem: Many user space programs don't handle SIGHUP. Thus they are using the default sig handler and terminate immediately without any cleanup. What is the reason why SIGHUP is sent? How to disable it to get the known behavior? cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Show replies by date

Andrei Borzenkov

10 Aug 10 Aug

07:46

On Tue, Aug 9, 2016 at 10:33 PM, Ruediger Meier wrote:

...

Hi,

I'm debugging why my processes do not terminate gracefully at shutdown and found that openSUSE's systemd setup seems to send SIGHUP to user's processes. This is the output of my test program while "shutdown -h"

2016-08-09 21:05:00.556999 catched SIGNAL 1 (SIGHUP) 2016-08-09 21:05:00.557086 catched SIGNAL 15 (SIGTERM) 2016-08-09 21:05:00.557572 catched SIGNAL 1 (SIGHUP)

On other sysvinit systems I get only SIGTERM _one_ time before it would send the final SIGKILL.

Problem: Many user space programs don't handle SIGHUP. Thus they are using the default sig handler and terminate immediately without any cleanup.

What is the reason why SIGHUP is sent? How to disable it to get the known behavior?

What are "your processes" - processes belonging to your login session or processes belonging to some service you created? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Ruediger Meier

08:42

On Wednesday 10 August 2016, Andrei Borzenkov wrote:

...

On Tue, Aug 9, 2016 at 10:33 PM, Ruediger Meier wrote:

...
Hi,

I'm debugging why my processes do not terminate gracefully at shutdown and found that openSUSE's systemd setup seems to send SIGHUP to user's processes. This is the output of my test program while "shutdown -h"

2016-08-09 21:05:00.556999 catched SIGNAL 1 (SIGHUP) 2016-08-09 21:05:00.557086 catched SIGNAL 15 (SIGTERM) 2016-08-09 21:05:00.557572 catched SIGNAL 1 (SIGHUP)

On other sysvinit systems I get only SIGTERM _one_ time before it would send the final SIGKILL.

Problem: Many user space programs don't handle SIGHUP. Thus they are using the default sig handler and terminate immediately without any cleanup.

What is the reason why SIGHUP is sent? How to disable it to get the known behavior?

What are "your processes" - processes belonging to your login session or processes belonging to some service you created?

Just any processes I've sent into background manually via ssh or console login. I'm talking about any "usual" processes which are started by "normal" users during uptime. $ pstree -lp systemd(1)─┬─agetty(533) .... ├─sigtest1(2944) ├─sigtest2(2943) .... But thanks for the "session" hint: $ systemctl status 2944 session-56.scope - Session 56 of user rudi Loaded: loaded (/run/systemd/system/session-56.scope; static) Drop-In: /run/systemd/system/session-56.scope.d └─90-After-systemd-logind\x2eservice.conf, 90-After-systemd-user-sessions\x2eservice.conf, 90-Description.conf, 90-KillMode.conf, 90-SendSIGHUP.conf, 90-Slice.conf Active: active (abandoned) since Wed 2016-08-10 10:28:11 CEST; 5min ago CGroup: /user.slice/user-1000.slice/session-56.scope ├─2943 ./sigtest2 └─2944 ./sigtest1 $ cat /run/systemd/system/session-56.scope.d/90-SendSIGHUP.conf [Scope] SendSIGHUP=yes So, where does the "90-SendSIGHUP.conf" comes from? cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Andrei Borzenkov

09:20

On Wed, Aug 10, 2016 at 11:42 AM, Ruediger Meier wrote:

...

$ systemctl status 2944 session-56.scope - Session 56 of user rudi Loaded: loaded (/run/systemd/system/session-56.scope; static) Drop-In: /run/systemd/system/session-56.scope.d └─90-After-systemd-logind\x2eservice.conf, 90-After-systemd-user-sessions\x2eservice.conf, 90-Description.conf, 90-KillMode.conf, 90-SendSIGHUP.conf, 90-Slice.conf Active: active (abandoned) since Wed 2016-08-10 10:28:11 CEST; 5min ago CGroup: /user.slice/user-1000.slice/session-56.scope ├─2943 ./sigtest2 └─2944 ./sigtest1

$ cat /run/systemd/system/session-56.scope.d/90-SendSIGHUP.conf [Scope] SendSIGHUP=yes

So, where does the "90-SendSIGHUP.conf" comes from?

...

From logind creating session scope.

src/login/logind-dbus.c:manager_start_scope() /* Make sure that the session shells are terminated with * SIGHUP since bash and friends tend to ignore SIGTERM */ r = sd_bus_message_append(m, "(sv)", "SendSIGHUP", "b", true); if (r < 0) return r; As far as I can tell it is not configurable. You may consider using systemd-run to start those processes outside of normal session scope. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Ruediger Meier

10:25

On Wednesday 10 August 2016, Andrei Borzenkov wrote:

...

On Wed, Aug 10, 2016 at 11:42 AM, Ruediger Meier wrote:

...
$ systemctl status 2944 session-56.scope - Session 56 of user rudi Loaded: loaded (/run/systemd/system/session-56.scope; static) Drop-In: /run/systemd/system/session-56.scope.d └─90-After-systemd-logind\x2eservice.conf, 90-After-systemd-user-sessions\x2eservice.conf, 90-Description.conf, 90-KillMode.conf, 90-SendSIGHUP.conf, 90-Slice.conf Active: active (abandoned) since Wed 2016-08-10 10:28:11 CEST; 5min ago CGroup: /user.slice/user-1000.slice/session-56.scope ├─2943 ./sigtest2 └─2944 ./sigtest1

$ cat /run/systemd/system/session-56.scope.d/90-SendSIGHUP.conf [Scope] SendSIGHUP=yes

So, where does the "90-SendSIGHUP.conf" comes from?

From logind creating session scope.

src/login/logind-dbus.c:manager_start_scope()

/* Make sure that the session shells are terminated with * SIGHUP since bash and friends tend to ignore SIGTERM */ r = sd_bus_message_append(m, "(sv)", "SendSIGHUP", "b", true); if (r < 0) return r;

This makes no sense to me, processes which want to ignore SIGTERM get what they deserve later: SIGKILL (after a ridiculous long timeout, default 90s). BTW other processes may reload their configs on SIGHUP ... while shutdown. Moreover why it sends SIGHUP two times plus SIGTERM one time? Just to check if any program has a rock-solid signal handler implemented?

...

As far as I can tell it is not configurable. You may consider using systemd-run to start those processes outside of normal session scope.

This means I have to start basically _any_ process with systemd-run? Should I try to set my login shells to "systemd-run --user /bin/bash" BTW in future systemd-logind seems to be even more agressive: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Hope that openSUSE does not use this insane default. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Martin Wilck

12:31

On Wed, 2016-08-10 at 12:25 +0200, Ruediger Meier wrote:

...

On Wednesday 10 August 2016, Andrei Borzenkov wrote:

...
From logind creating session scope.

src/login/logind-dbus.c:manager_start_scope()

/* Make sure that the session shells are terminated with * SIGHUP since bash and friends tend to ignore SIGTERM */ r = sd_bus_message_append(m, "(sv)", "SendSIGHUP", "b", true); if (r < 0) return r;

This makes no sense to me, processes which want to ignore SIGTERM get what they deserve later: SIGKILL (after a ridiculous long timeout, default 90s). BTW other processes may reload their configs on SIGHUP ... while shutdown.

SIGHUP is sent immediately after SIGTERM, to avoid the long timeout you complain about. (Btw the timeout may be just long enough for some services that need to shut down cleanly - the 90s default timeout applies to any service, not just user sessions). The shell sends SIGHUP to children at exit, so most processes that run interactively from a shell will be able to deal with that. Processes that re-read configuration at SIGHUP will normally be daemons started by their on unit files.

...

Moreover why it sends SIGHUP two times plus SIGTERM one time? Just to check if any program has a rock-solid signal handler implemented?

...
As far as I can tell it is not configurable. You may consider using systemd-run to start those processes outside of normal session scope.

This means I have to start basically _any_ process with systemd-run? Should I try to set my login shells to "systemd-run --user /bin/bash"

Why that? /bin/bash is supposed to be sent a SIGHUP, that's the whole point.

...

BTW in future systemd-logind seems to be even more agressive: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Hope that openSUSE does not use this insane default.

KillUserProcesses is configurable in logind.conf. But traditionally, it's normal behaviour that background processes are killed when you log out. If you don't want that, use nohup or at - this allows you to be specific about the processes you don't want to be killed. That's a good thing IMO, otherwise the system is likely to be burdened with forgotten/abandoned user processes. Martin

...

cu, Rudi

-- Dr. Martin Wilck , Tel. +49 (0)911 74053 2107 SUSELinux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Ruediger Meier

14:26

On Wednesday 10 August 2016, Martin Wilck wrote:

...

On Wed, 2016-08-10 at 12:25 +0200, Ruediger Meier wrote:

...
On Wednesday 10 August 2016, Andrei Borzenkov wrote:

...
From logind creating session scope.

src/login/logind-dbus.c:manager_start_scope()

/* Make sure that the session shells are terminated with * SIGHUP since bash and friends tend to ignore SIGTERM */ r = sd_bus_message_append(m, "(sv)", "SendSIGHUP", "b", true); if (r < 0) return r;

This makes no sense to me, processes which want to ignore SIGTERM get what they deserve later: SIGKILL (after a ridiculous long timeout, default 90s). BTW other processes may reload their configs on SIGHUP ... while shutdown.

SIGHUP is sent immediately after SIGTERM,

Actually I get 3 signals within the same millisecond.

...

to avoid the long timeout you complain about.

It could also try all other kind of signals ... to increase the chance that the process will run into SIG_DFL ...

...

(Btw the timeout may be just long enough for some services that need to shut down cleanly - the 90s default timeout applies to any service, not just user sessions).

The shell sends SIGHUP to children at exit, so most processes that run interactively from a shell will be able to deal with that.

...

Processes that re-read configuration at SIGHUP will normally be daemons started by their on unit files.

What is normal? Here at my office most users are running all kinds of daemons. None of them ever heard something about "unit files". We also have a few gnome or kde users. They have even many daemons running automatically after login.

...

...
Moreover why it sends SIGHUP two times plus SIGTERM one time? Just to check if any program has a rock-solid signal handler implemented?

...
As far as I can tell it is not configurable. You may consider using systemd-run to start those processes outside of normal session scope.

This means I have to start basically _any_ process with systemd-run? Should I try to set my login shells to "systemd-run --user /bin/bash"

Why that? /bin/bash is supposed to be sent a SIGHUP, that's the whole point.

...
BTW in future systemd-logind seems to be even more agressive: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Hope that openSUSE does not use this insane default.

KillUserProcesses is configurable in logind.conf. But traditionally, it's normal behaviour that background processes are killed when you log out.

No, my ssh-agent and many other progs have always run forever after starting them once. bash even seem to automatically "nohup" child processes (may depend on the used terminal). zsh only send SIGHUP to background childs if stdin or stdout is connected to the terminal. systemd again changes known behavior.

...

If you don't want that, use nohup or at - this allows you to be specific about the processes you don't want to be killed. That's a good thing IMO, otherwise the system is likely to be burdened with forgotten/abandoned user processes.

You have not understood that in systemd-230 especially nohup, screen and friends will be killed by default too ... the traditional way how to run background processes is not the default anymore ... cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Martin Wilck

15:01

On Wed, 2016-08-10 at 16:26 +0200, Ruediger Meier wrote:

...

On Wednesday 10 August 2016, Martin Wilck wrote:

...
SIGHUP is sent immediately after SIGTERM,

Actually I get 3 signals within the same millisecond.

two would be expected, the 3rd one is perhaps a bug?

...

...
to avoid the long timeout you complain about.

It could also try all other kind of signals ... to increase the chance that the process will run into SIG_DFL ...

Please - treating the shell in a special way does make some sense.

...

...
...
BTW in future systemd-logind seems to be even more agressive: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Hope that openSUSE does not use this insane default.

KillUserProcesses is configurable in logind.conf. But traditionally, it's normal behaviour that background processes are killed when you log out.

No, my ssh-agent and many other progs have always run forever after starting them once.

Hm, is that really a good idea?

...

bash even seem to automatically "nohup" child processes (may depend on the used terminal).

Only if you "disown" them, to my understanding.

...

You have not understood that in systemd-230 especially nohup, screen and friends will be killed by default too ... the traditional way how to run background processes is not the default anymore ...

Indeed, I didn't read the debian bug far enough. Breaking nohup is sort of disconcerting, I agree. I wonder if "at" would be affected, too - using "at" has been my personal preferred method for long running jobs for a while. At first sight, it seems that "at" jobs end up in a different scope than the starting shell, so it's looking good thus far. Cheers Martin -- Dr. Martin Wilck , Tel. +49 (0)911 74053 2107 SUSELinux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Ruediger Meier

15:35

On Wednesday 10 August 2016, Martin Wilck wrote:

...

On Wed, 2016-08-10 at 16:26 +0200, Ruediger Meier wrote:

...
On Wednesday 10 August 2016, Martin Wilck wrote:

...
SIGHUP is sent immediately after SIGTERM,

Actually I get 3 signals within the same millisecond.

two would be expected, the 3rd one is perhaps a bug?

...
...
to avoid the long timeout you complain about.

It could also try all other kind of signals ... to increase the chance that the process will run into SIG_DFL ...

Please - treating the shell in a special way does make some sense.

...
...
...
BTW in future systemd-logind seems to be even more agressive: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Hope that openSUSE does not use this insane default.

KillUserProcesses is configurable in logind.conf. But traditionally, it's normal behaviour that background processes are killed when you log out.

No, my ssh-agent and many other progs have always run forever after starting them once.

Hm, is that really a good idea?

...

...
bash even seem to automatically "nohup" child processes (may depend on the used terminal).

Only if you "disown" them, to my understanding.

...

...
You have not understood that in systemd-230 especially nohup, screen and friends will be killed by default too ... the traditional way how to run background processes is not the default anymore ...

Indeed, I didn't read the debian bug far enough. Breaking nohup is sort of disconcerting, I agree.

BTW then systemd folks tried to add systemd specific code to existing programs (like tmux, screen, x2go) to "fix" the non-sense they have introduced. https://github.com/tmux/tmux/issues/428 https://news.ycombinator.com/item?id=11797075

...

I wonder if "at" would be affected, too - using "at" has been my personal preferred method for long running jobs for a while. At first sight, it seems that "at" jobs end up in a different scope than the starting shell, so it's looking good thus far. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Martin Wilck

16:37

On Wed, 2016-08-10 at 12:25 +0200, Ruediger Meier wrote:

...

BTW in future systemd-logind seems to be even more agressive: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Hope that openSUSE does not use this insane default.

Not sure whether or not it is insane, but using the words of https://bu gs.freedesktop.org/show_bug.cgi?id=94508#c3, it comes down to whether OpenSUSE intends to support the "login-session-centric" or "user- session-centric" views of the world, or both ... Has this question been adressed in the board already? Martin -- Dr. Martin Wilck , Tel. +49 (0)911 74053 2107 SUSELinux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Per Jessen

17:39

Martin Wilck wrote:

...

On Wed, 2016-08-10 at 12:25 +0200, Ruediger Meier wrote:

...
BTW in future systemd-logind seems to be even more agressive: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825394 Hope that openSUSE does not use this insane default.

Not sure whether or not it is insane, but using the words of https://bu gs.freedesktop.org/show_bug.cgi?id=94508#c3, it comes down to whether OpenSUSE intends to support the "login-session-centric" or "user- session-centric" views of the world, or both ...

Has this question been adressed in the board already?

Is the Board in charge of the technical direction of openSUSE? -- Per Jessen, Zürich (17.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Dominique Leuenberger / DimStar

17:48

On Wed, 2016-08-10 at 19:39 +0200, Per Jessen wrote:

...

Is the Board in charge of the technical direction of openSUSE?

As I'm sure you know the answer to this question (or you'd probably not even have asked it) - please make this obvious to the not so seasoned contributors that might actually believe this question is meant serious. You could for example add 'rhetorical' tags or similar to your question. <sarcasm>Thank you for considering this in the future</sarcasm> Cheers, Dominique

Per Jessen

17:56

Dominique Leuenberger / DimStar wrote:

...

On Wed, 2016-08-10 at 19:39 +0200, Per Jessen wrote:

...
Is the Board in charge of the technical direction of openSUSE?

As I'm sure you know the answer to this question (or you'd probably not even have asked it) - please make this obvious to the not so seasoned contributors that might actually believe this question is meant serious.

You could for example add 'rhetorical' tags or similar to your question.

<sarcasm>Thank you for considering this in the future</sarcasm>

Cheers, Dominique

Haha, good one Dominique. (I was unable to detect any sarcasm in Martin's question, so I felt I had to ask). -- Per Jessen, Zürich (17.1°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Martin Wilck

18:32

On Wed, 2016-08-10 at 19:56 +0200, Per Jessen wrote:

...

Dominique Leuenberger / DimStar wrote:

...
On Wed, 2016-08-10 at 19:39 +0200, Per Jessen wrote:

...
Is the Board in charge of the technical direction of openSUSE?

As I'm sure you know the answer to this question (or you'd probably not even have asked it) - please make this obvious to the not so seasoned contributors that might actually believe this question is meant serious.

You could for example add 'rhetorical' tags or similar to your question.

<sarcasm>Thank you for considering this in the future</sarcasm>

Cheers, Dominique

Haha, good one Dominique.

(I was unable to detect any sarcasm in Martin's question, so I felt I had to ask).

In fact I wasn't being sarcastic. IMO the issue at hand isn't a technical issue. I'm rather new to this community so I may not have fully understood how the decision process works, and may have used the wrong terms. Sorry about that. Yet I'd be interested to know where we are heading to, or what the status of the discussion is. Martin -- Dr. Martin Wilck , Tel. +49 (0)911 74053 2107 SUSELinux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

2820

Age (days ago)

2821

Last active (days ago)

List overview

Download

13 comments

5 participants

participants (5)

Andrei Borzenkov
Dominique Leuenberger / DimStar
Martin Wilck
Per Jessen
Ruediger Meier