http://bugzilla.suse.com/show_bug.cgi?id=954765 Mel Gorman changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fbui@suse.com, | |mfleming@suse.com, | |mgalbraith@suse.com, | |okir@suse.com Assignee|bnc-team-screening@forge.pr |systemd-maintainers@suse.de |ovo.novell.com | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=954765 http://bugzilla.suse.com/show_bug.cgi?id=954765#c2 --- Comment #2 from Mel Gorman --- (In reply to Dr. Werner Fink from comment #1)

Maybe it's worth to watch the talk about control groups at systemd.conf 2015 on

https://www.youtube.com/watch?v=7CWmuhkgZWs

So I watched it but I still don't see why the activation of the resource controllers is necessary via Delegate=yes. Note that I see no problem with grouping related processes together, it's the controller activation I'm confused by. The talk opens by saying that cgroups are a means of hierarchically labelling processes and uses this to manage service lifetime. cgroups are primarily about resource control and guaranteeing of fairness which is what some of the controllers do. As a side-effect, this can map PIDs to services and systemd uses this for service managmement and notification about service shutdown. While I can see why the grouping of processes is desirable to cleanly startup and shutdown services, I cannot see why the controllers get activated even when full resource control is not required. It's known the resource control enforcement incurs an 83% penalty on a scheduler microbenchmark due to the cpuacct controller. I observed myself a case where dbench4 regressed 80% due to the blkio controller as the journalling kernel thread and IO submitter were in separate cgroups. Even if the unified hierarchy was fully in place (and I understand why the separate cgroup hierarchies is difficult), it would not remove the overhead. For example, any process in the user slice with the cpuacct controller activated is going to force all processes to update what is essentially global data (the cumulative cpu usage for all processes in the user slice). Multiple processes updating the same data incurs a high penalty due to cache misses. Even if the overhead was zero (and I don't know anyone who is working specifically on eliminating the overhead), there is a semantic difference when controllers are enabled. For example, the memory controller creates per-memcg LRU lists. In the event of global memory pressure, those lists are reclaimed proportionally to each other. Their existence alters the order memory is reclaimed in and now a relatively new process can get reclaimed prematurely because it's being aged relative to other older cgroups that are idle. Now I can see why such enabling resource control would be a good idea in some cases. For example, virtual machines or containers could justify being resource controlled to avoid interfering with each other but that's a special case. It seems like a very bad idea to incur the same overhead and semantic differences for a single user on a single machine running basic workloads that do not require strict resource control. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=954765 Mel Gorman changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mel.gorman@microfocus.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=954765 http://bugzilla.suse.com/show_bug.cgi?id=954765#c5 --- Comment #5 from Franck Bui --- (In reply to Mel Gorman from comment #4)

It is possible this was an oversight. I do not have the full picture of what was planned but it appears that the intent was to activate this only for new containers. It's just the case that in Tumbleweed that normal sessions (or at least ssh sessions from root) are also impacted. If only processes within a container or a VM were resource controlled then it would limit the impact of this bug.

I think the intent was to really activate this for 'root' user sessions too. Regular user sessions are left alone for now unless unified hierarchy is supported and used by the kernel. See commit which introduced the Delegate property for some details: https://github.com/systemd/systemd/commit/a931ad47a8623163a29d898224d8a8c117... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=954765 http://bugzilla.suse.com/show_bug.cgi?id=954765#c7 --- Comment #7 from Franck Bui --- (In reply to Mel Gorman from comment #6)

(In reply to Franck Bui from comment #5)

...

(In reply to Mel Gorman from comment #4)

...

It is possible this was an oversight. I do not have the full picture of what was planned but it appears that the intent was to activate this only for new containers. It's just the case that in Tumbleweed that normal sessions (or at least ssh sessions from root) are also impacted. If only processes within a container or a VM were resource controlled then it would limit the impact of this bug.

I think the intent was to really activate this for 'root' user sessions too. Regular user sessions are left alone for now unless unified hierarchy is supported and used by the kernel.

This means that any process launched by root be it manual launch of a server or a benchmark is going to be throttled. It cannot be what they really intended or if they did, it's a massive overhead to occur for very little, if any, gain.

...

See commit which introduced the Delegate property for some details: https://github.com/systemd/systemd/commit/ a931ad47a8623163a29d898224d8a8c1177ffdaf

The commit to me seems to say that it was only intended to be activated for containers :(

From the commit message:

"Delegate=yes should also be set for user@.service, so that systemd --user can run, controlling its own cgroup tree." user@.service also handles root user session (in that case it's named 'user@0.service) and this unit is a priviliged one when executed for root user. And still from the commit message: "For priviliged units this resource control property ensures that the processes have all controllers systemd manages enabled." -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=954765 http://bugzilla.suse.com/show_bug.cgi?id=954765#c9 --- Comment #9 from Franck Bui --- (In reply to Mel Gorman from comment #8)

Given the level of performance damage this decision incurs, can we not disable it until such time as either the controllers are all fixed to reduce the overhead? It seems to gain us very little except in specific corner cases but incurs a penalty for basic scenarios. As it is, LEAP is going to be damaged long term in comparison to other distributions.

Hi Mel, Leap is following SLE12 so it should be disabled unless it hasn't got the latest updates yet. I don't know how Leap is updated, but I'll make sure it gets the workaround. Regarding the rest of the distros (TW, Factory, ...) based on a newer version of systemd, we're still waiting for upstream to see what will be the decision taken for this matter. AFAIK we havent got any feedback to the bug you opened upstream unfortunately. Thanks, -- You are receiving this mail because: You are on the CC list for the bug.