On 11.08.2023 19:59, Manfred Hollstein wrote:

Moin,

to be honest, I'm afraid the whole UsrMerge activitiy told us something completely different!

On Fri, 11 Aug 2023, 18:40:44 +0200, Franck Bui wrote:

...

Hi,

The way the systemd daemons (such as PID1, logind) parse their configuration file is specified in systemd-system.conf(5) man page (section CONFIGURATION DIRECTORIES AND PRECEDENCE).

Basically the daemons parse first their "main" configuration file (e.g /etc/systemd/system.conf for PID1) and then read and apply the optional drop-in files located in the following directories:

/usr/lib/systemd/*.conf.d/ /usr/local/lib/systemd/*.conf.d/ /run/systemd/*.conf.d/ /etc/systemd/*.conf.d/

*If* systemd (PID1) has access to /usr/lib/systemd/system.conf, it *must* use that file - unless there is a file /etc/systemd/system.conf.

During the process to migrate all /{bin,etc,lib}/* files, we were told to leave the file at /usr/* untouched and make our local modifications to the files at /etc/...

Now systemd wants to tell us a different logic :(

...

It's important to note that drop-in files have higher precedence than the main configuration file. Hence if a package customizes an option by shipping a drop-in file in /usr/lib/systemd/*.conf.d/, it might override the value already set by the user if the same option was customized by the sysadmin in the main configuration.

There were various discussions [1] about addressing this concern but at the end upstream preferred keeping the existing logic. This implies that using the main configuration files is not 100% safe (even if it's been documented since a long time) especially since openSUSE distros relies on drop-in files to tune a few upstream defaults, see systemd-default-settings package to get the list.

To avoid inconvenient divergences with upstream, we'll follow the existing logic but this implies that:

- use of main config files, such as /etc/systemd/{system.conf,logind.conf}, should be avoided and instead customization should be done via drop-in files, such as /etc/systemd/system.conf.d/50-local.conf.

- Some drop-in priority ranges should be defined to make sure that drop-in created by users have always higher precedence over the ones shipped by packages. The priority ranges reserved for users and downstream will be defined and documented in the relevant wiki page.

- at installation, systemd packages will stop installing (empty/template) main configuration files and instead users will find the templates in drop-in directories (such as /etc/systemd/*.conf.d/50-{local,main,xxx}.conf, exact filename to be defined).

- optionally, the packaging stuff will take care of converting existing main configuration files into drop-in files during package update.

If there's no objection with the above, this change will be probably take place during the upgrade to systemd v254 or quickly after.

Again, to be honest, we had a lot of effort and work to teach the various packages to be able to deal with two different configuration directories below /usr and /etc for overriding stuff, now systemd wants to revert that? I strongly vote NO!

NO to what? Doing nothing and leaving it as is will not magically make systemd honor /usr/lib/systemd/system.conf, nor will it stop it from overriding settings from /etc/systemd/system.conf with something from /usr/lib/sytemd/system.conf.d. That said, I do not think this change will be beneficial short term. Users will continue to edit /etc/systemd/system.conf because they found description on Internet. Having link /etc/systemd/system.conf -> /etc/systemd/system.conf.d/50-main.conf mai mitigate it as long as systemd allows it in the first place.

...

Thanks.

[1] https://github.com/systemd/systemd/pull/17161

Cheers.

l8er manfred

On Fri, Aug 11, Manfred Hollstein wrote:

Moin,

to be honest, I'm afraid the whole UsrMerge activitiy told us something completely different!

No, it does not.

On Fri, 11 Aug 2023, 18:40:44 +0200, Franck Bui wrote:

...

Hi,

The way the systemd daemons (such as PID1, logind) parse their configuration file is specified in systemd-system.conf(5) man page (section CONFIGURATION DIRECTORIES AND PRECEDENCE).

Basically the daemons parse first their "main" configuration file (e.g /etc/systemd/system.conf for PID1) and then read and apply the optional drop-in files located in the following directories:

/usr/lib/systemd/*.conf.d/ /usr/local/lib/systemd/*.conf.d/ /run/systemd/*.conf.d/ /etc/systemd/*.conf.d/

*If* systemd (PID1) has access to /usr/lib/systemd/system.conf, it *must* use that file - unless there is a file /etc/systemd/system.conf.

During the process to migrate all /{bin,etc,lib}/* files, we were told to leave the file at /usr/* untouched and make our local modifications to the files at /etc/...

Now systemd wants to tell us a different logic :(

No, it is not about /usr/lib/systemd/system.conf vs /etc/systemd/system.conf, it is about the logic how to handle ".d" directories and snippets. You are mixing up two differnt things. Read the next sentence again:

...

It's important to note that drop-in files have higher precedence than the main configuration file. Hence if a package customizes an option by shipping a drop-in file in /usr/lib/systemd/*.conf.d/, it might override the value already set by the user if the same option was customized by the sysadmin in the main configuration.

Again, to be honest, we had a lot of effort and work to teach the various packages to be able to deal with two different configuration directories below /usr and /etc for overriding stuff, now systemd wants to revert that? I strongly vote NO!

No, systemd does not revert anything and none of our effords is void with this. Between, libeconf behaves in the same way with with drop-in files as systemd. And with drop-in files, the main configuration file should no longer be used at all. For other parts is already normal, or do you know anybody who is still using /etc/pam.conf? So systemd is only consequent here. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect, Future Technologies SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)

On Sat, Aug 12, Olaf Hering wrote:

Fri, 11 Aug 2023 18:40:44 +0200 Franck Bui :

...

- at installation, systemd packages will stop installing (empty/template) main configuration files and instead users will find the templates in drop-in directories (such as /etc/systemd/*.conf.d/50-{local,main,xxx}.conf, exact filename to be defined).

systemd is well documented, systemd.directives(7) is one potential entry point.

Please stop polluting /etc with empty and rpm-owned configuration files.

+1 Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect, Future Technologies SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)

On 8/14/23 12:34, Martin Wilck via openSUSE Factory wrote:

But having to copy stuff from /usr/lib to /etc has become commonplace, so perhaps storing the default config file as /usr/lib/systemd/xxx.conf or /usr/lib/systemd/xxx.d/00-defaults.conf should be sufficient.

Wouldn't that suggest to override the defaults with /etc/systemd/xxx.conf or /etc/systemd/xxx.d/00-defaults.conf ?

On 8/18/23 11:42, Martin Wilck via openSUSE Factory wrote:

In a way, yes. I suppose the file would have comments at the top, hinting how to override the settings. Actually the files contain such hints already, e.g. in /etc/systemd.journald.conf:

# Entries in this file show the compile time defaults. Local configuration # should be created by either modifying this file, or by creating "drop-ins" in # the journald.conf.d/ subdirectory. The latter is generally recommended. # Defaults can be restored by simply deleting this file and all drop-ins. # # Use 'systemd-analyze cat-config systemd/journald.conf' to display the full config.

The text "by either modifying this file" would need to be adapted, and a hint would need to be added that drop-ins under /usr/lib override settings from /etc/journald.conf. Otherwise this should be fine.

Since the goal is to not install files in /etc anymore that sounds the best approach, yes.

OTOH, if people memorize "systemd-analyze cat-config", we won't need these  hints any more. Will they?

No they won't indeed but I would still keep the hints for those who don't have a clue about systemd-analyze yet. Thanks.

On Thu, Aug 17, Franck Bui wrote:

On 8/12/23 09:01, Olaf Hering wrote:

...

Fri, 11 Aug 2023 18:40:44 +0200 Franck Bui :

...

- at installation, systemd packages will stop installing (empty/template) main configuration files and instead users will find the templates in drop-in directories (such as /etc/systemd/*.conf.d/50-{local,main,xxx}.conf, exact filename to be defined).

systemd is well documented, systemd.directives(7) is one potential entry point.

Please stop polluting /etc with empty and rpm-owned configuration files.

That would be the simplest way to switch to drop-in files only from a packager POV.

However I think that a lot of users would be confused by the fact that the main config files are missing. And even worse would be tempted to reintroduce them when they would need to override the defaults as the use of drop-ins is not wide spread yet.

I disagree. Other tools did that in the past, too (most prominent one: Linux-PAM). It did take some time until people did start learning, but in the end it was quite easy. Much simpler then having the pain to update this config files later.

The advantages of shipping /etc/systemd/*.conf.d/50-local.conf are:

- it implicitly teaches users to use a drop-in file to override defaults

If they are not aware of drop-ins they will follow the search engine hits and create the main config files again.

- it list all config options and their default in a concise form (as pointed out by Martin) which is convenient.

This is not convenient, this is the major pain for updates. Everytime, a config changes upstram and an admin made a local change, we will either have rpmsave or rpmnew files, which makes the whole thing void and all advantages are gone. And you will never get this cleaned up again.

So I would still be tempted to ship the "template" drop-ins at least for now and make them go away in the future when drop-ins will be common practice. That would be a more user-friendly transition IMHO.

Sorry, but with todays RPM you will never get rid of them again, we have enough other packages with exactly the same problem. Don't repeat all the mistakes we made in the past because somebody wrongly claims "it's more user friendly", in the end it is not. But learn from it and make it better. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect, Future Technologies SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)

On Thu, Aug 17, Michal Suchánek wrote:

On Thu, Aug 17, 2023 at 02:38:42PM +0000, Thorsten Kukuk wrote:

...

I disagree. Other tools did that in the past, too (most prominent one: Linux-PAM). It did take some time until people did start learning, but in the end it was quite easy. Much simpler then having the pain to update this config files later.

These are much less likely to be changed.

It doesn't matter how often they get changed, once is already one time too often. And if they should contain the default settings as comment, they will be changed very frequently.

...

...

- it list all config options and their default in a concise form (as pointed out by Martin) which is convenient.

This is not convenient, this is the major pain for updates. Everytime, a config changes upstram and an admin made a local change, we will either have rpmsave or rpmnew files, which makes the whole thing void and all advantages are gone. And you will never get this cleaned up again.

With the fifes shipping only documentation and no actual settings there is absolutely no problem with rpmnew files.

It is a problem because it contradicts the original idea to have the current settings there to make it easier for the admin. If you don't care if the defaults are valid, complete and correct, then we don't need them at all. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect, Future Technologies SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)

Thu, 17 Aug 2023 16:51:10 +0200 Michal Suchánek :

With the fifes shipping only documentation and no actual settings there is absolutely no problem with rpmnew files.

So you are saying we will never reach the goal of "no rpm-owned files in /etc"? Was that even a goal? Perhaps it was just an idea, and the current state has no practical downside. Olaf

On 8/18/23 13:04, Richard Brown wrote:

We should work towards that ideal, and not accept the current status quo as acceptable.

Ok then we'll head toward this goal by stopping shipping the various configuration files from systemd in /etc. Should we still consider migrating the current main config files into drop-ins ? Thanks all for the inputs.

On 8/21/23 15:27, Richard Brown wrote:

Else, I think it's best to leave them there, with the assumption that the current user is happy with their current config as defined in /etc

Leaving them might be the best option indeed but is there a way to instruct rpm to not convert them into .rpmsave backup files when the package will be updated? IMU an .rpmsave file will be created each time rpm notices that a %config file has been modified by the user and is no more part of the package file list of the new package version. Thanks.

Franck Bui wrote:

On 8/21/23 15:27, Richard Brown wrote:

...

Else, I think it's best to leave them there, with the assumption that the current user is happy with their current config as defined in /etc

Leaving them might be the best option indeed but is there a way to instruct rpm to not convert them into .rpmsave backup files when the package will be updated?

IMU an .rpmsave file will be created each time rpm notices that a %config file has been modified by the user and is no more part of the package file list of the new package version.

You can mark the files in /etc as %ghost. That way they are still listed in the file list so rpm won't touch existing ones. The downside is that rpm also won't remove unmodified versions of such config files of a previous package. A missing upstream rpm feature IMO. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH; GF: Ivo Totev, Andrew McDonald, Werner Knoblich; HRB 36809 (AG Nürnberg)

On 12.08.2023 12:14, Jan Engelhardt wrote:

On Friday 2023-08-11 18:40, Franck Bui wrote:

...

Basically the daemons parse first their "main" configuration file (e.g /etc/systemd/system.conf for PID1) and then read and apply the optional drop-in files located in the following directories:

/usr/lib/systemd/*.conf.d/ /usr/local/lib/systemd/*.conf.d/ /run/systemd/*.conf.d/ /etc/systemd/*.conf.d/

There were various discussions [1] about addressing this concern but at the end upstream preferred keeping the existing logic. [1] https://github.com/systemd/systemd/pull/17161

The reasoning there in [1] is sound; it's all getting too complicated and it is best to stick to just fragments (drop-ins).

The use of "early config" files that you proposed makes it even harder. How about an alternate proposal:

* Remember the "priority" where the main config file was found; /usr (low), /run (medium), /etc (high)

* _Only_ read drop-ins from locations with same-or-higher priority.

IOW,

* if /usr/xx.conf was used, read fragments from /usr/xx.d,/run/xx.d,/etc/xx.d * if /run/xx.conf was used, read fragments from /run/xx.d,/etc/xx.d * if /etc/xx.conf was used, read fragments from /etc/xx.d

Which makes it impossible for packages to install drop-ins in /usr if user edited file in /etc.

(This is _also_ not backwards-compatible obviously, but at least it's better than reading /usr/xx.d at unexpected times.)

On Sat, 2023-08-12 at 15:55 +0200, Jan Engelhardt wrote:

On Saturday 2023-08-12 13:10, Andrei Borzenkov wrote:

...

...

IOW,

  * if /usr/xx.conf was used, read fragments from   /usr/xx.d,/run/xx.d,/etc/xx.d   * if /run/xx.conf was used, read fragments from /run/xx.d,/etc/xx.d   * if /etc/xx.conf was used, read fragments from /etc/xx.d

Which makes it impossible for packages to install drop-ins in /usr if user edited file in /etc.

If the user placed /etc/xx.conf (rather than /etc/xx.d/xxx), they didn't want anything from /usr anyway.

I don't follow. Even if a user made some customizations in xx.conf, she may want still to follow package maintainer's customizations in /usr/lib/xx.d/. Your proposal would mean that she'd have to copy all files with directives she wants applied from /usr/lib to /etc. Such files might be added by package updates, and she might not even be aware of them. I guess what could be done (with additional complexity) would be something like /etc/xx.d/00-directories.conf, where a user could say that she wants /usr/lib and /run (or whatever else) to be ignored. Martin

On Monday, 14 August 2023 8:01:02 PM ACST Rodney Baker wrote:

On Monday, 14 August 2023 7:44:03 PM ACST Martin Wilck via openSUSE Factory [...]

...

I don't follow. Even if a user made some customizations in xx.conf, she may want still to follow package maintainer's customizations in /usr/lib/xx.d/. Your proposal would mean that she'd have to copy all files with directives she wants applied from /usr/lib to /etc. Such files might be added by package updates, and she might not even be aware of them.

I guess what could be done (with additional complexity) would be something like /etc/xx.d/00-directories.conf, where a user could say that she wants /usr/lib and /run (or whatever else) to be ignored.

Martin

Not so. If one uses systemctl edit <unit file>, it creates an override file in /etc/systemd/system/<unit>.d into which the admin can place directives that override those specific directives in the primary unit file in /lib/systemd/ system/. Only those directives specified in the override.conf file are affected - all other directives in the primary unit file stil apply.

For example, from one of my systems, I have /lib/systemd/system/graylog- server.service, which has the default directives, but in /etc/systemd/system/ graylog-server.service.d/override.conf there are 2 directives:

[Unit] After=opensearch.service

[Service] AmbientCapabilities=CAP_NET_BIND_SERVICE

These override the defaults in /lib/systemd/system/graylog-server.service.

I was not a fan of systemd when it first appeared, but things like this actually make system admistration easier once you learn how to use them (and the logic behind them).

It's also worth noting that systemctl status <unit> shows the unit file and drop-ins in use... root@vm-srv-graylog ~ # systemctl status graylog-server.service ● graylog-server.service - Graylog server Loaded: loaded (/lib/systemd/system/graylog-server.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/graylog-server.service.d └─override.conf Active: active (running) since Sun 2023-08-06 15:58:08 ACST; 1 weeks 1 days ago [...output truncated...] -- ================================================================================================================== Rodney Baker rodney.baker@iinet.net.au ==================================================================================================================

On Mon, Aug 14, Joe Salmeri wrote:

Robert, Is there some config option that must be enabled to for systemctl status to show the Drop-In items ?

I have a drop in for the journal configuration in /etc/systemd/journald.conf.d/ but it does not show up when I check the status...

That's for journald.conf, which is not a systemd unit.

systemctl status systemd-journald.service

That's for systemd units... No idea if journalctl can/will show you drop in files, too, but systemctl only works on systemd units, not something else. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect, Future Technologies SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)

On Mon, Aug 14, Joe Salmeri wrote:

Thanks very much for the explanation.   It is confusing that something that is not a systemd unit would have their configuration file in /etc/systemd.

/etc/systemd is for the full systemd eco system (which also includes e.g. systemd-network). /etc/systemd/system is for systemd units. It's not confusing at all. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect, Future Technologies SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nuernberg, Germany Managing Director: Ivo Totev, Andrew McDonald, Werner Knoblich (HRB 36809, AG Nürnberg)

On Mon, 2023-08-14 at 12:44 +0200, Jan Engelhardt wrote:

On Monday 2023-08-14 12:14, Martin Wilck via openSUSE Factory wrote:

...

...

...

Which makes it impossible for packages to install drop-ins in /usr if user edited file in /etc.

If the user placed /etc/xx.conf (rather than /etc/xx.d/xxx), they didn't want anything from /usr anyway.

I don't follow. Even if a user made some customizations in xx.conf, she may want still to follow package maintainer's customizations in /usr/lib/xx.d/.

No. If you wanted to follow package maintainer's customizations, you would use drop-ins (/etc/xx.d in my spelling), not a full config (xx.conf).

This assumes that every user understands and adopts the drop-in policy immediately. Also, it adds a non-obvious assumption about the user's intentions to the existence of the xx.conf file. More often than not, xx.conf will be a leftover from the past, created with no other intention than just modifying one specific setting. The current scheme with both conf files and drop-ins in different directories is complex, but it has a concise logic: files in one hierarchy override files _of the same name_ in another. Your scheme further complicates this by adding a rule "file X in one hierarchy overrides file Y in another". I can see that some admins may want to configure their systems in a way that no package-installed drop-in would ever override their own settings. But I think this must be achieved differently.

This follows directly from when you eliminate drop-ins from the picture and consider a system that uses just "full" config files. Like we used to in SUSE 6.x or something.

I don't remember 6.x, sorry. Martin

On 8/12/23 11:14, Jan Engelhardt wrote:

How about an alternate proposal:

The point of this discussion is to stop diverging from upstream and other distros when it comes to parsing config files. So unless your proposal gets accepted by upstream, which is unlikely since it's not backwards compatible, I don't think we should try to explore new paths. Thanks.