On Mon, Nov 04, Imobach González Sosa wrote:

On Mon, 2019-11-04 at 09:39 +0100, Thorsten Kukuk wrote:

...

On Mon, Nov 04, Andrei Borzenkov wrote:

Hi all,

...

...

I wonder was it coordinated with YaST?

Yes, it was, but as nobody really knows which files YaST is reading we clearly missed some files.

Yes, YaST reads or writes quite a lot of files[1], especially under /etc/sysconfig.

...

YaST should use libeconf to read login.defs, like shadow, util-linux and pam do already.

The problem I see with libeconf is that, as far as I understood, it only supports key=value configuration files (as it is stated in the project's README[2]). That's fine for many configuration files (like sysconfig, login.defs, etc.), but I am not sure whether it will be good enough for other cases (ini files -having sections into account-, YAML- like (Salt), JSON (docker), etc.).

The whole stuff makes only sense with ini style files.

Why not using augeas (or any other similar library) to parse the files instead of rolling our own? libaugeas and its lenses are quite small.

The parser is far the smallest part, and we haven't found anything which is merging the configuration files in the way we need it (the systemd developer were not willing to provide their code as standalone library or make the interface public). Thorsten

The YaST team has a Ruby gem[3] that offers a high level API to configuration files using augeas to parse them. As a proof of concept, we already implemented support to behave more or less like libeconf. But I would be great if we could join forces and reduce code duplication. Of course, we could rely on libeconf when possible and use augeas when needed...

Regards, Imo

[1] https://etherpad.suse.de/p/yast-usr-etc [2] https://github.com/openSUSE/libeconf/blob/master/README.md [3] https://github.com/config-files-api/config_files_api [4] https://github.com/config-files-api/config_files_api/pull/32

...

-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg) -- Imobach González Sosa YaST Team at SUSE LINUX GmbH https://imobachgs.github.io/

-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

V Mon, 4 Nov 2019 13:19:29 +0100 Thorsten Kukuk napsáno:

On Mon, Nov 04, Imobach González Sosa wrote:

...

On Mon, 2019-11-04 at 09:39 +0100, Thorsten Kukuk wrote:

...

On Mon, Nov 04, Andrei Borzenkov wrote:

Hi all,

...

...

I wonder was it coordinated with YaST?

Yes, it was, but as nobody really knows which files YaST is reading we clearly missed some files.

Yes, YaST reads or writes quite a lot of files[1], especially under /etc/sysconfig.

...

YaST should use libeconf to read login.defs, like shadow, util-linux and pam do already.

The problem I see with libeconf is that, as far as I understood, it only supports key=value configuration files (as it is stated in the project's README[2]). That's fine for many configuration files (like sysconfig, login.defs, etc.), but I am not sure whether it will be good enough for other cases (ini files -having sections into account-, YAML- like (Salt), JSON (docker), etc.).

The whole stuff makes only sense with ini style files.

Hi thorsten, I am a bit lost here. Do you mean with this statement that libeconf makes sense only with ini files? Or that whole /usr/etc is intended for ini files? Or that merging makes sense only for ini files?

...

Why not using augeas (or any other similar library) to parse the files instead of rolling our own? libaugeas and its lenses are quite small.

The parser is far the smallest part, and we haven't found anything which is merging the configuration files in the way we need it (the systemd developer were not willing to provide their code as standalone library or make the interface public).

As Imo write, our approach with augeas is that we get configuration tree and do operation on top of that tree like merging and we get final result and it also allows various strategy for merging as we learned, that sometimes logic is quite complicated and merging is not just about /etc and /usr/etc, but in fact also various conf.d directory that do config snippets that can overwrite. We pick augeas due to number of existing parsers and also ability to edit with minimal changes. That allows us to share with upstream various fixes and faster react to changes in configuration. I just worry that your parser is small only as long as you not handle all possible corner cases of configuration formats. Know problems even with simple ini files we face so far are windows line ending, strange characters, escaping and various format extensions. Josef

Thorsten

...

The YaST team has a Ruby gem[3] that offers a high level API to configuration files using augeas to parse them. As a proof of concept, we already implemented support to behave more or less like libeconf. But I would be great if we could join forces and reduce code duplication. Of course, we could rely on libeconf when possible and use augeas when needed...

Regards, Imo

[1] https://etherpad.suse.de/p/yast-usr-etc [2] https://github.com/openSUSE/libeconf/blob/master/README.md [3] https://github.com/config-files-api/config_files_api [4] https://github.com/config-files-api/config_files_api/pull/32

...

-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg) -- Imobach González Sosa YaST Team at SUSE LINUX GmbH https://imobachgs.github.io/

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

V Tue, 5 Nov 2019 19:40:08 +0100 Thorsten Kukuk napsáno:

On Tue, Nov 05, Josef Reidinger wrote:

...

V Mon, 4 Nov 2019 13:19:29 +0100 Thorsten Kukuk napsáno:

...

...

The whole stuff makes only sense with ini style files.

Hi thorsten, I am a bit lost here. Do you mean with this statement that libeconf makes sense only with ini files? Or that whole /usr/etc is intended for ini files? Or that merging makes sense only for ini files?

The merging of several snippets to one file makes only sense for some formats, most likely ini files. It's pretty hard for some other files like yaml or json.

What is exactly difference in data format that prevent merging? For it is not much difference if file1: A => 1, B => 2 file2: A => 3 file3: C => 4 result: A => 3, B => 2, C => 4 depends on format. So I still do not see how merging and configuration format is related together.

So while the /usr/etc move makes sense for all configuration files, the best solution for every single file really depends on the format of the configuration file, there is no "this library will solve all use cases".

That is why I think that best approach is to provide reasonable default and allow to redefine strategy for merging when it makes sense for user. In our POC we basically do simple replace as I did above when merging multiple files, but allow user to redefine it if there is specific behavior needed ( like concating values for keys, strange precedence, etc ). This way it is easy to use for us and also does not block us when we need more complicated strategy. Josef

Thorsten

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

06.11.2019 16:43, Bernhard M. Wiedemann пишет:

On 06/11/2019 14.07, Josef Reidinger wrote:

...

What is exactly difference in data format that prevent merging? For it is not much difference if

file1: A => 1, B => 2 file2: A => 3 file3: C => 4

result: A => 3, B => 2, C => 4

depends on format. So I still do not see how merging and configuration format is related together.

yaml and json support nested structures and there can be

A: B: 1

and A: C: 2

How is it different from [A] B=1 [A] C=2 in .ini file?

and merging them could either give A: C: 2

Can you explain how merging can give that result?

or

A: B: 1 C: 2

and if the merge produce the latter, it is not clear how to unset values

Neither is it possible to unset keys with .ini files. You can only set them to empty value and treat empty value as unset value (that is what systemd does) but then you can do it with any serialization format.

- most likely you have to explicitly include them again and set them again.

-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org