On 7/31/21 1:26 PM, Philipp Wagner wrote:

...

Someone submitted an update to OpenLDAP 2.5. https://build.opensuse.org/request/show/909449

That someone was me.

Personally I have no objections to simply accept your request. But I'm pretty sure, someone (not me!), has to deal with the fallout. Therefore I wanted to draw some attention to your request.

Here's a bit of background: I need the client side (libldap/ldapsearch) of OpenLDAP 2.5 to support SNI (Server Name Indication) over TLS 1.3, configuration that is needed to access "Google Secure LDAP" with the OpenSSL version we have in Tumbleweed. There are more libldap options I'd also like to see in Tumbleweed.

So here's the request: If you're currently running an OpenLDAP server on Tumbleweed, or have a more sophisticated LDAP-related setup, could you give the new OpenLDAP 2.5 packages a try?

I have two flavors of my own OpenLDAP 2.5 packages I'm already testing for a while, e.g. for my Æ-DIR: https://build.opensuse.org/project/show/home:stroeder:openldap25 IMHO Æ-DIR counts as a more complex OpenLDAP setup and therefore I know that there are breaking changes. Personally I'm still using static config (aka slapd.conf) and thus I can always easily adapt this static config for 2.5, bringing up slapd again without nasty blocker issues. But if people are using back-config (aka cn=config) they would e.g. first have to remove ppolicy module, overlay config and schema from cn=config, restart slapd, and then re-add ppolicy module and overlay config. If people are still using back-bdb or back-hdb it has to be decided whether that should be still built. Still building back-bdb/-hdb contradicts SUSE's attempt to get rid of Berkeley-DB and has no real upstream support anymore. And personally I don't want to waste my time implementing repair scripts like this for smooth upgrade: https://build.opensuse.org/package/view_file/network:ldap/openldap2/fixup-mo...

Another thing to check for: OpenLDAP used to provide a thread-safe library "libldap_r". In OpenSUSE (and Fedora at least), we symlinked this library to "libldap" for a while. In OpenLDAP 2.5 the "libldap_r" library is gone and one should always use "libldap". [..] [..] we can introduce a compat-symlink to libldap_r.

I don't see a reason why not to create a compat-symlink to libldap_r. That's what I already did in my home:stroeder:openldap25/openldap2. Ciao, Michael.

Hi Andreas, On 05.08.21 08:38, Andreas Vögele wrote:

This change in OpenLDAP 2.5 is scary as it affects most users who run slapd:

"Due to database format changes between OpenLDAP 2.4 and OpenLDAP 2.5 it is mandatory for existing slapd-mdb(5) databases to be exported via an OpenLDAP 2.4 slapcat prior to upgrade. They must then be reloaded via an OpenLDAP 2.5 slapadd after the binaries are updated."

I think that OpenLDAP 2.4 cannot be replaced by version 2.5. There have to be packages for both versions. At least a statically linked slapcat from version 2.4 has to be provided.

Good point, but I'm not sure what the solution is. To widen the question a bit: What do we do in Tumbleweed generally in a case where an update requires manual intervention from the user? (In this case *before* the update is actually performed.) 1. Somehow notify the user before doing the update, and giving them a choice to not do the upgrade and e.g. pin the old version. 2. Buy us time by providing two versions of the same software. (I'd note that this solution isn't sustainable long-term, so we'll need to have the discussion about "what then" one way or the other.) 3. Try to come up with a automatic migration solution downstream (in this case, dump the db and try to re-import it). 4. Simply do the upgrade and assume users will somehow deal with it (e.g. under the assumption that "this is Tumbleweed"). I personally would go for option (1) (but I don't know how we can implement that, I'm looking for input on the packaging side here). Providing an automatic migration downstream seems risky, and solution (2) is only deferring the problem until OpenLDAP 2.4 runs out of support. I agree with you Andreas that what we have right now (4) isn't a good user experience and something I'd like to avoid if possible. Philipp

On 05/08/2021 15.39, Philipp Wagner wrote:

Hi Andreas,

On 05.08.21 08:38, Andreas Vögele wrote:

...

This change in OpenLDAP 2.5 is scary as it affects most users who run slapd:

"Due to database format changes between OpenLDAP 2.4 and OpenLDAP 2.5 it is mandatory for existing slapd-mdb(5) databases to be exported via an OpenLDAP 2.4 slapcat prior to upgrade. They must then be reloaded via an OpenLDAP 2.5 slapadd after the binaries are updated."

I think that OpenLDAP 2.4 cannot be replaced by version 2.5. There have to be packages for both versions. At least a statically linked slapcat from version 2.4 has to be provided.

Good point, but I'm not sure what the solution is. To widen the question a bit: What do we do in Tumbleweed generally in a case where an update requires manual intervention from the user? (In this case *before* the update is actually performed.)

1. Somehow notify the user before doing the update, and giving them a choice to not do the upgrade and e.g. pin the old version.

2. Buy us time by providing two versions of the same software. (I'd note that this solution isn't sustainable long-term, so we'll need to have the discussion about "what then" one way or the other.)

3. Try to come up with a automatic migration solution downstream (in this case, dump the db and try to re-import it).

4. Simply do the upgrade and assume users will somehow deal with it (e.g. under the assumption that "this is Tumbleweed").

5. Do not provide 2.5 as update of 2.4. Forces manual action to install 2.5 when 2.4 dissapears, but for a limited time both versions should be available. Somehow tell users in an update of 2.4 to 2.4 something that it is end of the road, read some link. Just some out of the box thinking, I have no idea if it is feasible :-) -- Cheers / Saludos, Carlos E. R. (from oS Leap 15.2 x86_64 (Minas Tirith))

On 8/6/21 9:33 AM, Simon Lees wrote:

On 8/6/21 5:28 AM, Carlos E. R. wrote:

...

On 05/08/2021 15.39, Philipp Wagner wrote:

...

On 05.08.21 08:38, Andreas Vögele wrote:

...

This change in OpenLDAP 2.5 is scary as it affects most users who run slapd:

"Due to database format changes between OpenLDAP 2.4 and OpenLDAP 2.5 it is mandatory for existing slapd-mdb(5) databases to be exported via an OpenLDAP 2.4 slapcat prior to upgrade. They must then be reloaded via an OpenLDAP 2.5 slapadd after the binaries are updated."

I think that OpenLDAP 2.4 cannot be replaced by version 2.5. There have to be packages for both versions. At least a statically linked slapcat from version 2.4 has to be provided.

Good point, but I'm not sure what the solution is. To widen the question a bit: What do we do in Tumbleweed generally in a case where an update requires manual intervention from the user? (In this case *before* the update is actually performed.)

1. Somehow notify the user before doing the update, and giving them a choice to not do the upgrade and e.g. pin the old version.

2. Buy us time by providing two versions of the same software. (I'd note that this solution isn't sustainable long-term, so we'll need to have the discussion about "what then" one way or the other.)

3. Try to come up with a automatic migration solution downstream (in this case, dump the db and try to re-import it).

4. Simply do the upgrade and assume users will somehow deal with it (e.g. under the assumption that "this is Tumbleweed").

5. Do not provide 2.5 as update of 2.4.

Forces manual action to install 2.5 when 2.4 dissapears, but for a limited time both versions should be available. Somehow tell users in an update of 2.4 to 2.4 something that it is end of the road, read some link.

Just some out of the box thinking, I have no idea if it is feasible :-)

This is practically the same as 2, the question is how long can we sustain both if at all.

At the end of the day whatever solution is chosen someone has to do the work and take responsibility for the custom migration tools. So the most important question is: Who? Although I'm regularly updating the openldap2 package I won't work on this because I use custom OpenLDAP packages instead anyway and I don't need such migration scripts myself. Ciao, Michael.

On Thursday 2021-08-05 09:38, Andreas Vögele wrote:

Michael Ströder writes:

...

On 7/31/21 1:26 PM, Philipp Wagner wrote:

...

...

Someone submitted an update to OpenLDAP 2.5. https://build.opensuse.org/request/show/909449

That someone was me.

Personally I have no objections to simply accept your request. But I'm pretty sure, someone (not me!), has to deal with the fallout. Therefore I wanted to draw some attention to your request.

This change in OpenLDAP 2.5 is scary as it affects most users who run slapd:

"Due to database format changes between OpenLDAP 2.4 and OpenLDAP 2.5 it is mandatory for existing slapd-mdb(5) databases to be exported via an OpenLDAP 2.4 slapcat prior to upgrade. They must then be reloaded via an OpenLDAP 2.5 slapadd after the binaries are updated."

Just for the record, that is from https://www.openldap.org/doc/admin25/appendix-upgrading.html Which... isn't something the user ever goes to see when doing rpm -U/zypper. It's 2021. mysql just autoupgrades its tables (did so for years). ffmpeg just reads everything under the sun, no matter the age. Both options should be ample inspiration for openldap, but I guess I'm just praying to the user choir. Perhaps I should preach 389ds instead. "Change my mind." Also: I'm running a 2.4 mdb file with 2.5, not large, but reading it with slapcat and/or an active slapd seems to work. (Only until I hit an impasse, I guess.)

I think that OpenLDAP 2.4 cannot be replaced by version 2.5. There have to be packages for both versions. At least a statically linked slapcat from version 2.4 has to be provided.

Duplicate the spec, add Conflicts. Don't worry about installing both side-by-side. That is the least amount of work. After all, you only need to slapcat the old db, then you can replace openldap (hence no concurrent presence), then un-slapcat it.

On 05.08.21 21:17, Jan Engelhardt wrote:

It's 2021. mysql just autoupgrades its tables (did so for years).

...and it broke badly during the leap 15.1 -> 15.2 update, so that I was unable to repair it easily. Even the db dump created with 15.1 could not be restored with 15.2. This was not too hard to fix in the end, but without lots of help from other users I would have been left out in the cold and started everything from scratch. (I don't run mariadb because I want to or am even remotely interested in it, just because owncloud and volkszaehler need it. I am not sure if anyone runs openldap because something else needs it or just because they like to feel the pain ;-) Personally I would have preferred something just preventing the update, telling me that I need to read up on the subject first and then continuing with the update. But to be fair, even mariadb upstream was not aware of that particular update failure (a relatively old database that was upgraded over the years and one particular migration path was missed, ending up with an empty global_priv table), so nobody expected this to fail. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman

On 07.08.21 09:03, Michael Ströder wrote:

On 8/5/21 9:17 PM, Jan Engelhardt wrote:

...

Also: I'm running a 2.4 mdb file with 2.5, not large, but reading it with slapcat and/or an active slapd seems to work. (Only until I hit an impasse, I guess.)

More information in this thread:

https://lists.openldap.org/hyperkitty/list/openldap-technical@openldap.org/t...

TL;DR: - DB reload is not necessary if you don't use the new back-mdb options. - In release 2.5.7 slapcat will be able to dump 2.4 MDB files without prior start of slapd.

Thanks for sorting this out, Michael. I'd then suggest to just wait for the 2.5.7 release to come out, and then we can provide OpenLDAP 2.5 as normal update to OpenLDAP 2.4. If users have been running 2.4 before, they won't be using new back-mdb options, so the update should always work. And if things still go wrong, the newer slapcat provides a convenient way to recover. I'll close the update SR for now, and open a new one with 2.5.7 once available. Philipp

On Monday 2021-08-09 14:43, Michael Ströder wrote:

If users are still running back-hdb or even the ancient back-bdb they won't be able upgrade because in your build these deprecated back-ends are disabled. They won't even be able to slapcat their databases.

Personally I think those users should have migrated their DBs to back-mdb years ago. But what's the position of the openSUSE project on this?

I could propose something like %pretrans if grep -Pq 'database\s+hdb' /etc/openldap/slapd.conf 2>/dev/null; then echo "slapd.conf: You should convert from hdb to mdb first before starting the upgrade." exit 1 fi

On 09.08.21 14:43, Michael Ströder wrote:

On 8/9/21 3:28 PM, Jan Engelhardt wrote:

...

On Monday 2021-08-09 14:43, Michael Ströder wrote:

...

If users are still running back-hdb or even the ancient back-bdb they won't be able upgrade because in your build these deprecated back-ends are disabled. They won't even be able to slapcat their databases.

Personally I think those users should have migrated their DBs to back-mdb years ago. But what's the position of the openSUSE project on this?

I could propose something like

%pretrans if grep -Pq 'database\s+hdb' /etc/openldap/slapd.conf 2>/dev/null; then echo "slapd.conf: You should convert from hdb to mdb first before starting the upgrade." exit 1 fi

Detail: The regex pattern should also cover back-bdb.

Slightly harder: The installation could use back-config and user root does not have LDAP search privileges on cn=config (or whatever the config is).

So the %pretrans task would also have to find out which file directory is used to store back-config's LDIF files and search therein.

Is this scenario common enough that we have to cover it in the upgrade? I would expect that OpenLDAP users are fairly advanced users, and users using back-config even more so. They should have a robust update procedure in place with maybe a staging setup, a rollback mechanism, etc. It seems to me that being helpful in the most common case with something like the pretrans script Jan suggested is sufficient? After all, this is an upgrade only Tumbleweed users get directly, everybody else only when they do an explicit distro update. Philipp

On Sat, Jul 31, 2021 at 7:27 AM Philipp Wagner wrote:

Another thing to check for: OpenLDAP used to provide a thread-safe library "libldap_r". In OpenSUSE (and Fedora at least), we symlinked this library to "libldap" for a while. In OpenLDAP 2.5 the "libldap_r" library is gone and one should always use "libldap". To check if you can migrate easily, check for code that links against libldap_r/-lldap_r and replace it with "-lldap". That already works today and shouldn't make a difference, apart from being forward-compatible. If that turns out infeasible we can introduce a compat-symlink to libldap_r.

No, don't . if one library is no longer provided one cannot replace it with symlinks and expect it to work. the best case is the program failing to start.. worst case, the program pretending to work and doing crazy things with your data.