obs-build standalone dist auto detection broken with Leap 15.3+
Hi I'm probably one of three users worldwide that use obs-build as a standalone solution and out of these three the only one that is actually using the distribution auto-detection feature.[0] I'll have to give a bit context for that. If you use obs-build standalone (i.e., without any obs-server integration) and do not pass the --dist parameter, it will try to auto-detect the distribution in expanddeps[1] using some magic in Build.pm[2] by inspecting the DISTRIBUTION RPM tag of the "rpm" package (yes, literally the binary "rpm" package that is falling out of the "rpm" source package). This makes sense, since the "rpm" package is a prime-time member of any distribution release and we know that it will always be there (unless SUSE decides to switch to a different low-level package manager). This method has been working fine for decades. I've had to tune the code slightly in 2016 to get Leap support[3], but generally it has been smooth sailing. Starting with Leap 15.3 (and partially the testing version of Leap 15.2, which was used as a testing ground for Leap 15.3), which copies binaries directly from SLE, this method now breaks: $ rpm -q --queryformat '%{NAME}-%{VERSION}-%{RELEASE}: %{DISTRIBUTION}\n' *.rpm rpm-4.14.1-29.46 [Leap 15.4]: SUSE Linux Enterprise 15 rpm-4.14.1-lp152.17.5 [Leap 15.2]: openSUSE Leap 15.2 rpm-4.14.3-150300.46.1 [Leap 15.3]: SUSE Linux Enterprise 15 (I manually amended the nvr output with the repository I fetched the package from for this post.) "SUSE Linux Enterprise 15" will be automatically mapped to sles15sp2 or sles15 depending on rpm's dependency upon libgcrypt - neither of which is suitable to detect Leap versions and it certainly wouldn't detect Leap 15.3 or 15.4 in any case. The other issue is that, if rpm was really copied from SLE 15 SP2 verbatim[4], the distribution RPM tag used in SLE is omitting any service pack information, so it's impossible to determine the actual operating system version based on that. Hacks, such as looking at the dependencies and figuring out what service pack it's coming from via newly introduced dependencies only work the first time around, i.e., to distinguish between sle15 and sle15sp2, but not between, e.g., sle15 and sle15sp1 or sle15sp2 and sle15sp3. Fixing these two issues is really easy, but requires the introduction of two new policies: 1.) Always rebuild the "rpm" package for Leap - don't just copy it from SLE. While I understand that rebuilds are costly, especially if the sources and dependencies are unchanged, I'm only asking to make an exception for the core package that will be used in auto-detection - "rpm". All other packages can happily be copied from SLE. 2.) For SLE itself, the distribution RPM tag should also include service pack information, and the "rpm" package likewise always be rebuild for any new service pack release - no matter whether the source or direct dependencies actually changed. Is this something that could be done? It's too late for Leap 15.3 now (and by extension sle15sp3), since it's already EOL, but it's not too late to fix Leap 15.4 and higher (also sle15sp4+). Best regards, Mihai [0] The numbers are completely made up, but sound believable. [1] https://github.com/openSUSE/obs-build/blob/eaf56f9af8e4bf4c4cf61560ce2454030... [2] https://github.com/openSUSE/obs-build/blob/eaf56f9af8e4bf4c4cf61560ce2454030... [3] https://github.com/openSUSE/obs-build/commit/66328f9e93716fb2c24eeebe5057a49... [4] I haven't downloaded the actual SLE media to cross-check this assumption, but I bet that the distribution tag would be the same for the rpm package on the actual SLE installation media/repository.
On 18.02.23 13:28, Mihai Moldovan wrote:
Hi
I'm probably one of three users worldwide that use obs-build as a standalone solution and out of these three the only one that is actually using the distribution auto-detection feature.[0]
I'll have to give a bit context for that. If you use obs-build standalone (i.e., without any obs-server integration) and do not pass the --dist parameter, it will try to auto-detect the distribution in expanddeps[1] using some magic in Build.pm[2] by inspecting the DISTRIBUTION RPM tag of the "rpm" package (yes, literally the binary "rpm" package that is falling out of the "rpm" source package). This makes sense, since the "rpm" package is a prime-time member of any distribution release and we know that it will always be there (unless SUSE decides to switch to a different low-level package manager).
This method has been working fine for decades. I've had to tune the code slightly in 2016 to get Leap support[3], but generally it has been smooth sailing.
Starting with Leap 15.3 (and partially the testing version of Leap 15.2, which was used as a testing ground for Leap 15.3), which copies binaries directly from SLE, this method now breaks:
$ rpm -q --queryformat '%{NAME}-%{VERSION}-%{RELEASE}: %{DISTRIBUTION}\n' *.rpm rpm-4.14.1-29.46 [Leap 15.4]: SUSE Linux Enterprise 15 rpm-4.14.1-lp152.17.5 [Leap 15.2]: openSUSE Leap 15.2 rpm-4.14.3-150300.46.1 [Leap 15.3]: SUSE Linux Enterprise 15
(I manually amended the nvr output with the repository I fetched the package from for this post.)
"SUSE Linux Enterprise 15" will be automatically mapped to sles15sp2 or sles15 depending on rpm's dependency upon libgcrypt - neither of which is suitable to detect Leap versions and it certainly wouldn't detect Leap 15.3 or 15.4 in any case.
The other issue is that, if rpm was really copied from SLE 15 SP2 verbatim[4], the distribution RPM tag used in SLE is omitting any service pack information, so it's impossible to determine the actual operating system version based on that.
Well, for 15.4 and SLES15-SP4, it is actually the case: server:~ # rpm -q rpm rpm-4.14.3-150300.52.1.x86_64 server:~ # rpm -q --queryformat %{disturl}\\n rpm obs://build.suse.de/SUSE:Maintenance:26687/SUSE_SLE-15-SP3_Update/f0914b2c279f3c55f9d8c41439b2e804-rpm.SUSE_SLE-15-SP3_Update
Hacks, such as looking at the dependencies and figuring out what service pack it's coming from via newly introduced dependencies only work the first time around, i.e., to distinguish between sle15 and sle15sp2, but not between, e.g., sle15 and sle15sp1 or sle15sp2 and sle15sp3.
Fixing these two issues is really easy, but requires the introduction of two new policies:
1.) Always rebuild the "rpm" package for Leap - don't just copy it from SLE. While I understand that rebuilds are costly, especially if the sources and dependencies are unchanged, I'm only asking to make an exception for the core package that will be used in auto-detection - "rpm". All other packages can happily be copied from SLE. 2.) For SLE itself, the distribution RPM tag should also include service pack information, and the "rpm" package likewise always be rebuild for any new service pack release - no matter whether the source or direct dependencies actually changed.
This would bring no benefit, except for your fringe usecase and I'm still feeling like "you're doing it wrong"... If I wanted to do what I'm thinking you are wanting to do, I'd probably implement somehthing like "check which of the packages contains /etc/os-release, then check what distribution it is coming from" (or just unpack it and parse os-release). This will also work once you try to build for RHEL-x.y :-) Personally I'd skip all that and just run a proper build service setup. Good luck, -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
* On 2/18/23 14:24, Stefan Seyfried wrote:
On 18.02.23 13:28, Mihai Moldovan wrote:
The other issue is that, if rpm was really copied from SLE 15 SP2 verbatim[4], the distribution RPM tag used in SLE is omitting any service pack information, so it's impossible to determine the actual operating system version based on that.
Well, for 15.4 and SLES15-SP4, it is actually the case:
server:~ # rpm -q rpm rpm-4.14.3-150300.52.1.x86_64 server:~ # rpm -q --queryformat %{disturl}\\n rpm obs://build.suse.de/SUSE:Maintenance:26687/SUSE_SLE-15-SP3_Update/f0914b2c279f3c55f9d8c41439b2e804-rpm.SUSE_SLE-15-SP3_Update
True, but the disturl tag is rather new and... I'm assuming that this is really SLE or Leap 15.3? If it were 15.4, it would mean that the package was copied from SUSE:SLE-15-SP3:Update, which again would make it impossible to differentiate between sle15sp3 and sle15sp4. Parsing the disturl tag also looks like a nightmare.
This would bring no benefit, except for your fringe usecase [...]
Copying binary packages from SLE instead of rebuilding them from source brings no benefit, except to lower stress on the build infrastructure. Throwing around straw man arguments is not helpful. Avoiding rebuilds wherever possible is really useful[0] and I welcome the "Closing the Leap Gap" approach. This said, making an exception for one package only would fix a currently broken feature within obs-build at a negligible cost. I've already admitted straight up that this feature is a real niche, since obs-server will probably always spawn obs-build with a --dist argument (although this is difficult to know for sure, since the obs-build invocation does not usually show up in OBS build logs).
and I'm still feeling like "you're doing it wrong"...
If I wanted to do what I'm thinking you are wanting to do, I'd probably implement somehthing like [...]
I'm really not just "holding it wrong" or doing anything too crazy. My use case is to build software for a lot of different operating systems, including OpenSUSE and SLE versions. For SUSE-based systems, I'm first spawning builds to create a source package using something like that: obs-build --nosignature --repo https://download.opensuse.org/distribution/leap/x.y/repo/oss// --repo https://download.opensuse.org/update/leap/x.y/oss/ [--repo other repositories managed by me] --root /var/cache/obs-build/opensuse/x.y/x86_64/ --clean --stage=-bs --define '%vendor ...' /path/to/file.spec and afterwards build binary packages using pretty much the same approach: obs-build --nosignature --repo https://download.opensuse.org/distribution/leap/x.y/repo/oss// --repo https://download.opensuse.org/update/leap/x.y/oss/ [--repo other repositories managed by me] --root /var/cache/obs-build/opensuse/x.y/x86_64/ --arch [actual arch] --clean --stage=-bb --debug --define '%vendor ...' /path/to/file.spec My goal is to just avoid having to pass "--dist ..." directly, since obs-build can usually figure out the distribution by itself, if the RPM tags of the rpm's package, coming directly from either the official main or update repositories, RPM tags allow that. Mihai [0] Although it can make Reproducible Builds a lot more difficult, since the binaries are coming from a non-public build system and a lot of metadata about the builds is hence unavailable. I don't know how critical or interesting RB is for OpenSUSE (or SUSE in general).
On 18.02.23 17:00, Mihai Moldovan wrote:
* On 2/18/23 14:24, Stefan Seyfried wrote:
On 18.02.23 13:28, Mihai Moldovan wrote:
The other issue is that, if rpm was really copied from SLE 15 SP2 verbatim[4], the distribution RPM tag used in SLE is omitting any service pack information, so it's impossible to determine the actual operating system version based on that.
Well, for 15.4 and SLES15-SP4, it is actually the case:
server:~ # rpm -q rpm rpm-4.14.3-150300.52.1.x86_64 server:~ # rpm -q --queryformat %{disturl}\\n rpm obs://build.suse.de/SUSE:Maintenance:26687/SUSE_SLE-15-SP3_Update/f0914b2c279f3c55f9d8c41439b2e804-rpm.SUSE_SLE-15-SP3_Update
True, but the disturl tag is rather new and... I'm assuming that this is really SLE or Leap 15.3? If it were 15.4, it would mean that the package was copied from SUSE:SLE-15-SP3:Update, which again would make it impossible to differentiate between sle15sp3 and sle15sp4. Parsing the disturl tag also looks like a nightmare.
1.) the %disturl is not new. I don't have sufficiently old systems at hand to look at but it's certainly in SLES12 from the beginning and I'm pretty sure also in SLES11. 2) you did misinterpret what I wanted to say: 15.4 is getting rpm from sles15-sp3-update channel (as is sles15-sp4). I'm pretty happy (being a huge SLES customer) to have as much identical packages / package versions across all SLES15 variants still in use (hint: all variants...) that I have to take care of. Quite some packages are actually getting their updates from the SLES15-GA update repo. I like that. On my 15.4 server machine at home, it's 244 packages right now: server:~ # rpm -qa | grep -c -- -150000 244
My use case is to build software for a lot of different operating systems, including OpenSUSE and SLE versions.
I have the same use case.
For SUSE-based systems, I'm first spawning builds to create a source package using something like that:
obs-build --nosignature --repo https://download.opensuse.org/distribution/leap/x.y/repo/oss// --repo https://download.opensuse.org/update/leap/x.y/oss/ [--repo other repositories managed by me] --root /var/cache/obs-build/opensuse/x.y/x86_64/ --clean --stage=-bs --define '%vendor ...' /path/to/file.spec
and afterwards build binary packages using pretty much the same approach:
obs-build --nosignature --repo https://download.opensuse.org/distribution/leap/x.y/repo/oss// --repo https://download.opensuse.org/update/leap/x.y/oss/ [--repo other repositories managed by me] --root /var/cache/obs-build/opensuse/x.y/x86_64/ --arch [actual arch] --clean --stage=-bb --debug --define '%vendor ...' /path/to/file.spec
For rpm based distributions, I just put them into our obs instance and be done with that. Builds automatically without any magic.
My goal is to just avoid having to pass "--dist ..." directly, since obs-build
This is certainly a worthy goal, but as you need to know your dist parameter anyway for the "--repo" argument, I'm not that sure how much this saves.
can usually figure out the distribution by itself, if the RPM tags of the rpm's package, coming directly from either the official main or update repositories, RPM tags allow that.
Just use the content of the package of whatever provides /etc/os-release for your magic distro detection? Just put that into obs-build and issue a pull request, I'm pretty sure that there might be a good chance to get this accepted as it is no worse a hack than parsing rpm's rpm tags :-)
[0] Although it can make Reproducible Builds a lot more difficult, since the binaries are coming from a non-public build system and a lot of metadata about the builds is hence unavailable. I don't know how critical or interesting RB is for OpenSUSE (or SUSE in general).
There are SUSE people working on reproducible builds for openSUSE. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On 2/19/23 02:30, Mihai Moldovan wrote:
* On 2/18/23 14:24, Stefan Seyfried wrote:
On 18.02.23 13:28, Mihai Moldovan wrote:
The other issue is that, if rpm was really copied from SLE 15 SP2 verbatim[4], the distribution RPM tag used in SLE is omitting any service pack information, so it's impossible to determine the actual operating system version based on that.
Well, for 15.4 and SLES15-SP4, it is actually the case:
server:~ # rpm -q rpm rpm-4.14.3-150300.52.1.x86_64 server:~ # rpm -q --queryformat %{disturl}\\n rpm obs://build.suse.de/SUSE:Maintenance:26687/SUSE_SLE-15-SP3_Update/f0914b2c279f3c55f9d8c41439b2e804-rpm.SUSE_SLE-15-SP3_Update
True, but the disturl tag is rather new and... I'm assuming that this is really SLE or Leap 15.3? If it were 15.4, it would mean that the package was copied from SUSE:SLE-15-SP3:Update, which again would make it impossible to differentiate between sle15sp3 and sle15sp4. Parsing the disturl tag also looks like a nightmare.
This would bring no benefit, except for your fringe usecase [...]
Copying binary packages from SLE instead of rebuilding them from source brings no benefit, except to lower stress on the build infrastructure.
Unfortunately that's not quite true, with our current maintenance process in order to "rebuild" the package we'd have to create a separate copy of the sources and every time a bug / security issue was created the maintainer would also have to update that copy, where as currently when the maintainer updates the package for SLE-15-SP3 it automatically flows onto SLE-15-SP4 and probably SLE-15-SP5 (unless there was other changes for that service pack) as well as there corresponding Leap versions so as well as lower stress on the build infra it also creates significantly lower stress on the package maintainer. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
participants (3)
-
Mihai Moldovan -
Simon Lees -
Stefan Seyfried