[opensuse-packaging] Splitting up binary packages for large SDKs
Hello! I am currently working on the AWS SDK for C++ by Amazon, the package is called "aws-sdk-cpp" [1]. After some fiddling and disabling the testsuite for the time being due to an upstream bug [2], I am at the point where I have to decide how to distribute the SDK files among binary packages. Since I haven't built any larger packages for openSUSE before, in particular not any SDKs or library packages which include development files such static libraries, cmake files and header files, I'm wondering whether anyone can give me a heads-up what the common policy is. In Debian, we usually put all shared library files into a lib$NAME$SOVER package where $NAME is the library name and $SOVER it's SO version. Development files such as headers, cmake files and static library archives go into a lib$NAME-dev package. What's the policy for openSUSE? Also, while the package built fine on my AMD EPYC machine (except for the failing installation process due to the missing %files statements), on OBS the package occasionally fails with out-of-memory errors or even the compiler segfaulting. Is there any chance to schedule large packages onto beefier build instances? In case anyone wonders, the package is required within the Public Cloud Team. Adrian
[1] https://build.opensuse.org/package/show/home:glaubitz:branches:Cloud:Tools/a... [2] https://github.com/aws/aws-sdk-cpp/issues/929 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Tue, 7 Aug 2018, John Paul Adrian Glaubitz wrote:
Since I haven't built any larger packages for openSUSE before, in particular not any SDKs or library packages which include development files such static libraries, cmake files and header files, I'm wondering whether anyone can give me a heads-up what the common policy is.
In Debian, we usually put all shared library files into a lib$NAME$SOVER package where $NAME is the library name and $SOVER it's SO version. Development files such as headers, cmake files and static library archives go into a lib$NAME-dev package. What's the policy for openSUSE?
Specifically for shared libraries: https://en.opensuse.org/openSUSE:Shared_library_packaging_policy which also talks about -devel packages. Of relevance: * There is no reason why a -devel package cannot provide development files for multiple shared library packages. Also the -devel package doesn't have to (but can) be named lib$NAME-devel. It might make sense to call it aws-devel, or even package the devel files (headers, .so links and such) into the main package (if that exists). If the SDK supports the option of disabling static libraries do that. If not, package the static libs separately (into xxx-devel-static for instance), if the SDK then at least has the option of not using static libs. So, for something like that SDK (without knowing its details) I'd expect these binary rpms: - libfoo1 - libfoo2 - libfoo3 (and so on for all individual shared libs in {,/usr}/lib{,64}) - awk-sdk-cpp-devel-static (if really needed and indeed optional) - aws-sdk-cpp-devel (if not folded into aws-sdk-cpp) - aws-sdk-cpp (the rest) See exceptions 4a/4b in the above policy for when also the -devel packages need to be versioned (I'd hope it's not supported to install multiple AWK-SDK version onto the same system).
Also, while the package built fine on my AMD EPYC machine (except for the failing installation process due to the missing %files statements), on OBS the package occasionally fails with out-of-memory errors or even the compiler segfaulting.
Compiler segfaulting is no resource problem, but either a compiler bug, or a flaky machine (usually when the compiler segfault can't be reproduced).
Is there any chance to schedule large packages onto beefier build instances?
You want build constraints: https://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.build_... See e.g. % osc cat devel:gcc/gcc8/_constraints Careful with specifying too strong constraints, not all architectures provide build hosts with much RAM, if you're using too large numbers your package won't ever be scheduled for building. For curren resource usage of build e.g.: % osc api /build/openSUSE:Factory/standard/x86_64/gcc8/_statistics Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Aug 07 2018, Michael Matz <matz@suse.de> wrote:
Careful with specifying too strong constraints, not all architectures provide build hosts with much RAM, if you're using too large numbers your package won't ever be scheduled for building.
Though nowadays the build service may be able to detect when no build worker exists to fulfill the constraints, and forcefully fails the package then. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi Michael! On 08/07/2018 04:56 PM, Michael Matz wrote:
In Debian, we usually put all shared library files into a lib$NAME$SOVER package where $NAME is the library name and $SOVER it's SO version. Development files such as headers, cmake files and static library archives go into a lib$NAME-dev package. What's the policy for openSUSE?
Specifically for shared libraries: https://en.opensuse.org/openSUSE:Shared_library_packaging_policy which also talks about -devel packages. Of relevance: * There is no reason why a -devel package cannot provide development files for multiple shared library packages.
Since the SDK has a large number of submodules, I wasn't planning on creating individual -devel packages anyway. The number of binary packages would have just been too large.
Also the -devel package doesn't have to (but can) be named lib$NAME-devel. It might make sense to call it aws-devel, or even package the devel files (headers, .so links and such) into the main package (if that exists).
Yes, I just found that information as well. An additional important point that probably applies here is unversioned .so files:
https://en.opensuse.org/openSUSE:Specfile_guidelines#Development_files
Unversioned shared libraries (e.g. libfoo.so). Versioned shared libraries, e.g. libfoo.so.3, libfoo.so.3.0.0 should not be in -devel.
If the SDK supports the option of disabling static libraries do that. If not, package the static libs separately (into xxx-devel-static for instance), if the SDK then at least has the option of not using static libs.
I currently use the default cmake setting in the SDK which is BUILD_SHARED_LIBS=ON, so I'm not getting any static libraries. Is there a way to have the %cmake macro build both variants? Although I don't think we need both of them.
So, for something like that SDK (without knowing its details) I'd expect these binary rpms:
- libfoo1 - libfoo2 - libfoo3 (and so on for all individual shared libs in {,/usr}/lib{,64})
That would result in a large number of libfoo packages due to the large number of .so files. Also, since the .so files are unversioned, I'm not sure whether the naming scheme applies here.
- awk-sdk-cpp-devel-static (if really needed and indeed optional) - aws-sdk-cpp-devel (if not folded into aws-sdk-cpp)
Isn't the -devel package mandatory when packaging cmake modules, pkgconfig files and header files? At least rpmlint is complaining about the files not yet in the -devel package.
- aws-sdk-cpp (the rest)
See exceptions 4a/4b in the above policy for when also the -devel packages need to be versioned (I'd hope it's not supported to install multiple AWK-SDK version onto the same system).
Multiple versions don't seem supported. But installing the SDK for multiple platforms.
Also, while the package built fine on my AMD EPYC machine (except for the failing installation process due to the missing %files statements), on OBS the package occasionally fails with out-of-memory errors or even the compiler segfaulting.
Compiler segfaulting is no resource problem, but either a compiler bug, or a flaky machine (usually when the compiler segfault can't be reproduced).
Yes, I'm aware of that. I shouldn't have put the segfault into the same sentence.
Is there any chance to schedule large packages onto beefier build instances?
You want build constraints: https://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.build_...
See e.g. % osc cat devel:gcc/gcc8/_constraints
Careful with specifying too strong constraints, not all architectures provide build hosts with much RAM, if you're using too large numbers your package won't ever be scheduled for building. For curren resource usage of build e.g.: % osc api /build/openSUSE:Factory/standard/x86_64/gcc8/_statistics
Ah, that's handy. Thanks for the hint. Adrian -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tuesday 2018-08-07 17:11, John Paul Adrian Glaubitz wrote:
On 08/07/2018 04:56 PM, Michael Matz wrote:
In Debian, we usually put all shared library files into a lib$NAME$SOVER package where $NAME is the library name and $SOVER it's SO version. Development files such as headers, cmake files and static library archives go into a lib$NAME-dev package. What's the policy for openSUSE?
Specifically for shared libraries: https://en.opensuse.org/openSUSE:Shared_library_packaging_policy which also talks about -devel packages. Of relevance: * There is no reason why a -devel package cannot provide development files for multiple shared library packages.
Since the SDK has a large number of submodules, I wasn't planning on creating individual -devel packages anyway. The number of binary packages would have just been too large.
npm-leftpad anyone? Ask upstream to not write a gazillion micromodules. ;-)
I currently use the default cmake setting in the SDK which is BUILD_SHARED_LIBS=ON, so I'm not getting any static libraries.
Is there a way to have the %cmake macro build both variants? Although I don't think we need both of them.
Static libs are not allowed by default.
See exceptions 4a/4b in the above policy for when also the -devel packages need to be versioned (I'd hope it's not supported to install multiple AWK-SDK version onto the same system).
Multiple versions don't seem supported. But installing the SDK for multiple platforms.
There is just one platform as far as openSUSE is concerned. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 08/07/2018 05:21 PM, Jan Engelhardt wrote:
Since the SDK has a large number of submodules, I wasn't planning on creating individual -devel packages anyway. The number of binary packages would have just been too large.
npm-leftpad anyone? Ask upstream to not write a gazillion micromodules. ;-)
I think asking upstream would be a futile attempt :-(.
I currently use the default cmake setting in the SDK which is BUILD_SHARED_LIBS=ON, so I'm not getting any static libraries.
Is there a way to have the %cmake macro build both variants? Although I don't think we need both of them.
Static libs are not allowed by default.
Why would that be the case? The static libraries are supposed to be used for the user to build static binaries, not for building any of the packages in the distribution with static binaries. The zlib package, for example, does include a static library package:
https://build.opensuse.org/package/view_file/devel:libraries:c_c++/zlib/zlib...
See exceptions 4a/4b in the above policy for when also the -devel packages need to be versioned (I'd hope it's not supported to install multiple AWK-SDK version onto the same system).
Multiple versions don't seem supported. But installing the SDK for multiple platforms.
There is just one platform as far as openSUSE is concerned.
Well, the SDK supports Android so it might be useful to include the Android version of the AWS SDK in the future as well depending on whether building AWS applications for Android is a popular feature. Adrian -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Tue, 7 Aug 2018, John Paul Adrian Glaubitz wrote:
Static libs are not allowed by default.
Why would that be the case?
*) often they are unused (if static and shared libs are installed the link editor will by default use the shared ones) *) see last mail: they create problems that shared libs don't have, from a software distribution perspective
The static libraries are supposed to be used for the user to build static binaries, not for building any of the packages in the distribution with static binaries.
The zlib package, for example, does include a static library package:
https://build.opensuse.org/package/view_file/devel:libraries:c_c++/zlib/zlib...
That's one of the historic exceptions I was talking about. It's not a model for good packaging :-)
See exceptions 4a/4b in the above policy for when also the -devel packages need to be versioned (I'd hope it's not supported to install multiple AWK-SDK version onto the same system).
Multiple versions don't seem supported. But installing the SDK for multiple platforms.
There is just one platform as far as openSUSE is concerned.
Well, the SDK supports Android so it might be useful to include the Android version of the AWS SDK in the future as well depending on whether building AWS applications for Android is a popular feature.
Well, supposedly if the same SDK is installed for multiple targets the respective (shared or static) libraries aren't installed into the same directory by the SDKs build system. So just package them into foo-android-suffix subpacks (for all foo-suffix) I'd say, once the need for them arises. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Tue, 7 Aug 2018, John Paul Adrian Glaubitz wrote:
If the SDK supports the option of disabling static libraries do that. If not, package the static libs separately (into xxx-devel-static for instance), if the SDK then at least has the option of not using static libs.
I currently use the default cmake setting in the SDK which is BUILD_SHARED_LIBS=ON, so I'm not getting any static libraries.
Is there a way to have the %cmake macro build both variants? Although I don't think we need both of them.
No static libs please, then. They aren't allowed by default because they create more problems than they solve: if you have a bug in them (say a security problem) you need to not only deliver an updated library to customers, but actually re-link all applications making use of them and deliver all _those_ to customers. There are cases where we deliver static libs, they are usually historic cases, or where upstream doesn't provide shared libs (and they can't be made easily). But no new static libs without Very Good Reasons (tm).
So, for something like that SDK (without knowing its details) I'd expect these binary rpms:
- libfoo1 - libfoo2 - libfoo3 (and so on for all individual shared libs in {,/usr}/lib{,64})
That would result in a large number of libfoo packages due to the large number of .so files. Also, since the .so files are unversioned, I'm not sure whether the naming scheme applies here.
Urgs, they are unversioned? Sigh. Are they using symbol versioning? If not they shouldn't be in /usr/lib64 then. Unversioned shared libraries without symbol versioning can't ever be nively updated. But yes, one package per shared lib by default. There are exceptions to that, and this might be one (e.g. because they all belong intricately together and could just as well be merged into one large AWS shared library). But as soon as they can be used sensibly separately (e.g. if the programs envisioned to be created with the SDK sometimes use this and sometimes another features, and therefore link against different subsets of the provided shared libs) they have to go into separate packages. How many libs are we talking about?
- awk-sdk-cpp-devel-static (if really needed and indeed optional) - aws-sdk-cpp-devel (if not folded into aws-sdk-cpp)
Isn't the -devel package mandatory when packaging cmake modules, pkgconfig files and header files? At least rpmlint is complaining about the files not yet in the -devel package.
If there is a -devel package those files need to be placed in there. If there's no -devel package to start with those files can go into the main package. (And I sure hope that this is what rpmlint is checking). Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 08/07/2018 05:39 PM, Michael Matz wrote:
No static libs please, then. They aren't allowed by default because they create more problems than they solve: if you have a bug in them (say a security problem) you need to not only deliver an updated library to customers, but actually re-link all applications making use of them and deliver all _those_ to customers.
Yes, I know what the ramifications are when building static libraries.
There are cases where we deliver static libs, they are usually historic cases, or where upstream doesn't provide shared libs (and they can't be made easily). But no new static libs without Very Good Reasons (tm).
Well, the point of the static libraries is usually not to use them for building distribution packages with statically linked binaries but to allow customers to build their own binaries as static. I also disagree that static binaries is a historic thing. In fact, static linking is very popular these days for container applications, for example, and the SDK is specifically targeting cloud applications. Languages like Go even default to static linking by default.
So, for something like that SDK (without knowing its details) I'd expect these binary rpms:
- libfoo1 - libfoo2 - libfoo3 (and so on for all individual shared libs in {,/usr}/lib{,64})
That would result in a large number of libfoo packages due to the large number of .so files. Also, since the .so files are unversioned, I'm not sure whether the naming scheme applies here.
Urgs, they are unversioned? Sigh. Are they using symbol versioning? If not they shouldn't be in /usr/lib64 then. Unversioned shared libraries without symbol versioning can't ever be nively updated.
Haven't checked for versioned symbols yet. I will examine the .so files with objdump later to find out.
But yes, one package per shared lib by default. There are exceptions to that, and this might be one (e.g. because they all belong intricately together and could just as well be merged into one large AWS shared library). But as soon as they can be used sensibly separately (e.g. if the programs envisioned to be created with the SDK sometimes use this and sometimes another features, and therefore link against different subsets of the provided shared libs) they have to go into separate packages.
How many libs are we talking about?
168 :-).
- awk-sdk-cpp-devel-static (if really needed and indeed optional) - aws-sdk-cpp-devel (if not folded into aws-sdk-cpp)
Isn't the -devel package mandatory when packaging cmake modules, pkgconfig files and header files? At least rpmlint is complaining about the files not yet in the -devel package.
If there is a -devel package those files need to be placed in there. If there's no -devel package to start with those files can go into the main package. (And I sure hope that this is what rpmlint is checking).
Well, then I would need an override for rpmlint. I just pushed an updated version of the package into my home project:
https://build.opensuse.org/package/show/home:glaubitz:branches:Cloud:Tools/a...
Adrian -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Aug 07 2018, John Paul Adrian Glaubitz <adrian.glaubitz@suse.com> wrote:
Go even default to static linking by default.
Yes, that's a very silly thing to do. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 08/07/2018 05:52 PM, Andreas Schwab wrote:
On Aug 07 2018, John Paul Adrian Glaubitz <adrian.glaubitz@suse.com> wrote:
Go even default to static linking by default.
Yes, that's a very silly thing to do.
I agree. But that's what cool web developers want these days -.-. Adrian -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 8/7/18 5:52 PM, Andreas Schwab wrote:
On Aug 07 2018, John Paul Adrian Glaubitz <adrian.glaubitz@suse.com> wrote:
Go even default to static linking by default.
Yes, that's a very silly thing to do.
As usual general statements like the above can be easily proven to be wrong by a counter example: In automated container deployments you won't do any updates. You deliver an update of your service in a newly built container anyway (at least every two weeks or so). The advantage of having one statically linked binary is that you have a defined run-time environment delivered to production without all dependency hell on normal systems. Same reason why people use Python virtual envs or build everything into Java WARs. Of course you must have a decent CI/CD pipeline in place to make that a real advantage. And everything has pros and cons. Note: The above is not meant as statement about how to package libs in openSUSE. Ciao, Michael.
On Tuesday 2018-08-07 18:16, Michael Ströder wrote:
On 8/7/18 5:52 PM, Andreas Schwab wrote:
On Aug 07 2018, John Paul Adrian Glaubitz <adrian.glaubitz@suse.com> wrote:
Go even default to static linking by default.
Yes, that's a very silly thing to do.
As usual general statements like the above can be easily proven to be wrong by a counter example:
In automated container deployments you won't do any updates. You deliver an update of your service in a newly built container anyway
It does not matter jack if your container is updated by means of `zypper up` or by rebuilding the disk (`zypper in -R ...`). The final state will be the same, and it will match what's on the download server. And that's decided by the maintenance team, and the current practice for :Update is to only (re-)build e.g. libX11, not all libX11-using programs. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Tue, 7 Aug 2018, John Paul Adrian Glaubitz wrote:
There are cases where we deliver static libs, they are usually historic cases, or where upstream doesn't provide shared libs (and they can't be made easily). But no new static libs without Very Good Reasons (tm).
Well, the point of the static libraries is usually not to use them for building distribution packages with statically linked binaries but to allow customers to build their own binaries as static.
I also disagree that static binaries is a historic thing. In fact, static linking is very popular these days for container applications, for example, and the SDK is specifically targeting cloud applications. Languages like Go even default to static linking by default.
They use static linking by default because the designers of Go couldn't be bothered to think about software modularization on the binary level and implement it in their initial go compiler but instead decided to ignore advancements from the last 25 years on that topic. The same as the container people, but for different reasons. In these times static libs are the get-results-fast-without-thinking-hard equivalent to throw-away perl scripts that become unmaintainable after five years. You could move the static libs (if you chose to enable them, but why would you when upstream is at least somewhat sensible and provides shared libs) outside /usr/lib64 into a private directory. Then the library policy is of no concern to you anymore (for the static libs). Some small adjustments to the SDK should make the SDK build tools (when users are using the SDK) use the new directory as the library search path.
How many libs are we talking about?
168 :-).
Sigh. Proper modularization doesn't seem en vouge anymore. :-/ How is upstream even thinking about updating these unversioned shared libs? Are they thinking?
Isn't the -devel package mandatory when packaging cmake modules, pkgconfig files and header files? At least rpmlint is complaining about the files not yet in the -devel package.
If there is a -devel package those files need to be placed in there. If there's no -devel package to start with those files can go into the main package. (And I sure hope that this is what rpmlint is checking).
Well, then I would need an override for rpmlint.
So you're saying that rpmlint complains about e.g. cmake files in the main package even though you didn't even have a -devel subpackage? Seems a regression in rpmlint :-/ Well, so be it, a -devel package it is then :)
I just pushed an updated version of the package into my home project:
https://build.opensuse.org/package/show/home:glaubitz:branches:Cloud:Tools/a...
So all .so files into a -libs package. Pragmatic, but you will have problems getting this into Factory. If you'd consider putting all the shared libs into one (versioned) subdirectory of /usr/lib64 (and make similar adjustments to the SDKs build wrappers for library search paths; for shared libs you also need the RPATH) you'd avoid all problems the shared library policy poses to you _and_ would solve the update-libraries problem as well, at least somewhat. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tuesday 2018-08-07 18:09, Michael Matz wrote:
You could move the static libs (if you chose to enable them, but why would you when upstream is at least somewhat sensible and provides shared libs) outside /usr/lib64 into a private directory. Then the library policy is of no concern to you anymore (for the static libs). Some small adjustments to the SDK should make the SDK build tools (when users are using the SDK) use the new directory as the library search path.
How many libs are we talking about?
168 :-).
Sigh. Proper modularization doesn't seem en vouge anymore. :-/
How is upstream even thinking about updating these unversioned shared libs? Are they thinking?
Isn't the -devel package mandatory when packaging cmake modules, pkgconfig files and header files? At least rpmlint is complaining about the files not yet in the -devel package.
If there is a -devel package those files need to be placed in there. If there's no -devel package to start with those files can go into the main package. (And I sure hope that this is what rpmlint is checking).
Well, then I would need an override for rpmlint.
So you're saying that rpmlint complains about e.g. cmake files in the main package even though you didn't even have a -devel subpackage? Seems a regression in rpmlint :-/ Well, so be it, a -devel package it is then :)
I just pushed an updated version of the package into my home project:
https://build.opensuse.org/package/show/home:glaubitz:branches:Cloud:Tools/a...
So all .so files into a -libs package. Pragmatic, but you will have problems getting this into Factory. If you'd consider putting all the shared libs into one (versioned) subdirectory of /usr/lib64 (and make similar adjustments to the SDKs build wrappers for library search paths; for shared libs you also need the RPATH) you'd avoid all problems the shared library policy poses to you _and_ would solve the update-libraries problem as well, at least somewhat.
RPM dependency autotracking uses only the file's basename and the file's content. If you solely rely on a directory to have a version number, you will almost certainly have the same ABI problems as without the extra directory. The only time versions in directory names are useful is when you dlopen libraries so placed. But DT_SONAME (the things you see with `ldd`; the things rpm sees) work without directories. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Tue, 7 Aug 2018, Jan Engelhardt wrote:
So all .so files into a -libs package. Pragmatic, but you will have problems getting this into Factory. If you'd consider putting all the shared libs into one (versioned) subdirectory of /usr/lib64 (and make similar adjustments to the SDKs build wrappers for library search paths; for shared libs you also need the RPATH) you'd avoid all problems the shared library policy poses to you _and_ would solve the update-libraries problem as well, at least somewhat.
RPM dependency autotracking uses only the file's basename and the file's content. If you solely rely on a directory to have a version number, you will almost certainly have the same ABI problems as without the extra directory.
All true about RPM deps, they can't be solved this way (but see below). That's the "somewhat" in my remark :) (Depending on your point of view it might translate to "not very much", but at least it'd be possible to install multiple such sets of libs in parallel; the dependence tracking wouldn't at least not get in the way, it won't see a problem, and the dependency on the version could be expressed explicitely without autodeps on shared libs).
The only time versions in directory names are useful is when you dlopen libraries so placed. But DT_SONAME (the things you see with `ldd`; the things rpm sees) work without directories.
That's not fully correct. non-standard libs aren't search by default, so will either need LD_LIBRARY_NAME, or (which was my suggestion) a DT_RUNPATH entry in the apps, which is the point at which versioned directories make sense again, because DT_RUNPATH will contain that version and hence apps will only run when the correct set is installed. We could even extend the rpm AutoReqProv mechanism to look at DT_RPATH/DT_RUNPATH and create a Requires on, say, "libpath(/usr/lib64/foobar3)", which could be Provided by the respective package (and even that could be AutoProvided when a package contains shared libs in a non-standard subdir). The whole reason we created the shared lib policy was to be able to install multiple versions in parallel (so as to not loose old ones on partly updates). The above versioned-directory solution would still give this. Both approaches need a bit manual work (shared lib policy: putting each lib into separate package; version-dir: putting libs there and ensure that libs therein are found at linkedit- and run-time, which is more work, but still). Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Wednesday 2018-08-08 14:17, Michael Matz wrote:
The only time versions in directory names are useful is when you dlopen libraries so placed. But DT_SONAME (the things you see with `ldd`; the things rpm sees) work without directories.
That's not fully correct. [...] a DT_RUNPATH entry in the apps
yeah but my point was that - AFAIR - rpm does not look at DT_RUNPATH/RPATH. If you have /usr/lib64/amazon-2.0/libfoo.so, the rpms only carry a dependency on "libfoo.so", and if some randomother package happens to provide that too, then boom. Well, it won't start and cripple data, but not starting is an issue in its own right, too. So, if you can, /usr/lib64/libfoo-2.0.so. Then the default rpm modus operandi should be good enough. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi! On 08/07/2018 06:09 PM, Michael Matz wrote:
Well, then I would need an override for rpmlint.
So you're saying that rpmlint complains about e.g. cmake files in the main package even though you didn't even have a -devel subpackage? Seems a regression in rpmlint :-/ Well, so be it, a -devel package it is then :)
I can re-test this and file an upstream bug report if necessary.
I just pushed an updated version of the package into my home project:
https://build.opensuse.org/package/show/home:glaubitz:branches:Cloud:Tools/a...
So all .so files into a -libs package. Pragmatic, but you will have problems getting this into Factory. If you'd consider putting all the shared libs into one (versioned) subdirectory of /usr/lib64 (and make similar adjustments to the SDKs build wrappers for library search paths; for shared libs you also need the RPATH) you'd avoid all problems the shared library policy poses to you _and_ would solve the update-libraries problem as well, at least somewhat.
I'll look into this. But for now, the package is supposed to stay in Cloud:Tools anyway. Adrian -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 8/7/18 5:39 PM, Michael Matz wrote:
There are cases where we deliver static libs, they are usually historic cases, or where upstream doesn't provide shared libs (and they can't be made easily). But no new static libs without Very Good Reasons (tm).
Some people consider security issues with dynamic linking of setuid executables a good reason to link those statically (or completely avoid certain shared libs). I'm no expert on this though. Ciao, Michael.
On Tue, Aug 07, 2018 at 06:00:21PM +0200, Michael Ströder wrote:
On 8/7/18 5:39 PM, Michael Matz wrote:
There are cases where we deliver static libs, they are usually historic cases, or where upstream doesn't provide shared libs (and they can't be made easily). But no new static libs without Very Good Reasons (tm).
Some people consider security issues with dynamic linking of setuid executables a good reason to link those statically (or completely avoid certain shared libs).
I'm no expert on this though.
THis is superstition. If you static link libraries, we need to release updates for the library for both the library and the setuid binary ... Easy to overlook such cases, thats why we strive to keep everything shared only. Ciao, Marcus -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 8/7/18 6:04 PM, Marcus Meissner wrote:
On Tue, Aug 07, 2018 at 06:00:21PM +0200, Michael Ströder wrote:
On 8/7/18 5:39 PM, Michael Matz wrote:
There are cases where we deliver static libs, they are usually historic cases, or where upstream doesn't provide shared libs (and they can't be made easily). But no new static libs without Very Good Reasons (tm).
Some people consider security issues with dynamic linking of setuid executables a good reason to link those statically (or completely avoid certain shared libs).
I'm no expert on this though.
THis is superstition.
The libvirt developers were concerned about one such case: "Also note when building the setuid libvirt pieces we must never use GNUTLS because its library constructors do very bad things leading to CVEs." https://www.redhat.com/archives/libvirt-users/2018-June/msg00001.html
If you static link libraries, we need to release updates for the library for both the library and the setuid binary ...
Hmm, but with the OBS processing build dependencies this should not be such a big deal for a couple of setuid binaries. Or? Ciao, Michael.
participants (6)
-
Andreas Schwab -
Jan Engelhardt -
John Paul Adrian Glaubitz -
Marcus Meissner -
Michael Matz -
Michael Ströder