[opensuse-factory] Shared library policy - multiple providers for a single API/ABI
Hi, just stumbled over an omission in the shared library packaging policy [1]. The policy gives several examples for SONAMEs and package names when there are several versions of a library with incompatible API/ABI. But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI. One such library is openblas, which is available as singlethreaded ("serial"), multithreaded using pthreads, and multithreaded using openmp. These three libraries have the same ABI and thus are interchangeable, but unfortunately each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0). Any program/library linking to libopenblas.so (linker flag -lopenblas) will end up with the default implementation, which is either openblas_pthreads (on x86*) or openblas_openmp (else). The library name then is fixed in both the program/ library dynamic linking as well as the rpm dependencies. Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants. IMHO the policy and rpmlint should be extended to cover these cases. My RFC: - if there are multiple ABI compatible variants, all should have the same SONAME (e.g. libopenblas.so.0) - the package name starts with a package name according to the current policy, *suffixed the the variant name* (e.g. libopenblas0_openmp). - in case of name changes, Provides: for backwards compatibilty must be specified. Kind regards, Stefan [1] https://en.opensuse.org/ openSUSE:Shared_library_packaging_policy#Package_naming -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Saturday 2017-06-03 19:38, Stefan Bruens wrote:
But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI.
You are correct in that it is not specified. One could say that we were still experimenting with what works. One arm of that experiment was Mesa-libGL1.
openblas [...] three libraries [..] each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0).
Different SONAME => different package names. It's that simple.
Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants.
The pattern is [vendor-][libblah1][-variant].rpm, stemming from the "other" arm of the experiment that is libatomic1-gcc6, libblasc2-openmp. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, 3 Jun 2017, Jan Engelhardt wrote:
On Saturday 2017-06-03 19:38, Stefan Bruens wrote:
But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI.
You are correct in that it is not specified. One could say that we were still experimenting with what works. One arm of that experiment was Mesa-libGL1.
openblas [...] three libraries [..] each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0).
Different SONAME => different package names. It's that simple.
Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants.
The pattern is [vendor-][libblah1][-variant].rpm, stemming from the "other" arm of the experiment that is libatomic1-gcc6, libblasc2-openmp.
With the important part that each of the variants shall
Provide: libblah1 = %version
Conflict: libblah1
to explicitely disallow installing multiple variants at the same time
(you'll get file conflicts) plus make requiring them via sth other
than the SONAME auto generated provides possible.
Note that in case of libopenblas_serial vs. libopenblas_pthreads vs.
libopenblas_openmp it's also a matter of how an application was built
thus using the same SONAME for this might not be what you want as
you do want the possibility to run a OpenMP app at the same time
as a serial one both using openblas.
Richard.
--
Richard Biener
On Dienstag, 6. Juni 2017 09:31:59 CEST Richard Biener wrote:
On Sat, 3 Jun 2017, Jan Engelhardt wrote:
On Saturday 2017-06-03 19:38, Stefan Bruens wrote:
But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI.> You are correct in that it is not specified. One could say that we were still experimenting with what works. One arm of that experiment was Mesa-libGL1.> openblas [...] three libraries [..] each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0).> Different SONAME => different package names. It's that simple.
Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants.
The pattern is [vendor-][libblah1][-variant].rpm, stemming from the "other" arm of the experiment that is libatomic1-gcc6, libblasc2-openmp.
With the important part that each of the variants shall
Provide: libblah1 = %version Conflict: libblah1
to explicitely disallow installing multiple variants at the same time (you'll get file conflicts) plus make requiring them via sth other than the SONAME auto generated provides possible.
A conflict is only required if the variants have the same paths. This can be avoided by either having a filename different from the soname, or by creating directories for each variant.
Note that in case of libopenblas_serial vs. libopenblas_pthreads vs. libopenblas_openmp it's also a matter of how an application was built thus using the same SONAME for this might not be what you want as you do want the possibility to run a OpenMP app at the same time as a serial one both using openblas.
Thats not exactly true - I *may* want different applications running at the same time to use different variants, but more likely I want all applications to use the same variant. Currently, the variant is fixed at link time. If I want to switch, I have to rebuild the whole stack. For a real world example: OpenCV links against openblas_pthreads on x86(_64) (and openblas_openmp else, go figure). openblas claims to have support for switching the variants using update-alternatives, but this has no effect for OpenCV and any of its users. If I want to change the openblas variant, I have to rebuild: - openblas - opencv - all of the packages depending on openblas/opencv Kind regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019 work: +49 2405 49936-424 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, 6 Jun 2017, Stefan Bruens wrote:
On Dienstag, 6. Juni 2017 09:31:59 CEST Richard Biener wrote:
On Sat, 3 Jun 2017, Jan Engelhardt wrote:
On Saturday 2017-06-03 19:38, Stefan Bruens wrote:
But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI.> You are correct in that it is not specified. One could say that we were still experimenting with what works. One arm of that experiment was Mesa-libGL1.> openblas [...] three libraries [..] each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0).> Different SONAME => different package names. It's that simple.
Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants.
The pattern is [vendor-][libblah1][-variant].rpm, stemming from the "other" arm of the experiment that is libatomic1-gcc6, libblasc2-openmp.
With the important part that each of the variants shall
Provide: libblah1 = %version Conflict: libblah1
to explicitely disallow installing multiple variants at the same time (you'll get file conflicts) plus make requiring them via sth other than the SONAME auto generated provides possible.
A conflict is only required if the variants have the same paths. This can be avoided by either having a filename different from the soname, or by creating directories for each variant.
Not sure this is easily possible, you need LD_LIBRARY_PATH for the latter and ldconfig will screw up the former.
Note that in case of libopenblas_serial vs. libopenblas_pthreads vs. libopenblas_openmp it's also a matter of how an application was built thus using the same SONAME for this might not be what you want as you do want the possibility to run a OpenMP app at the same time as a serial one both using openblas.
Thats not exactly true - I *may* want different applications running at the same time to use different variants, but more likely I want all applications to use the same variant.
Currently, the variant is fixed at link time. If I want to switch, I have to rebuild the whole stack.
For a real world example: OpenCV links against openblas_pthreads on x86(_64) (and openblas_openmp else, go figure). openblas claims to have support for switching the variants using update-alternatives, but this has no effect for OpenCV and any of its users. If I want to change the openblas variant, I have to rebuild: - openblas - opencv - all of the packages depending on openblas/opencv
But I'm quite sure -openmp and -pthreads are not equivalent when used
from an OpenMP application.
Richard.
--
Richard Biener
On Jun 06 2017, Richard Biener
Not sure this is easily possible, you need LD_LIBRARY_PATH for the latter and ldconfig will screw up the former.
You can add additional library directories to /etc/ld.so.conf. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Dienstag, 6. Juni 2017 15:59:35 CEST Richard Biener wrote:
On Tue, 6 Jun 2017, Stefan Bruens wrote:
On Dienstag, 6. Juni 2017 09:31:59 CEST Richard Biener wrote:
On Sat, 3 Jun 2017, Jan Engelhardt wrote:
On Saturday 2017-06-03 19:38, Stefan Bruens wrote:
But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI.>
You are correct in that it is not specified. One could say that we were still experimenting with what works. One arm of that experiment was Mesa-libGL1.>
openblas [...] three libraries [..] each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0).>
Different SONAME => different package names. It's that simple.
Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants.
The pattern is [vendor-][libblah1][-variant].rpm, stemming from the "other" arm of the experiment that is libatomic1-gcc6, libblasc2-openmp.
With the important part that each of the variants shall
Provide: libblah1 = %version Conflict: libblah1
to explicitely disallow installing multiple variants at the same time (you'll get file conflicts) plus make requiring them via sth other than the SONAME auto generated provides possible.
A conflict is only required if the variants have the same paths. This can be avoided by either having a filename different from the soname, or by creating directories for each variant.
Not sure this is easily possible, you need LD_LIBRARY_PATH for the latter and ldconfig will screw up the former.
I have modified the openblas package and built opencv and os-autoinst packages on top. Now I can select my variant either: - by installing only one variant - selecting one using "update-alternatives --config libopenblas.so.0"
Note that in case of libopenblas_serial vs. libopenblas_pthreads vs. libopenblas_openmp it's also a matter of how an application was built thus using the same SONAME for this might not be what you want as you do want the possibility to run a OpenMP app at the same time as a serial one both using openblas.
Thats not exactly true - I *may* want different applications running at the same time to use different variants, but more likely I want all applications to use the same variant.
Currently, the variant is fixed at link time. If I want to switch, I have to rebuild the whole stack.
For a real world example: OpenCV links against openblas_pthreads on x86(_64) (and openblas_openmp else, go figure). openblas claims to have support for switching the variants using update-alternatives, but this has no effect for OpenCV and any of its users. If I want to change the openblas variant, I have to rebuild: - openblas - opencv - all of the packages depending on openblas/opencv
But I'm quite sure -openmp and -pthreads are not equivalent when used from an OpenMP application.
For openblas, all three variants are completely equivalent from a functional point of view. OpenMP vs pthreads is an implementation detail. Of course pthreads will ignore e.g. OPENMP_NUM_THREADS, but if you really care, openmp vs pthreads should be selected at your discretion whats best. Currently, you can not select. Regards, Stefan -- Stefan Brüns / Bergstraße 21 / 52062 Aachen home: +49 241 53809034 mobile: +49 151 50412019 work: +49 2405 49936-424 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, 6 Jun 2017, Stefan Bruens wrote:
On Dienstag, 6. Juni 2017 15:59:35 CEST Richard Biener wrote:
On Tue, 6 Jun 2017, Stefan Bruens wrote:
On Dienstag, 6. Juni 2017 09:31:59 CEST Richard Biener wrote:
On Sat, 3 Jun 2017, Jan Engelhardt wrote:
On Saturday 2017-06-03 19:38, Stefan Bruens wrote:
But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI.>
You are correct in that it is not specified. One could say that we were still experimenting with what works. One arm of that experiment was Mesa-libGL1.>
openblas [...] three libraries [..] each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0).>
Different SONAME => different package names. It's that simple.
Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants.
The pattern is [vendor-][libblah1][-variant].rpm, stemming from the "other" arm of the experiment that is libatomic1-gcc6, libblasc2-openmp.
With the important part that each of the variants shall
Provide: libblah1 = %version Conflict: libblah1
to explicitely disallow installing multiple variants at the same time (you'll get file conflicts) plus make requiring them via sth other than the SONAME auto generated provides possible.
A conflict is only required if the variants have the same paths. This can be avoided by either having a filename different from the soname, or by creating directories for each variant.
Not sure this is easily possible, you need LD_LIBRARY_PATH for the latter and ldconfig will screw up the former.
I have modified the openblas package and built opencv and os-autoinst packages on top. Now I can select my variant either: - by installing only one variant - selecting one using "update-alternatives --config libopenblas.so.0"
Note that in case of libopenblas_serial vs. libopenblas_pthreads vs. libopenblas_openmp it's also a matter of how an application was built thus using the same SONAME for this might not be what you want as you do want the possibility to run a OpenMP app at the same time as a serial one both using openblas.
Thats not exactly true - I *may* want different applications running at the same time to use different variants, but more likely I want all applications to use the same variant.
Currently, the variant is fixed at link time. If I want to switch, I have to rebuild the whole stack.
For a real world example: OpenCV links against openblas_pthreads on x86(_64) (and openblas_openmp else, go figure). openblas claims to have support for switching the variants using update-alternatives, but this has no effect for OpenCV and any of its users. If I want to change the openblas variant, I have to rebuild: - openblas - opencv - all of the packages depending on openblas/opencv
But I'm quite sure -openmp and -pthreads are not equivalent when used from an OpenMP application.
For openblas, all three variants are completely equivalent from a functional point of view. OpenMP vs pthreads is an implementation detail. Of course pthreads will ignore e.g. OPENMP_NUM_THREADS, but if you really care, openmp vs pthreads should be selected at your discretion whats best.
Well, if your program uses OpenMP and inside a OpenMP region calls a blas function then behavior will be different dependent on whether blas is OpenMP aware or not. OpenMP aware blas should be safe, eventually the pthreads variant has lower overhead.
Currently, you can not select.
Yes. But I don't believe in selecting in this case. Richard.
Regards,
Stefan
--
Richard Biener
On 04/06/17 03:08, Stefan Bruens wrote:
Hi,
just stumbled over an omission in the shared library packaging policy [1].
The policy gives several examples for SONAMEs and package names when there are several versions of a library with incompatible API/ABI.
But there is no specification for libraries available in different variants, i.e. different implementations or optimizations, with compatible ABI.
One such library is openblas, which is available as singlethreaded ("serial"), multithreaded using pthreads, and multithreaded using openmp. These three libraries have the same ABI and thus are interchangeable, but unfortunately each one carries its own SONAME (libopenblas_serial.so.0, libopenblas_pthreads.so.0, libopenblas_openmp.so.0). Any program/library linking to libopenblas.so (linker flag -lopenblas) will end up with the default implementation, which is either openblas_pthreads (on x86*) or openblas_openmp (else). The library name then is fixed in both the program/ library dynamic linking as well as the rpm dependencies.
Now, it is trivial to set the SONAME for all three implementations to libopenblas.so.0, but unfortunately rpmlint insist on the package name to be libopenblas0 in this case, with no possibility to have different package names for the three variants.
IMHO the policy and rpmlint should be extended to cover these cases.
Later this week i'll be starting a topic on what rpmlint should be doing better so I guess we can cover it then. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
participants (5)
-
Andreas Schwab -
Jan Engelhardt -
Richard Biener -
Simon Lees -
Stefan Bruens