Working ansible and ansible-lint packages for Leap 15.4 / SLES15SP4: Open questions on the fine details of python packaging
TL;DR There are working packages for ansible, ansible-core and ansible-lint (and all of their dependencies) for Leap 15.4 and SLES15SP4 with python3.10. Hi all, as I had the request for ansible and ansible-lint packages for SLES15 SP4 I started the journey, and some 100 packages later I got it working.
https://build.opensuse.org/project/show/home:ojkastl_buildservice:ansible_fo... https://build.opensuse.org/project/show/home:ojkastl_buildservice:ansible-li...
Which depend on and build against these projects:
https://build.opensuse.org/project/show/home:ojkastl_buildservice:cblas_for_... https://build.opensuse.org/project/show/home:ojkastl_buildservice:pytest_dep... https://build.opensuse.org/project/show/home:ojkastl_buildservice:pytest_for...
Due to my limited understanding of the python macros I used a simple prjconf to only build against the python3.10 version in the Python3 module of SLES15SP4.
%define pythons python310 %define primary_python python310
Substitute: python3-base python310-base
Macros: %pythons python310 %primary_python python310 :Macros
That did the trick for most of the packages, but I have two questions: 1. Several packages failed as they correctly BuildRequire python310-setuptools, but not python310-base. I added a line to their spec file and all was well. Example:
https://build.opensuse.org/package/show/home:ojkastl_buildservice:pytest_dep...
Question: Is this something that should be fixed, i.e. should I open SRs for those packages? Or this due to an error I made somehow in the prjconf? Should the setuptools package BuildRequire the corresponding python3*-base package? I do not want to frequently repair broken links in the packages, but I did not want to use Preinstall or similar from the prjconf. 2. I had to refine the spec files for ansible, ansible-core, ansible-lint. As those are just building against the default python in Tumbleweed (i.e. they are not available as python310-ansible-...). I made them kind-of-singlespec, while defining only one python version in the "%pythons" macro. But I had to use the %primary_python macro to properly set the Requires, as singlespec and the macros did not help there. And just using "%pythons" would make this fail on Tumbleweed where there are multiple python versions. And as the %pythons macro needs to be set inside the spec file to overwrite this on Tumbleweed, I had to define it again inside the package. Example:
I would like to forward the changes to the ansible packages in Tumbleweed. Question to the python packaging experts: Any hints or objections? Any flaws that might make this fail for $REASON? And: Is there a macro that contains the executable name for the primary python? The ansible-lint packages contains calls to python3.10 for %build and %install, not sure if those could be replaced by some macro-foo. Thanks in advance! Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi Johannes, we already had some discussions about this. I don't have the time to go too much into the details again, just some remarks. Am 21.11.22 um 15:28 schrieb Johannes Kastl:
TL;DR There are working packages for ansible, ansible-core and ansible-lint (and all of their dependencies) for Leap 15.4 and SLES15SP4 with python3.10.
Congratulations!
Hi all,
as I had the request for ansible and ansible-lint packages for SLES15 SP4 I started the journey, and some 100 packages later I got it working.
https://build.opensuse.org/project/show/home:ojkastl_buildservice:ansible_fo...
https://build.opensuse.org/project/show/home:ojkastl_buildservice:ansible-li...
Which depend on and build against these projects:
https://build.opensuse.org/project/show/home:ojkastl_buildservice:cblas_for_...
https://build.opensuse.org/project/show/home:ojkastl_buildservice:pytest_dep...
https://build.opensuse.org/project/show/home:ojkastl_buildservice:pytest_for...
Due to my limited understanding of the python macros I used a simple prjconf to only build against the python3.10 version in the Python3 module of SLES15SP4.
%define pythons python310 %define primary_python python310
Substitute: python3-base python310-base
Macros: %pythons python310 %primary_python python310 :Macros
That did the trick for most of the packages, but I have two questions:
`%primary_python` affects the behavior and contents of the python310{,-base} packages. The definition for the primary interpreter is where /usr/bin/python3 points to. Thus, for having any effect, you have to rebuild pythin310{,-base} too. Once you have this, every python module in your stack which you rebuild yourself will have a `Provides: python3-foo` and override the python3-foo for Python 3.6. OTOH, this will require a lot of "Prefer:" lines, because the distro will now have multiple providers of python3-foo. Otherwise I do not see why `%primary_python` would need to be set at all. See: https://github.com/openSUSE/python-rpm-macros/blob/868757846f707ffcf351720b9...
1. Several packages failed as they correctly BuildRequire python310-setuptools, but not python310-base. I added a line to their spec file and all was well. Example:
https://build.opensuse.org/package/show/home:ojkastl_buildservice:pytest_dep...
Question: Is this something that should be fixed, i.e. should I open SRs for those packages? Or this due to an error I made somehow in the prjconf? Should the setuptools package BuildRequire the corresponding python3*-base package?
The requirement on python310-base comes automatically from the requirement on `python(abi) = 3.10` which is normally automatically added by /usr/lib/rpm/pythondistdeps.py. Leap's python310-setuptools does not have it but probably should. Wothout checking I do not know, whether it will be added when you build it yourself with above mentioned change of the primary interpreter link.
2. I had to refine the spec files for ansible, ansible-core, ansible-lint. As those are just building against the default python in Tumbleweed (i.e. they are not available as python310-ansible-...). I made them kind-of-singlespec, while defining only one python version in the "%pythons" macro. But I had to use the %primary_python macro to properly set the Requires, as singlespec and the macros did not help there. And just using "%pythons" would make this fail on Tumbleweed where there are multiple python versions.
And as the %pythons macro needs to be set inside the spec file to overwrite this on Tumbleweed, I had to define it again inside the package.
Example:
The %fdupes and the mkdir lines are wrong. %{python_sitelib} is not expanded in %prep,%build,%install, or %check. And it is only expanded in the header tags and %files when you use %python_subpackages. The way you are using it, it is only by happy coincidence pointing to the python310_sitelib.
And: Is there a macro that contains the executable name for the primary python? The ansible-lint packages contains calls to python3.10 for %build and %install, not sure if those could be replaced by some macro-foo.
Remember: "primary" is always where /usr/bin/python3 links to. Setting "%primary_python" without python310-base containing the file is wrong. - Ben
Am 21.11.22 um 16:19 schrieb Ben Greiner:
Hi Johannes,
we already had some discussions about this. I don't have the time to go too much into the details again, just some remarks.
I am sorry, I might have confused this with a similar project by Pete. You might want to check https://build.opensuse.org/project/show/home:frispete:blender:15.4 for an (incomplete) attempt of doing something similar without replacing the system Python 3.6 as primary interpreter. - Ben
Hi Ben, On 21.11.22 at 16:19 Ben Greiner wrote:
Am 21.11.22 um 15:28 schrieb Johannes Kastl:
Due to my limited understanding of the python macros I used a simple prjconf to only build against the python3.10 version in the Python3 module of SLES15SP4.
%define pythons python310 %define primary_python python310
Substitute: python3-base python310-base
Macros: %pythons python310 %primary_python python310 :Macros
That did the trick for most of the packages, but I have two questions:
`%primary_python` affects the behavior and contents of the python310{,-base} packages. The definition for the primary interpreter is where /usr/bin/python3 points to. Thus, for having any effect, you have to rebuild pythin310{,-base} too. Once you have this, every python module in your stack which you rebuild yourself will have a `Provides: python3-foo` and override the python3-foo for Python 3.6. OTOH, this will require a lot of "Prefer:" lines, because the distro will now have multiple providers of python3-foo. Otherwise I do not see why `%primary_python` would need to be set at all.
Objections, your honour! :-) I am not building the python310 packages, those are delivered by Leap/SLES. And they do not change the default python (i.e. where /usr/bin/python3 links to) and should not. I am only piggybacking on the macro, as it is set properly in both Tumbleweed (which I cannot influence) and my projects. So while %pythons contains multiple python versions on Tumbleweed, the ansible/ansible-core/ansible-lint packages only build for one version. %primary_python is just used to set the proper Requires, as that did not work on Leap/SLES. On Tumbleweed python3-foo is pointing to python310-foo, so it works there...
See: https://github.com/openSUSE/python-rpm-macros/blob/868757846f707ffcf351720b9...
I need to have a good read on the whole python macros, once I have some time to dig in. This is not for the faint of heart... :-(
1. Several packages failed as they correctly BuildRequire python310-setuptools, but not python310-base. I added a line to their spec file and all was well. Example:
https://build.opensuse.org/package/show/home:ojkastl_buildservice:pytest_dep...
Question: Is this something that should be fixed, i.e. should I open SRs for those packages? Or this due to an error I made somehow in the prjconf? Should the setuptools package BuildRequire the corresponding python3*-base package?
The requirement on python310-base comes automatically from the requirement on `python(abi) = 3.10` which is normally automatically added by /usr/lib/rpm/pythondistdeps.py. Leap's python310-setuptools does not have it but probably should. Wothout checking I do not know, whether it will be added when you build it yourself with above mentioned change of the primary interpreter link.
As stated, I do not want to rebuild python310/python310-base. Not sure if that can be changed in Leap 15.4, maybe it is worth investigating for 15.5, whether this functionality should be added?
The %fdupes and the mkdir lines are wrong. %{python_sitelib} is not expanded in %prep,%build,%install, or %check. And it is only expanded in the header tags and %files when you use %python_subpackages. The way you are using it, it is only by happy coincidence pointing to the python310_sitelib.
Thanks for the hints, I'll take a look. Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Am 22.11.22 um 09:27 schrieb Johannes Kastl:
Hi Ben,
On 21.11.22 at 16:19 Ben Greiner wrote:
Am 21.11.22 um 15:28 schrieb Johannes Kastl:
Due to my limited understanding of the python macros I used a simple prjconf to only build against the python3.10 version in the Python3 module of SLES15SP4.
%define pythons python310 %define primary_python python310
Substitute: python3-base python310-base
Macros: %pythons python310 %primary_python python310 :Macros
That did the trick for most of the packages, but I have two questions:
`%primary_python` affects the behavior and contents of the python310{,-base} packages. The definition for the primary interpreter is where /usr/bin/python3 points to. Thus, for having any effect, you have to rebuild pythin310{,-base} too. Once you have this, every python module in your stack which you rebuild yourself will have a `Provides: python3-foo` and override the python3-foo for Python 3.6. OTOH, this will require a lot of "Prefer:" lines, because the distro will now have multiple providers of python3-foo. Otherwise I do not see why `%primary_python` would need to be set at all.
Objections, your honour! :-)
I am not building the python310 packages, those are delivered by Leap/SLES. And they do not change the default python (i.e. where /usr/bin/python3 links to) and should not.
Then you should not redefine %primary_python to python310! Instread set %primary_python to python3, so that the Factory specfiles you are reusing are not confused. (It is not defined in vanilla Leap/SLE). For everything else, change python3- to python310-, either directly or thourgh a macro-redirection like %mypython. - Ben
Hi Ben, thanks for taking the time to answer. Appreciated! On 22.11.22 at 09:50 Ben Greiner wrote:
`%primary_python` affects the behavior and contents of the python310{,-base} packages. The definition for the primary interpreter is where /usr/bin/python3 points to. Thus, for having any effect, you have to rebuild pythin310{,-base} too. Once you have this, every python module in your stack which you rebuild yourself will have a `Provides: python3-foo` and override the python3-foo for Python 3.6. OTOH, this will require a lot of "Prefer:" lines, because the distro will now have multiple providers of python3-foo. Otherwise I do not see why `%primary_python` would need to be set at all.
Objections, your honour! :-)
I am not building the python310 packages, those are delivered by Leap/SLES. And they do not change the default python (i.e. where /usr/bin/python3 links to) and should not.
Then you should not redefine %primary_python to python310!
I still do not get why? If this only affects the python310 and python310-base packages during build and I am not rebuilding them, there is no negative impact? And inside "my packages", everything just seems to work? I do not want those packages to use the default python (which is 3.6) and I do not want to replace it with 3.10. The SLES-packages are apparently built in a way, that they can be installed in parallel without changing the default python.
For everything else, change python3- to python310-, either directly or thourgh a macro-redirection like %mypython.
Then I would need to also add some if-condition to set this mypython macro inside the spec for Tumbleweed, where I cannot change the prjconf. And hardcoding python310- will break, once Tumbleweed switches to python311 as its default python. But piggybacking on %primary_python everything works automagically (at least I hope so...). Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Am 22.11.22 um 10:03 schrieb Johannes Kastl:
Hi Ben,
thanks for taking the time to answer. Appreciated!
On 22.11.22 at 09:50 Ben Greiner wrote:
`%primary_python` affects the behavior and contents of the python310{,-base} packages. The definition for the primary interpreter is where /usr/bin/python3 points to. Thus, for having any effect, you have to rebuild pythin310{,-base} too. Once you have this, every python module in your stack which you rebuild yourself will have a `Provides: python3-foo` and override the python3-foo for Python 3.6. OTOH, this will require a lot of "Prefer:" lines, because the distro will now have multiple providers of python3-foo. Otherwise I do not see why `%primary_python` would need to be set at all.
Objections, your honour! :-)
I am not building the python310 packages, those are delivered by Leap/SLES. And they do not change the default python (i.e. where /usr/bin/python3 links to) and should not.
Then you should not redefine %primary_python to python310!
I still do not get why?
If this only affects the python310 and python310-base packages during build and I am not rebuilding them, there is no negative impact?
It depends which specfiles from Factory you did reuse: bmwiedemann/openSUSE> git grep primary_python packages/_/_project/_config:%define primary_pythonpython310 packages/_/_project/_config:%primary_pythonpython310 packages/p/python-ctypeslib2/python-ctypeslib2.spec:%define primary_python3 python%{python3_version_nodots} packages/p/python-ctypeslib2/python-ctypeslib2.spec:Provides: %{primary_python3}-ctypeslib2 = %{version}-%{release} packages/p/python-ctypeslib2/python-ctypeslib2.spec:Obsoletes: %{primary_python3}-ctypeslib2 < %{version}-%{release} packages/p/python-flit-core/.rev: <comment>- Use prjconf provided primary_pythonif available packages/p/python-flit-core/python-flit-core.changes:- Use prjconf provided primary_pythonif available packages/p/python-flit-core/python-flit-core.spec:# fallback if primary_pythonis not available from the project configuration packages/p/python-flit-core/python-flit-core.spec:%{?!primary_python:%define primary_pythonpython3%{?!sle_version:10}} packages/p/python-flit-core/python-flit-core.spec:%define pprefix %{primary_python} packages/p/python-flit-core/python-flit-core.spec:%define pythons %{primary_python} packages/p/python-flit-core/python-flit-core.spec:%{expand:%%define skip_%{primary_python} 1} packages/p/python-flit-core/python-flit-core.spec:#!BuildIgnore: %{primary_python}-packaging packages/p/python-packaging/python-packaging.spec:# fallback if primary_pythonis not available from the project configuration packages/p/python-packaging/python-packaging.spec:%{?!primary_python:%define primary_pythonpython3%{?!sle_version:10}} packages/p/python-packaging/python-packaging.spec:%define pprefix %{primary_python} packages/p/python-packaging/python-packaging.spec:%define pythons %{primary_python} packages/p/python-packaging/python-packaging.spec:%{expand:%%define skip_%{primary_python} 1} packages/p/python-packaging/python-packaging.spec:Requires: %{primary_python}-pyparsing >= 2.0.2 packages/p/python-pyparsing/.rev:- Remove hardcoded primary_pythonvariable. packages/p/python-pyparsing/python-pyparsing.changes:- Remove hardcoded primary_pythonvariable. packages/p/python-pyparsing/python-pyparsing.spec:%define pprefix %{primary_python} packages/p/python-pyparsing/python-pyparsing.spec:%define pythons %{primary_python} packages/p/python-pyparsing/python-pyparsing.spec:%{expand:%%define skip_%{primary_python} 1} packages/p/python-qtconsole/python-qtconsole.spec:%if "%{python_flavor}" == "%{primary_python}" packages/p/python310/.rev: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python310/python310.changes: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python310/python310.spec:%if "%{python_pkg_name}" == "%{primary_python}" packages/p/python311/.rev: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python311/python311.changes: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python311/python311.spec:%if "%{python_pkg_name}" == "%{primary_python}" packages/p/python38/.rev: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python38/python38.changes: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python38/python38.spec:%if "%{python_pkg_name}" == "%{primary_python}" packages/p/python39/.rev: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python39/python39.changes: %primary_python(gh#openSUSE/python-rpm-macros#127). packages/p/python39/python39.spec:%if "%{python_pkg_name}" == "%{primary_python}" If you did not branch any of these, don't bother. %primary_python is not used by any of the python-rpm-macros.
And inside "my packages", everything just seems to work? I do not want those packages to use the default python (which is 3.6) and I do not want to replace it with 3.10. The SLES-packages are apparently built in a way, that they can be installed in parallel without changing the default python.
The crucial part is the redefinition of %pythons and that you are properly expanding "singlespec" through %python_subpackages, even though the expansion only contains one entry.
For everything else, change python3- to python310-, either directly or thourgh a macro-redirection like %mypython.
Then I would need to also add some if-condition to set this mypython macro inside the spec for Tumbleweed, where I cannot change the prjconf.
And hardcoding python310- will break, once Tumbleweed switches to python311 as its default python. But piggybacking on %primary_python everything works automagically (at least I hope so...).
Hence the non-singlespec packages in Tumbleweed using the python3- alias for the primary interpreter. Change your non-singlespec packages to: %if 0%{?suse_version} < 1550 %define mypython python310 %define __mypython %__python310 %define mypython_sitelib %python310_sitelib # and so on, whatever is used in the specfile %else %define mypython python3 %define __mypython %__python3 %define mypython_sitelib %python3_sitelib # and so on, whatever is used in the specfile %endif BuildRequires: %{mypython}-foo Requires: %{mypython}-foo %files %{mypython_sitelib}/foo %{mypython_sitelib}/foo-%{version}*-info The singlespec stuff works by just changing %pythons in the prjconf. - Ben
Hi Ben, On 22.11.22 at 10:25 Ben Greiner wrote:
If you did not branch any of these, don't bother. %primary_python is not used by any of the python-rpm-macros.
The crucial part is the redefinition of %pythons and that you are properly expanding "singlespec" through %python_subpackages, even though the expansion only contains one entry.
Main use case is ansible, ansible-core and ansible-lint, which are only built for the primary python. Using *full* singlespec would mean the packages are no longer called "ansible-core", but rather "python3-ansible-core" or "python310-ansible-core", if I am not mistaken. Which might cause problems during updates, once Tumbleweed switches to 3.11. But it seems to work even without the %python_subpackages.
Change your non-singlespec packages to:
%if 0%{?suse_version} < 1550 %define mypython python310 %define __mypython %__python310
Out of interest, what exactly does __mypython do?
%define mypython_sitelib %python310_sitelib # and so on, whatever is used in the specfile %else %define mypython python3 %define __mypython %__python3 %define mypython_sitelib %python3_sitelib # and so on, whatever is used in the specfile %endif
BuildRequires: %{mypython}-foo Requires: %{mypython}-foo
%files %{mypython_sitelib}/foo %{mypython_sitelib}/foo-%{version}*-info
The singlespec stuff works by just changing %pythons in the prjconf.
This seems to work, I'll add this to ansible, ansible-core and ansible-lint and then remove my redefinition of the primary_python. Thanks for the help, Ben! Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
participants (2)
-
Ben Greiner -
Johannes Kastl