Python 3 parallel install (Re: Tumbleweed - Review of the week 2020/52)
Hello, my apologies in advance if this mail reaches you up to three times, but in my opinion the topic is relevant and important to reach people subscribed to all three mailing lists. Am 25.12.20 um 18:34 schrieb Dominique Leuenberger / DimStar an factory@lists.opensuse.org:
* Multiple python 3 versions parallel installable. Adding to python 3.8, version 3.6 will be reintroduced. Python modules will be built for both versions.
This line sounds rather innocent, but it has severe implications: Every package python-foo, which uses the so called singlespec macro %python_subpackages, will also try to build a python36-foo package. The whole build process will fail if a step in that particular flavor fails or if building for more than one flavor will create conflicts. We took care [1] of all the packages in Ring0 and Ring1, which popped up as problematic in the staging project. But there are many more packages, which will see the change the first time, when the staging project will be accepted into Factory. You can refer to the documented changes in [1] for examples how to fix a package. There are also some updates to the wiki [2] and the documentation of python-rpm-macros [3]. In particular, look out for: - Packages already skipping Python 2 will start to produce more than one package. Avoid duplicate files by using update-alternatives (there are macros for it in python-rpm-macros), or put common files into unflavored subpackages. - If you provide or require python3-foo, it will only do that for the primary python interpreter. Replace `BuildRequires python3-foo` by `BuildRequires %{python_module foo}` and let the %python_subpackages macro do the work for all Tags and macros where it can replace the unflavored variants e.g.: * `Requires: python-bar` instead of python3-bar. * `%python_sitelib` instead of `%python3_sitelib` in `%files` - Some packages need extra packages backporting stuff to Python 3.6, e.g. importlib-metadata or dataclasses. See black as an example how to deal with this [4] - If upstream does not support python 3.6 anymore (Numpy already deprecates to support it in new releases, according to NEP 29 [5]) you can use `%define skip_python36 1`. All depending packages need to do the same. Alternatively a branched package can provide an older version for this flavor, e.g. python-ipython715 has been created to provide the last version which support Python 3.6. - Python "apps" not following the flavored python-foo naming scheme, will only collect files for one (hopefully the primary python3 providing) flavor during the packaging phase. Remove %python_modules, %python_build, %python_install, %python_expand and the sort, as it unnecessarily pulls in packages for all python flavors and tries to build and install into multiple sitelib/sitearch directories. Replace by calling python3 directly or use the equivalent %python3_* macros. - Some non-singlespec packages only create python3-foo. They will either need an update to include python36-foo, python38-foo (and python39-foo in the near future), or all depending packages must only build for the primary python3, too. `%define pythons python3` or skip all non-primary interpreters. - The previous item is also true for many packages, which have a python bindings subpackage but are not Python only. We introduced %python_subpackage_only so that these packages now can take advantage of the singlespec macros as well. The tracker comment on GitHub [6] has links to a few examples how to use it. - %ifpython3 and %python3_only are evil. They have been deprecated for some months already. Essentially, they fail to recognize the new flavors python36 and python38 which replace python3. OTOH, older distributions will continue to use the python3 flavor. We have introduced a new %python_provides macro, but did not modify %ifpython3 or %python3_only to incorporate this, because it is not automatically clear whether the conditioned code block shall apply to all flavors of Python3 packages or only to the primary interpreter. Replace by something like * `%if "%{python_flavor}" == "python3" || "%{?python_provides}" == "python3"` for sections only provided by the primary flavor. * `%if 0%{python_version_nodots} > 34` or `%if "%{python_flavor}" != "python2"` for section applicable to all Python 3 flavors. Cheers and happy coding, Ben [1] https://github.com/openSUSE/python-rpm-macros/issues/66 [2] https://en.opensuse.org/openSUSE:Packaging_Python [3] https://github.com/openSUSE/python-rpm-macros/blob/master/README.md [4] https://build.opensuse.org/request/show/853412 [5] https://numpy.org/neps/nep-0029-deprecation_policy.html [6] https://github.com/openSUSE/python-rpm-macros/issues/66#issuecomment-7155383...
Dear Python packagers, Am 26.12.20 um 16:01 schrieb Ben Greiner:
Am 25.12.20 um 18:34 schrieb Dominique Leuenberger / DimStar an factory@lists.opensuse.org:
* Multiple python 3 versions parallel installable. Adding to python 3.8, version 3.6 will be reintroduced. Python modules will be built for both versions.
Last Friday evening, all checks for Staging:N were successful and the associated submissions were accepted into openSUSE:Factory. As of now, no snapshot with the changes has been released, and many packages show as unresolvable because the appropriate python36-foo and python38-foo have not been published yet. Hopefully this will be fixed after the maintainers come out of their well deserved weekend. This leaves us with a pressing task:
- If upstream does not support python 3.6 anymore (Numpy already deprecates to support it in new releases, according to NEP 29 [5]) you can use `%define skip_python36 1`. All depending packages need to do the same. Alternatively a branched package can provide an older version for this flavor, e.g. python-ipython715 has been created to provide the last version which support Python 3.6.
SciPy's recent update to 1.6.0 already dropped support for Python 3.6. Luckily for us, the update is stuck in Staging:I, and :N got merged before it. But in order to move on in the spirit of Tumbleweed, we need to `%define skip_python36 1` in python-scipy. As a consequence, all packages requiring python-scipy need to do the same, unless someone steps up to create and maintain a legacy python-scipy15 package (and one for numpy119, see below). Expect a lot of submit request in the next few days, and please help with proactively making the changes in packages, which you care about. For an even larger set of packages, the same will be required for NumPy in the near future. NumPy 1.20 already has has an rc2 and will also drop Python 3.6 following its own NEP 29. Note that from Tumbleweed's point of view, this does not mean to *drop* Python 3.6 support, as Python 3.6 packages have not been on Tumbleweed since the arrival of Python 3.7. It rather prevents (or reverses the very short period of) the reintroduction of those legacy version packages. If your package's dependency on scipy is only optional and upstream still officially supports python 3.6, please consider providing the python36 flavor as long as upstream does it, and just remove the requirement for python <=3.6 while deselecting scipy specific tests. This way, you don't break the package in d:l:p:backports (if it is not already broken). Regards, Ben
Ben Greiner píše v So 16. 01. 2021 v 21:43 +0100:
packages requiring python-scipy need to do the same, unless someone steps up to create and maintain a legacy python-scipy15 package (and one for numpy119, see below). Expect a lot of submit request in the next few days, and please help with proactively making the changes in packages, which you care about.
If somebody wants to do this, please, let us know beforehand. python36 is kept in Tumbleweed mostly for legacy packages (and because we can now with the parallel installable versions), so I would be against putting whole lot of these packages without any serious reasons (after all python38-* scientific packages should be present without any hitch). THANK YOU, AGAIN, EVERYBODY WHO HELPED MAKING THE PARALLEL INSTALLABLE PACKAGES DONE!!! THANK YOU! Best, Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl@ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 If you are not paying for it, you're not the customer; you're the product being sold. -- blue_beetle (2010-08-26) http://www.metafilter.com/95152/Userdriven- discontent#3256046
Hi On 1/17/21 7:13 AM, Ben Greiner wrote:
Dear Python packagers,
This leaves us with a pressing task:
- If upstream does not support python 3.6 anymore (Numpy already deprecates to support it in new releases, according to NEP 29 [5]) you can use `%define skip_python36 1`. All depending packages need to do the same. Alternatively a branched package can provide an older version for this flavor, e.g. python-ipython715 has been created to provide the last version which support Python 3.6.
do we have the ability to do something like `%define skip_python_legacy 1` or `%define skip_python_non_default 1`, I can think of use cases such as bindings for graphical toolkits where it doesn't make sense to support multiple python versions and having to go through and explicitly disable things every time something changes will get painful. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
Hi Am 18.01.21 um 00:01 schrieb Simon Lees:
Hi ... do we have the ability to do something like `%define skip_python_legacy 1` or `%define skip_python_non_default 1`, I can think of use cases such as bindings for graphical toolkits where it doesn't make sense to support multiple python versions and having to go through and explicitly disable things every time something changes will get painful.
We don't. We could create one, but it also has to go the configuration files of all projects using it, just as python_module does. But you can `%define pythons python3` in order to only provide a package through the default python3 flavor, which is always linked to the primary interpreter. Some packages originating from openstack do this. (See comments in gh#openSUSE/python-rpm-macros#66) Ben
Dear Python packagers, Am 16.01.21 um 21:43 schrieb Ben Greiner:
Am 26.12.20 um 16:01 schrieb Ben Greiner:
- If upstream does not support python 3.6 anymore (Numpy already deprecates to support it in new releases, according to NEP 29 [5]) you can use `%define skip_python36 1`. All depending packages need to do the same. Alternatively a branched package can provide an older version for this flavor, e.g. python-ipython715 has been created to provide the last version which support Python 3.6.
SciPy's recent update to 1.6.0 already dropped support for Python 3.6. Luckily for us, the update is stuck in Staging:I, and :N got merged before it. But in order to move on in the spirit of Tumbleweed, we need to `%define skip_python36 1` in python-scipy. As a consequence, all packages requiring python-scipy need to do the same.
This just happened: python-scipy 1.6.0 has been merged into openSUSE:Factory today. There is no longer a python36-scipy , and packages depending on SciPy, which are in Ring1, also already skip the build. Scikit-image and networkx are notable mentions. So do many packages in devel:languages:python:numeric. Due to the drop, the count of unresolvable packages in Factory rose from just below 100 to 184. Please check you python packages now and skip the python36 build if necessary. Please keep your singlespec definitions, so that a new python39 flavor in the (hopefully near) future can pick up, where we just left.
For an even larger set of packages, the same will be required for NumPy in the near future. NumPy 1.20 already has has an rc2 and will also drop Python 3.6 following its own NEP 29.
NumPy 1.20 is out and on it's way into Factory (https://build.opensuse.org/request/show/870996).
If your package's dependency on scipy is only optional and upstream still officially supports python 3.6, please consider providing the python36 flavor as long as upstream does it, and just remove the requirement for python <=3.6 while deselecting scipy specific tests. This way, you don't break the package in d:l:p:backports (if it is not already broken).
The same statement is valid for NumPy. Regards, Ben
participants (3)
-
Ben Greiner
-
Matěj Cepl
-
Simon Lees