Hello,
my apologies in advance if this mail reaches you up to three times, but
in my opinion the topic is relevant and important to reach people
subscribed to all three mailing lists.
Am 25.12.20 um 18:34 schrieb Dominique Leuenberger / DimStar an
factory(a)lists.opensuse.org:
> * Multiple python 3 versions parallel installable. Adding to python
> 3.8, version 3.6 will be reintroduced. Python modules will be built
> for both versions.
This line sounds rather innocent, but it has severe implications:
Every package python-foo, which uses the so called singlespec macro
%python_subpackages, will also try to build a python36-foo package. The
whole build process will fail if a step in that particular flavor fails
or if building for more than one flavor will create conflicts. We took
care [1] of all the packages in Ring0 and Ring1, which popped up as
problematic in the staging project. But there are many more packages,
which will see the change the first time, when the staging project will
be accepted into Factory. You can refer to the documented changes in [1]
for examples how to fix a package. There are also some updates to the
wiki [2] and the documentation of python-rpm-macros [3].
In particular, look out for:
- Packages already skipping Python 2 will start to produce more than one
package. Avoid duplicate files by using update-alternatives (there are
macros for it in python-rpm-macros), or put common files into unflavored
subpackages.
- If you provide or require python3-foo, it will only do that for the
primary python interpreter. Replace `BuildRequires python3-foo` by
`BuildRequires %{python_module foo}` and let the %python_subpackages
macro do the work for all Tags and macros where it can replace the
unflavored variants e.g.:
* `Requires: python-bar` instead of python3-bar.
* `%python_sitelib` instead of `%python3_sitelib` in `%files`
- Some packages need extra packages backporting stuff to Python 3.6,
e.g. importlib-metadata or dataclasses. See black as an example how to
deal with this [4]
- If upstream does not support python 3.6 anymore (Numpy already
deprecates to support it in new releases, according to NEP 29 [5]) you
can use `%define skip_python36 1`. All depending packages need to do the
same. Alternatively a branched package can provide an older version for
this flavor, e.g. python-ipython715 has been created to provide the last
version which support Python 3.6.
- Python "apps" not following the flavored python-foo naming scheme,
will only collect files for one (hopefully the primary python3
providing) flavor during the packaging phase. Remove %python_modules,
%python_build, %python_install, %python_expand and the sort, as it
unnecessarily pulls in packages for all python flavors and tries to
build and install into multiple sitelib/sitearch directories. Replace by
calling python3 directly or use the equivalent %python3_* macros.
- Some non-singlespec packages only create python3-foo. They will either
need an update to include python36-foo, python38-foo (and python39-foo
in the near future), or all depending packages must only build for the
primary python3, too. `%define pythons python3` or skip all non-primary
interpreters.
- The previous item is also true for many packages, which have a python
bindings subpackage but are not Python only. We introduced
%python_subpackage_only so that these packages now can take advantage of
the singlespec macros as well. The tracker comment on GitHub [6] has
links to a few examples how to use it.
- %ifpython3 and %python3_only are evil. They have been deprecated for
some months already. Essentially, they fail to recognize the new flavors
python36 and python38 which replace python3. OTOH, older distributions
will continue to use the python3 flavor. We have introduced a new
%python_provides macro, but did not modify %ifpython3 or %python3_only
to incorporate this, because it is not automatically clear whether the
conditioned code block shall apply to all flavors of Python3 packages or
only to the primary interpreter. Replace by something like
* `%if "%{python_flavor}" == "python3" || "%{?python_provides}" ==
"python3"` for sections only provided by the primary flavor.
* `%if 0%{python_version_nodots} > 34` or `%if "%{python_flavor}" !=
"python2"` for section applicable to all Python 3 flavors.
Cheers and happy coding,
Ben
[1] https://github.com/openSUSE/python-rpm-macros/issues/66
[2] https://en.opensuse.org/openSUSE:Packaging_Python
[3] https://github.com/openSUSE/python-rpm-macros/blob/master/README.md
[4] https://build.opensuse.org/request/show/853412
[5] https://numpy.org/neps/nep-0029-deprecation_policy.html
[6]
https://github.com/openSUSE/python-rpm-macros/issues/66#issuecomment-715538…
Neal Gompa píše v So 26. 12. 2020 v 10:21 -0500:
> On Sat, Dec 26, 2020 at 10:02 AM Ben Greiner <code(a)bnavigator.de> wrote:
> > my apologies in advance if this mail reaches you up to three times, but
> > in my opinion the topic is relevant and important to reach people
> > subscribed to all three mailing lists.
First of all, I would like to thank Ben for the gigantic amount
of work he spent helping to make this working. Understanding our
packaging macros is feat which is not simple, and he made it
working mostly on his own. THANK YOU!
> I'm still not convinced this is a good idea. This fundamental
> change to how the Python stack works means that we will have
> to deal with this for all the different modules, and there are
> packages that deliberately don't use the "singlespec" model
> because it doesn't work for them (e.g. they don't use
> setuptools or they are an application).
>
> What is the compelling reason to fully build out the stack
> this way?
What Ben was saying: we have just too many stakeholders in
Tumbleweed. We market OpenSUSE as “The makers’ choice for
sysadmins, developers and desktop users.”, but that's slightly
disingenuous because each of these groups wants something
different. Let me consider these examples:
* I would love to have well-tested python3* package released on
the same day as it is released upstream. Not as the primary
interpreter of Python, but at least with minimal set of
packages (at least setuptools, pip, and pytest). That would
require to have separate Staging for it or perhaps even beta
version in Factory, and ability to keep those packages
separate from the normal primary interpreter. We cannot do
this now, obviously, because even now not all python-*
packages are well working with 3.8, where the current wave of
removing long-deprecated features started (the trend continues
with 3.9, and it will certainly continue in 3.10). We were forced
in past to patch in hurry all those packages ourselves, but
the result (even after testing with full d:l:p of packages)
was that every major release was minor disaster in Factory.
* Obviously most people don't care about version of Python
interpreter that much and they just want something reasonably
recent and reasonably stable. Python 3.8 is probably the
version for them at this moment.
* There is however also small group of people who grumble that
Factory goes too fast, and they have some good reasons why
they want the oldest upstream supported version (3.6 at the
moment). In OpenSUSE these are just minor grumblings, in SLE
there are our customers who are paying good money for having
these older versions. The same technology of using parallel
versions in OpenSUSE will be used in SLE (e.g., currently we
have fully synchronized SPEC files of python36 in SLE-12,
SLE-15, and in Tumbleweed).
* OpenSUSE is currently as far as I know the only major Linux
distribution which doesn’t have PyPy at all (the package in
https://build.opensuse.org/package/show/devel:languages:python:Factory/pypy3
has been broken for long time; anybody willing to pick up that
challenge?). With the separate parallel packages we could put
some good effort to package that as well without fear of
breaking everything.
* I don’t know, if it would be possible, but I still dream about
having Jython as the equivalent Python interpreter (if they
ever manage to have Python 3 version).
This new version of packaging macros allows us to support (to
some extent) these different groups at the same time. Without
this substantial investment in packaging macros further
development of Python packaging in Factory would be severely
hampered.
Thank again to Ben, and Merry Christmas to all OpenSUSE friends
of good will!
Matěj
--
https://matej.ceplovi.cz/blog/, Jabber: mcepl(a)ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8
According to the Franciscan priest Richard Rohr, spirituality is
not for people who are trying to avoid hell; it is for people
who have been through hell. In many ways, spirituality is about
what we do with our pain. And the truth is, if we don't
transform it, we will transmit it.
-- Al Gustafson
Am 26.12.20 um 16:21 schrieb Neal Gompa:
> On Sat, Dec 26, 2020 at 10:02 AM Ben Greiner <code(a)bnavigator.de> wrote:
>>
>> - Python "apps" not following the flavored python-foo naming scheme,
>> will only collect files for one (hopefully the primary python3
>> providing) flavor during the packaging phase. Remove %python_modules,
>> %python_build, %python_install, %python_expand and the sort, as it
>> unnecessarily pulls in packages for all python flavors and tries to
>> build and install into multiple sitelib/sitearch directories. Replace by
>> calling python3 directly or use the equivalent %python3_* macros.
>>
>>
>> [...]
>>
>>
>>
>> - %ifpython3 and %python3_only are evil. They have been deprecated for
>> some months already. Essentially, they fail to recognize the new flavors
>> python36 and python38 which replace python3. OTOH, older distributions
>> will continue to use the python3 flavor. We have introduced a new
>> %python_provides macro, but did not modify %ifpython3 or %python3_only
>> to incorporate this, because it is not automatically clear whether the
>> conditioned code block shall apply to all flavors of Python3 packages or
>> only to the primary interpreter. Replace by something like
>>
>> * `%if "%{python_flavor}" == "python3" || "%{?python_provides}" ==
>> "python3"` for sections only provided by the primary flavor.
>>
>> * `%if 0%{python_version_nodots} > 34` or `%if "%{python_flavor}" !=
>> "python2"` for section applicable to all Python 3 flavors.
>>
>>
> I'm still not convinced this is a good idea. This fundamental change
> to how the Python stack works means that we will have to deal with
> this for all the different modules, and there are packages that
> deliberately don't use the "singlespec" model because it doesn't work
> for them (e.g. they don't use setuptools or they are an application).
>
Nobody has to use the singlespec model. Thankfully, nobody needs to
convince you, or anybody else. But if you do use the macros, you have to
use them correctly. This change has nothing to do with setuptools, and
for applications see my comment about not using "%python_module" and the
flavorless macros, which are -- and always have been -- singlespec.
> What is the compelling reason to fully build out the stack this way?
Nothing changed in the "stack". You can continue to build against the
primary python interpreter only.
One reason for parallel install, which I can personally imagine, is to
provide future python versions faster without switching to them as the
primary python3 interpreter immediately. Python 3.9 has been released
some weeks ago, but virtually no packages are built against it because
many upstream projects are not ready yet. If we build against python39
in devel projects, we can detect early that a package needs work.
Ben
We (SUSE) have a client complaining about our python-base pulling
in unnecessary requirement python-rpm-macros. I will probably
remove this requirement all over the board from all Python -base
packages, but I would like to have some feedback, whether anybody
sees anything wrong with this removal.
And Merry Christmas and less crazy and happy new year 2021!
Matěj
--
https://matej.ceplovi.cz/blog/, Jabber: mcepl(a)ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8
Monday, December 9th. We skip the bus tour of Stockholm to attend
the economics lecture. Our guest status is again good for front
row seats. We hear about the theory of auctions. There are
integrals and derivatives. It's like physics except physics
works.
Hello,
At the time of writing we have 538 packages in d:l:python which
are out of date and require updating.
There are only two reasons why any Python package should be
packaged for OpenSUSE: either it is dependency of another
package in OpenSUSE, or we want to maintain it. Otherwise, if an
user wants an unmaintained package, every user has pip available
and they can install a package from PyPI directly.
The conclusion of these two points is that every package in Factory
carries with itself some (small) cost, and there is no point in
trying to push all of PyPI into Factory.
Therefore I suggest these limitations on putting new packages
into Factory:
1. Every new package submitted to d:l:p (or any other official
OpenSUSE project) SHALL include in its submit request message
“business reasons” for including the package into OpenSUSE
(either because it is dependency of some other package, or
some other reason, why it is needed).
2. Everybody who wants to submit new package to OpenSUSE, MUST
submit two updates of packages already in Factory from the
list delivered to this list every week.
3. Packages which fail to build for sufficiently long time SHALL
be removed from Factory and d:l:python (or moved to
d:l:p:misc).
John, let me address you directly, because you do by far the
most work for OpenSUSE Python packages. I really do appreciate
how incredibly much you do for OpenSUSE, but I just don’t think
it is right, when from 30+ requests on any work day, twenty of
them are submissions of your new packages. Moreover, some of
those packages are really questionable. For example, do we
really want to maintain in OpenSUSE complete set of packages
needed for accessing Fedora infrastructure? Why anybody who
wants to maintain Fedora packages on OpenSUSE systems (and that
include me, I have still rights for some Fedora packages) cannot
just use `pip install --user copr-cli`?
Any queries and comments are welcome,
Matěj
--
https://matej.ceplovi.cz/blog/, Jabber: mcepl(a)ceplovi.cz
GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8
The truth is a beautiful and terrible thing, and should therefore
be treated with caution.
-- Albus Dumbledore