[opensuse-packaging] singlespec RFC: python3-only builds, %have_python, %skip_python
Fellow Python packagers, while trying to solve the issues surrounding python3-only singlespec builds, I finally figured out that the OBS is not *at all* doing what I thought it was doing, and everything is now lost forever and now seriously. For various technical reasons that I will discuss at the end of this e-mail, using "%undefine have_python2" for removing python2 from the build set is not viable. I will have to redesign the macros that are present in prjconf and use some other method to accomplish the same. The current WIP solution is as follows: 1. %pythons macro will contain the build set, i.e., list of all flavors that we build for. in Factory, "%pythons" will expand to "python2 python3". 2. contents of %pythons will be conditional on %skip_python macros. If you "%define skip_python2 1" in the spec file, python2 will be removed from %pythons. the "%skip_python" symbols should not be defined ANYWHERE ELSE BUT the current spec file. For prjconf, it is more suitable to change contents of %pythons. 3. %python_module will iterate over %pythons, through a crazy convoluted scheme that Ruby is also using. So for the end packager: * if you want to build everything by distro defaults, nothing changes * if you want to *exclude* a particular flavor from the build, you "%define skip_$flavor 1" * if you want to list build flavors explicitly, you can redefine %pythons themselves: %define pythons python2 jython2 pypy3 * you can BuildRequire: %pythons, because %pythons will come from prjconf For prjconf, we need: * %pythons python2 python3 (This is an opportunity to mess things up, because the *last* item in %pythons actually specifies, through a side effect, the "default" python that overwrites conflicting files. If you specify "python3 python2" and then "%python3_only /usr/bin/foo", the /usr/bin/foo will actually be a python2 version. OTOH, this allows per-package customization of the default, and python-rpm-macros will contain the canonical order.) * the set of python_module expanders Installed Python packages will still define %have_$flavor macros, so you can switch build options based on which pythons are present. However, this is no longer tied to the build set: especially in a local build, you could get %have_python2 true but not actually build it for Python2. That shouldn't matter in practice. comments, questions, thoughts? regards m. ----- p.s.: For those interested, here are the technical details. The original intention was that if you wanted to skip a particular build requirement, you could do: %undefine have_python2 This would remove "python2" from the build set and you wouldn't get any python2-related wording in the spec file. One problem with this approach is that definitions in RPM are placed on a stack and so you need to %undefine as many times as you %defined. Hence the current bug where you need to %undefine have_python2 twice (at least). Another is that OBS doesn't understand %undefine at all! So this declaration actually doesn't do anything, and your %python_modules still install *all* python versions. The double-%undefine is then needed because you get one have_python2 from prjconf and another from the installed python2 package. The new solution doesn't rely on undefines, and redefinitions work about as well as we would expect.
Hi Jan, On Dienstag, 13. Juni 2017, 16:52:45 CEST wrote jan matejek:
Fellow Python packagers,
while trying to solve the issues surrounding python3-only singlespec builds, I finally figured out that the OBS is not *at all* doing what I thought it was doing, and everything is now lost forever
do I want to know more? ;)
and now seriously. For various technical reasons that I will discuss at the end of this e-mail, using "%undefine have_python2" for removing python2 from the build set is not viable. I will have to redesign the macros that are present in prjconf and use some other method to accomplish the same.
The current WIP solution is as follows:
1. %pythons macro will contain the build set, i.e., list of all flavors that we build for. in Factory, "%pythons" will expand to "python2 python3".
2. contents of %pythons will be conditional on %skip_python macros. If you "%define skip_python2 1" in the spec file, python2 will be removed from %pythons. the "%skip_python" symbols should not be defined ANYWHERE ELSE BUT the current spec file. For prjconf, it is more suitable to change contents of %pythons.
3. %python_module will iterate over %pythons, through a crazy convoluted scheme that Ruby is also using.
So for the end packager:
* if you want to build everything by distro defaults, nothing changes * if you want to *exclude* a particular flavor from the build, you "%define skip_$flavor 1" * if you want to list build flavors explicitly, you can redefine %pythons themselves: %define pythons python2 jython2 pypy3 * you can BuildRequire: %pythons, because %pythons will come from prjconf
For prjconf, we need:
* %pythons python2 python3 (This is an opportunity to mess things up, because the *last* item in %pythons actually specifies, through a side effect, the "default" python that overwrites conflicting files. If you specify "python3 python2" and then "%python3_only /usr/bin/foo", the /usr/bin/foo will actually be a python2 version. OTOH, this allows per-package customization of the default, and python-rpm-macros will contain the canonical order.) * the set of python_module expanders
Installed Python packages will still define %have_$flavor macros, so you can switch build options based on which pythons are present. However, this is no longer tied to the build set: especially in a local build, you could get %have_python2 true but not actually build it for Python2. That shouldn't matter in practice.
comments, questions, thoughts?
I think it is a valid solution when you want to enable/disable python version builds per project or repository. Just for reference, here is an example where one spec file builds for both python worlds in parallel. It is IMHO a good solution when you want to sort out it per python package: https://build.opensuse.org/package/view_file/home:adrianSuSE:branches:scienc... Note: The @BUILD_FLAVOR@ will be set by OBS based on the _multibuild file. The main package is excluded here and two sub packages are building. Check the build state box here: https://build.opensuse.org/package/show/home:adrianSuSE:branches:science:uns... ...
Another is that OBS doesn't understand %undefine at all! So this declaration actually doesn't do
this is actually a rpm limitation ;)
anything, and your %python_modules still install *all* python versions. The double-%undefine is then needed because you get one have_python2 from prjconf and another from the installed python2 package.
The new solution doesn't rely on undefines, and redefinitions work about as well as we would expect.
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 13.6.2017 16:58, Adrian Schröter wrote:
Another is that OBS doesn't understand %undefine at all! So this declaration actually doesn't do
this is actually a rpm limitation ;)
RPM understands %undefine just fine ... for some values of "fine" anyway ;) But for the purpose of expanding BuildRequires, OBS is running its own spec parser (build/Build/Rpm.py) and that supports %define but not %undefine. At least as far as I was able to figure out; this also closely matches the observation that OBS's macro expansion step behaves very differently from everything else. regards m.
Hi Jan,
[...] For various technical reasons that I will discuss at the end of this e-mail, using "%undefine have_python2" for removing python2 from the build set is not viable. I will have to redesign the macros that are present in prjconf and use some other method to accomplish the same.
The current WIP solution is as follows: [...] comments, questions, thoughts?
Thanks Jan for all your efforts! Are these nice and fancy topics documented somewhere? ;) -- Gruß/Regards, Thomas Schraitle ---------------------------------------------------------------------- SUSE LINUX GmbH (o< Maxfeldstrasse 5 /\\ Documentation Specialist 90409 Nuernberg, Germany _\_v http://www.suse.com http://lizards.opensuse.org/author/thomas-schraitle/ GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
On Tue, Jun 13, 2017 at 10:52 AM, jan matejek <jmatejek@suse.com> wrote:
Fellow Python packagers,
while trying to solve the issues surrounding python3-only singlespec builds, I finally figured out that the OBS is not *at all* doing what I thought it was doing, and everything is now lost forever
and now seriously. For various technical reasons that I will discuss at the end of this e-mail, using "%undefine have_python2" for removing python2 from the build set is not viable. I will have to redesign the macros that are present in prjconf and use some other method to accomplish the same.
The current WIP solution is as follows:
1. %pythons macro will contain the build set, i.e., list of all flavors that we build for. in Factory, "%pythons" will expand to "python2 python3".
2. contents of %pythons will be conditional on %skip_python macros. If you "%define skip_python2 1" in the spec file, python2 will be removed from %pythons. the "%skip_python" symbols should not be defined ANYWHERE ELSE BUT the current spec file. For prjconf, it is more suitable to change contents of %pythons.
3. %python_module will iterate over %pythons, through a crazy convoluted scheme that Ruby is also using.
So for the end packager:
* if you want to build everything by distro defaults, nothing changes * if you want to *exclude* a particular flavor from the build, you "%define skip_$flavor 1" * if you want to list build flavors explicitly, you can redefine %pythons themselves: %define pythons python2 jython2 pypy3 * you can BuildRequire: %pythons, because %pythons will come from prjconf
For prjconf, we need:
* %pythons python2 python3 (This is an opportunity to mess things up, because the *last* item in %pythons actually specifies, through a side effect, the "default" python that overwrites conflicting files. If you specify "python3 python2" and then "%python3_only /usr/bin/foo", the /usr/bin/foo will actually be a python2 version. OTOH, this allows per-package customization of the default, and python-rpm-macros will contain the canonical order.) * the set of python_module expanders
Installed Python packages will still define %have_$flavor macros, so you can switch build options based on which pythons are present. However, this is no longer tied to the build set: especially in a local build, you could get %have_python2 true but not actually build it for Python2. That shouldn't matter in practice.
comments, questions, thoughts?
regards m.
-----
p.s.: For those interested, here are the technical details.
The original intention was that if you wanted to skip a particular build requirement, you could do: %undefine have_python2 This would remove "python2" from the build set and you wouldn't get any python2-related wording in the spec file.
One problem with this approach is that definitions in RPM are placed on a stack and so you need to %undefine as many times as you %defined. Hence the current bug where you need to %undefine have_python2 twice (at least).
Another is that OBS doesn't understand %undefine at all! So this declaration actually doesn't do anything, and your %python_modules still install *all* python versions. The double-%undefine is then needed because you get one have_python2 from prjconf and another from the installed python2 package.
The new solution doesn't rely on undefines, and redefinitions work about as well as we would expect.
Overall this seems good. The only issue with the current approach from what I can see is that it seems you can only add new pythons by completely replacing the default set. There doesn't seem to be a way to add individual non-default pythons while otherwise keeping the project defaults. This would be important for pypy or jython, where I think we would want to keep it disabled by default and only explicitly turn it on in a per-package basis. So perhaps a "%add_python foo" macro that adds "foo" to "%pythons" if it isn't already there. Perhaps by default it adds it to the beginning of the list if it isn't already there, while "%add_python foo 1" will always add it to the end of the list (promoting it to being the "primary" python version, even if it is already in "%pythons" at a different position). -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 14.6.2017 18:14, Todd Rme wrote:
Overall this seems good.
The only issue with the current approach from what I can see is that it seems you can only add new pythons by completely replacing the default set. There doesn't seem to be a way to add individual non-default pythons while otherwise keeping the project defaults.
True, but unlikely to be fixed. OBS's limited macro processing when evaluating buildrequires probably wouldn't run anything remotely as clever as you're proposing... But it doesn't matter, because of dependencies. It doesn't make much sense to make an individual package supporting "system pythons + jython", because then the package's jython dependencies wouldn't be satisfied. Adding jython to the buildset is something suitable on project level. And on project level, we can afford to change the buildset once in a blue moon if the distro buildset changes... On project level, you could also add the "have_python" condition, and then allow individual packages to "%define have_jython 1" to only build some of them and not the whole project. This is what is done with pypy3 right now: the pypy3 interpreter doesn't build, but you can %define have_pypy3 in your spec today and you get build instructions for pypy3. regards m.
This would be important for pypy or jython, where I think we would want to keep it disabled by default and only explicitly turn it on in a per-package basis. So perhaps a "%add_python foo" macro that adds "foo" to "%pythons" if it isn't already there. Perhaps by default it adds it to the beginning of the list if it isn't already there, while "%add_python foo 1" will always add it to the end of the list (promoting it to being the "primary" python version, even if it is already in "%pythons" at a different position).
jan matejek-4 wrote
and everything is now lost forever
That bad, huh? ;)
comments, questions, thoughts?
Two questions: * Will python_module() have to be defined differently in python3-only specfiles? * Any chance it can be transformed into a macro and moved to the prjconf? It's kind of impossible to remember. Regards -- View this message in context: http://opensuse.14.x6.nabble.com/singlespec-RFC-python3-only-builds-have-pyt... Sent from the opensuse-packaging mailing list archive at Nabble.com. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
hi, On 14.6.2017 21:15, Luigi Baldoni wrote:
Two questions: * Will python_module() have to be defined differently in python3-only specfiles? * Any chance it can be transformed into a macro and moved to the prjconf? It's kind of impossible to remember.
You can transfer it to prjconf in your own projects :) The reason for the redefinition in spec is that we can't modify prjconfs of released distros. In d:l:py, we could add the macros everywhere, but then you'd get specs that fail to build in other projects. That's not desirable. I'm looking into modifying Leap's prjconf, but I'm not sure if it is a good idea to add untested things this close to a release. So. For now, if you want a package for anything other than Tumbleweed, then you need to keep the redefinition, and yes, you need to modify it for a python3-only package. m.
Thanks for all your efforts! Could you add these explanations to the existing wiki page please? Where in the specfile do I need to define %skip_python2? Do you have a working example? I tried to fix d:l:py/mypy but it didn't work out (same errors as before). Sebastian -- python programming - mail server - photo - video - https://sebix.at cryptographic key at https://sebix.at/DC9B463B.asc and on public keyservers
hello, On 15.6.2017 16:40, Sebastian wrote:
Thanks for all your efforts!
Could you add these explanations to the existing wiki page please? Where in the specfile do I need to define %skip_python2? Do you have a working example? I tried to fix d:l:py/mypy but it didn't work out (same errors as before).
this is a proposal! nothing about this works just yet! :) i'll let you (and the list) know when this is deployed. m.
Sebastian
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, I used the new approach for python-typed-ast and unfortunately I now I get this build error: [ 54s] found conflict of python-typed-ast-1.0.4-2.1.x86_64 with python3-typed-ast-1.0.4-2.1.x86_64: for all distros. Sebastian https://build.opensuse.org/package/show/devel%3Alanguages%3Apython/python-ty... On 06/15/2017 04:46 PM, jan matejek wrote:
hello,
Thanks for all your efforts!
Could you add these explanations to the existing wiki page please? Where in the specfile do I need to define %skip_python2? Do you have a working example? I tried to fix d:l:py/mypy but it didn't work out (same errors as before).
On 15.6.2017 16:40, Sebastian wrote: this is a proposal! nothing about this works just yet! :) i'll let you (and the list) know when this is deployed.
m.
Sebastian
-- python programming - mail server - photo - video - https://sebix.at cryptographic key at https://sebix.at/DC9B463B.asc and on public keyservers
Hi, as of today, even with %define python_module() python3-%{**} and %define skip_python2 1 my python3-only module is still trying to create both a python- and a python3- package. But speaking of the change to the default, is the known header: %{?!python_module:%define python_module() python-%{**} python3-%{**}} supposed to become: %{?!python_module:%define python_module() python2-%{**} python3-%{**}} or even: %{?!python_module:%define python_module() python2-%{**} python-%{**}} now ? Regards -- View this message in context: http://opensuse.14.x6.nabble.com/singlespec-RFC-python3-only-builds-have-pyt... Sent from the opensuse-packaging mailing list archive at Nabble.com. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 29.6.2017 11:29, Luigi Baldoni wrote:
Hi, as of today, even with %define python_module() python3-%{**} and %define skip_python2 1 my python3-only module is still trying to create both a python- and a> python3- package.
hm, let's see which three new bugs were introduced by squashing the previous one.
But speaking of the change to the default, is the known header:
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
supposed to become:
%{?!python_module:%define python_module() python2-%{**} python3-%{**}}
definitely not! this header applies only for old distributions. Anything based on today's TW will have a definition of %python_module coming from prjconf and this line will not execute. Old distributions have python-foo for python2, python3-foo for python3. Do not mess with compatibility line. ;) (if you don't plan on supporting Leap 42.x, you can drop this line entirely) regards m.
jan matejek-4 wrote
On 29.6.2017 11:29, Luigi Baldoni wrote:
But speaking of the change to the default, is the known header: supposed to become:
definitely not!
this header applies only for old distributions. Anything based on today's TW will have a definition of %python_module coming from prjconf and this line will not execute.
Good to know. But what about python3-only packages? Will anything have to be redefined in TW? And what of openstack python modules? The ones that were about to supply a Provides: python2-foo in each. This kind of throws a wrench in the works. Unification doesn't seem imminent there, what's the best option, in your opinion? Regards -- View this message in context: http://opensuse.14.x6.nabble.com/singlespec-RFC-python3-only-builds-have-pyt... Sent from the opensuse-packaging mailing list archive at Nabble.com. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 29.6.2017 13:46, Luigi Baldoni wrote:
Good to know. But what about python3-only packages? Will anything have to be redefined in TW?
the TW definition recognizes "skip_python", so no. this is designed to work as is.
And what of openstack python modules? The ones that were about to supply a Provides: python2-foo in each. This kind of throws a wrench in the works. Unification doesn't seem imminent there, what's the best option, in your opinion?
well, I could really use input from people who actually work on OpenStack. But this seems unrelated? If OpenStack packages can't work on python3, that is a problem that OpenStack needs to solve ASAP? m.
jan matejek-4 wrote
On 29.6.2017 13:46, Luigi Baldoni wrote:
Good to know. But what about python3-only packages? Will anything have to be redefined in TW?
the TW definition recognizes "skip_python", so no. this is designed to work as is.
Excellent jan matejek-4 wrote
And what of openstack python modules? The ones that were about to supply a Provides: python2-foo in each. This kind of throws a wrench in the works. Unification doesn't seem imminent there, what's the best option, in your opinion?
well, I could really use input from people who actually work on OpenStack.
But this seems unrelated? If OpenStack packages can't work on python3, that is a problem that OpenStack needs to solve ASAP?
They have meetings every (well, almost) Thursday in #openstack-meeting-alt at 12:00 UTC if you wish to partake. Regards -- View this message in context: http://opensuse.14.x6.nabble.com/singlespec-RFC-python3-only-builds-have-pyt... Sent from the opensuse-packaging mailing list archive at Nabble.com. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
hello, as of today, the new macros are live in Factory, documented on the wiki page, and Sebastian's python-typed-ast bug is also resolved in d:l:py. (still present in factory, I'll submit the updated python-rpm-macros later today) regards m. On 13.6.2017 16:52, jan matejek wrote:
Fellow Python packagers,
while trying to solve the issues surrounding python3-only singlespec builds, I finally figured out that the OBS is not *at all* doing what I thought it was doing, and everything is now lost forever
and now seriously. For various technical reasons that I will discuss at the end of this e-mail, using "%undefine have_python2" for removing python2 from the build set is not viable. I will have to redesign the macros that are present in prjconf and use some other method to accomplish the same.
The current WIP solution is as follows:
1. %pythons macro will contain the build set, i.e., list of all flavors that we build for. in Factory, "%pythons" will expand to "python2 python3".
2. contents of %pythons will be conditional on %skip_python macros. If you "%define skip_python2 1" in the spec file, python2 will be removed from %pythons. the "%skip_python" symbols should not be defined ANYWHERE ELSE BUT the current spec file. For prjconf, it is more suitable to change contents of %pythons.
3. %python_module will iterate over %pythons, through a crazy convoluted scheme that Ruby is also using.
So for the end packager:
* if you want to build everything by distro defaults, nothing changes * if you want to *exclude* a particular flavor from the build, you "%define skip_$flavor 1" * if you want to list build flavors explicitly, you can redefine %pythons themselves: %define pythons python2 jython2 pypy3 * you can BuildRequire: %pythons, because %pythons will come from prjconf
For prjconf, we need:
* %pythons python2 python3 (This is an opportunity to mess things up, because the *last* item in %pythons actually specifies, through a side effect, the "default" python that overwrites conflicting files. If you specify "python3 python2" and then "%python3_only /usr/bin/foo", the /usr/bin/foo will actually be a python2 version. OTOH, this allows per-package customization of the default, and python-rpm-macros will contain the canonical order.) * the set of python_module expanders
Installed Python packages will still define %have_$flavor macros, so you can switch build options based on which pythons are present. However, this is no longer tied to the build set: especially in a local build, you could get %have_python2 true but not actually build it for Python2. That shouldn't matter in practice.
comments, questions, thoughts?
regards m.
-----
p.s.: For those interested, here are the technical details.
The original intention was that if you wanted to skip a particular build requirement, you could do: %undefine have_python2 This would remove "python2" from the build set and you wouldn't get any python2-related wording in the spec file.
One problem with this approach is that definitions in RPM are placed on a stack and so you need to %undefine as many times as you %defined. Hence the current bug where you need to %undefine have_python2 twice (at least).
Another is that OBS doesn't understand %undefine at all! So this declaration actually doesn't do anything, and your %python_modules still install *all* python versions. The double-%undefine is then needed because you get one have_python2 from prjconf and another from the installed python2 package.
The new solution doesn't rely on undefines, and redefinitions work about as well as we would expect.
participants (7)
-
Adrian Schröter -
jan matejek -
jan matejek -
Luigi Baldoni -
Sebastian -
Thomas Schraitle -
Todd Rme