[opensuse-packaging] Factory submission policy in light of default-python switch
fellow pythonistas, i'd like to propose a new review rule: do not accept new python modules into Factory that are not either singlespec or explicitly versioned. So "python2-foo" is fine, "python3-foo" is fine, "python-foo" which is not singlespec is rejected. For apps that have python as a requirement, do not accept anything that requires "python", and look hard on apps that require "python2". (they should require python3 unless the app actually doesn't work with it) This would apply to *new* packages. Anyone opposed? I'd like to apply to existing packages too... as in, if you touch a thing that requires python, also make sure that it is versioned. But perhaps this is not a reason for rejection, more like, leave a comment "can you please also do this"? Thoughts? m.
On Thu, Jun 29, 2017 at 7:44 AM, jan matejek <jmatejek@suse.com> wrote:
fellow pythonistas,
i'd like to propose a new review rule: do not accept new python modules into Factory that are not either singlespec or explicitly versioned.
So "python2-foo" is fine, "python3-foo" is fine, "python-foo" which is not singlespec is rejected.
For apps that have python as a requirement, do not accept anything that requires "python", and look hard on apps that require "python2". (they should require python3 unless the app actually doesn't work with it)
This would apply to *new* packages. Anyone opposed?
I'd like to apply to existing packages too... as in, if you touch a thing that requires python, also make sure that it is versioned. But perhaps this is not a reason for rejection, more like, leave a comment "can you please also do this"?
Thoughts?
m.
Jan, I'm not a pythonista, but I maintain a large python based package - python-plaso. Upstream is working on python 3 compatibility, but it isn't there yet. It is dependant on a couple dozen (or more) python modules: https://build.opensuse.org/package/binary/security:forensics/python-plaso?arch=x86_64&filename=python-plaso-1.5.1-26.3.noarch.rpm&repository=openSUSE_Tumbleweed Note none of the modules are exclusively available as python3. What happens when the first of those dependencies goes python 3 only? Thanks Greg -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hello,
Note none of the modules are exclusively available as python3. What happens when the first of those dependencies goes python 3 only?
there is no reason to think any of them would go python3-only, unless upstream drops support. If/when the packages are singlespecced, python2 versions will still be available. If this does happen, then your package would break, and you would need to separately maintain an old version of that package which is still compatible with python 2. does this answer your question? regards m.
Thanks Greg
-- Greg Freemyer Advances are made by answering questions. Discoveries are made by questioning answers. — Bernard Haisch On Thu, Jun 29, 2017 at 8:39 AM, jan matejek <jmatejek@suse.cz> wrote:
Hello,
Note none of the modules are exclusively available as python3. What happens when the first of those dependencies goes python 3 only?
there is no reason to think any of them would go python3-only, unless upstream drops support. If/when the packages are singlespecced, python2 versions will still be available.
If this does happen, then your package would break, and you would need to separately maintain an old version of that package which is still compatible with python 2.
does this answer your question?
I fear so. My question related to your earlier statement:
So "python2-foo" is fine, "python3-foo" is fine, "python-foo" which is not singlespec is rejected.
On its face I could address that requirement in python-liblnk by modifying the specfile to appropriately produce python3-liblnk instead. Unfortunately I take your answer to say that will break python-plaso currently. Thus I now know the only viable solution is to do what I've done in libsmdev as an example and ensure both python2-smdev and python3-smdev are available, the create a version free tag (pysmdev) that both of them provide and have python-plaso require: pysmdev. There's about 20 c library packages with python bindings I maintain outside of d:l:p. None have been adjusted for singlespec. They all have similar spec file structures. Perhaps I can ask you to adjust libsmraw in security:forensics to use singlespec, then I can replicate it to the others. Alternatively, libsmdev was my pre-singlespec effort to add dual python2/python3 support. Thanks Greg -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 29.6.2017 15:49, Greg Freemyer wrote:
So "python2-foo" is fine, "python3-foo" is fine, "python-foo" which is not singlespec is rejected.
On its face I could address that requirement in python-liblnk by modifying the specfile to appropriately produce python3-liblnk instead.
Actually in your case, you would modify it to be python2-liblnk, requiring python2, and the package requiring python2-devel. That is also valid under this proposal. This would mean that you need to touch all packages when you convert python-plaso to python3. But you probably need that anyway because when we switch the default, this package is likely to break.
Thus I now know the only viable solution is to do what I've done in libsmdev as an example and ensure both python2-smdev and python3-smdev are available, the create a version free tag (pysmdev) that both of them provide and have python-plaso require: pysmdev.
this looks like a wrong approach to me. why can't you require python2-smdev? having a version-free provides seems to ask for provider conflicts, iiuc?
Perhaps I can ask you to adjust libsmraw in security:forensics to use singlespec, then I can replicate it to the others.
this is probably not supported by singlespec at the moment. But it is a good point, we need to have some sort of support for packages like this before the python3 switch.
Alternatively, libsmdev was my pre-singlespec effort to add dual python2/python3 support.
This looks good to me, actually. Singlespec could save you some typing here, but not really much, esp. given that you don't need special build steps. All in all, i'm not sure I understand the problems you are trying to solve. As I see it, you have a package, python-plaso, that requires Python 2 at the moment, but you expect to switch to Python 3 in the near future. The singlespec approach would mean that you convert the package to singlespec using the usual instructions, set %pythons to python2, and produce python2-plaso. You would need to make sure that all your requirements exist as "python2-foo", and ideally that they are also singlespec (so both python2-foo and python3-foo (and possibly pypy3-foo, jython-foo etc.) also exist). This part is more difficult at the moment because of some missing pieces, but can be done manually, or we can figure out what the missing pieces are and add them to the singlespec system. Then when Python3 support arrives, you would switch the package to build for all pythons. Alternately, if you don't expect python3 support, you should make sure that the package is explicitly versioned for python2, requires python2-* etc. This way, when we switch the default, your package will still build fine for python2. If you want your package to switch pythons based on, say, which distro it is in, then you can leave it as-is and make sure that the requirement under "python-libsm", for instance, will be pulled in the right version. Maybe this is the missing piece? Currently singlespec says "Provides: python-foo" in package "python2-foo". This will probably change when the default is switched, and instead the provides will come from python3-foo. please explain more, i guess :) m.
Thanks Greg
More below, but as I said I'm not a pythonista. On Thu, Jun 29, 2017 at 10:14 AM, jan matejek <jmatejek@suse.cz> wrote:
On 29.6.2017 15:49, Greg Freemyer wrote:
So "python2-foo" is fine, "python3-foo" is fine, "python-foo" which is not singlespec is rejected.
On its face I could address that requirement in python-liblnk by modifying the specfile to appropriately produce python3-liblnk instead.
Actually in your case, you would modify it to be python2-liblnk, requiring python2, and the package requiring python2-devel. That is also valid under this proposal.
This would mean that you need to touch all packages when you convert python-plaso to python3. But you probably need that anyway because when we switch the default, this package is likely to break.
Okay
Thus I now know the only viable solution is to do what I've done in libsmdev as an example and ensure both python2-smdev and python3-smdev are available, the create a version free tag (pysmdev) that both of them provide and have python-plaso require: pysmdev.
this looks like a wrong approach to me. why can't you require python2-smdev? having a version-free provides seems to ask for provider conflicts, iiuc?
I likely made a rash assumption in the past. I assumed that if as an example I did a provides pysmdev in both python2-smdev and python3-smdev, and then did a "requires smdev" in python-plaso, zypper would be smart enough to give me the version appropriate to the python version in use. Is there a python neutral "requires" statement? Or does it have to be explicit from the top down?
Perhaps I can ask you to adjust libsmraw in security:forensics to use singlespec, then I can replicate it to the others.
this is probably not supported by singlespec at the moment. But it is a good point, we need to have some sort of support for packages like this before the python3 switch.
Alternatively, libsmdev was my pre-singlespec effort to add dual python2/python3 support.
This looks good to me, actually. Singlespec could save you some typing here, but not really much, esp. given that you don't need special build steps.
I guess I will move forward without singlespec's functionality. After all python3 will likely be around a long time.
All in all, i'm not sure I understand the problems you are trying to solve. As I see it, you have a package, python-plaso, that requires Python 2 at the moment, but you expect to switch to Python 3 in the near future.
I don't know when. Upstream provides most of the various python-* packages plaso uses. But there are multiple front-end packages, so it is a many-to-many relationship between the packages. My plan is to get all the underlying packages available in both python2 and python3, then convert the front-end packages one at a time.
The singlespec approach would mean that you convert the package to singlespec using the usual instructions, set %pythons to python2, and produce python2-plaso. You would need to make sure that all your requirements exist as "python2-foo", and ideally that they are also singlespec (so both python2-foo and python3-foo (and possibly pypy3-foo, jython-foo etc.) also exist). This part is more difficult at the moment because of some missing pieces, but can be done manually, or we can figure out what the missing pieces are and add them to the singlespec system. Then when Python3 support arrives, you would switch the package to build for all pythons.
From earlier comment, I think I will just ignore singlespec for some years.
Alternately, if you don't expect python3 support, you should make sure that the package is explicitly versioned for python2, requires python2-* etc. This way, when we switch the default, your package will still build fine for python2.
Many / most of the pieces already pass python3 self-test per upstream. It's just that I don't have the python3 packages in place yet. And if I did, the python3 support isn't yet in 100 of the packages.
If you want your package to switch pythons based on, say, which distro it is in, then you can leave it as-is and make sure that the requirement under "python-libsm", for instance, will be pulled in the right version.
Maybe this is the missing piece? Currently singlespec says "Provides: python-foo" in package "python2-foo". This will probably change when the default is switched, and instead the provides will come from python3-foo.
I think I got it. The biggest issue is I assumed there was a python neutral "Provides" mechanism to notify zypper of a packages availability in multiple flavors.
please explain more, i guess :) m.
Thanks Greg -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 29.6.2017 20:45, Greg Freemyer wrote:
this looks like a wrong approach to me. why can't you require python2-smdev? having a version-free provides seems to ask for provider conflicts, iiuc?
I likely made a rash assumption in the past.
I assumed that if as an example I did a provides pysmdev in both python2-smdev and python3-smdev, and then did a "requires smdev" in python-plaso, zypper would be smart enough to give me the version appropriate to the python version in use.
Is there a python neutral "requires" statement? Or does it have to be explicit from the top down?
In general, it needs to be explicit from the top down. Your end app knows which version it's using and should require the appropriate libraries. With singlespec, you can specify "Requires: python-something" and it gets rewritten to the appropriate version. There's a vague plan to expand this functionality to non-library packages, but nothing that works today. I'm not even sure if rpm/zypper can express this sort of dynamic dependency. What you did, with the unversioned symbol, seems to me to lead to a bunch of "Prefer" lines in the prjconf. Perhaps for you it would be easiest to say something like: %define mypython python2 BuildRequires: %{mypython}-foo Requires: %{mypython}-foo and then switch %mypython to the appropriate flavor? Or if you want to keep with the distro preferred language, you can use %system_python from python-rpm-macros. regards m.
Jan, On Thu, 2017-06-29 at 13:44 +0200, jan matejek wrote:
fellow pythonistas,
i'd like to propose a new review rule: do not accept new python modules into Factory that are not either singlespec or explicitly versioned.
So "python2-foo" is fine, "python3-foo" is fine, "python-foo" which is not singlespec is rejected.
sounds reasonable to me - but as this seems to be something a bot can check upon, I would rather see the bot take care of that: If it's cheap / easy to check : https://github.com/openSUSE/osc-plugin-factory/blob/master/source-check er.pl if more complex or time consuming, then we should probably rather add it to repo-checker - which already spends quite some time on many submissions. Reason why a bot: the bot is just less 'stressed' any makes less mistakes for such obvious things. And people can discuss with the bot as much as they want.
For apps that have python as a requirement, do not accept anything that requires "python", and look hard on apps that require "python2". (they should require python3 unless the app actually doesn't work with it)
That one is probably harder to have in a bot, so will have to be cast off to the review team.
This would apply to *new* packages. Anyone opposed?
I'd like to apply to existing packages too... as in, if you touch a thing that requires python, also make sure that it is versioned. But perhaps this is not a reason for rejection, more like, leave a comment "can you please also do this"?
imho, this kind of change should be implemented in rpmlint checks first; so that we can get an overview of how much break we curretnly have (remember, we have http://rpmlint.opensuse.org where we casn filter on them) Cheers, Dominique
participants (4)
-
Dominique Leuenberger / DimStar -
Greg Freemyer -
jan matejek -
jan matejek