[opensuse-packaging] Python 2 + 3 from single spec
hello folks, as this got somehow tangled in the big python-python-* thing, I'd like to announce a working prototype of the new macroset that will let us easily build Python 2 and Python 3 packages from a single spec file. Note that this has been possible most of the time, for anyone willing to manually write two filelists, two descriptions, two sets of requirements, etc. etc. But with this new development, it is now possible to run a spec file through a set of small conversions and get working packages with correct dependencies (hopefully). The changes you need to make: - include `%python_subpackages` macro somewhere in your specfile, preferably after the `%description` section - convert `python setup.py build` to `%python_build` and `python setup.py install` to `%python_install`. Or copy your python2-specific build instructions and change instances of python2 to python3 (or vice versa). - convert all `BuildRequires: python-$something` to `BuildRequires: %{python_module $something}` Limitations: - filelists must be the same. I have not figured out a way to make this work otherwise, except maybe by forcing the packager to write out both (all) %files sections by hand, which doesn't seem satisfactory. - subpackages are not supported just yet. They are on the roadmap. - no automated handling of update-alternatives, but that's a whole different story - i have no idea what happens to packages that don't build with distutils/setuptools Anyone interested can have a look at home:matejcik:messing-with-macros repository [1]. I'll probably use rest of today to fork off devel:languages:python:singlespec sub-repository and start copying packages to it. Once a critical mass of packages is there, and unless there are major objections, I'll go ahead and delete with maximum prejudice the ill-considered devel:languages:python3 repository which is a mistake that burdens us to this very day... It would also be nice to get other distros (by which i really mean Fedora) in on this. Anyone here knows how to make that happen? regards m.
Hi, On Fri, Aug 19, 2016 at 04:01:22PM +0200, jan matejek wrote:
hello folks,
as this got somehow tangled in the big python-python-* thing, I'd like to announce a working prototype of the new macroset that will let us easily build Python 2 and Python 3 packages from a single spec file.
Great news!
Note that this has been possible most of the time, for anyone willing to manually write two filelists, two descriptions, two sets of requirements, etc. etc.
But with this new development, it is now possible to run a spec file through a set of small conversions and get working packages with correct dependencies (hopefully).
The changes you need to make:
- include `%python_subpackages` macro somewhere in your specfile, preferably after the `%description` section
- convert `python setup.py build` to `%python_build` and `python setup.py install` to `%python_install`.
Is this based on %py2_build, %py3_build, %py2_install and %py3_install? Would be nice to have theses macros, too (they are avaialble in Fedora).
Or copy your python2-specific build instructions and change instances of python2 to python3 (or vice versa).
- convert all `BuildRequires: python-$something` to `BuildRequires: %{python_module $something}`
What if a BuildRequires or Requires is only needed for a specfic version (described via marker, see pep-0496 in setup.cfg or requirements.txt) ?
Limitations: - filelists must be the same. I have not figured out a way to make this work otherwise, except maybe by forcing the packager to write out both (all) %files sections by hand, which doesn't seem satisfactory. - subpackages are not supported just yet. They are on the roadmap. - no automated handling of update-alternatives, but that's a whole different story - i have no idea what happens to packages that don't build with distutils/setuptools
Anyone interested can have a look at home:matejcik:messing-with-macros repository [1]. I'll probably use rest of today to fork off devel:languages:python:singlespec sub-repository and start copying packages to it. Once a critical mass of packages is there, and unless there are major objections, I'll go ahead and delete with maximum prejudice the ill-considered devel:languages:python3 repository which is a mistake that burdens us to this very day...
It would also be nice to get other distros (by which i really mean Fedora) in on this. Anyone here knows how to make that happen?
Fedora already has single specs for py2/py3 packages. Or what exactly do you mean? Best, Tom -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 19.8.2016 16:40, Thomas Bechtold wrote:
- convert `python setup.py build` to `%python_build` and `python setup.py install` to `%python_install`.
Is this based on %py2_build, %py3_build, %py2_install and %py3_install? Would be nice to have theses macros, too (they are avaialble in Fedora).
No, it's a simple shell-based for-loop now. I'll look into the Fedora macros and possibly port them over.
What if a BuildRequires or Requires is only needed for a specfic version (described via marker, see pep-0496 in setup.cfg or requirements.txt) ?
BuildRequires is common for the whole spec, so it has to include everything. %{python_module} will make sure you get both/all versions (i'm thinking of adding pypy and possibly other pythons later). If you only need a particular version, say, python3-something, you can write BuildRequires: that-specific-thing, without the %{python_module} macro. Different requires for different python versions are not supported at the moment; all resulting packages will have the same Requires set, except with the relevant s/python/python3/ or vice versa. Also thank you rubber duck, in the course of writing this e-mail, i figured out how to accomplish it :) In the next version of the prototype, you will be able to specify: Requires: python-common-for-all-version %ifpython3 Requires: python3-only-required-in-python3 %endif
It would also be nice to get other distros (by which i really mean Fedora) in on this. Anyone here knows how to make that happen?
Fedora already has single specs for py2/py3 packages. Or what exactly do you mean?
Same autogenerating macros for subpackages and filelists. AFAICT Fedora is now doing the thing with listing the different subpackages by hand. m.
Best,
Tom
On vendredi, 19 août 2016 16.01:22 h CEST jan matejek wrote:
Once a critical mass of packages is there, and unless there are major objections, I'll go ahead and delete with maximum prejudice the ill-considered devel:languages:python3 repository which is a mistake that burdens us to this very day...
Beware that some package (python3 only can exist there) so a way to track then is needed ;-) We don't want to make those maintainer less happier than before :-) -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Fri, Aug 19, 2016 at 10:01 AM, jan matejek <jmatejek@suse.com> wrote:
hello folks,
as this got somehow tangled in the big python-python-* thing, I'd like to announce a working prototype of the new macroset that will let us easily build Python 2 and Python 3 packages from a single spec file.
Note that this has been possible most of the time, for anyone willing to manually write two filelists, two descriptions, two sets of requirements, etc. etc.
But with this new development, it is now possible to run a spec file through a set of small conversions and get working packages with correct dependencies (hopefully).
One thing I think that would be really important is that we support more than just Python 2 and Python 3. There are other python implementations out there like pypy that are becoming popular so I hink that it would be great to be able to support them down the road. So I think designing the macros now so that this support could be easily added later will make us much more flexible. So I think rather than having this in terms of "python2" and "python3", we should use something like "cpy2", "cpy3", "pypy2", "pypy3", "jpy2", etc. format for macros. So for example "%have_cpy2", "%have_pyp3", etc. There could also be shortcuts for packages that only support python versions or implementions, so example "py2" would match any python 2 implemention, or "pypy" would match any version of pypy, and just "py" would match any python. These patterns would be followed in every macro, so once you know that pattern you know how to use every macro.
The changes you need to make:
- include `%python_subpackages` macro somewhere in your specfile, preferably after the `%description` section
- convert `python setup.py build` to `%python_build` and `python setup.py install` to `%python_install`. Or copy your python2-specific build instructions and change instances of python2 to python3 (or vice versa).
What about unit tests? I could see either just doing it manually using %if_foo commands, executing an arbitrary python command for each python version, allowing packages to provide their own macros, or tying into update-alternatives somewhow.
- convert all `BuildRequires: python-$something` to `BuildRequires: %{python_module $something}`
Limitations: - filelists must be the same. I have not figured out a way to make this work otherwise, except maybe by forcing the packager to write out both (all) %files sections by hand, which doesn't seem satisfactory.
What about having, for example: %{cpy2_file} file Which expands to: %if_cpy2 file %endif
- subpackages are not supported just yet. They are on the roadmap. - no automated handling of update-alternatives, but that's a whole different story
This should probably be implemented for all packages, not just Python.
- i have no idea what happens to packages that don't build with distutils/setuptools
What about a %{python_exec} macro that executes the following line in a version-specific context for each Python version?
Anyone interested can have a look at home:matejcik:messing-with-macros repository [1]. I'll probably use rest of today to fork off devel:languages:python:singlespec sub-repository and start copying packages to it. Once a critical mass of packages is there, and unless there are major objections, I'll go ahead and delete with maximum prejudice the ill-considered devel:languages:python3 repository which is a mistake that burdens us to this very day...
It would also be nice to get other distros (by which i really mean Fedora) in on this. Anyone here knows how to make that happen?
So here are the macros I was thinking about. (of course pypy and jpy are not supported now, that is just the format they would take if any when they are supported down the road.). Specify which python versions to build: %have_cpy2 (defaults to 1) %have_cpy3 (defaults to 1) %have_pypy2 (defaults to 0) %have_pypy3 (defaults to 0) %have_jpy2 (defaults to 0) General tests: %if_cpy %if_cpy2 %if_cpy3 %if_pypy %if_pypy2 %if_pypy3 %if_jpy %if_jpy2 BuildRequires: %py_buildrequires %py2_buildrequires %py3_buildrequires %cpy_buildrequires %cpy2_buildrequires %cpy3_buildrequires %pypy_buildrequires %pypy2_buildrequires %pypy3_buildrequires %jpy_buildrequires %jpy2_buildrequires Requires: %py_requires %py2_requires %py3_requires %cpy_requires %cpy2_requires %cpy3_requires %pypy_requires %pypy2_requires %pypy3_requires %jpy_requires %jpy2_requires Build: %py_build %py2_build %py3_build %cpy_build %cpy2_build %cpy3_build %pypy_build %pypy2_build %pypy3_build %jpy_build %jpy2_build Install: %py_install %py2_install %py3_install %cpy_install %cpy2_install %cpy3_install %pypy_install %pypy2_install %pypy3_install %jpy_install %jpy2_install Execute an arbitrary python command %py_exec %py2_exec %py3_exec %cpy_exec %cpy2_exec %cpy3_exec %pypy_exec %pypy2_exec %pypy3_exec %jpy_exec %jpy2_exec Files: %py2_file %py3_file %cpy_file %cpy2_file %cpy3_file %pypy_file %pypy2_file %pypy3_file %jpy_file %jpy2_file Excludes/not present: %py2_exclude %py3_exclude %cpy_exclude %cpy2_exclude %cpy3_exclude %pypy_exclude %pypy2_exclude %pypy3_exclude %jpy_exclude %jpy2_exclude -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On 08/19/2016 06:41 PM, Todd Rme wrote:
- convert `python setup.py build` to `%python_build` and `python setup.py install` to `%python_install`. Or copy your python2-specific build instructions and change instances of python2 to python3 (or vice versa). What about unit tests? I could see either just doing it manually using %if_foo commands, executing an arbitrary python command for each python version, allowing packages to provide their own macros, or tying into update-alternatives somewhow. Ideally packages should support `python setup.py test` (and many do although they advice users to run e.g. nosetests), could be executed just by %{python_check} Alternatively, one could use the proposed %{python_exec}, e.g. %{python_exec} -m nose; or %{python_exec} -m pytest; or ...
Sebastian -- python programming - mail server - photo - video - https://sebix.at cryptographic key at https://sebix.at/DC9B463B.asc and on public keyservers
hello, On 19.8.2016 18:41, Todd Rme wrote:
One thing I think that would be really important is that we support more than just Python 2 and Python 3. There are other python
that is what i'm trying to do, yes.
So I think rather than having this in terms of "python2" and "python3", we should use something like "cpy2", "cpy3", "pypy2", "pypy3", "jpy2", etc. format for macros. So for example "%have_cpy2", "%have_pyp3", etc. There could also be shortcuts for packages that only support python versions or implementions, so example "py2" would match any python 2 implemention, or "pypy" would match any version of pypy, and just "py" would match any python. These patterns would be followed in every macro, so once you know that pattern you know how to use every macro.
It is reasonable to have "if_cpython2" etc., but for all the rest, I'd much rather have a common set of python_* macros that would take on the meaning of whatever is currently necessary -- regardless of which python(s) is/are used to build the package. But I think that we will see how much specificity is actually needed.
What about unit tests? I could see either just doing it manually using %if_foo commands, executing an arbitrary python command for each python version, allowing packages to provide their own macros, or tying into update-alternatives somewhow.
at the moment my plan is to do this manually with %if_foo, or alternately through something like %{py_exec_all nosetests} using the %py_exec machinery you proposed.
Limitations: - filelists must be the same. I have not figured out a way to make this work otherwise, except maybe by forcing the packager to write out both (all) %files sections by hand, which doesn't seem satisfactory.
What about having, for example:
%{cpy2_file} file
Unfortunately, i could not manage to write a macro that would work like this. Best I could do was %{cpy2_file file} and there were some unresolved problems too.
Which expands to:
%if_cpy2 file %endif
Which I now figured out how to generate and write, so this is going to be the preferred syntax, unless someone steps up as a better RPM macro writer than I am ;)
What about a %{python_exec} macro that executes the following line in a version-specific context for each Python version?
I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.
So here are the macros I was thinking about. (of course pypy and jpy are not supported now, that is just the format they would take if any when they are supported down the road.).
Specify which python versions to build: %have_cpy2 (defaults to 1) %have_cpy3 (defaults to 1)
yes, although i'd call them full "cpython"
%have_pypy2 (defaults to 0) %have_pypy3 (defaults to 0)
I don't think introducing "pypy2" support now is a good idea. Given that pypy is not established, I say we go ahead and only support the Python3 variant.
General tests: %if_cpy %if_cpy2 %if_cpy3
I don't see the usecase of testing for both "cpython-any" and "cpython-particular". Why would we ever want that? Also there is an interesting technical dilemma with this syntax: either %if_cpython2 means "if cpython2 is in the buildset", or "if the following piece of code applies to the cpython2 subpackage". These are very different in the requires/filelist context: %files %{python_sitelib}/foomodule %if_cpython3 %{python_sitelib}/foomodule3 %endif will either include "foomodule3" in filelist for package presumably called "python-foo", or leave it out. For now I am building the %if_cpython macros in a way that works with the latter; if you want the former, you would explicitly write "%if %have_cpython3"
BuildRequires: %py_buildrequires
With my current approach this doesn't seem useful at all? Either you use %{python_module foo} to get module for all pythons that are in %have_flavor, or you can list the buildrequires explicitly.
Build: %py_build (...)
Install: %py_install (...)
Why would we need this? instead of "python setup.py build", but for particular pythons? That doesn't seem practical IMHO, better to have "%python_build" that builds all, or "%cpy3_exec setup.py build" if you need specific instructions? I suppose this could be something like "do nothing unless %have_cpy3", but even then you'd need to manually list all variants. Perhaps a better way would be %{python_build -x "cpy3"} as in build all except cpy3. I'm not entirely sure how the macros would handle multiple values for the parameter however.
On Mon, Aug 22, 2016 at 9:23 AM, jan matejek <jmatejek@suse.com> wrote:
hello,
On 19.8.2016 18:41, Todd Rme wrote:
One thing I think that would be really important is that we support more than just Python 2 and Python 3. There are other python
that is what i'm trying to do, yes.
So I think rather than having this in terms of "python2" and "python3", we should use something like "cpy2", "cpy3", "pypy2", "pypy3", "jpy2", etc. format for macros. So for example "%have_cpy2", "%have_pyp3", etc. There could also be shortcuts for packages that only support python versions or implementions, so example "py2" would match any python 2 implemention, or "pypy" would match any version of pypy, and just "py" would match any python. These patterns would be followed in every macro, so once you know that pattern you know how to use every macro.
It is reasonable to have "if_cpython2" etc., but for all the rest, I'd much rather have a common set of python_* macros that would take on the meaning of whatever is currently necessary -- regardless of which python(s) is/are used to build the package.
The idea would be to have common macros, as well as more specific macros for special cases.
But I think that we will see how much specificity is actually needed.
What about unit tests? I could see either just doing it manually using %if_foo commands, executing an arbitrary python command for each python version, allowing packages to provide their own macros, or tying into update-alternatives somewhow.
at the moment my plan is to do this manually with %if_foo, or alternately through something like %{py_exec_all nosetests} using the %py_exec machinery you proposed.
Limitations: - filelists must be the same. I have not figured out a way to make this work otherwise, except maybe by forcing the packager to write out both (all) %files sections by hand, which doesn't seem satisfactory.
What about having, for example:
%{cpy2_file} file
Unfortunately, i could not manage to write a macro that would work like this. Best I could do was %{cpy2_file file} and there were some unresolved problems too.
Which expands to:
%if_cpy2 file %endif
Which I now figured out how to generate and write, so this is going to be the preferred syntax, unless someone steps up as a better RPM macro writer than I am ;)
What about a %{python_exec} macro that executes the following line in a version-specific context for each Python version?
I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.
So here are the macros I was thinking about. (of course pypy and jpy are not supported now, that is just the format they would take if any when they are supported down the road.).
Specify which python versions to build: %have_cpy2 (defaults to 1) %have_cpy3 (defaults to 1)
yes, although i'd call them full "cpython"
I was trying to keep things concise and consistent. Having %buildrequires_cpython3 would mess up the alignment currently being used in spec files.
%have_pypy2 (defaults to 0) %have_pypy3 (defaults to 0)
I don't think introducing "pypy2" support now is a good idea. Given that pypy is not established, I say we go ahead and only support the Python3 variant.
I am not suggesting we support it now, just that this is the pattern it would take once it is supported. That being said, it is pretty popular and quite a few python packages already explicitly list support for it.
General tests: %if_cpy %if_cpy2 %if_cpy3
I don't see the usecase of testing for both "cpython-any" and "cpython-particular". Why would we ever want that?
Because some dependencies or actions will be needed for cpython variants but not, for example, pypi variants, or vice versus. Easier to have one macro that can be used rather than writing everything twice.
Also there is an interesting technical dilemma with this syntax: either %if_cpython2 means "if cpython2 is in the buildset", or "if the following piece of code applies to the cpython2 subpackage". These are very different in the requires/filelist context:
This was intended for the latter case. I can't think of a situation where it would be useful for one python variant to know that some other python variant is also being built. I thought the whole idea was to keep them separate.
BuildRequires: %py_buildrequires
With my current approach this doesn't seem useful at all? Either you use %{python_module foo} to get module for all pythons that are in %have_flavor, or you can list the buildrequires explicitly.
Some buildrequires are only relevant, or only available, for certain python variants. So there needs to be some way to specify buildrequires on a per-variant basis. You could wrap them in an %if_foo clause, but that seems much more verbose to me.
Build: %py_build (...)
Install: %py_install (...)
Why would we need this? instead of "python setup.py build", but for particular pythons? That doesn't seem practical IMHO, better to have "%python_build" that builds all, or "%cpy3_exec setup.py build" if you need specific instructions?
This would essentially be a shortcut of the latter. I guess it isn't entirely necessary. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 22.8.2016 20:21, Todd Rme wrote:
Also there is an interesting technical dilemma with this syntax: either %if_cpython2 means "if cpython2 is in the buildset", or "if the following piece of code applies to the cpython2 subpackage". These are very different in the requires/filelist context:
This was intended for the latter case. I can't think of a situation where it would be useful for one python variant to know that some other python variant is also being built. I thought the whole idea was to keep them separate.
The problem becomes apparent when you want a spec file that can build for "all available pythons". You might want to include instructions in %build that apply to python3 only, and you might want the spec file to pass even when python3 is not in the build set. Of course, we can simply say that this configuration is not supported and that the spec itself defines required python versions. That's one way to resolve this. Another is to rely on "%if %have_flavor" for the former purpose, and reserve "%if_cpython2" for the latter.
BuildRequires: %py_buildrequires
With my current approach this doesn't seem useful at all? Either you use %{python_module foo} to get module for all pythons that are in %have_flavor, or you can list the buildrequires explicitly.
Some buildrequires are only relevant, or only available, for certain python variants. So there needs to be some way to specify buildrequires on a per-variant basis. You could wrap them in an %if_foo clause, but that seems much more verbose to me.
BuildRequires are common for the whole spec file. All of them must be present. Say you have a package that needs %flavor-setuptools and %flavor-nose for all %flavors, but pypy-featuresupport for pypy only, and python3-legacy-prints for python3 only: BuildRequires: %{python_module setuptools} BuildRequires: %{python_module nose} BuildRequires: pypy-featuresupport BuildRequires: python3-legacy-prints If we want to support building for "all available pythons", then pypy and python3 requires have to be conditional, yes. But in such case, i'd prefer to use "%if %have_pypy" conditional to signal the intent.
On Tue, Aug 23, 2016 at 9:47 AM, jan matejek <jmatejek@suse.com> wrote:
On 22.8.2016 20:21, Todd Rme wrote:
Also there is an interesting technical dilemma with this syntax: either %if_cpython2 means "if cpython2 is in the buildset", or "if the following piece of code applies to the cpython2 subpackage". These are very different in the requires/filelist context:
This was intended for the latter case. I can't think of a situation where it would be useful for one python variant to know that some other python variant is also being built. I thought the whole idea was to keep them separate.
The problem becomes apparent when you want a spec file that can build for "all available pythons". You might want to include instructions in %build that apply to python3 only, and you might want the spec file to pass even when python3 is not in the build set.
I am still not clear why, for example, the python2 build would need to know whether a python 3 build is happening. At least my thinking was that if a spec file is not going to be built against python3, for example, any python3-specific macros would just be skipped.
BuildRequires: %py_buildrequires
With my current approach this doesn't seem useful at all? Either you use %{python_module foo} to get module for all pythons that are in %have_flavor, or you can list the buildrequires explicitly.
Some buildrequires are only relevant, or only available, for certain python variants. So there needs to be some way to specify buildrequires on a per-variant basis. You could wrap them in an %if_foo clause, but that seems much more verbose to me.
BuildRequires are common for the whole spec file. All of them must be present. Say you have a package that needs %flavor-setuptools and %flavor-nose for all %flavors, but pypy-featuresupport for pypy only, and python3-legacy-prints for python3 only:
That is only true if all supported variants are being built. My thinking was that the whole point of being able to define, for example, %have_cpy2, is to be able to disable that build option without having to modify the rest of the spec file. That way it would be easy for projects to apply a patch to single line in a spec file to disable variants they don't care about. That made me think of another point. Would it be possible to make the %have_foo macros wrap %bcond_with and %bcond_without to be able to easily override it at build time? That way individual projects could even more easily disable particular python variants. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hello, Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:
I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.
You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ? Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please? (I used python3 for the example, but the same question applies to python 2.) Regards, Christian Boltz -- Es gibt Leute, bei denen man im Killfile sein sollte, oder man hat etwas flashc gemacht. [Henning Sponbiel in doc] -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tuesday 2016-08-23 13:54, Christian Boltz wrote:
Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:
I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.
You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?
Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?
Well, env will search $PATH. But of course, there are systems where it is /bin/env, not /usr/bin/env. So you are back to square one. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tue, 23 Aug 2016 13:54, Christian Boltz wrote:
Hello,
Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:
I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.
You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?
Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?
(I used python3 for the example, but the same question applies to python 2.)
+1 for direct use (#!/usr/bin/python3 or #!/usr/bin/python2) Details: Well, theoretically you could define a variable "python3" with a value of, say "/opt/python3-head/python" and mark that variable for export. With "env python3" the content of the variable "python3" would be prefered over the content of the variable "PATH" (see "which python3"). IMHO, for programs that come from distro-core repos (e.g. Version-oss and Version-update) a "direct" call (#!/usr/bin/python3) scould be prefered, esp. for system-tools. For out-lying "leaf" packages, it's more a matter of making sure that either python2 or python3 are used than anythin else. A call to "#!/usr/bin/python" should only bre allowed for fully and proven version agnostic programs. Another positive effect of using direct calls is a much clearer output of tools like "ps" "pstree" "top" and consorts. Try starting 15 different python programs, all with "env python" and look at the output of those tools to see the humbug that the use of "env" causes. But that's me and my 2ct. - Yamaban. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Aug 23 2016, Yamaban <foerster@lisas.de> wrote:
Try starting 15 different python programs, all with "env python" and look at the output of those tools to see the humbug that the use of "env" causes.
Once env has sucessfully called execve you cannot see it any more. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
hi, On 23.8.2016 13:54, Christian Boltz wrote:
Hello,
Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:
I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.
You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?
Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?
This is rather clear-cut: for (almost) everything *we* as the distribution provide, "#!/usr/bin/python3" is correct and the "env" variant is incorrect. The reason being, "/usr/bin/env python3" will happily pick up whichever hand-built version of python the user has placed in their $PATH (or $python3). This is also the purpose of using the "env" construction: as an ISV you can't know whether /usr/bin/python3 exists, or is it in /usr/local, or somewhere in /opt, or maybe in the user's home... However, the programs that we provide must run against the runtimes that we provide -- hence, explicit path. (The users still have the option of executing /opt/myworkspace/bin/python3 script.py, skipping the shebang line altogether.) ...the situation in the build environment is less clear-cut, as the scripts that run during the build are not yet "provided by us"... that's why i was considering messing with the result of env.
(I used python3 for the example, but the same question applies to python 2.)
At this point in time, we really should be turning the scripts from /usr/bin/python to /usr/bin/python2. Or making sure that they can run with python3, which is bound to become /usr/bin/python sooner or later. hope this helps m.
Regards,
Christian Boltz
On Tue, 23 Aug 2016 15:20:34 +0200 jan matejek wrote:
You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?
Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?
This is rather clear-cut: for (almost) everything *we* as the distribution provide, "#!/usr/bin/python3" is correct and the "env" variant is incorrect.
Is this the official openSUSE requirement? If so, I need to fix some of my packages. -- WBR Kyrill
Am 25. August 2016 00:20:35 MESZ, schrieb Kyrill Detinov <lazy.kent@opensuse.org>:
Is this the official openSUSE requirement?
Unless your packages are in Factory using env is fine. Olaf -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hello, On Aug 25 08:52 Olaf Hering wrote (fullquote):
Unless your packages are in Factory using env is fine.
Could you please be a bit more explanatory to help people like me who do not have direct access into your brain ;-) I thought all packages of the openSUSE distribution are in Factory so "using env is fine" only for private packages? Kind Regards Johannes Meixner -- SUSE LINUX GmbH - GF: Felix Imendoerffer, Jane Smithard, Graham Norton - HRB 21284 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, 25 Aug 2016 08:52:11 +0200 Olaf Hering wrote:
Is this the official openSUSE requirement?
Unless your packages are in Factory using env is fine.
I mean the packages in Factory. And I see a change in the recently updated "i18nspector" program (the package is in Factory): ... * Use /usr/bin/env in shebangs. ... https://github.com/jwilk/i18nspector/blob/master/doc/changelog We need a strict rule then. -- WBR Kyrill
On Thu, 25 Aug 2016 17:23, Kyrill Detinov wrote:
On Thu, 25 Aug 2016 08:52:11 +0200 Olaf Hering wrote:
Is this the official openSUSE requirement?
Unless your packages are in Factory using env is fine.
I mean the packages in Factory.
And I see a change in the recently updated "i18nspector" program (the package is in Factory):
... * Use /usr/bin/env in shebangs. ...
https://github.com/jwilk/i18nspector/blob/master/doc/changelog
We need a strict rule then.
May I make a proposal for such a rule? ANYTHING that is in the "Testing-Rings" (is that the whole DVD?) should NOT use /usr/bin/env in shebangs at all. Here is why: To much change for faulty user environment to make things go awry or even open up pathes for attack software. I can understand 3-party software (directly downloaded from the offering project) to make use of such constructs, but in the 'inner' circle of a distro? -- To much risk. - Yamaban. -- PS: I read the list, no direct answers please. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, 25 Aug 2016 17:37:39 +0200 (CEST) Yamaban wrote:
We need a strict rule then.
May I make a proposal for such a rule?
ANYTHING that is in the "Testing-Rings" (is that the whole DVD?) should NOT use /usr/bin/env in shebangs at all.
Found Debian Python Policy: …the form #!/usr/bin/env interpreter_name is deprecated… https://www.debian.org/doc/packaging-manuals/python-policy/ch-programs.html https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html#s-... -- WBR Kyrill
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 08/25/2016 07:16 PM, Kyrill Detinov wrote:
…the form #!/usr/bin/env interpreter_name is deprecated…
hehe, one could still use #!/usr/bin/env /usr/bin/python3 to ensure that a specific install of python3 is used. I don't remember the reason, but for some tools it was easier to check that an "env" script is something executable, wasn't it? Have a nice day, Berny -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJXv02nAAoJEEZQLveWkXGVxoEIAJ6nl6HPjX5AQk78rmmeCYw1 kJpcw0CtXqDuTls1Wb0DHHQVlfJgLGTDAjow7xkEisBRb8dvqSpLPxFSSMY2jtRT FUXX7BsGG0Z5uZ0NkTfD6J4zciqj5o0k3jNIB6YfcuGngRvopnZejeaMAAG7bGrZ nSZbe+zfQTGFhynEH/gZ+cCpULjBGEYcpCOZC1oI3Y38/qPBHTWpkDRfksVfEFur EHHXDQh3DaVy2hgLxHA+hNw6oL8ZsByvDMQf3zwbsxFwvLvd6LnOeeHHUI4PGvVl zMQ59fducTRckov/gUJhkq0drDJ9/EMfO7LdbGtyfdyCV0vMK168jtryFAc9SiU= =s2Gx -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 08/26/2016 01:07 AM, Yamaban wrote:
On Thu, 25 Aug 2016 17:23, Kyrill Detinov wrote:
On Thu, 25 Aug 2016 08:52:11 +0200 Olaf Hering wrote:
Is this the official openSUSE requirement?
Unless your packages are in Factory using env is fine.
I mean the packages in Factory.
And I see a change in the recently updated "i18nspector" program (the package is in Factory):
... * Use /usr/bin/env in shebangs. ...
https://github.com/jwilk/i18nspector/blob/master/doc/changelog
We need a strict rule then.
May I make a proposal for such a rule?
ANYTHING that is in the "Testing-Rings" (is that the whole DVD?) should NOT use /usr/bin/env in shebangs at all.
Why just the "Testing-Rings"? It should cover everything we ship as part of an openSUSE Distribution (Leap/Tumbleweed), third party repo's especially ones on OBS should also be strongly encouraged to do this as well (we can't really force it) as people may want to take those packages and add them to the distro oneday. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Tue, Aug 23, 2016 at 9:20 AM, jan matejek <jmatejek@suse.com> wrote:
hi,
On 23.8.2016 13:54, Christian Boltz wrote:
Hello,
Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:
I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.
You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?
Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?
This is rather clear-cut: for (almost) everything *we* as the distribution provide, "#!/usr/bin/python3" is correct and the "env" variant is incorrect.
The reason being, "/usr/bin/env python3" will happily pick up whichever hand-built version of python the user has placed in their $PATH (or $python3). This is also the purpose of using the "env" construction: as an ISV you can't know whether /usr/bin/python3 exists, or is it in /usr/local, or somewhere in /opt, or maybe in the user's home...
However, the programs that we provide must run against the runtimes that we provide -- hence, explicit path. (The users still have the option of executing /opt/myworkspace/bin/python3 script.py, skipping the shebang line altogether.)
...the situation in the build environment is less clear-cut, as the scripts that run during the build are not yet "provided by us"... that's why i was considering messing with the result of env.
So if we have a third-party package that provides python scripts, should we modify the scripts to change this, or does this only apply if we add it ourselves? If the former, is there any way to automate this since it would require changing a huge number of packages. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 25.8.2016 16:49, Todd Rme wrote:
So if we have a third-party package that provides python scripts, should we modify the scripts to change this, or does this only apply if we add it ourselves? If the former, is there any way to automate this since it would require changing a huge number of packages.
The former -- we should ensure this for all scripts that are in Factory. This was never enforced, but now that we are talking about it, maybe it is a good time to make a rpmlint check out of it and write a macro, something like %fix_shebangs, that would replace "env bla" with "bla"... Could someone please file a bug / openfate feature for this? m.
participants (15)
-
Andreas Schwab -
Bernhard Voelker -
Bruno Friedmann -
Christian Boltz -
Jan Engelhardt -
jan matejek -
jan matejek -
Johannes Meixner -
Kyrill Detinov -
Olaf Hering -
Sebastian -
Simon Lees -
Thomas Bechtold -
Todd Rme -
Yamaban