On vendredi, 19 août 2016 16.01:22 h CEST jan matejek wrote:

Once a critical mass of packages is there, and unless there are major objections, I'll go ahead and delete with maximum prejudice the ill-considered devel:languages:python3 repository which is a mistake that burdens us to this very day...

Beware that some package (python3 only can exist there) so a way to track then is needed ;-) We don't want to make those maintainer less happier than before :-) -- Bruno Friedmann Ioda-Net Sàrl www.ioda-net.ch Bareos Partner, openSUSE Member, fsfe fellowship GPG KEY : D5C9B751C4653227 irc: tigerfoot -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Fri, Aug 19, 2016 at 10:01 AM, jan matejek wrote:

hello folks,

as this got somehow tangled in the big python-python-* thing, I'd like to announce a working prototype of the new macroset that will let us easily build Python 2 and Python 3 packages from a single spec file.

Note that this has been possible most of the time, for anyone willing to manually write two filelists, two descriptions, two sets of requirements, etc. etc.

But with this new development, it is now possible to run a spec file through a set of small conversions and get working packages with correct dependencies (hopefully).

One thing I think that would be really important is that we support more than just Python 2 and Python 3. There are other python implementations out there like pypy that are becoming popular so I hink that it would be great to be able to support them down the road. So I think designing the macros now so that this support could be easily added later will make us much more flexible. So I think rather than having this in terms of "python2" and "python3", we should use something like "cpy2", "cpy3", "pypy2", "pypy3", "jpy2", etc. format for macros. So for example "%have_cpy2", "%have_pyp3", etc. There could also be shortcuts for packages that only support python versions or implementions, so example "py2" would match any python 2 implemention, or "pypy" would match any version of pypy, and just "py" would match any python. These patterns would be followed in every macro, so once you know that pattern you know how to use every macro.

The changes you need to make:

- include `%python_subpackages` macro somewhere in your specfile, preferably after the `%description` section

- convert `python setup.py build` to `%python_build` and `python setup.py install` to `%python_install`. Or copy your python2-specific build instructions and change instances of python2 to python3 (or vice versa).

What about unit tests? I could see either just doing it manually using %if_foo commands, executing an arbitrary python command for each python version, allowing packages to provide their own macros, or tying into update-alternatives somewhow.

- convert all `BuildRequires: python-$something` to `BuildRequires: %{python_module $something}`

Limitations: - filelists must be the same. I have not figured out a way to make this work otherwise, except maybe by forcing the packager to write out both (all) %files sections by hand, which doesn't seem satisfactory.

What about having, for example: %{cpy2_file} file Which expands to: %if_cpy2 file %endif

- subpackages are not supported just yet. They are on the roadmap. - no automated handling of update-alternatives, but that's a whole different story

This should probably be implemented for all packages, not just Python.

- i have no idea what happens to packages that don't build with distutils/setuptools

What about a %{python_exec} macro that executes the following line in a version-specific context for each Python version?

Anyone interested can have a look at home:matejcik:messing-with-macros repository [1]. I'll probably use rest of today to fork off devel:languages:python:singlespec sub-repository and start copying packages to it. Once a critical mass of packages is there, and unless there are major objections, I'll go ahead and delete with maximum prejudice the ill-considered devel:languages:python3 repository which is a mistake that burdens us to this very day...

It would also be nice to get other distros (by which i really mean Fedora) in on this. Anyone here knows how to make that happen?

So here are the macros I was thinking about. (of course pypy and jpy are not supported now, that is just the format they would take if any when they are supported down the road.). Specify which python versions to build: %have_cpy2 (defaults to 1) %have_cpy3 (defaults to 1) %have_pypy2 (defaults to 0) %have_pypy3 (defaults to 0) %have_jpy2 (defaults to 0) General tests: %if_cpy %if_cpy2 %if_cpy3 %if_pypy %if_pypy2 %if_pypy3 %if_jpy %if_jpy2 BuildRequires: %py_buildrequires %py2_buildrequires %py3_buildrequires %cpy_buildrequires %cpy2_buildrequires %cpy3_buildrequires %pypy_buildrequires %pypy2_buildrequires %pypy3_buildrequires %jpy_buildrequires %jpy2_buildrequires Requires: %py_requires %py2_requires %py3_requires %cpy_requires %cpy2_requires %cpy3_requires %pypy_requires %pypy2_requires %pypy3_requires %jpy_requires %jpy2_requires Build: %py_build %py2_build %py3_build %cpy_build %cpy2_build %cpy3_build %pypy_build %pypy2_build %pypy3_build %jpy_build %jpy2_build Install: %py_install %py2_install %py3_install %cpy_install %cpy2_install %cpy3_install %pypy_install %pypy2_install %pypy3_install %jpy_install %jpy2_install Execute an arbitrary python command %py_exec %py2_exec %py3_exec %cpy_exec %cpy2_exec %cpy3_exec %pypy_exec %pypy2_exec %pypy3_exec %jpy_exec %jpy2_exec Files: %py2_file %py3_file %cpy_file %cpy2_file %cpy3_file %pypy_file %pypy2_file %pypy3_file %jpy_file %jpy2_file Excludes/not present: %py2_exclude %py3_exclude %cpy_exclude %cpy2_exclude %cpy3_exclude %pypy_exclude %pypy2_exclude %pypy3_exclude %jpy_exclude %jpy2_exclude -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

hello, On 19.8.2016 18:41, Todd Rme wrote:

One thing I think that would be really important is that we support more than just Python 2 and Python 3. There are other python

that is what i'm trying to do, yes.

So I think rather than having this in terms of "python2" and "python3", we should use something like "cpy2", "cpy3", "pypy2", "pypy3", "jpy2", etc. format for macros. So for example "%have_cpy2", "%have_pyp3", etc. There could also be shortcuts for packages that only support python versions or implementions, so example "py2" would match any python 2 implemention, or "pypy" would match any version of pypy, and just "py" would match any python. These patterns would be followed in every macro, so once you know that pattern you know how to use every macro.

It is reasonable to have "if_cpython2" etc., but for all the rest, I'd much rather have a common set of python_* macros that would take on the meaning of whatever is currently necessary -- regardless of which python(s) is/are used to build the package. But I think that we will see how much specificity is actually needed.

What about unit tests? I could see either just doing it manually using %if_foo commands, executing an arbitrary python command for each python version, allowing packages to provide their own macros, or tying into update-alternatives somewhow.

at the moment my plan is to do this manually with %if_foo, or alternately through something like %{py_exec_all nosetests} using the %py_exec machinery you proposed.

...

Limitations: - filelists must be the same. I have not figured out a way to make this work otherwise, except maybe by forcing the packager to write out both (all) %files sections by hand, which doesn't seem satisfactory.

What about having, for example:

%{cpy2_file} file

Unfortunately, i could not manage to write a macro that would work like this. Best I could do was %{cpy2_file file} and there were some unresolved problems too.

Which expands to:

%if_cpy2 file %endif

Which I now figured out how to generate and write, so this is going to be the preferred syntax, unless someone steps up as a better RPM macro writer than I am ;)

What about a %{python_exec} macro that executes the following line in a version-specific context for each Python version?

I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.

So here are the macros I was thinking about. (of course pypy and jpy are not supported now, that is just the format they would take if any when they are supported down the road.).

Specify which python versions to build: %have_cpy2 (defaults to 1) %have_cpy3 (defaults to 1)

yes, although i'd call them full "cpython"

%have_pypy2 (defaults to 0) %have_pypy3 (defaults to 0)

I don't think introducing "pypy2" support now is a good idea. Given that pypy is not established, I say we go ahead and only support the Python3 variant.

General tests: %if_cpy %if_cpy2 %if_cpy3

I don't see the usecase of testing for both "cpython-any" and "cpython-particular". Why would we ever want that? Also there is an interesting technical dilemma with this syntax: either %if_cpython2 means "if cpython2 is in the buildset", or "if the following piece of code applies to the cpython2 subpackage". These are very different in the requires/filelist context: %files %{python_sitelib}/foomodule %if_cpython3 %{python_sitelib}/foomodule3 %endif will either include "foomodule3" in filelist for package presumably called "python-foo", or leave it out. For now I am building the %if_cpython macros in a way that works with the latter; if you want the former, you would explicitly write "%if %have_cpython3"

BuildRequires: %py_buildrequires

With my current approach this doesn't seem useful at all? Either you use %{python_module foo} to get module for all pythons that are in %have_flavor, or you can list the buildrequires explicitly.

Build: %py_build (...)

Install: %py_install (...)

Why would we need this? instead of "python setup.py build", but for particular pythons? That doesn't seem practical IMHO, better to have "%python_build" that builds all, or "%cpy3_exec setup.py build" if you need specific instructions? I suppose this could be something like "do nothing unless %have_cpy3", but even then you'd need to manually list all variants. Perhaps a better way would be %{python_build -x "cpy3"} as in build all except cpy3. I'm not entirely sure how the macros would handle multiple values for the parameter however.

Hello, Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:

I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.

You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ? Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please? (I used python3 for the example, but the same question applies to python 2.) Regards, Christian Boltz -- Es gibt Leute, bei denen man im Killfile sein sollte, oder man hat etwas flashc gemacht. [Henning Sponbiel in doc] -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Tue, 23 Aug 2016 13:54, Christian Boltz wrote:

Hello,

Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:

...

I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.

You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?

Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?

(I used python3 for the example, but the same question applies to python 2.)

+1 for direct use (#!/usr/bin/python3 or #!/usr/bin/python2) Details: Well, theoretically you could define a variable "python3" with a value of, say "/opt/python3-head/python" and mark that variable for export. With "env python3" the content of the variable "python3" would be prefered over the content of the variable "PATH" (see "which python3"). IMHO, for programs that come from distro-core repos (e.g. Version-oss and Version-update) a "direct" call (#!/usr/bin/python3) scould be prefered, esp. for system-tools. For out-lying "leaf" packages, it's more a matter of making sure that either python2 or python3 are used than anythin else. A call to "#!/usr/bin/python" should only bre allowed for fully and proven version agnostic programs. Another positive effect of using direct calls is a much clearer output of tools like "ps" "pstree" "top" and consorts. Try starting 15 different python programs, all with "env python" and look at the output of those tools to see the humbug that the use of "env" causes. But that's me and my 2ct. - Yamaban. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

hi, On 23.8.2016 13:54, Christian Boltz wrote:

Hello,

Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:

...

I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.

You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?

Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?

This is rather clear-cut: for (almost) everything *we* as the distribution provide, "#!/usr/bin/python3" is correct and the "env" variant is incorrect. The reason being, "/usr/bin/env python3" will happily pick up whichever hand-built version of python the user has placed in their $PATH (or $python3). This is also the purpose of using the "env" construction: as an ISV you can't know whether /usr/bin/python3 exists, or is it in /usr/local, or somewhere in /opt, or maybe in the user's home... However, the programs that we provide must run against the runtimes that we provide -- hence, explicit path. (The users still have the option of executing /opt/myworkspace/bin/python3 script.py, skipping the shebang line altogether.) ...the situation in the build environment is less clear-cut, as the scripts that run during the build are not yet "provided by us"... that's why i was considering messing with the result of env.

(I used python3 for the example, but the same question applies to python 2.)

At this point in time, we really should be turning the scripts from /usr/bin/python to /usr/bin/python2. Or making sure that they can run with python3, which is bound to become /usr/bin/python sooner or later. hope this helps m.

Regards,

Christian Boltz

On Tue, Aug 23, 2016 at 9:47 AM, jan matejek wrote:

On 22.8.2016 20:21, Todd Rme wrote:

...

...

Also there is an interesting technical dilemma with this syntax: either %if_cpython2 means "if cpython2 is in the buildset", or "if the following piece of code applies to the cpython2 subpackage". These are very different in the requires/filelist context:

This was intended for the latter case. I can't think of a situation where it would be useful for one python variant to know that some other python variant is also being built. I thought the whole idea was to keep them separate.

The problem becomes apparent when you want a spec file that can build for "all available pythons". You might want to include instructions in %build that apply to python3 only, and you might want the spec file to pass even when python3 is not in the build set.

I am still not clear why, for example, the python2 build would need to know whether a python 3 build is happening. At least my thinking was that if a spec file is not going to be built against python3, for example, any python3-specific macros would just be skipped.

...

...

...

BuildRequires: %py_buildrequires

With my current approach this doesn't seem useful at all? Either you use %{python_module foo} to get module for all pythons that are in %have_flavor, or you can list the buildrequires explicitly.

Some buildrequires are only relevant, or only available, for certain python variants. So there needs to be some way to specify buildrequires on a per-variant basis. You could wrap them in an %if_foo clause, but that seems much more verbose to me.

BuildRequires are common for the whole spec file. All of them must be present. Say you have a package that needs %flavor-setuptools and %flavor-nose for all %flavors, but pypy-featuresupport for pypy only, and python3-legacy-prints for python3 only:

That is only true if all supported variants are being built. My thinking was that the whole point of being able to define, for example, %have_cpy2, is to be able to disable that build option without having to modify the rest of the spec file. That way it would be easy for projects to apply a patch to single line in a spec file to disable variants they don't care about. That made me think of another point. Would it be possible to make the %have_foo macros wrap %bcond_with and %bcond_without to be able to easily override it at build time? That way individual projects could even more easily disable particular python variants. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Am 25. August 2016 00:20:35 MESZ, schrieb Kyrill Detinov :

Is this the official openSUSE requirement?

Unless your packages are in Factory using env is fine. Olaf -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Tue, Aug 23, 2016 at 9:20 AM, jan matejek wrote:

hi,

On 23.8.2016 13:54, Christian Boltz wrote:

...

Hello,

Am Montag, 22. August 2016, 15:23:08 CEST schrieb jan matejek:

...

I like this idea. I'll see what I can do about making it work. ISTM you basically only need to substitute the correct executable? Possibly mess with what "env python" returns, but OTOH perhaps we should run the scripts with an explicit interpreter, and consider "env python" a bug... which we've been quietly doing since forever.

You are raising an interesting detail here - shoud python scripts have #!/usr/bin/python3 or #!/usr/bin/env python3 ?

Both work, but I wonder if (and why) one of them is "better". Can you give some insights, please?

This is rather clear-cut: for (almost) everything *we* as the distribution provide, "#!/usr/bin/python3" is correct and the "env" variant is incorrect.

The reason being, "/usr/bin/env python3" will happily pick up whichever hand-built version of python the user has placed in their $PATH (or $python3). This is also the purpose of using the "env" construction: as an ISV you can't know whether /usr/bin/python3 exists, or is it in /usr/local, or somewhere in /opt, or maybe in the user's home...

However, the programs that we provide must run against the runtimes that we provide -- hence, explicit path. (The users still have the option of executing /opt/myworkspace/bin/python3 script.py, skipping the shebang line altogether.)

...the situation in the build environment is less clear-cut, as the scripts that run during the build are not yet "provided by us"... that's why i was considering messing with the result of env.

So if we have a third-party package that provides python scripts, should we modify the scripts to change this, or does this only apply if we add it ourselves? If the former, is there any way to automate this since it would require changing a huge number of packages. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Thu, 25 Aug 2016 17:23, Kyrill Detinov wrote:

On Thu, 25 Aug 2016 08:52:11 +0200 Olaf Hering wrote:

...

...

Is this the official openSUSE requirement?

Unless your packages are in Factory using env is fine.

I mean the packages in Factory.

And I see a change in the recently updated "i18nspector" program (the package is in Factory):

... * Use /usr/bin/env in shebangs. ...

https://github.com/jwilk/i18nspector/blob/master/doc/changelog

We need a strict rule then.

May I make a proposal for such a rule? ANYTHING that is in the "Testing-Rings" (is that the whole DVD?) should NOT use /usr/bin/env in shebangs at all. Here is why: To much change for faulty user environment to make things go awry or even open up pathes for attack software. I can understand 3-party software (directly downloaded from the offering project) to make use of such constructs, but in the 'inner' circle of a distro? -- To much risk. - Yamaban. -- PS: I read the list, no direct answers please. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Thu, 25 Aug 2016 17:37:39 +0200 (CEST) Yamaban wrote:

...

We need a strict rule then.

May I make a proposal for such a rule?

ANYTHING that is in the "Testing-Rings" (is that the whole DVD?) should NOT use /usr/bin/env in shebangs at all.

Found Debian Python Policy: …the form #!/usr/bin/env interpreter_name is deprecated… https://www.debian.org/doc/packaging-manuals/python-policy/ch-programs.html https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html#s-... -- WBR Kyrill

On 08/26/2016 01:07 AM, Yamaban wrote:

On Thu, 25 Aug 2016 17:23, Kyrill Detinov wrote:

...

On Thu, 25 Aug 2016 08:52:11 +0200 Olaf Hering wrote:

...

...

Is this the official openSUSE requirement?

Unless your packages are in Factory using env is fine.

I mean the packages in Factory.

And I see a change in the recently updated "i18nspector" program (the package is in Factory):

... * Use /usr/bin/env in shebangs. ...

https://github.com/jwilk/i18nspector/blob/master/doc/changelog

We need a strict rule then.

May I make a proposal for such a rule?

ANYTHING that is in the "Testing-Rings" (is that the whole DVD?) should NOT use /usr/bin/env in shebangs at all.

Why just the "Testing-Rings"? It should cover everything we ship as part of an openSUSE Distribution (Leap/Tumbleweed), third party repo's especially ones on OBS should also be strongly encouraged to do this as well (we can't really force it) as people may want to take those packages and add them to the distro oneday. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B