[opensuse-packaging] New thoughts on python singlespec macros

Todd Rme

14 Nov 2017 14 Nov '17

18:29

Thinking about what we need for python3 conversion long-term, a few issues have come to mind that currently aren't addressed well in the macros or previous discussions: 1. We really need to get rid of using unversioned "python" in shebangs. 2. We need to switch everything over to use versioned rpm names (particularly "python2-foo"), while still supporting older SLE/Leap versions. 3. We need to be able to handle builds that don't include python2. 4. pytest often checks the directories being used, which breaks when using "%python_expand" Regarding 1, there are currently two pain points. First and most obvious is the fact that the shebangs have to be edit manually. The second is that if we want versioned shebangs, we need to manually recompile the files to avoid mtime issues. Ideally changing the shebangs could be handled automatically during %python_build, but there could still be a separate macro that could be called in cases where manual changes are necessary. Recompiling is needed in other situations as well, so having a macro to handle that would be useful in general. Regarding 2, this is more of a policy issue. I think all packages, even python2-only ones, should be switched over to using singlespec. This makes sure that they have the correct name on all platforms. Backports packages should have a conditional "skip_python3" that is set to disable building for python versions equal to or greater than the version they were backported from, even when those versions are no longer shipped by openSUSE. There should also be a policy that, when an upstream package drops python2 support, the last-supported python2 version should be made available in the "python2-foo" package. Only "python-foo-doc" and "python-foo-data" can have the unversioned "python-" name, and that is only for version-independent documentation and assets, respectively. Regarding 3, there are again two main pain points. The first is that there is no macro I am aware of to reliable determine if a given python version is being built. %have_python2 and %have_python3 no longer work reliably AFAIK. The second is that build dependencies for particular python versions cannot be reliably disabled if that python version is not built. Currently most packages that have a python2-only build dependency just unconditionally require it, which means it will be pulled in even if python2 builds are disabled. Ideally I would like to see "%{python2_module foo}" and "%{python3_module foo}" that will only pull in that dependency if that version of python is being used. This also has the advantage of not needing to care about backwards-compatibility issues of "python-foo" vs. "python2-foo" names, which is handled inconsistently right now. If that is not feasible, just having a reliable check would be an improvement. Regarding 4, I don't really understand what "%python_expand" is doing behind the scenes, and whether there is a simple workaround or whether the macro needs to do something different internally. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Show replies by date

Jan Engelhardt

14 Nov 14 Nov

18:43

On Tuesday 2017-11-14 19:29, Todd Rme wrote:

...

1. We really need to get rid of using unversioned "python" in shebangs.

Regarding 1, there are currently two pain points. First and most obvious is the fact that the shebangs have to be edit manually. The second is that if we want versioned shebangs, we need to manually recompile the files to avoid mtime issues.

Ideally changing the shebangs could be handled automatically during %python_build, but there could still be a separate macro that could be called in cases where manual changes are necessary. Recompiling is needed in other situations as well, so having a macro to handle that would be useful in general.

Aside from unversioned pythons in the #! line, "#!/usr/bin/env" has also been made out to be a problem, so there is an opportunity here to kill two birds with one stone. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Henne Vogelsang

15 Nov 15 Nov

13:30

...

Thinking about what we need for python3 conversion long-term, a few issues have come to mind that currently aren't addressed well in the macros or previous discussions:

1. -> 4.

Hey, On 14.11.2017 19:29, Todd Rme wrote: 5. packages that include files outside of sitelib/arch. This fails during verification because of the file conflicts. Henne -- Henne Vogelsang http://www.opensuse.org Everybody has a plan, until they get hit. - Mike Tyson -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Todd Rme

15:33

On Wed, Nov 15, 2017 at 8:30 AM, Henne Vogelsang  wrote:
> Hey,
>
> On 14.11.2017 19:29, Todd Rme wrote:
>
>> Thinking about what we need for python3 conversion long-term, a few
>> issues have come to mind that currently aren't addressed well in the
>> macros or previous discussions:
>>
>> 1. -> 4.
> 5. packages that include files outside of sitelib/arch. This fails
> during verification because of the file conflicts.
>
> Henne

Those are usually python version-independent.  That is the idea of
having a "python-foo-data" package convention for such files.
-- 
To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org
To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Henne Vogelsang

15:45

Hey,

On 15.11.2017 16:33, Todd Rme wrote:
> On Wed, Nov 15, 2017 at 8:30 AM, Henne Vogelsang  wrote:
>> On 14.11.2017 19:29, Todd Rme wrote:
>>
>>> Thinking about what we need for python3 conversion long-term, a few
>>> issues have come to mind that currently aren't addressed well in the
>>> macros or previous discussions:
>>>
>>> 1. -> 4.
>> 5. packages that include files outside of sitelib/arch. This fails
>> during verification because of the file conflicts.
> 
> Those are usually python version-independent. That is the idea of
> having a "python-foo-data" package convention for such files.

And who decides which python*-foo package you use? :-)

Henne

-- 
Henne Vogelsang
http://www.opensuse.org
Everybody has a plan, until they get hit.
	- Mike Tyson
-- 
To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org
To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

Todd Rme

15:51

On Wed, Nov 15, 2017 at 10:45 AM, Henne Vogelsang  wrote:
> Hey,
>
> On 15.11.2017 16:33, Todd Rme wrote:
>> On Wed, Nov 15, 2017 at 8:30 AM, Henne Vogelsang  wrote:
>>> On 14.11.2017 19:29, Todd Rme wrote:
>>>
>>>> Thinking about what we need for python3 conversion long-term, a few
>>>> issues have come to mind that currently aren't addressed well in the
>>>> macros or previous discussions:
>>>>
>>>> 1. -> 4.
>>> 5. packages that include files outside of sitelib/arch. This fails
>>> during verification because of the file conflicts.
>>
>> Those are usually python version-independent. That is the idea of
>> having a "python-foo-data" package convention for such files.
>
> And who decides which python*-foo package you use? :-)
>
> Henne

>From all the examples I have seen the "python*-foo" packages require
the "python-foo-data" package (the name varies). If the files are
version-independent it doesn't matter which one they are pulled from.

In cases I have seen where they are version-dependent, they can be
renamed to use "%{$python_bin_suffix}".  That is typically the case
when .desktop files are involved, but those need to use
"%{$python_bin_suffix}" anyway.

The only case I am aware of where neither of these approaches work is
with jupyter notebook extensions, but since we only ship the python3
version of the notebook executable, we also only ship the python3
version of the notebook extensions.

Do you have a specific situation where none of these approaches work?
-- 
To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org
To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

jan matejek

14:43

New subject: [opensuse-packaging] Re: New thoughts on python singlespec macros

hello, On 14.11.2017 19:29, Todd Rme wrote:

...

Regarding 1, there are currently two pain points. First and most obvious is the fact that the shebangs have to be edit manually. The second is that if we want versioned shebangs, we need to manually recompile the files to avoid mtime issues.

The second is sort of a "wrong point". Files that need shebangs are different from files that need bytecode. Scripts that you directly execute don't create pyc files. Of course, some upstreams put shebangs everywhere, and in rare cases you get a file that is both a module (gets a pyc when used by others) and an executable. But in general i wouldn't want to bend over backwards for this.

...

Ideally changing the shebangs could be handled automatically during %python_build,

It is for files declared as executables. Distutils / setuptools do that.

...

but there could still be a separate macro that could be called in cases where manual changes are necessary. Recompiling is needed in other situations as well, so having a macro to handle that would be useful in general.

yes, we need $flavor_compile. I've had some blocking issues with it, i don't recall what they were... Also yes, we need the shebang macros. I'll just, uh, go ahead and put them in python-rpm-macros.

...

Regarding 3, there are again two main pain points. The first is that there is no macro I am aware of to reliable determine if a given python version is being built. %have_python2 and %have_python3 no longer work reliably AFAIK.

Never worked in the first place ;) But yes. I've been setting %bcond_without python2, then wrapping python2 parts in %if %{with python2}, same for other flavors if necessary. This works even for buildrequires. This is a possibly good convention. I'll look into if we can do the %with part without explicitly setting the bcond in every package.

...

python2 builds are disabled. Ideally I would like to see "%{python2_module foo}" and "%{python3_module foo}" that will only pull in that dependency if that version of python is being used. This also has the advantage of not needing to care about backwards-compatibility issues of "python-foo" vs. "python2-foo" names, which is handled inconsistently right now. If that is not feasible, just having a reliable check would be an improvement.

The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of %python2_module. We'd need to have something like %python2_buildrequires, but that sounds too specific and impractical. Having a reliable guard conditions, like the %with thing, is probably a better way.

...

Regarding 4, I don't really understand what "%python_expand" is doing behind the scenes, and whether there is a simple workaround or whether the macro needs to do something different internally.

I suspect this might be better fixed by a patch to pytest.... (setuptools create a "build" directory and all python_expanding macros move this directory around to "_build.$flavor", so that python is not reusing pycs from the wrong version. This should not be necessary in theory, but in practice it is. pytest knows to ignore "build" directory, but doesn't know about "_build". Probably easiest to teach pytest about its existence.)

Todd Rme

15:45

New subject: [opensuse-packaging] Re: New thoughts on python singlespec macros

On Wed, Nov 15, 2017 at 9:43 AM, jan matejek wrote:

...

hello,

On 14.11.2017 19:29, Todd Rme wrote:

...
Regarding 1, there are currently two pain points. First and most obvious is the fact that the shebangs have to be edit manually. The second is that if we want versioned shebangs, we need to manually recompile the files to avoid mtime issues.

The second is sort of a "wrong point". Files that need shebangs are different from files that need bytecode. Scripts that you directly execute don't create pyc files. Of course, some upstreams put shebangs everywhere, and in rare cases you get a file that is both a module (gets a pyc when used by others) and an executable.

But in general i wouldn't want to bend over backwards for this.

It is pretty common for tests.

...

...
python2 builds are disabled. Ideally I would like to see "%{python2_module foo}" and "%{python3_module foo}" that will only pull in that dependency if that version of python is being used. This also has the advantage of not needing to care about backwards-compatibility issues of "python-foo" vs. "python2-foo" names, which is handled inconsistently right now. If that is not feasible, just having a reliable check would be an improvement.

The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of %python2_module. We'd need to have something like %python2_buildrequires, but that sounds too specific and impractical.

Would it be possible to replace disabled requires with some dummy package that is a buildrequires by default anyway?

...

...
Regarding 4, I don't really understand what "%python_expand" is doing behind the scenes, and whether there is a simple workaround or whether the macro needs to do something different internally.

I suspect this might be better fixed by a patch to pytest....

(setuptools create a "build" directory and all python_expanding macros move this directory around to "_build.$flavor", so that python is not reusing pycs from the wrong version. This should not be necessary in theory, but in practice it is. pytest knows to ignore "build" directory, but doesn't know about "_build". Probably easiest to teach pytest about its existence.)

Can the macros move "_build.$flavor" to "build" during the corresponding part of "python_expand"? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

jan matejek

17:04

New subject: [opensuse-packaging] Re: New thoughts on python singlespec macros

On 15.11.2017 16:45, Todd Rme wrote:

...

On Wed, Nov 15, 2017 at 9:43 AM, jan matejek wrote:

...
...
python2 builds are disabled. Ideally I would like to see "%{python2_module foo}" and "%{python3_module foo}" that will only pull in that dependency if that version of python is being used. This also has the advantage of not needing to care about backwards-compatibility issues of "python-foo" vs. "python2-foo" names, which is handled inconsistently right now. If that is not feasible, just having a reliable check would be an improvement.

The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of %python2_module. We'd need to have something like %python2_buildrequires, but that sounds too specific and impractical.

Would it be possible to replace disabled requires with some dummy package that is a buildrequires by default anyway?

python-rpm-macros for instance? :) That's certainly possible, although rather kludgey. It would solve the inconsistency. OTOH another possible inconsistency is when python2 version requires something that doesn't start with "python-". That's why I like guard conditions better.

...

Can the macros move "_build.$flavor" to "build" during the corresponding part of "python_expand"?

That is precisely what happens. The problems you see exist because the *other* versions are lying around. An alternate solution would be to hide the directories better. I'm not sure where though. m.

Todd Rme

16 Nov 16 Nov

15:49

New subject: [opensuse-packaging] Re: New thoughts on python singlespec macros

On Wed, Nov 15, 2017 at 12:04 PM, jan matejek wrote:

...

On 15.11.2017 16:45, Todd Rme wrote:

...
On Wed, Nov 15, 2017 at 9:43 AM, jan matejek wrote:

...
...
python2 builds are disabled. Ideally I would like to see "%{python2_module foo}" and "%{python3_module foo}" that will only pull in that dependency if that version of python is being used. This also has the advantage of not needing to care about backwards-compatibility issues of "python-foo" vs. "python2-foo" names, which is handled inconsistently right now. If that is not feasible, just having a reliable check would be an improvement.

The issue with "BuildRequires: %python2_module" is that you can't put empty string in place of %python2_module. We'd need to have something like %python2_buildrequires, but that sounds too specific and impractical.

Would it be possible to replace disabled requires with some dummy package that is a buildrequires by default anyway?

python-rpm-macros for instance? :) That's certainly possible, although rather kludgey. It would solve the inconsistency. OTOH another possible inconsistency is when python2 version requires something that doesn't start with "python-". That's why I like guard conditions better.

Yes, the guard conditions would still be needed. The reason I like the approach is because it fits well with the existing pattern, where "%python_foo" is the multi-python version, "%python2_foo" is the python2 version, and "%python3_foo". There are only a couple cases where this pattern doesn't hold.

...

...
Can the macros move "_build.$flavor" to "build" during the corresponding part of "python_expand"?

That is precisely what happens. The problems you see exist because the *other* versions are lying around. An alternate solution would be to hide the directories better. I'm not sure where though.

What about moving it into buildroot? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

2355

Age (days ago)

2357

Last active (days ago)

List overview

Download

9 comments

4 participants

participants (4)

Henne Vogelsang
Jan Engelhardt
jan matejek
Todd Rme

[opensuse-packaging] New thoughts on python singlespec macros

tags

participants (4)