[opensuse-packaging] Some more python singlespec questions/suggestions
A few questions and ideas have popped up for me while doing the singlespec conversion: First, is there a way to use the macros without actually setting up the multiple packages? Particularly I need to use %{python_module foo}, but %python_expand and %python_exec would also be useful. The main use-case is for packages where the unit tests have been moved into a separate -doc spec file to avoid dependency cycles. These packages need to import both python2 and python3 dependencies to for the unit tests, and ideally would use %python_expand or %python_exec to run the tests. But the documentation itself would only be built once, so we don't need multiple rpms. I tried %_python_macro_init but it didn't work. Second, is there a single-spec version of %py(3)_compile? Ideally it would be nice to have %python2_compile and %python3_compile that could be used with %python_expand, as well as a macro that automatically run them both on %{buildroot}%{python(2/3)_sitelib} and %{buildroot}%{python(2/3)_sitearch} (whichever exists). Third, is there a macros to get just the current python major version number? So "2" for python2 and "3" for python3? So projects, such as the jupyter suite, use a "foo2-bar"/"foo3-bar" pattern that we need to support, so having a macro to get the major number is pretty critical especially since IPython is dropping python2 support soon so we need to be able to adjust the build targets easily. Third, the wiki recommends defining %oldpython when you need a specific python-foo dependency. Considering how often the backports are used these days in python2 packages, might this be a macro that would be good to include so people don't have to manually define it? Fourth, I notice the python2-foo versions of the packages (for Tumbleweed) provide python-foo, but the python-foo versions (for older releases) do not provide python2-foo. For maximum simplicity and compatibility, might it be good to have the python-foo versions also provide python2-foo? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10.4.2017 17:09, Todd Rme wrote:
A few questions and ideas have popped up for me while doing the singlespec conversion:
First, is there a way to use the macros without actually setting up the multiple packages? Particularly I need to use %{python_module foo}, but %python_expand and %python_exec would also be useful. The main use-case is for packages where the unit tests have been moved into a separate -doc spec file to avoid dependency cycles. These packages need to import both python2 and python3 dependencies to for the unit tests, and ideally would use %python_expand or %python_exec to run the tests. But the documentation itself would only be built once, so we don't need multiple rpms. I tried %_python_macro_init but it didn't work.
AFAIK the macros you list work fine by themselves. (of course, you still need to require python-rpm-macros; and if you are using %python_module in BuildRequires, you still need to shim the definition) If you're seeing failures, show me :) The only macros whose presence depend on each other are %python_subpackages and %python_files. Either will cause an error if the other is not used.
Second, is there a single-spec version of %py(3)_compile? Ideally it would be nice to have %python2_compile and %python3_compile that could be used with %python_expand, as well as a macro that automatically run them both on %{buildroot}%{python(2/3)_sitelib} and %{buildroot}%{python(2/3)_sitearch} (whichever exists).
Not as of now; so far the singlespec macros were geared towards setuptools-based installations where you don't use %py_compile. We can add it.
Third, is there a macros to get just the current python major version number? So "2" for python2 and "3" for python3? So projects, such as the jupyter suite, use a "foo2-bar"/"foo3-bar" pattern that we need to support, so having a macro to get the major number is pretty critical especially since IPython is dropping python2 support soon so we need to be able to adjust the build targets easily.
what will happen to such packages when they encounter pypy3? In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on "major version of python", as opposed to "is this python2 or something newer", does not look like a good practice. (and if it's limited to one package, you can easily get, e.g. from %python_version_nodots) OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?
Third, the wiki recommends defining %oldpython when you need a specific python-foo dependency. Considering how often the backports are used these days in python2 packages, might this be a macro that would be good to include so people don't have to manually define it?
If a backport doesn't provide python2-name, it's a bug and should be fixed. As for adding %oldpython to the macro set for other cases (obsoletes/provides usually), maybe? (Or maybe find a different spelling; %oldpython is good because it is easy to fit it into any spec file, but it is a hack. Something like "Provides: %{literally python-foo}" would be nicer)
Fourth, I notice the python2-foo versions of the packages (for Tumbleweed) provide python-foo, but the python-foo versions (for older releases) do not provide python2-foo. For maximum simplicity and compatibility, might it be good to have the python-foo versions also provide python2-foo?
Not as easy as it looks, unfortunately .e( I can add auto-provides to the main package, but I can't touch subpackages. That way the "maximum simplicity and compatibility" is a half-promise. However it seems that practicality does beat purity here, as most packages don't have subpackages anyway, so I guess i'll put it on my TODO list. regards m.
On Wed, Apr 12, 2017 at 8:39 AM, jan matejek <jmatejek@suse.com> wrote:
On 10.4.2017 17:09, Todd Rme wrote:
A few questions and ideas have popped up for me while doing the singlespec conversion:
First, is there a way to use the macros without actually setting up the multiple packages? Particularly I need to use %{python_module foo}, but %python_expand and %python_exec would also be useful. The main use-case is for packages where the unit tests have been moved into a separate -doc spec file to avoid dependency cycles. These packages need to import both python2 and python3 dependencies to for the unit tests, and ideally would use %python_expand or %python_exec to run the tests. But the documentation itself would only be built once, so we don't need multiple rpms. I tried %_python_macro_init but it didn't work.
AFAIK the macros you list work fine by themselves. (of course, you still need to require python-rpm-macros; and if you are using %python_module in BuildRequires, you still need to shim the definition) If you're seeing failures, show me :)
Try here: https://build.opensuse.org/package/show/home:TheBlackCat:branches:devel:lang... I get errors like this outside Tumbleweed: "nothing provides %{python_module, nothing provides testtools = %{version}}"
Second, is there a single-spec version of %py(3)_compile? Ideally it would be nice to have %python2_compile and %python3_compile that could be used with %python_expand, as well as a macro that automatically run them both on %{buildroot}%{python(2/3)_sitelib} and %{buildroot}%{python(2/3)_sitearch} (whichever exists).
Not as of now; so far the singlespec macros were geared towards setuptools-based installations where you don't use %py_compile. We can add it.
Even in setuptools, you sometimes need to run %py_compile due to, for example, "inconsistent mtime" rpmlint warnings. But more and more packages are shipping wheels only, so having support for %py_compile, and less importantly "pip install", is becoming important.
Third, is there a macros to get just the current python major version number? So "2" for python2 and "3" for python3? So projects, such as the jupyter suite, use a "foo2-bar"/"foo3-bar" pattern that we need to support, so having a macro to get the major number is pretty critical especially since IPython is dropping python2 support soon so we need to be able to adjust the build targets easily.
what will happen to such packages when they encounter pypy3?
No idea, that is up to the upstream project to decide.
In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on "major version of python", as opposed to "is this python2 or something newer", does not look like a good practice. (and if it's limited to one package, you can easily get, e.g. from %python_version_nodots) OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?
I agree with you, but this is an upstream decision.
Third, the wiki recommends defining %oldpython when you need a specific python-foo dependency. Considering how often the backports are used these days in python2 packages, might this be a macro that would be good to include so people don't have to manually define it?
If a backport doesn't provide python2-name, it's a bug and should be fixed.
As for adding %oldpython to the macro set for other cases (obsoletes/provides usually), maybe? (Or maybe find a different spelling; %oldpython is good because it is easy to fit it into any spec file, but it is a hack. Something like "Provides: %{literally python-foo}" would be nicer)
I think that whatever is recommended on the wiki should be supported out-of-the-box. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 12.4.2017 17:01, Todd Rme wrote:
Try here: https://build.opensuse.org/package/show/home:TheBlackCat:branches:devel:lang...
I get errors like this outside Tumbleweed: "nothing provides %{python_module, nothing provides testtools = %{version}}"
You're missing this line: %{?!python_module:%define python_module() python-%{**} python3-%{**}} Because it's used in BuildRequires, it must be either in prjconf (in Factory) or in the spec file.
Even in setuptools, you sometimes need to run %py_compile due to, for example, "inconsistent mtime" rpmlint warnings.
If you get "inconsistent mtime" from setup.py install, it's either the installer's or your packaging bug. Usually you're running tests that touch things they shouldn't. (But yes, %py_compiling it again is often easier than fixing it properly.)
But more and more packages are shipping wheels only, so having support for %py_compile, and less importantly "pip install", is becoming important.
wouldn't "pip install" take care of it too?
In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on "major version of python", as opposed to "is this python2 or something newer", does not look like a good practice. (and if it's limited to one package, you can easily get, e.g. from %python_version_nodots) OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?
I agree with you, but this is an upstream decision.
And also easily resolvable inside the spec file. The question here is whether this should be directly supported by a singlespec macro. One more point against it that it's not general: for instance, in LXML, you run "make test" for python2 tests and "make test3" for python3. Here you would need a variable that is "" for python2 and "3" for python3. In other cases, you need "2" for python2 and "3" for python3. Next time it might be "2" for python2, "pp" for PyPy and "" for python3? ...and as long as you're if'ing it anyway, you can do this: if [ %{$python_version_nodots} -ge 30 ]; then mysuffix=3 else mysuffix=2 fi or even: mysuffix=`expr %{$python_version_nodots} / 10` m.
On Thu, Apr 13, 2017 at 9:06 AM, jan matejek <jmatejek@suse.com> wrote:
On 12.4.2017 17:01, Todd Rme wrote:
Try here: https://build.opensuse.org/package/show/home:TheBlackCat:branches:devel:lang...
I get errors like this outside Tumbleweed: "nothing provides %{python_module, nothing provides testtools = %{version}}"
You're missing this line: %{?!python_module:%define python_module() python-%{**} python3-%{**}} Because it's used in BuildRequires, it must be either in prjconf (in Factory) or in the spec file.
Next time I will wait until after my 101 degree fever is over before asking ;) Sorry.
Even in setuptools, you sometimes need to run %py_compile due to, for example, "inconsistent mtime" rpmlint warnings.
If you get "inconsistent mtime" from setup.py install, it's either the installer's or your packaging bug. Usually you're running tests that touch things they shouldn't.
(But yes, %py_compiling it again is often easier than fixing it properly.)
I have seen it in other situations as well, and it isn't necessarily easy to track down.
But more and more packages are shipping wheels only, so having support for %py_compile, and less importantly "pip install", is becoming important.
wouldn't "pip install" take care of it too?
Once pypi starts providing consistent URLs for wheels we can do that when they are provided. But even then there are packages with their own home-made build systems that still need to be byte-compiled manually.
In any case I'm not convinced that this should be a part of the "singlespec API". Basing decision on "major version of python", as opposed to "is this python2 or something newer", does not look like a good practice. (and if it's limited to one package, you can easily get, e.g. from %python_version_nodots) OTOH, maybe this distinction will be useful again with Python 4 somewhere down the road, so perhaps?
I agree with you, but this is an upstream decision.
And also easily resolvable inside the spec file. The question here is whether this should be directly supported by a singlespec macro.
One more point against it that it's not general: for instance, in LXML, you run "make test" for python2 tests and "make test3" for python3. Here you would need a variable that is "" for python2 and "3" for python3. In other cases, you need "2" for python2 and "3" for python3. Next time it might be "2" for python2, "pp" for PyPy and "" for python3?
...and as long as you're if'ing it anyway, you can do this: if [ %{$python_version_nodots} -ge 30 ]; then mysuffix=3 else mysuffix=2 fi
or even: mysuffix=`expr %{$python_version_nodots} / 10`
I agree that it isn't a desirable situation, but if we want our users to be able to use upstream documentation then we need to support upstream naming conventions. We can create links with our own naming patterns, but if upstream provides executable with certain names we need to have those executables as well. And I think this is a common enough upstream naming pattern that there should be support for it. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (2)
-
jan matejek -
Todd Rme