[opensuse-factory] Python for after 12.2
Considering all of the talk about improving openSUSE quality, and with 12.2 branched, I would like to discuss somewhat how to move forward with Python packaging in openSUSE. There are three big things happening right now with Python for openSUSE. First, and most prominently, is Python 3. Python 3 has been supported on openSUSE since at the latest 11.4. However, Python 2.x still remains the default, but it will no longer be getting upstream updates. At the present time I don't see that being quite ready to change, since some critical packages (like django, wxpython, and matplotlib) are still Python 2-only. Unless all of those, and probably some others, support Python 3 by the next openSUSE release, I don't think switching will work. We probably need a list of packages that need python 3 support before we switch to python 3 by default. That doesn't mean Python 3 doesn't matter. I and several others have been working to get Python 3 support for most critical Python packages available for openSUSE 12.2. I think we have largely succeeded. For the next openSUSE release, I think we should have a policy that, if a new python package is submitted that has upstream Python 3 support, it must be submitted with both Python 2 and Python 3 versions of the package. There might even be a policy that any packaging of a new version of a python package that supports Python 3 should also have the Python 3 version packaged, but that may be going too far. Further, I think we should have an embargo on any new Python packages that do not have Python 3 versions and don't appear to have been updated within the last couple of years, since it is unlikely such packages would ever get a Python 3 version. Of course exceptions can be made on a case-by-case basis, for instance if the package is a dependency of an important Python package. In the end I don't think we really want unmaintained software to begin with, but this is all the more serious when it is unmaintained software that depends on deprecated, unmaintained software. There is an issue that some Python packages install non-versions software, such as in /usr/bin. This makes installing Python 2 and 3 versions in parallel difficult. For the next openSUSE release I intend to implement update-alternatives for these files, so users can easily switch between running Python 2 and Python 3 versions. This also solves some issues where Python 3 versions of packages depend on the Python 2 versions. I know there is a Python Package Index Reader that scans pypi for updated packages. I don't know how it works myself, but is there a way to also scan to see if the package supports Python 3? They should have one or more of the following categories: "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.0", "Programming Language :: Python :: 3.1", "Programming Language :: Python :: 3.2", or "Programming Language :: Python :: 3.3". Sending regular email alerts about such packages that do not yet have Python 3 versions would be nice. Also, currently most Python 3 packages that use the same tarballs as the Python 2 versions have the Python 3 version link to the Python 2 version, which has Python 2 and Python 3 spec files. I would like to turn this around for the next openSUSE release, so the Python 2 version links to the Python 3 version and the Python 3 version is the official factory devel package. The point of all of this is to ease the transition to having Python 3 as the default easier, which I expect will likely will happen wit2 release after 12.2, judging by the rate of progress of upstream Python 3 porting. Basically most or all of the tasks that are needed before we would be ready to switch the default Python version should be done for the next openSUSE release. There are 3 other issues, which are more openSUSE-specific. First is the new python autodeps system, which should allow the requires of Python 2 and 3 packages to be automatically detected. This will make packaging python easier and installing python packages more reliable. It still needs testing. I suggest a tiered approach: first, which I am doing now, I am testing it against all the python packages in devel:languages:python3. If that seems to work, I will then switch to implementing it directly in devel:languages:python3. If that works, I also implement it in devel:languages:python. If that works, it can be implemented globally (at least for Factory build targets). The second is --record-rpm. This allows python to automatically list files that are then used by rpm file lists. As far as I understand this is an openSUSE-specific hack whose use is now discouraged. It also leads to some problems, especially if the upstream package name changed. If this understanding is correct, I think this should be officially deprecated, with a plan to remove it for 2 releases after 12.2. For the release after 12.2, I think it should go through a few stages. First, add an rpmlint warning about the use of this build option. After a couple months, upgrade the badness of this warning so it causes an automatic package failure. A few months after that, change the warning to an error. No packages in Factory should use the option. Finally, there is the use of the common %{python_sitelib}/* and %{python_sitearch}/* items in the rpm files list. These have been used a lot because they make packaging easier. However, they have the disadvantage, in my opinion, that there can be (and are) major changes in the upstream package, even complete name changes, that go undetected because these commands essentially match every possible python file. I think we should discourage this in the wiki, reject new packages that use it, and add an rpmlint warning for it, with the goal of removing this entirely from the next openSUSE release. So does anyone have opinions on these proposals? I know this is a big wall of text, so I can break it up into individual messages with different issues you think that would simplify reading or discussion. -Todd -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 06/25/2012 11:12 AM, todd rme wrote:
Considering all of the talk about improving openSUSE quality, and with 12.2 branched, I would like to discuss somewhat how to move forward with Python packaging in openSUSE.
There are three big things happening right now with Python for openSUSE.
[snip]
There are 3 other issues, which are more openSUSE-specific.
First is the new python autodeps system, which should allow the requires of Python 2 and 3 packages to be automatically detected. This will make packaging python easier and installing python packages more reliable. It still needs testing. I suggest a tiered approach: first, which I am doing now, I am testing it against all the python packages in devel:languages:python3. If that seems to work, I will then switch to implementing it directly in devel:languages:python3. If that works, I also implement it in devel:languages:python. If that works, it can be implemented globally (at least for Factory build targets). I'm glad it's picked up, but be careful about the "requires ... automatically detected". Actually, only provides are automatic and I strongly prefer that requires remain generated by scripts such as py2pack and are explicitly stated in the spec file (so that you can comment them out in case of ...). This is the same as our Perl friends to BTW. :-)
The second is --record-rpm. This allows python to automatically list files that are then used by rpm file lists. As far as I understand this is an openSUSE-specific hack whose use is now discouraged. It also leads to some problems, especially if the upstream package name changed. If this understanding is correct, I think this should be officially deprecated, with a plan to remove it for 2 releases after 12.2. For the release after 12.2, I think it should go through a few stages. First, add an rpmlint warning about the use of this build option. After a couple months, upgrade the badness of this warning so it causes an automatic package failure. A few months after that, change the warning to an error. No packages in Factory should use the option.
I think we shouldn't have much packages left that still use '--record-rpm'. Also, at least I remove that whenever I touch a package. Similarly, there is really no need for '--skip-build' when you do 'python setup.py install' in the %install phase when you already have a "python setup.py build" in %build. Setuptools is clever enough to not re-generate byte-code. Lastly, "-O1" doesn't do anything useful either.
Finally, there is the use of the common %{python_sitelib}/* and %{python_sitearch}/* items in the rpm files list. These have been used a lot because they make packaging easier. However, they have the disadvantage, in my opinion, that there can be (and are) major changes in the upstream package, even complete name changes, that go undetected because these commands essentially match every possible python file. I think we should discourage this in the wiki, reject new packages that use it, and add an rpmlint warning for it, with the goal of removing this entirely from the next openSUSE release.
Yes, actually they have been used to ease cross-distro building (due to inconsistent *.egg-info directory names), which unfortunately never took off in d:l:p. I am fine with doing it either way but the exact file list isn't always easy to generate in scripts (in contrast to %{python_sitelib}/*).
So does anyone have opinions on these proposals? I know this is a big wall of text, so I can break it up into individual messages with different issues you think that would simplify reading or discussion. %py_requires is another one, it is still needed for SLE_11_SP2 (rpm is too old). But as I was basically the only one fixing build errors in d:l:p for quite a while and most of them where (and still are) for SLE, I think it would be ok to continue ignoring that... -- Viele Grüße, Sascha Peilicke
On Mon, Jun 25, 2012 at 1:55 PM, Sascha Peilicke <saschpe@gmx.de> wrote:
On 06/25/2012 11:12 AM, todd rme wrote:
Considering all of the talk about improving openSUSE quality, and with 12.2 branched, I would like to discuss somewhat how to move forward with Python packaging in openSUSE.
There are three big things happening right now with Python for openSUSE.
[snip] The second is --record-rpm. This allows python to automatically list files that are then used by rpm file lists. As far as I understand this is an openSUSE-specific hack whose use is now discouraged. It also leads to some problems, especially if the upstream package name changed. If this understanding is correct, I think this should be officially deprecated, with a plan to remove it for 2 releases after 12.2. For the release after 12.2, I think it should go through a few stages. First, add an rpmlint warning about the use of this build option. After a couple months, upgrade the badness of this warning so it causes an automatic package failure. A few months after that, change the warning to an error. No packages in Factory should use the option. I think we shouldn't have much packages left that still use '--record-rpm'. Also, at least I remove that whenever I touch a package.
Similarly, there is really no need for '--skip-build' when you do 'python setup.py install' in the %install phase when you already have a "python setup.py build" in %build. Setuptools is clever enough to not re-generate byte-code.
Lastly, "-O1" doesn't do anything useful either.
So add rpmlint warnings for all of these?
Finally, there is the use of the common %{python_sitelib}/* and %{python_sitearch}/* items in the rpm files list. These have been used a lot because they make packaging easier. However, they have the disadvantage, in my opinion, that there can be (and are) major changes in the upstream package, even complete name changes, that go undetected because these commands essentially match every possible python file. I think we should discourage this in the wiki, reject new packages that use it, and add an rpmlint warning for it, with the goal of removing this entirely from the next openSUSE release. Yes, actually they have been used to ease cross-distro building (due to inconsistent *.egg-info directory names), which unfortunately never took off in d:l:p.
I am fine with doing it either way but the exact file list isn't always easy to generate in scripts (in contrast to %{python_sitelib}/*).
Which in some ways may be a good thing. If the person who submitted the package can't handle making a %files list, the chances of them being able to fix the package later is slim.
So does anyone have opinions on these proposals? I know this is a big wall of text, so I can break it up into individual messages with different issues you think that would simplify reading or discussion. %py_requires is another one, it is still needed for SLE_11_SP2 (rpm is too old). But as I was basically the only one fixing build errors in d:l:p for quite a while and most of them where (and still are) for SLE, I think it would be ok to continue ignoring that...
There shouldn't be many simple fixes like this left for SLE 11 at least in d:l:p, I went through and fixed most such problems a while back. -Todd -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 06/25/2012 11:12 AM, todd rme wrote:
Considering all of the talk about improving openSUSE quality, and with 12.2 branched, I would like to discuss somewhat how to move forward with Python packaging in openSUSE.
Hi, some other things I saw recently related to Python3 packaging: BuildRequires: python3 BuildRequires: python3-devel You need only one, not both. Generally, we recommend using the latter (as it should always work). For Python-only packages (i.e. no arch-dependant code contained) the former should suffice but it's more consistent to always use the -devel package. Requires: python3 >= %{py3_ver} Is only needed on SLE-11, check "rpm -q --requires $PKG" (or "rpm -qp --requires $PKG"). You get those automatically, i.e. like this: % rpm -qp --requires /var/tmp/build-root/openSUSE_Factory-x86_64/home/abuild/rpmbuild/RPMS/noarch/python3-pep8-1.3.1-0.noarch.rpm /usr/bin/python3 python(abi) = 3.2 ... As you recommended, we should probably update http://en.opensuse.org/openSUSE:Packaging_Python. Maybe a special paragraph should deal with SLE-11 compatibility issues, i.e. mentioning this beauty: %if 0%{?suse_version} <= 1140 %{!?python3_sitelib: %global python3_sitelib %(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")} %{!?py3_ver: %global py3_ver %(python3 -c "import sys; version=str(sys.version_info[0]) + '.' + str(sys.version_info[1]); print(version)" 2>/dev/null || echo PYTHON-NOT-FOUND)} Requires: python3 >= %{py3_ver} %endif Or we just decide to not care at all about Python-3 and SLE-11 :-) Besides, thank you for actually getting things done! -- Viele Grüße, Sascha Peilicke
On Tue, Jun 26, 2012 at 11:27 AM, Sascha Peilicke <saschpe@gmx.de> wrote:
On 06/25/2012 11:12 AM, todd rme wrote:
Considering all of the talk about improving openSUSE quality, and with 12.2 branched, I would like to discuss somewhat how to move forward with Python packaging in openSUSE.
Hi,
some other things I saw recently related to Python3 packaging:
BuildRequires: python3 BuildRequires: python3-devel
You need only one, not both. Generally, we recommend using the latter (as it should always work). For Python-only packages (i.e. no arch-dependant code contained) the former should suffice but it's more consistent to always use the -devel package.
This is not really the case. python3-devel depends on python3-base, not python3. Many packages, however, require the full python3 to build, because it contains additional functionality these packages depend on (often md5 or sha hash creation).
Requires: python3 >= %{py3_ver}
Is only needed on SLE-11, check "rpm -q --requires $PKG" (or "rpm -qp --requires $PKG"). You get those automatically, i.e. like this:
% rpm -qp --requires /var/tmp/build-root/openSUSE_Factory-x86_64/home/abuild/rpmbuild/RPMS/noarch/python3-pep8-1.3.1-0.noarch.rpm /usr/bin/python3 python(abi) = 3.2 ...
SLE-11 doesn't have python3 at all, that it why building is disabled for this target for all python3 packages in d:l:p (and for the d:l:p3 project completely). Pretty much all of the workarounds you see in the python3 packages are for openSUSE 11.4, which had a lot of problems with its python3 implementation (more on that later). If it isn't required for openSUSE 11.4 it can be removed, I will have to check.
As you recommended, we should probably update http://en.opensuse.org/openSUSE:Packaging_Python. Maybe a special paragraph should deal with SLE-11 compatibility issues, i.e. mentioning this beauty:
%if 0%{?suse_version} <= 1140 %{!?python3_sitelib: %global python3_sitelib %(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")} %{!?py3_ver: %global py3_ver %(python3 -c "import sys; version=str(sys.version_info[0]) + '.' + str(sys.version_info[1]); print(version)" 2>/dev/null || echo PYTHON-NOT-FOUND)} Requires: python3 >= %{py3_ver} %endif
This is an issue with openSUSE 11.4, not SLE 11. There were no python3 rpm macros shipped with openSUSE 11.4, so they have to be manually specified or the packages won't build. openSUSE 11.4 also has some problems with directory ownership, with the 64bit version of python3 not owning %python3_sitelib, which means these directories have to be manually owned by every noarch python3 package. Once openSUSE 11.4 is EOL we can remove a ton of workarounds in python3 packages. -Todd -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tuesday 2012-06-26 12:18, todd rme wrote:
BuildRequires: python3 BuildRequires: python3-devel
You need only one, not both. Generally, we recommend using the latter (as it should always work). For Python-only packages (i.e. no arch-dependant code contained) the former should suffice but it's more consistent to always use the -devel package.
This is not really the case. python3-devel depends on python3-base, not python3. Many packages, however, require the full python3 to build, because it contains additional functionality these packages depend on (often md5 or sha hash creation).
In that case, should not it better depend on "python(md5) >= 3" rather than python3-base? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Jun 26, 2012 at 12:24 PM, Jan Engelhardt <jengelh@inai.de> wrote:
On Tuesday 2012-06-26 12:18, todd rme wrote:
BuildRequires: python3 BuildRequires: python3-devel
You need only one, not both. Generally, we recommend using the latter (as it should always work). For Python-only packages (i.e. no arch-dependant code contained) the former should suffice but it's more consistent to always use the -devel package.
This is not really the case. python3-devel depends on python3-base, not python3. Many packages, however, require the full python3 to build, because it contains additional functionality these packages depend on (often md5 or sha hash creation).
In that case, should not it better depend on "python(md5) >= 3" rather than python3-base?
Nothing seems to provide that. There are provides for the specific libraries missing from python3-base, but their names vary depending on the python3 version (like _hashlib.cpython-32mu.so). -Todd -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 06/26/2012 12:18 PM, todd rme wrote:
On Tue, Jun 26, 2012 at 11:27 AM, Sascha Peilicke <saschpe@gmx.de> wrote:
On 06/25/2012 11:12 AM, todd rme wrote:
Considering all of the talk about improving openSUSE quality, and with 12.2 branched, I would like to discuss somewhat how to move forward with Python packaging in openSUSE.
Hi,
some other things I saw recently related to Python3 packaging:
BuildRequires: python3 BuildRequires: python3-devel
You need only one, not both. Generally, we recommend using the latter (as it should always work). For Python-only packages (i.e. no arch-dependant code contained) the former should suffice but it's more consistent to always use the -devel package.
This is not really the case. python3-devel depends on python3-base, not python3. Many packages, however, require the full python3 to build, because it contains additional functionality these packages depend on (often md5 or sha hash creation).
Requires: python3 >= %{py3_ver}
Is only needed on SLE-11, check "rpm -q --requires $PKG" (or "rpm -qp --requires $PKG"). You get those automatically, i.e. like this:
% rpm -qp --requires /var/tmp/build-root/openSUSE_Factory-x86_64/home/abuild/rpmbuild/RPMS/noarch/python3-pep8-1.3.1-0.noarch.rpm /usr/bin/python3 python(abi) = 3.2 ...
SLE-11 doesn't have python3 at all, that it why building is disabled for this target for all python3 packages in d:l:p (and for the d:l:p3 project completely). Pretty much all of the workarounds you see in the python3 packages are for openSUSE 11.4, which had a lot of problems with its python3 implementation (more on that later). Uhm, you're right. I forgot about that one ;-)
If it isn't required for openSUSE 11.4 it can be removed, I will have to check.
As you recommended, we should probably update http://en.opensuse.org/openSUSE:Packaging_Python. Maybe a special paragraph should deal with SLE-11 compatibility issues, i.e. mentioning this beauty:
%if 0%{?suse_version} <= 1140 %{!?python3_sitelib: %global python3_sitelib %(python3 -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")} %{!?py3_ver: %global py3_ver %(python3 -c "import sys; version=str(sys.version_info[0]) + '.' + str(sys.version_info[1]); print(version)" 2>/dev/null || echo PYTHON-NOT-FOUND)} Requires: python3 >= %{py3_ver} %endif
This is an issue with openSUSE 11.4, not SLE 11. There were no python3 rpm macros shipped with openSUSE 11.4, so they have to be manually specified or the packages won't build.
openSUSE 11.4 also has some problems with directory ownership, with the 64bit version of python3 not owning %python3_sitelib, which means these directories have to be manually owned by every noarch python3 package.
Once openSUSE 11.4 is EOL we can remove a ton of workarounds in python3 packages.
-Todd
-- Viele Grüße, Sascha Peilicke
Am 25.06.2012 11:12, schrieb todd rme:
[...] We probably need a list of packages that need python 3 support before we switch to python 3 by default.
Upstream QEMU for one is still lacking (build) support for Python 3, affecting at least the qemu and kvm packages. Andreas -- SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (4)
-
Andreas Färber -
Jan Engelhardt -
Sascha Peilicke -
todd rme