[opensuse-factory] plan for 12.3 - multiple Python runtimes
fellow openSUSErs, with the advent of Python 3 and still almost unnoticeable decline of Python 2, i have come to a realization that the way we separate them into "python" and "python3" is less than ideal. The granularity of this can be often too coarse (you can't run two apps requiring two different python2 versions, unless they are specifically packaged and bring their own python runtime package), and too fine at the same time (there are already packages that don't actually care whether they run in python 2 or 3) In the spirit of fixing what ain't broken, I humbly propose a grand plan: let's turn "python" into "python2.7", "python3" into "python3.3", and tinker with Provides until it all works just fine. Then, as a last step, symlink /usr/bin/python into /etc/alternatives and let the users pick whichever runtime they like better. This would probably go better if broken into several phases. My current idea is this: Phase 1: - rename "python" to "python2.7", providing "python = 2.7" - rename "python3" to "python3.3", providing "python3 = 3.3" - install the update-alternatives symlink Impact: - Unsure of what would happen on package upgrade. Would we need Obsoletes/Provides combo for this? Perhaps we'd have to create the metapackages outlined in the following phases at this point already. Phase 2: - unify RPM macros - python3 will have %py_(something) macros as well as %py3_(something) macros, the %py3_ versions being aliases to the %py_ versions - make python2.7 provide "python2 = 2.7" instead of "python = 2.7" - make a "python" meta-package requiring, at this time, python2 - each pythonX.Y package now only installs /usr/bin/pythonX.Y binary, python2 and python3 are alternatives for python2.Y and python3.Y respectively, and python is an alternative for either python2, python3, or a specific version. I don't know update-alternatives too well - is this even viable? Impact: - this should mark a clear upgrade path for each of the python variants and for the systemwide "default python" as well - at this phase, i would like to make it possible to build a package for any python (2 or 3 or 50) from the same specfile. i'm not sure that these outlined steps are enough, though. Originally I thought that every python would provide "python = X.Y" and the packages could select which "Requires: python" to pick through package or project meta. But that would result in a horrible breakage, and we need "python" as a separate package for upgrading in any case. We could use "python(abi)" symbol for this, or do the same just with "python-devel"? Any other options? Phase 3A: - when python 3.4 comes out, introduce a new package "python3.4", new meta-package "python3" requiring "python3.4", and corresponding alternatives Impact: - This is the other difficult one. What about lifetime of the pythonX.Y packages? What happens with old systems on perpetual-upgrade? Such as tumbleweed - will the old python runtimes rot there forever-and-ever, or can we force uninstall at some point? Perhaps "Obsoletes: python3 < 3.x" in the "python3" metapackage? Which versions do we support? The newest and the most popular, probably? I know we do something similar with gcc, how does it work there? Phase 3B: - steal and improve Debian's python-central (or however they call it these days) - designate /usr/lib/python-central or something similar for packages that don't want to depend on a specific python runtime version - introduce %post/%postun scriptlet macros for packages that would compile bytecode in /usr/lib/python-central for all known python runtimes on the system, and for pythons so that they compile every package for their versions. This is now actually fully supported by upstream, we can have as many different versions of bytecode for the -same- .py file as we want. Impact: - package can specify "Requires: python" and Just Work(tm) from now until eternity, or until it runs into a deprecated API. Or specify a range of pythons it wants to support - by the time we get here, I'm sure RPM will allow that ;e) With a limited ABI in python3, this can even work for arch-based packages across python minor versions. Phase 4: - when we decide to switch to python3 system-wide, make "python" require "python3" instead of "python2", change default alternative, and be done with it Impact: - breakage of packages still incompatible at that time. that will help us see what needs fixing and what needs to explicitly specify that it wants python2. Somewhere in this is lurking a possibility of supporting different python runtimes such as Jython, Cython, PyPy and whatever else crops up. That's neat, too. Thanks for your attention. Comments, questions? regards m. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Sep 10, 2012 at 07:53:37PM +0200, Jan Matejek wrote:
I personally like the gcc approach more - they have one gcc package, which just maintain symlinks to the correct gccx.y version To illustrate it more python2 Provides: python = %{version}-%{release} # so zypper in python will select this package Obsoletes: python = some-version Requires: python27 /usr/bin/python -> /usr/bin/python27 python3 Requires: python33 /usr/bin/python -> /usr/bin/python33 I am not sure if it's sane to have /usr/lib/python/site-packages -> /usr/lib/python3.3/site-packages or if the python-central is intended to do the same. BTW: there are new features pyc repository directories and abi version tagged so files - are those already included in your proposal? http://docs.python.org/py3k/whatsnew/3.2.html#pep-3147-pyc-repository-direct... http://docs.python.org/py3k/whatsnew/3.2.html#pep-3149-abi-version-tagged-so... Regards Michal Vyskocil [snip]
Dne 11.9.2012 12:09, Michal Vyskocil napsal(a):
This would mean a conflict between python2 and python3 though. i'd like to avoid that. It does make sense to make it this way for /usr/bin/python2 -> python2.7 and /usr/bin/python3 -> python3.3 - then python apps can use #!/usr/bin/pythonX and be reasonably confident that they are running on the intended python version. I realized another thing: having "python" as an alias for either python2 or python3 could break packages if the user switched from python2 to python3 systemwide. We would have to change most python2 packages to use #!/usr/bin/python2 shebang, instead of just python. But this wouldn't have to be done all at once - we would state that switching system python to python3 is possible but unsupported, and change the packages gradually.
i'm not sure /usr/lib/python was ever used for anything else than lazily written install scripts ;e) so i'd probably just drop it and see what breaks. python-central would probably live in /usr/share/pyshared, for consistency with Debian. Who knows, at that point we might be able to convince upstream and FHS that it is -the- place for python modules
yes, this is what allows us to implement the python-central approach without debian's symlink craziness
http://docs.python.org/py3k/whatsnew/3.2.html#pep-3149-abi-version-tagged-so...
And no, I don't think this is useful for us, at least not at this time. While it's easy to generate all necessary .pycs at install-time, we can't do that with .so files. And as long as different versions of the .so module are in a package at build-time, we can as well install them into their respective site-packages directories. (building such package would be fun, too)
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Sep 11, 2012 at 11:33 AM, Jan Matejek <jan.matejek@suse.cz> wrote:
It wouldn't break if the symlink was managed with alternatives. And, the shebangs can be handled by an rpm macro that does the substitution (and an rpmlint warning if it's not done). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le lundi 10 septembre 2012, à 19:53 +0200, Jan Matejek a écrit :
Thanks for your attention. Comments, questions?
Is there any recommendation from upstream? It sounds to me like /usr/bin/python being python3 instead of python2 will simply break tons of things in the next 2 or 3 years :/ I would guess that nearly all apps using python3 are actually looking for a python3 binary because of this; so such a change wouldn't be needed. I'm also not sure I like using update-alternatives here, as people will believe they can change it and it will just work while it's simply not the case. Cheers, Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, Sep 12, 2012 at 3:30 AM, Vincent Untz <vuntz@opensuse.org> wrote:
Now that you mention, that's one of the main problems with treating pythonX.X as "alternatives". Each python installation has its own "prefix", and all the libraries installed in 2.6 are invisible for 2.7. Even worse, they can't be shared properly, because 2.6 uses a different bytecode than 2.7. So... not interchangeable at all. And, requiring a library for python2.6 may not imply the need to require it for 2.7 or 3.x, so you can't expect to have the same set of libraries in all versions. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Dne 12.9.2012 17:26, Claudio Freire napsal(a):
True, but this is not a problem in Python 3.x because of PEP3147. Simply put, you can have many different .pyc bytecode files for a single .py source file. So it is feasible to make a "shared prefix" with libraries working on multiple python runtimes. (See Phase 3B of my proposal) It's reasonably easy to backport this feature into Python 2.7, which would actually eliminate big part of this problem. You are right that from our point of view different pythons can't be really considered "alternatives". At this point, it only helps in scenarios where users install third-party python apps or modules outside of our package management - and if the users do this, it's not unreasonable to expect them to manage the alternative symlinks by hand, without us pretending to support that. m. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Sep 13, 2012 at 11:49 AM, Jan Matejek <jan.matejek@novell.com> wrote:
Ok, so you can share the prefix for python2, but python3 still needs to be separate (it needs different sources). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Dne 13.9.2012 16:53, Claudio Freire napsal(a):
Ok, so you can share the prefix for python2, but python3 still needs to be separate (it needs different sources).
It is also possible to write source-compatible packages for python2 and python3. (Obviously, some projects do this, others don't.) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Sep 13, 2012 at 11:55 AM, Jan Matějek <jmatejek@suse.cz> wrote:
Nah, there are some things that are impossible to make "version agnostic". Source-compatible packages mean they're auto-translatable by 2to3. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Dne 13.9.2012 17:11, Claudio Freire napsal(a):
Nope, "source-compatible" actually means that the same source runs the same way in python2 and python3. Often through future imports and similar tricks. I guess some things are impossible to do this way, which is probably why not everyone is doing it ;e) m. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
I would rather have only one python version on openSUSE but fear this is not feasible yet. Could you add this to http://en.opensuse.org/openSUSE:Goals_12.3, please? Thanks, Andreas -- Andreas Jaeger aj@{suse.com,opensuse.org} Twitter/Identica: jaegerandi SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn,Jennifer Guild,Felix Imendörffer,HRB16746 (AG Nürnberg) GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126 -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Jan Matejek <jan.matejek@suse.cz> writes:
Which program we ship do actually need python 3 now or in the near future? -- Karl Eichwalder SUSE LINUX Products GmbH R&D / Documentation Maxfeldstraße 5 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 13 Sep 2012 13:52:57 Karl Eichwalder wrote:
if we apply this logic there is no need to ship anything new... all we have to do is to teach the users not to complain. nobody argues, at the moment, to remove python2, just make it easy for average joe... to change to python3 as default... something in the styles of gcc maybe... progress is coming we cannot oppose it... Alin -- Without Questions there are no Answers! ______________________________________________________________________ Alin Marin ELENA Advanced Molecular Simulation Research Laboratory School of Physics, University College Dublin ---- Ardionsamblú Móilíneach Saotharlann Taighde Scoil na Fisice, An Coláiste Ollscoile, Baile Átha Cliath ----------------------------------------------------------------------------------- http://alin.elenaworld.net ______________________________________________________________________ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Alin M Elena <alinm.elena@gmail.com> writes:
if we apply this logic there is no need to ship anything new... all we have to do is to teach the users not to complain.
No, there is a difference. If you think, openSUSE users are mostly (python) developers, then offering python3 now would surely be a good thing. Otherwise for most python developers it would not be that difficult to install python 3 to /usr/local. These versioned packaged as regular openSUSE packages are often a PITA. Usually there setup is somehow broken, and in the end it is not working as planned. They are causing much trouble on all ends. (Note, I'm not talking about version libraries!) Of course, if you think it is worth to spend the time on developing parallel installs of python 2 and 3, then go for it! -- Karl Eichwalder SUSE LINUX Products GmbH R&D / Documentation Maxfeldstraße 5 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Dne 13.9.2012 14:36, Karl Eichwalder napsal(a):
please note that we _already have_ Python 3 in Factory and it's part of the main repo since 11.4, i think. Several python3 modules are building in dev:lang:py, and there's even dev:lang:python3 project in OBS. Also, unless things change significantly, Python 2.7 will be the last release in the 2.x series, and it will receive bugfixes for the next five or so years, but no new features ever. Upstream has stated very clearly that they want everyone to start moving to Python 3, and they're doing whatever they can to make the transition easier for the developers. So i want to do my part and make it easier for developers on openSUSE to support python3 and make the transition smoother ;e) Hence the proposal. Parallel installs of Python 2 and 3 work just fine today, because "python3" is a very separate package that pretends not to have anything to do with Python whatsoever. This is nice because it is trivial to maintain, but it will cause problems _when_ (not if) we decide to make python3 the default and python2 the legacy solution. I realize that this might come around to bite us in the rear, but i believe the benefits of supporting multiple Python runtimes outweigh the downsides. regards m. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Jan Matejek <jan.matejek@novell.com> writes:
This sounds pretty good, and thus far everything is fine :)
Ok, I see. -- Karl Eichwalder SUSE LINUX Products GmbH R&D / Documentation Maxfeldstraße 5 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le jeudi 13 septembre 2012, à 13:52 +0200, Karl Eichwalder a écrit :
GNOME is planning to switch to python 3 for GNOME 3.8 (which might or might not make it in 12.3, depending on the schedule we'll choose). Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Vincent Untz <vuntz@opensuse.org> writes:
GNOME is planning to switch to python 3 for GNOME 3.8 (which might or might not make it in 12.3, depending on the schedule we'll choose).
Ok, this could be a convincing argument for making python 3 available better sooner than later. -- Karl Eichwalder SUSE LINUX Products GmbH R&D / Documentation Maxfeldstraße 5 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 09/13/2012 01:52 PM, Karl Eichwalder wrote:
Interesting logic and by extension many people will be happy as they can also say which program we ship do actually systemd now or in future. Togan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Sep 10, 2012 at 07:53:37PM +0200, Jan Matejek wrote:
I personally like the gcc approach more - they have one gcc package, which just maintain symlinks to the correct gccx.y version To illustrate it more python2 Provides: python = %{version}-%{release} # so zypper in python will select this package Obsoletes: python = some-version Requires: python27 /usr/bin/python -> /usr/bin/python27 python3 Requires: python33 /usr/bin/python -> /usr/bin/python33 I am not sure if it's sane to have /usr/lib/python/site-packages -> /usr/lib/python3.3/site-packages or if the python-central is intended to do the same. BTW: there are new features pyc repository directories and abi version tagged so files - are those already included in your proposal? http://docs.python.org/py3k/whatsnew/3.2.html#pep-3147-pyc-repository-direct... http://docs.python.org/py3k/whatsnew/3.2.html#pep-3149-abi-version-tagged-so... Regards Michal Vyskocil [snip]
Dne 11.9.2012 12:09, Michal Vyskocil napsal(a):
This would mean a conflict between python2 and python3 though. i'd like to avoid that. It does make sense to make it this way for /usr/bin/python2 -> python2.7 and /usr/bin/python3 -> python3.3 - then python apps can use #!/usr/bin/pythonX and be reasonably confident that they are running on the intended python version. I realized another thing: having "python" as an alias for either python2 or python3 could break packages if the user switched from python2 to python3 systemwide. We would have to change most python2 packages to use #!/usr/bin/python2 shebang, instead of just python. But this wouldn't have to be done all at once - we would state that switching system python to python3 is possible but unsupported, and change the packages gradually.
i'm not sure /usr/lib/python was ever used for anything else than lazily written install scripts ;e) so i'd probably just drop it and see what breaks. python-central would probably live in /usr/share/pyshared, for consistency with Debian. Who knows, at that point we might be able to convince upstream and FHS that it is -the- place for python modules
yes, this is what allows us to implement the python-central approach without debian's symlink craziness
http://docs.python.org/py3k/whatsnew/3.2.html#pep-3149-abi-version-tagged-so...
And no, I don't think this is useful for us, at least not at this time. While it's easy to generate all necessary .pycs at install-time, we can't do that with .so files. And as long as different versions of the .so module are in a package at build-time, we can as well install them into their respective site-packages directories. (building such package would be fun, too)
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, Sep 11, 2012 at 11:33 AM, Jan Matejek <jan.matejek@suse.cz> wrote:
It wouldn't break if the symlink was managed with alternatives. And, the shebangs can be handled by an rpm macro that does the substitution (and an rpmlint warning if it's not done). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le lundi 10 septembre 2012, à 19:53 +0200, Jan Matejek a écrit :
Thanks for your attention. Comments, questions?
Is there any recommendation from upstream? It sounds to me like /usr/bin/python being python3 instead of python2 will simply break tons of things in the next 2 or 3 years :/ I would guess that nearly all apps using python3 are actually looking for a python3 binary because of this; so such a change wouldn't be needed. I'm also not sure I like using update-alternatives here, as people will believe they can change it and it will just work while it's simply not the case. Cheers, Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, Sep 12, 2012 at 3:30 AM, Vincent Untz <vuntz@opensuse.org> wrote:
Now that you mention, that's one of the main problems with treating pythonX.X as "alternatives". Each python installation has its own "prefix", and all the libraries installed in 2.6 are invisible for 2.7. Even worse, they can't be shared properly, because 2.6 uses a different bytecode than 2.7. So... not interchangeable at all. And, requiring a library for python2.6 may not imply the need to require it for 2.7 or 3.x, so you can't expect to have the same set of libraries in all versions. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Dne 12.9.2012 17:26, Claudio Freire napsal(a):
True, but this is not a problem in Python 3.x because of PEP3147. Simply put, you can have many different .pyc bytecode files for a single .py source file. So it is feasible to make a "shared prefix" with libraries working on multiple python runtimes. (See Phase 3B of my proposal) It's reasonably easy to backport this feature into Python 2.7, which would actually eliminate big part of this problem. You are right that from our point of view different pythons can't be really considered "alternatives". At this point, it only helps in scenarios where users install third-party python apps or modules outside of our package management - and if the users do this, it's not unreasonable to expect them to manage the alternative symlinks by hand, without us pretending to support that. m. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (10)
-
Alin M Elena -
Andreas Jaeger -
Claudio Freire -
Jan Matejek -
Jan Matejek -
Jan Matějek -
Karl Eichwalder -
Michal Vyskocil -
Togan Muftuoglu -
Vincent Untz