On Fri, Dec 2, 2011 at 2:34 PM, Thomas Schraitle wrote:

Hi,

...

[...]

2) Can you investigate how this issue solve in other distributions? It would be nice to have consistent solution and not reinvent wheel. Maybe some distribution have clean solution to it.

To give you some help, here is the Python packaging guide from Fedora and Ubuntu:

http://fedoraproject.org/wiki/Packaging:Python https://wiki.ubuntu.com/PackagingGuide/Python

To be more specific, here is the relevant portion of the fedora guidelines and the mailing list discussion on the issue: http://fedoraproject.org/wiki/Packaging:Python#Executables_in_.2Fusr.2Fbin http://lists.fedoraproject.org/pipermail/devel/2010-January/129217.html Their approach is the same as my suggestion 2, except that they only do it on a case-by-case basis rather than having general rules. I already explained the problems I have with this approach. Note that they made the decision to move entirely to python 3 and ditch python 2 support, which I don't think has been decided here yet. I don't see anything in the Ubuntu link about this issue, in fact it doesn't mention python3 at all, nor does a google search turn up anything obvious. I have not been able to find any ubuntu packages with files in /usr/bin at all, they might delete them or put them elsewhere. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Fri, Dec 2, 2011 at 4:51 PM, todd rme wrote:

On Fri, Dec 2, 2011 at 1:59 PM, Josef Reidinger wrote:

...

Hi, this is quite interesting problem. I see that you think about it quite lot. I just miss two piece of information: 1) why someone beside developers want have installed binary for sip for python2 and python3? From user POV I want one binary that just works, ignoring if it is python 2 or 3.

The problem is users who want to use both python 2 and python 3 versions of a program.  sip is a good example of this.  For example, python Qt (and thus KDE) programs require python-sip, and so far all or almost all are built using the python 2 version.  However, say you want to use the more up-to-date version of the Eric python IDE, which requires the Qt python bindings built against python 3.

So you have a lot of programs requiring the python 2 version of sip, but one or a few (at this point) requiring the python 3 version. Users should not need to even know this is going on, zypper should just pull in the necessary python 2 and python 3 versions of packages and install them cleanly.  That is not the case now, pulling in the python 3 version of sip could break programs requiring the python 2 version, or vice versa, depending on the order you install them.

The whole point of this is that users would not need to know which versions of the packages are installed or what they are called.  Those are advantages of putting the files in python version-specific directories, users would not need to even know multiple versions exist, their path would handle showing them the correct version on the command line if they need, and the RPM building would make sure applications know where to go to get the version they need to run properly.

...

2) Can you investigate how this issue solve in other distributions? It would be nice to have consistent solution and not reinvent wheel. Maybe some distribution have clean solution to it.

I don't consider Fedora's solution in my other email to be very clean, nor (judging by the mailing list discussion) was it intended to be. They are not planning to keep python 2 available longer than absolutely necessary, so their plan is basically a stop-gap measure until they can get rid of python 2.  I don't gather cleanliness was their top priority.

-Todd

It turns out this is a more serious issue in factory than in previous releases. Factory has checks for duplicate files in non-conflicting subpackages, and there is talk about potentially expanding this to checks between packages. Since many python2 and python3 versions of packages have such conflicts, this currently makes it impossible to build many of them in a single package, and will lead to many more errors if the check is implemented between packages. For this reason I think it is important to come up with an official policy on this, since it is standing in the way of creating more python3 packages. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 3:31 PM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 10:31 AM, todd rme wrote:

...

Since many python2 and python3 versions of packages have such conflicts, this currently makes it impossible to build many of them in a single package, and will lead to many more errors if the check is implemented between packages.

For this reason I think it is important to come up with an official policy on this, since it is standing in the way of creating more python3 packages.

Isn't that solved in single packages by fdupes?

Usually not, because the python 2 and python 3 versions are generally different (for instance a script that calls python 2 versus a script that calls python 3). fdupes only finds, and is only useful for, files that are identical. That is the reason a policy on this is needed. If the scripts were the same then this wouldn't be necessary. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 3:44 PM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 11:40 AM, todd rme wrote:

...

Usually not, because the python 2 and python 3 versions are generally different (for instance a script that calls python 2 versus a script that calls python 3).  fdupes only finds, and is only useful for, files that are identical.

Ok, but the warning isn't triggered either if the files are different.

Or are they talking of changing that?

I think you are thinking of a different issue. What I am talking about is not the rpmlint fdupes warning. Rather, it is an error that occurs when two sub-packages of the same package both own a file with the same name but do not conflict with each other. To provide a relevant example, both the python2 and python3 versions of python-cx_freeze create the file /usr/bin/cxfreeze. If you try to create a package that builds python2 and python3 subpackages for python-cx_freeze, the build will fail because they both contain a file with that name. The only wat to avoid the error is to make the subpackages conflict (which I have done temporarily), but this defeats the purpose of having python2 and python3 both available. There is talk about expanding this so that there is an error even if they are not subpackages of the same package, which will break a lot of existing python3 packages as well. The only way to avoid this error is to rename one or both packages, move one or both packages, or rename and move them. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 4:39 PM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 12:29 PM, todd rme wrote:

...

I think you are thinking of a different issue.  What I am talking about is not the rpmlint fdupes warning.  Rather, it is an error that occurs when two sub-packages of the same package both own a file with the same name but do not conflict with each other.

To provide a relevant example, both the python2 and python3 versions of python-cx_freeze create the file /usr/bin/cxfreeze.  If you try to create a package that builds python2 and python3 subpackages for python-cx_freeze, the build will fail because they both contain a file with that name.  The only wat to avoid the error is to make the subpackages conflict (which I have done temporarily), but this defeats the purpose of having python2 and python3 both available.

But this would indeed be a packaging error.

You cannot install two packages that own the same file, only one version can exist.

Instead, the python3 version should install cxfreeze3. Or install it to another path. Or something like that.

Certainly, two nonconflicting packages cannot install the a file on the same path. That's an error however you look at it, because one version will be stepped onto by the other.

Yes, that is the whole point of this discussion, to figure out what "something like that" we should actually do. I gave four possible solutions to this issue at the beginning of the discussion. I am bringing this issue up specifically to emphasize why it is important that we come to a decision on which of those options we should actually implement. Here are my suggested solutions again:

1. Add python version information to the beginning of the file name. For example, /usr/bin/sip becomes /usr/bin/python3-sip or /usr/bin/python3.2-sip. This has the benefit that it doesn't require any significant changes to current packaging. It has the disadvantage that anything looking for /usr/bin/sip will not find this file. Another disadvantage is that, as far as I have been able to find, python does not have a built-in way to do this, which would mean that the files have to be manually renamed. This is ugly and could potentially break things that are looking for specific file names, as well as making tab-completion on the command line less predictable. There is also a question of what would happen when python 2 support is ended, and a question of whether this would happen with python 3 packages or both python 2 and python 3 packages.

2. Add python version information at the end of the file name. So /usr/bin/sip becomes /usr/bin/sip-3 or /usr/bin/sip-3.2. This has similar issues to 1, but makes tab completion easier at the cost of making it unclear whether 3.2 is the version of sip or the version of python.

3. Create a new subdirectory in /usr/bin for python3 files. So for example /usr/bin/sip becomes /usr/bin/python3/sip or /usr/bin/python3.2/sip. This would require changing the path to include this directory. It has the advantage that the files in /usr/bin, which should be earlier in the path, should have priority. It also has the advantage that python has a built-in way to set what directory is used. It also means the file names will remain the same, so nothing will break. Python could also be patched to default to this directory so the install scriptlets would not need to be changed (although the files list scriptlets still would).

4. Create a new directory in %{python3_sitearch). So for example /usr/bin/sip becomes /usr/lib[64]/python3.2/bin/sip. This requires much more radical changes to how python files are handled, and it would be possible to have python2 files do the same. This has the advantage, however, of being more future-proof (i.e. python 4) and would allow multiple versions of the same python abi to be included (i.e. 3.2 and 3.3). Path priority issues could be solved by making sure /usr/lib[64]/python/bin is first in the path. This currently symlinks to /usr/lib[64]/python2, but simply by changing that /usr/lib[64]/python3 the python3 versions of the files would automatically become the defaults. It would also be easy for users to change by putting one ahead in their per-user path.

As I explained, I consider option 4 to be the best option, especially if we only do it for python3 and leave python2 alone. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 5:04 PM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 12:46 PM, todd rme wrote:

...

Yes, that is the whole point of this discussion, to figure out what "something like that" we should actually do.   I gave four possible solutions to this issue at the beginning of the discussion.  I am bringing this issue up specifically to emphasize why it is important that we come to a decision on which of those options we should actually implement.

Oh, ok... your mention of the validation was misleading, it looked as if you were complaining against the validation ;-)

Sorry about that.

...

Here are my suggested solutions again:

...

1. Add python version information to the beginning of the file name. For example, /usr/bin/sip becomes /usr/bin/python3-sip or /usr/bin/python3.2-sip.  This has the benefit that it doesn't require any significant changes to current  packaging.  It has the disadvantage that anything looking for /usr/bin/sip will not find this file.  Another disadvantage is that, as far as I have been able to find, python does not have a built-in way to do this, which would mean that the files have to be manually renamed.

First, if you symlink sip -> preferred version, this would be fully backwards-compatible.

Second, true. But most setuptools-based packages support --install-scripts, which at least would put them in a separate folder (you can symlink to them, manually, true).

But if you were doing this you would still need to make the other folder be in the global path, so there wouldn't be any point having it in /usr/bin.

...

...

 This is ugly and could potentially break things that are looking for specific file names, as well as making tab-completion on the command line less predictable.

Like I said, symlinks are your friend.

That won't help if the python3 version of a package looks in /usr/bin/ and finds the python2 version.

...

...

2. Add python version information at the end of the file name.  So /usr/bin/sip becomes /usr/bin/sip-3 or /usr/bin/sip-3.2.  This has similar issues to 1, but makes tab completion easier at the cost of making it unclear whether 3.2 is the version of sip or the version of python.

Ya, I prefer this one. Just MHO.

If we are creating new directories anyway, I think this adds a lot of additional complexity with no real benefit. I think we would only want to do this if we were not going to create new directories.

...

...

4. Create a new directory in %{python3_sitearch).  So for example /usr/bin/sip becomes /usr/lib[64]/python3.2/bin/sip.  This requires much more radical changes to how python files are handled, and it would be possible to have python2 files do the same.

I don't follow. Are we talking about apps or libs? Libs don't have an issue, they should reside wherever their python version expects them. Apps are the issue. SIP is both, IIRC. So part of SIP (the libs) should go with all the other python2/3 libs, and only the scripts (or symlinks) have special treatment.

I am talking about things currently installed in /usr/bin and /usr/share, so what python calls scripts and data, respectively. libs are not a problem because they are already in a versioned directory. The idea of suggestion 4 is to also put the scripts and data in a versioned directory So for example, libs currently go in /usr/lib[64]/python3.2/site-packages/. I would create additional directories: /usr/lib[64]/python3.2/scripts/ -- this would contain the python3 files currently in /usr/bin (the scripts), and would be in the global path (at the end of the path, of course, so python2 scripts currently in /usr/bin would remain the default for now). /usr/lib[64]/python3.2/data/ -- this would contain the python3 files currently in /usr/share (the data), and would NOT be in any path (these directories, of course, would use %{py3_ver} rather than a hard-coded version number). This could be set to happen automatically be changing the default for the --install-scripts and --install-data arguments for python. That would mean the %build and %install scriptlets would not need to be changed at all, only the %files scriplets would need to be changed (and with proper macros for these directories in macros.python3 the change should be very easy and only needs to be done once). This could also be done with python2, but that doesn't need to happen at this point (it would need to happen when python3 becomes the default, though, but that is the case with any of these suggestions). -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 3:44 PM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 3:34 PM, todd rme wrote:

...

...

Like I said, symlinks are your friend.

That won't help if the python3 version of a package looks in /usr/bin/ and finds the python2 version. ... I am talking about things currently installed in /usr/bin and /usr/share, so what python calls scripts and data, respectively.  libs are not a problem because they are already in a versioned directory.

Then your above point is moot. A python3 lib/app can only "look for" other python3 libs. Not apps (scripts). If a python3 lib *executes* another python app, it doesn't care if it's python3 or python2, as long as it's the same app.

Let me add (sorry for double posting), data should be shareable among versions. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 10:23 PM, todd rme wrote:

On Mon, Dec 5, 2011 at 7:44 PM, Claudio Freire wrote:

...

On Mon, Dec 5, 2011 at 3:34 PM, todd rme wrote:

...

...

Like I said, symlinks are your friend.

That won't help if the python3 version of a package looks in /usr/bin/ and finds the python2 version. ... I am talking about things currently installed in /usr/bin and /usr/share, so what python calls scripts and data, respectively.  libs are not a problem because they are already in a versioned directory.

Then your above point is moot. A python3 lib/app can only "look for" other python3 libs. Not apps (scripts). If a python3 lib *executes* another python app, it doesn't care if it's python3 or python2, as long as it's the same app.

That is only true if the scripts are identical between the python 2 and python 3 versions.  That may be the case but you can't assume it.

Sorry to double (or triple) post, but even if the scripts are identical, it still doesn't really solve the issue because we still have duplicates that need to be dealt with somehow. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 10:31 PM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 6:27 PM, todd rme wrote:

...

...

...

...

I am talking about things currently installed in /usr/bin and /usr/share, so what python calls scripts and data, respectively.  libs are not a problem because they are already in a versioned directory.

Then your above point is moot. A python3 lib/app can only "look for" other python3 libs. Not apps (scripts). If a python3 lib *executes* another python app, it doesn't care if it's python3 or python2, as long as it's the same app.

That is only true if the scripts are identical between the python 2 and python 3 versions.  That may be the case but you can't assume it.

Sorry to double (or triple) post, but even if the scripts are identical, it still doesn't really solve the issue because we still have duplicates that need to be dealt with somehow.

Somehow, I don't see your problem.

Can you point to an OBS project that fails because of that?

https://build.opensuse.org/package/show?package=python-cx_Freeze&project=home%3ATheBlackCat%3Abranches%3Adevel%3Alanguages%3Apython I am working around the problem right now by making the packages conflict, but if I didn't it would fail for factory and is overwriting files for everything else. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 10:50 PM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 6:40 PM, todd rme wrote:

...

I am working around the problem right now by making the packages conflict, but if I didn't it would fail for factory and is overwriting files for everything else.

Ok, why do you think the following would not work (regardless of prettiness):

%install python3 setup.py install --prefix=%{_prefix} --root=%{buildroot} mv %{buildroot}%{_bindir}/cxfreeze %{buildroot}%{_bindir}/cxfreeze3 python2 setup.py install --prefix=%{_prefix} --root=%{buildroot}

%files -n python3-%{mod_name} %defattr(-,root,root,-) %doc LICENSE.txt README.txt HISTORY.txt doc/%{mod_name}.html %{_bindir}/cxfreeze3

Even if it does work, we would have to do the same with many, if not most, other python packages, then change it again when python3 becomes the default, then change it yet again with python2 support is dropped. That doesn't seem like the most efficient approach, even ignoring what happens when python4 comes around and the whole process has to repeat. It looks like devel:languages:python alone has nearly a thousand packages, a significant fraction of which will need this. We are talking about a massive scalability issue here. Yeah, putting such a hack in single package is not a big deal, but putting it in hundreds of packages, and having to change it at least an additional 2 times for each package, is a much bigger deal. I haven't done a detailed check of how many packages have this issue, but amongst the admittedly small number of packages I have looked at all of them have it. So I think an approach that minimizes the amount of additional work needed would be a huge benefit. That is an advantage of putting things in a different directory, it means that the only things that need changing are python3 and/or python3-base and the handful of python3 packages that are currently available, and those changes will only be in the %file lists. All other packages will require no additional lines in the spec file beyond those that would be needed to build and list the python 3 versions of the files. At worst the python 2 versions of packages will require a one-time find/replace from %{_bindir} to %{python_bindir} or something like that (note that the name-change you suggest approach will also require that the python2 versions be renamed down the road). -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Tue, Dec 6, 2011 at 2:06 AM, Claudio Freire wrote:

On Mon, Dec 5, 2011 at 8:30 PM, todd rme wrote:

...

I haven't done a detailed check of how many packages have this issue, but amongst the admittedly small number of packages I have looked at all of them have it.

Let me save you some trouble: libs don't have the issue.

As per package naming convention, that means most (if not all) of the packages beginning with "python-".

That is simply not true. A great many "libs" also install files in /usr/bin. Just look in the spec file of devel:languages:python if you don't believe me. From what I have seen, a large fraction of the packages named "python-____" install something in /usr/bin, even if they are just supposed to be libs.

Of those that aren't libs, many don't need support for both python versions. mercurial for example. Any version you pick is OK, since it's just a standalone app. You'd only need both versions if they expose a programmatic API other packages depend on (like with sip and cxfreeze).

The vast majority of packages in devel:languages:python are libs.

I believe the number of packages won't be that huge.

Let me give some more concrete numbers. I opened the first 20 rows of python-___ packages in devel:languages:python. That is about 170 packages, and I exclude the about 45 packages that use files lists so I can't tell the contents. That leaves about 125 packages. Of those, about 50 have this issue, or about 40%. If that is representative, that is over 370 packages just amongst the python-___ packages in devel:languages:python alone.

Notice that you could easily build a macro for that rename, since it means renaming all the binaries installed in %{_bindir}. So the change isn't that "personalized" for each package. And if you decide to use a subdirectory within bindir that's even easier, since a macro can create the symlinks with one bash command.

So it *can* be made to scale. It only requires a bit of thought and prevision.

Even with that, it is still not as straightforward as the approach I propose, since it still requires additional lines in the spec file beyond what would be required just to add python3 support to the package. Besides scalability, I listed what I consider to be a number of other advantages to the approach I support. As for adding them to a subdirectory of bindir, that was another approach I suggested, but once again I only see disadvantages to this approach. What reasons do you have for choosing the approach you chose? Of course if there is a reason why this approach would be preferable I would support it. I am trying to figure out the best approach, by looking at the advantages and disadvantages to each approach. I don't see any advantages to this approach, but I very well may just not have thought of them. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Tue, Dec 6, 2011 at 3:25 PM, Claudio Freire wrote:

On Tue, Dec 6, 2011 at 6:03 AM, todd rme wrote:

...

On Tue, Dec 6, 2011 at 2:06 AM, Claudio Freire wrote: That is simply not true.  A great many "libs" also install files in /usr/bin.

They shouldn't...

...

...

I believe the number of packages won't be that huge.

Let me give some more concrete numbers.  I opened the first 20 rows of python-___ packages in devel:languages:python.  That is about 170 packages, and I exclude the about 45 packages that use files lists so I can't tell the contents.  That leaves about 125 packages.  Of those, about 50 have this issue, or about 40%.  If that is representative, that is over 370 packages just amongst the python-___ packages in devel:languages:python alone.

That *is* a problem. Python libs shouldn't be installing new binaries... what are they installing? Maybe it's an error. Maybe I just misjudged the number of libs performing dual functions (lib + app)

They aren't usually binaries, they are usually scripts. They appear to mostly be commandline (i.e. bash shell) wrappers around one or more python functions, letting you access one or more of the library's basic capabilities from within bash or other similar shell instead of needing to open a python shell. For instance numpy provides a fortran-to-python converter called f2py, cxfreeze has a script to convert a script to a stand-alone executable, pygment (a source code highlighting function) provides a script to highlight source code and store the highlights to a file, python-googletranslate of course provides a translator script, and so on. For things like pygment and googletranslate it doesn't really matter what is used, but for things like f2py and cxfreeze the version of python the script calls can have a major or even critical impact.

...

...

Notice that you could easily build a macro for that rename, since it means renaming all the binaries installed in %{_bindir}. So the change isn't that "personalized" for each package. And if you decide to use a subdirectory within bindir that's even easier, since a macro can create the symlinks with one bash command.

So it *can* be made to scale. It only requires a bit of thought and prevision.

Even with that, it is still not as straightforward as the approach I propose, since it still requires additional lines in the spec file beyond what would be required just to add python3 support to the package.  Besides scalability, I listed what I consider to be a number of other advantages to the approach I support.

As for adding them to a subdirectory of bindir, that was another approach I suggested, but once again I only see disadvantages to this approach.

What reasons do you have for choosing the approach you chose?

Only that I do not like the idea of fiddling with standard python paths. It will require a lot of care and testing to make things like setuptools, easy install and in general manually-installed packages to not break.

The problem is we either have to fiddle with paths, or fiddle with names. At least with paths, distutils (and presumably python) is aware of the change. And as you said, it can still be symlinked to the proper place.

Furthermore, when you suffix the binary with the python version, it is rather straightforward to choose a python version:

$ easy_install2 $ easy_install3

Or, the more standard

$ easy_install-2.6 $ easy_install-3.1

From the POV of a user, it is a lot better (a lot more explicit and clean) than fiddling with the PATH.

That is true, although on the other hand it leads to ambiguity about whether the number is the python version or the version of the script.

...

Of course if there is a reason why this approach would be preferable I would support it.  I am trying to figure out the best approach, by looking at the advantages and disadvantages to each approach.  I don't see any advantages to this approach, but I very well may just not have thought of them.

With your numbers about the number of lib packages "misbehaving", your option 4 is starting to sound better. I still maintain that it's a lot clearer and cleaner for a user to type "easy_install3" than to fiddle with the PATH in order to get easy_install for python3..

So we're down to what's better for a user vs. what's better for the packager.

It also depends on how likely users are to use the scripts, and how likely they are to care which one they use. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

I have been reading the discussion of this problem and I note 2 things: 1) The problem has hair. That is, there are many subtle nuances to the problem that would make it easy to take a bad solution to the problem resulting in disaster. 2) The problem is not opensuse specific. I can't see any aspect to the problem that would not apply equally to fedora, debian or any other GNU/Linux distro with python support. If separate distros adopt different solutions to this problem, it will be a headache to those who must support more than one distro. I suggest contacting the Python Software Foundation and other organizations with responsibility for the evolution of the python language as a whole. Hopefully, someone can suggest a solution that would apply to all distros. -- Paul Elliott 1(512)837-1096 pelliott@BlackPatchPanel.com PMB 181, 11900 Metric Blvd Suite J http://www.free.blackpatchpanel.com/pme/ Austin TX 78758-3117

----- Original Message -----

2) The problem is not opensuse specific. I can't see any aspect to the problem that would not apply equally to fedora, debian or any other GNU/Linux distro with python support.

That is why I suggest the solution all of the distros already agreed upon: http://wiki.debian.org/DebianAlternatives http://wiki.mandriva.com/en/Development/Howto/Alternatives I'd point to the Fedora page, but their wiki appears to be down ATM.

I suggest contacting the Python Software Foundation and other organizations with responsibility for the evolution of the python language as a whole. Hopefully, someone can suggest a solution that would apply to all distros.

I love Python, and the people involved in it, but historically, the approach to new packaging features has been fork-and-eventually-replace. Some of those forks included "features" that have been downright hostile to distribution packagers, and I still instinctively scan packages for pointless runtime dependency checks. (Thank you, setuptools...) It's gotten better for sure, but the consensus just isn't there, and I doubt we will see one any time soon. Even if they agreed on one-true-way, it would take years of discussion/implementation/deployment before we could use it. We need something we can implement today, using the packaging systems already in place. -- James Oakley jfunk@funktronics.ca -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

----- Original Message -----

Alternatives sounds OK at a first glance, but in contrast to the normal usage pattern of alternatives, the different python alternatives aren't interchangeable.

System scripts will *need* python2.7 (not 2.6, not 2.8 if there was one), because each python version has a different set of extension modules installed. Apps will perhaps *need* python3.1, and won't work with 2.x.

I'm not sure what you mean here. Are you talking about the shabang? IIRC, distutils is smart enough to replace it with the appropriate binary, so if you call setup.py like this: python3.1 setup.py The shabangs will have "python3.1" as well, and the associated libraries will be loaded from /usr/{lib,lib64}/python3.1. -- James Oakley jfunk@funktronics.ca -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Wednesday, December 07, 2011 05:27:55 PM Claudio Freire wrote:

On Wed, Dec 7, 2011 at 5:11 PM, James Oakley wrote:

...

I'm not sure what you mean here. Are you talking about the shabang? IIRC, distutils is smart enough to replace it with the appropriate binary, so if you call setup.py like this:

python3.1 setup.py

The shabangs will have "python3.1" as well, and the associated libraries will be loaded from /usr/{lib,lib64}/python3.1. What about easy_install?

(the example that always comes to mind)

Where does easy_install SQLAlchemy install to?

SQLAlchemy has no binaries, so that doesn't change in the slightest. If it did have binaries, they would go to /usr/local/lib/python$MAJ.$MIN/bin/ and presumably, we would have patched it to use update-alternatives to link from /usr/local/bin/. -- James Oakley jfunk@funktronics.ca -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Wednesday, December 07, 2011 05:38:35 PM Claudio Freire wrote:

On Wed, Dec 7, 2011 at 5:34 PM, James Oakley wrote:

...

SQLAlchemy has no binaries, so that doesn't change in the slightest.

If it did have binaries, they would go to /usr/local/lib/python$MAJ.$MIN/bin/ and presumably, we would have patched it to use update-alternatives to link from /usr/local/bin/.

I meant which easy_install is run.

Binaries *are* specific to a version, and it *is* required to execute binaries of a specific version in specific situations.

With alternatives, it gets really messy. With version-tagged symlinks, it's a lot cleaner.

Well, just like with Java, if you were working on Python 3, you would either: 1. Just install the python3 version, which would be set as the default, or 2. If multiple versions were installed, use update-alternatives to select the one you want, or 3. Call it as easy_install3.1. Alternatives easily supports this usage What I really don't want to see is an NIH solution, since we already have a system that packagers and users are familiar with, and is recommended by the guidelines: http://en.opensuse.org/openSUSE:Packaging_Multiple_Version_guidelines -- James Oakley jfunk@funktronics.ca -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Wed, Dec 7, 2011 at 6:08 PM, Claudio Freire wrote:

On Wed, Dec 7, 2011 at 5:59 PM, James Oakley wrote:

...

3. Call it as easy_install3.1. Alternatives easily supports this usage

Then *that*'s what's needed.

Didn't know alternatives supported it.

Only... would you need an alternatives entry for each executable script? (that would be a bit excessive perhaps) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Wed, Dec 7, 2011 at 10:21 PM, James Oakley wrote:

On Wednesday, December 07, 2011 06:09:28 PM Claudio Freire wrote:

...

On Wed, Dec 7, 2011 at 6:08 PM, Claudio Freire wrote:

...

On Wed, Dec 7, 2011 at 5:59 PM, James Oakley wrote:

...

3. Call it as easy_install3.1. Alternatives easily supports this usage

Then *that*'s what's needed.

Didn't know alternatives supported it.

Only... would you need an alternatives entry for each executable script? (that would be a bit excessive perhaps)

You'll be making symlinks regardless. Might as well use the established system for maintaining them. The management tools and spec macros are already there.

Does update-alternatives support putting files in subdirectories of the alternatives directory? If that is the case setting the scripts directory to a version-specific subdirectory within the alternatives directory then using alternatives to create a generic name and version-specific names in /usr/bin would probably cover all the angles. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Mon, Dec 5, 2011 at 10:12 PM, James Oakley wrote:

----- Original Message -----

...

...

Second, true. But most setuptools-based packages support --install-scripts, which at least would put them in a separate folder (you can symlink to them, manually, true).

But if you were doing this you would still need to make the other folder be in the global path, so there wouldn't be any point having it in /usr/bin.

Actually, you can use update-alternatives for this. It's precisely the situation it was intended to solve, and it's already in use by a number of packages in openSUSE, such as the verious Java packages, and vim/vi.

Yes, someone has suggested that, but we still need to figure out what to do with the original files. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org

On Tue, Dec 6, 2011 at 9:44 AM, Sascha Peilicke wrote:

On 12/05/2011 10:26 PM, todd rme wrote:

...

On Mon, Dec 5, 2011 at 10:12 PM, James Oakley wrote:

...

----- Original Message -----

...

...

Second, true. But most setuptools-based packages support --install-scripts, which at least would put them in a separate folder (you can symlink to them, manually, true).

But if you were doing this you would still need to make the other folder be in the global path, so there wouldn't be any point having it in /usr/bin.

Actually, you can use update-alternatives for this. It's precisely the situation it was intended to solve, and it's already in use by a number of packages in openSUSE, such as the verious Java packages, and vim/vi.

Yes, someone has suggested that, but we still need to figure out what to do with the original files. That would be me then. Simple, just postfix binaries with the Python ABI version they prefer (i.e. rename them). That would be option 2 of yours ;-) This way, we only have to touch packages that actually install binaries, the others can be left alone.

This would be the case with all of the approaches. None of the approaches would call for any changes to packages that do not include files /usr/bin (and maybe /usr/share), nor do any of the approaches require any changes to files already installed in versioned python directories (like python_sitelib or _sitearch). So that isn't an advantage to that approach. Although to be pedantic the things in /usr/bin are generally not binaries, they are usually scripts. -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org