[opensuse-packaging] python installing modules to /usr/local by default - good or bad?
Fellow packagers and pythonistas, i'd like to notify you about a proposed change to the way Python installs its packages, and give you a chance to object before it is put into effect. This has nothing to do with the "versioned python" proposal. TL;DR version: Currently, python is configured to install packages into /usr/local/lib/pythonX.Y/site-packages, unless you specify "--prefix=/usr". This mimicks autotools behavior, but diverges from upstream. If there are no significant objections, I'm going to restore the default behavior where "/usr" is the default prefix. Packages installed in /usr/local will still be recognized, it just won't be the default install location anymore. Detailed version: While investigating possible solutions to bnc#658604 (python user install scheme broken, [1]), i came to revisit the decision to make SUSE's python install modules into /usr/local by default. A little background: this feature came to be as a response to a pair of bugs, bnc#149809 [2] and bnc#149843 [3]. The original reporter argues that /usr/local is the right place for locally-installed software and that Python should not deviate from customary (i.e. autotools) behavior in this regard. Python should auto-recognize modules installed in the /usr/local hierarchy, and for consistency and convenience's sake, should install in there by default as well. The former request is reasonable, and back then i thought that the latter is reasonable as well. (also, back then i wasn't on the job for very long, i made the change, didn't tell anyone about it and broke about a million of other people's packages. oh those were the days. sorry again if you were affected.) Six years later, i still think it's the right thing to do, but i don't think we should actually be doing it. Arguments basically boil down to one of Python's core principles, "practicality beats purity". But for completeness, here's what i came up with. For installing to /usr/local: + it's what FHS tells us, with some good reasoning (read-only /usr and so on) + it's what autotools do + the /usr hierarchy remains untainted by files outside of rpm's control + and conversely, the user has all of her site-local packages in /usr/local, not mingled with "managed" installations (unlike e.g. Perl, Python has no official mechanism for separating "site" packages and "vendor" packages. this provides such distinction) Against: - AFAIK, we're the only ones doing it - upstream python doesn't recognize the distinction and there is no support for it in the libraries - to expand on that point, there is no "right" place to patch this behavior in - a correct patch for the new behavior wouldn't be too complex, but it would not be upstreamable. No patch is better than one patch. - modules don't like to be installed in locations they can't query through python's sysconfig methods - third-party python packagemanagers/installers might not recognize our intentions and would need to be patched separately Also, i'm not really aware of anyone reaping any benefits from our current config. So, if you have anything good to say about the way we do it now, here is your chance. I'll wait until next week, and if no significant objections are raised, i'll push this change into Factory. m. [1] "Python distutils setup does not allow user installation of Python packages" https://bugzilla.novell.com/show_bug.cgi?id=658604 [2] "python path does not contain /usr/local" https://bugzilla.novell.com/show_bug.cgi?id=149809 [3] "Python's site-packages should contain a directory in /usr/local" https://bugzilla.novell.com/show_bug.cgi?id=149843 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, Sep 27, 2012 at 12:12 PM, Jan Matejek <jan.matejek@suse.cz> wrote:
Fellow packagers and pythonistas,
i'd like to notify you about a proposed change to the way Python installs its packages, and give you a chance to object before it is put into effect.
This has nothing to do with the "versioned python" proposal.
TL;DR version: Currently, python is configured to install packages into /usr/local/lib/pythonX.Y/site-packages, unless you specify "--prefix=/usr". This mimicks autotools behavior, but diverges from upstream. If there are no significant objections, I'm going to restore the default behavior where "/usr" is the default prefix.
Packages installed in /usr/local will still be recognized, it just won't be the default install location anymore.
Jan, I worry about compatibility between what's out there and newly built packages. Is that going to break factory snapshots because they that have the old location vs. new packages having the new location? What about existing deployed releases that want to get a new python module. Will it be compatible? Greg -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Dne 27.9.2012 18:22, Greg Freemyer napsal(a):
Jan,
I worry about compatibility between what's out there and newly built packages.
Is that going to break factory snapshots because they that have the old location vs. new packages having the new location?
Our python packages will not be affected because they are already handling the non-default behavior and specifying "--prefix=/usr". Their install paths will not change. Some faulty 3rd party packages could be installing into /usr/local now (not in OBS, i think, there's a check for that) and would install into /usr when rebuilt with the new python. If some script was explicitly checking for files in /usr/local, that would fail, but i believe this scenario is far-fetched. Python will still be looking for modules in both locations.
What about existing deployed releases that want to get a new python module. Will it be compatible?
Yes. regards m.
Greg
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, Sep 27, 2012 at 1:12 PM, Jan Matejek <jan.matejek@suse.cz> wrote:
+ and conversely, the user has all of her site-local packages in /usr/local, not mingled with "managed" installations (unlike e.g. Perl, Python has no official mechanism for separating "site" packages and "vendor" packages. this provides such distinction)
Debian adds a "dist-packages" in tandem with site-packages, and that's also quite confusing. Maybe you're right, any deviation from upstream will be confusing in this regard, not only maintaining the patches that, as you say, are complicated by the fact that there's quite a great deal of places to patch, but also in the sense that users and modules both get confused by alternate search/install paths. I've certainly had to battle with such confusion in the past. Why do you think the patches aren't upstreamable? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Dne 27.9.2012 18:23, Claudio Freire napsal(a):
Why do you think the patches aren't upstreamable?
Because the change would be unsystematic - to do this "properly", parts of distutils would have to be redesigned and that's out of scope of any of our bugs ;e) The patch would be implementing a special behavior in order to conform to FHS, introducing possible compatibility problems (modules confused about their whereabouts), when all the while Python itself hits a couple of FHS gray areas anyway. Actual solution might be to cooperate with Debian on their "dist-packages" thing and try to upstream that. regards m. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, Sep 27, 2012 at 6:43 PM, Jan Matejek <jan.matejek@suse.cz> wrote:
Dne 27.9.2012 18:23, Claudio Freire napsal(a):
Why do you think the patches aren't upstreamable?
Because the change would be unsystematic - to do this "properly", parts of distutils would have to be redesigned and that's out of scope of any of our bugs ;e) The patch would be implementing a special behavior in order to conform to FHS, introducing possible compatibility problems (modules confused about their whereabouts), when all the while Python itself hits a couple of FHS gray areas anyway.
Actual solution might be to cooperate with Debian on their "dist-packages" thing and try to upstream that.
Why couldn't the patch merely implement a more flexible way to handle prefixes, and then we override the default in our packages? -Todd -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Dne 2.10.2012 15:09, todd rme napsal(a):
On Thu, Sep 27, 2012 at 6:43 PM, Jan Matejek <jan.matejek@suse.cz> wrote:
Dne 27.9.2012 18:23, Claudio Freire napsal(a):
Why do you think the patches aren't upstreamable?
Because the change would be unsystematic - to do this "properly", parts of distutils would have to be redesigned and that's out of scope of any of our bugs ;e) The patch would be implementing a special behavior in order to conform to FHS, introducing possible compatibility problems (modules confused about their whereabouts), when all the while Python itself hits a couple of FHS gray areas anyway.
Actual solution might be to cooperate with Debian on their "dist-packages" thing and try to upstream that.
Why couldn't the patch merely implement a more flexible way to handle prefixes, and then we override the default in our packages?
Because that's not "merely". The problem is that Python only recognizes one (*) location for site-specific packages, and we need to distinguish two: vendor-provided vs. user-provided. It is already easy enough to specify prefix on command line to force installation somewhere else, and as of now, our Pythons are preconfigured to set that prefix to /usr/local. This causes bnc#658604. I could do this differently so that this bug goes away and behavior remains the same, but original reasoning for "why do / not do this" remains the same as well. * - Actually two, but one is platform-dependent and one is platform independent. We use that to install and support noarch pythonic packages, but it doesn't help in this case.
-Todd
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, 2012-09-27 at 18:12 +0200, Jan Matejek wrote:
Fellow packagers and pythonistas,
i'd like to notify you about a proposed change to the way Python installs its packages, and give you a chance to object before it is put into effect.
This has nothing to do with the "versioned python" proposal.
TL;DR version: Currently, python is configured to install packages into /usr/local/lib/pythonX.Y/site-packages, unless you specify "--prefix=/usr". This mimicks autotools behavior, but diverges from upstream. If there are no significant objections, I'm going to restore the default behavior where "/usr" is the default prefix.
Packages installed in /usr/local will still be recognized, it just won't be the default install location anymore.
Jan, Two ways to look at it: - A user on a standard systems downloads a tarball and does the whole magic... likely with very few knowledge or just following bare documentation => This needs to go to /usr/local (as it's a locally built / locally installed package) - A packaged prepared a spec file with all the fancy stuff => this needs to go to /usr for autotools, this is nicely handled using the %configure macro, which includes --prefix and anything else the packager would need to pass, but, as every packager, is too lazy to type or even remember (me included). I tend to very often make good use of that: build something locally from source, have it install to /usr/local (which I tend to keep clean)... once my tests are completed, I can just wipe /usr/local again, do the right fix in the real packages and live happily ever after... now splitting it off to /usr/local would mean that I have to trace installs and remove them (ok, likely not an issue, as I tend not to do a lot of python stuff :P ) But of course, the argument against it, carrying a patch, is a very compelling one. The right decision? Tough one... The "Upstream" way is usually preferred by me (I also hate to carry patches).. so in this case... I don't know :( Dominique -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Le jeudi 27 septembre 2012, à 18:12 +0200, Jan Matejek a écrit :
Also, i'm not really aware of anyone reaping any benefits from our current config. So, if you have anything good to say about the way we do it now, here is your chance.
Does upstream have a nice way to avoid conflicts between packaged modules and modules manually installed? I guess it wouldn't be too unusual for someone to install a module from the package, to realize he needs another version, and to manually install the module. Or the other way around. I'm afraid that it might result in some weird situations. But maybe I'm worrying for no reason. Not sure what other distros are doing for this? Thanks, Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, Sep 27, 2012 at 1:36 PM, Vincent Untz <vuntz@opensuse.org> wrote:
Le jeudi 27 septembre 2012, à 18:12 +0200, Jan Matejek a écrit :
Also, i'm not really aware of anyone reaping any benefits from our current config. So, if you have anything good to say about the way we do it now, here is your chance.
Does upstream have a nice way to avoid conflicts between packaged modules and modules manually installed?
I guess it wouldn't be too unusual for someone to install a module from the package, to realize he needs another version, and to manually install the module. Or the other way around.
When one needs this, upstream usually tells us to use virtualenv -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Dne 27.9.2012 18:12, Jan Matejek napsal(a):
TL;DR version: Currently, python is configured to install packages into /usr/local/lib/pythonX.Y/site-packages, unless you specify "--prefix=/usr". This mimicks autotools behavior, but diverges from upstream. If there are no significant objections, I'm going to restore the default behavior where "/usr" is the default prefix.
Packages installed in /usr/local will still be recognized, it just won't be the default install location anymore.
just a notice: this change is now live in Factory. AFAICT, it did not (and should not) break any packages m. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (6)
-
Claudio Freire -
Dimstar / Dominique Leuenberger -
Greg Freemyer -
Jan Matejek -
todd rme -
Vincent Untz