[opensuse-packaging] python single-spec progress, questions
hello packagers, i'm happy to announce that i have a macroset that works pretty well, if i say so myself :) Have a look at the d:l:python:singlespec repository [1] and the github repo [2] for details. The gist of it: it is now very possible to build packages for both python 2 and python 3, using a pretty straightforward set of macros. See spec files in the linked repository, and documentation on github. So far this only reliably works on python 2 and 3. It can work with pypy too, but pypy is not building so I can't test against it. Now there is a number of remaining issues to be worked out, and I'd like your feedback on them: 1. Necessary changes to OBS configuration 2. Backwards compatibility 3. Transition to new macros 4. New policies for d:l:py 5. Naming scheme 1. Necessary changes to OBS configuration --- The macro set only works with the one particular repository, that has the macros in its prjconf. In particular, the %python_module macro is problematic, because it needs to be expanded *before* the build starts, by OBS. Usually, we would put the macro definitions inside /etc/rpm/macros.python, as a part of the python/python3/whichever package, but that won't work in this case. Before we can deploy these packages to Factory, either Factory's prjconf, or OBS global config, needs to include at least the %python_module macro. And if possible, I'd like to include all of them, because: 2. Backwards compatibility --- Basically, we would need to build for old distros using new macros. There's a number of options: a) macro definitions in OBS and everything will build everywhere b) macro definitions in python, everything will build with new enough version of python c) macro definitions in prjconf of particular projects (say d:l:py), only packages in those projects build. Personally, i'd go with (a). And if that's not possible, with (b) and (c) combined. The problem with anything other than (a) is that it is going to be difficult to write specs that work both with and without the macros. You could conditionally include the %python_packages macro, but you would have to shim all the others in every spec file, and that is going to be ugly. Which means that by using the new macros, we would be hard-forking the python packages from ALL currently released distributions, unless we could get the macro definitions into them. 3. Transition to new macros --- There is a script called 'process-spec.pl' in the github repo. I'm about to run it first against (some small subset of) python-* in d:l:py, then against python3-* in d:l:py3, and put the results in d:l:py:singlespec:staging. Then it will be necessary to review the packages and submit to d:l:py:singlespec. That's going to be a lot of work, and I would like to ask you to help with this. I'll put up a separate call for volunteers next week. In the meantime, feel free to grab the macro definitions from prjconf and play around. And if you want access rights to d:l:py:singlespec, just send a request. 4. New policies for d:l:py --- If the transition goes smoothly, I'd like to use the opportunity to clean out the devel:languages:python project. One, d:l:py is collecting applications that happen to be *written in* Python, but have nothing to do with python development, and should instead be placed in other topically appropriate projects. We've had some discussions about dependencies only present in d:l:py, but here's a policy proposal: If your package depends on d:l:py and is not appropriate for d:l:py, you can either push your dependencies to Factory, or link them to your topically appropriate project. Two, end-user applications should start to move to Python 3. This is already happening. The relevance here is that we can now build "dual-version" packages. This should be discouraged. There are packages where that is appropriate (pip, nose and similar), but most packages should only provide executables for python3. We have a great number of packages that act as modules (= dependencies for other packages) and also as command-line utilities. Policy proposal: Unless we actually need the executable for all versions of python, all executables should be %py3_only. Finally, three, d:l:py is collecting non-python dependencies of python modules. That should not be happening. Analogously to point one, policy proposal: If your package depends on something that is not appropriate for d:l:py, either get that dependency into Factory, or your package is also not appropriate for d:l:py. How about it? 5. Naming scheme --- Finally some bikeshedding: names of RPM macros for python-related things are inconsistent. We have %py_ver and %py_compile and %python_sitelib and %ifpython2 and %py2_only. I would like to keep "python" and "py" as *generic* versions of the macros, not specifically referring to a particular version or flavor of Python. So for every "py_" and "python_", equivalent "py2_" and "python2_" should be defined. This is partially true in the new macro set already. Another thing is with "short" and "long" versions. I would really prefer to use only one of them, and that's probably the short one. Keep %py_ver and %py2_ver, but make %py2_sitelib etc. The last thing is about executables for pypy. For instance, when installing pip for python2 and python3, you get pip-2.7 and pip-3.5. What should the pypy variant be called? Thoughts, questions, comments? thanks m. [1] https://build.opensuse.org/project/show/devel:languages:python:singlespec [2] https://github.com/openSUSE/multipython-macros
On Thu, Oct 27, 2016 at 12:51 PM, jan matejek <jmatejek@suse.com> wrote:
Some comments on the macros: 1. I think the version number should be explicit in all cases. So it should be "%have_python2", not "%have_python", and "%have_pypy2" and "%have_pyp3", not "%have_pypy". 2. Along those lines, I think pypy2 and pyp3 versions should be available, especially with a lot of progress now happening in pypy3 thanks to mozilla funding it.
When was this ever not the case? There are packages like that which haven't been updated in 4 or 5 years. I did some spot checks and the other devel:languages:___ repos also have a wide variety of software written in that language. What do you see as wrong with the current approach?
I think you are fundamentally changing the purpose of dlp here. So I think the first thing we need to decide is "what is appropriate for dlp"? I would personally keep dlp as a generic place for all python packages. This has worked very well up to this point and I don't see any advantage to changing it.
Agreed. I think the rule should be that if the executable is intended for use with python files or packages, it should provide both python 2 and python 3 versions when available. So this would include installers like pip, analysis and testing tools like pylint and pytest, IDEs like spyder and eric, and shells like bpython and ipython. It would not include general-purpose tools that happen to be written in Python like the python wget implementation. Also, in the case if bindings, when a package only supports building bindings for one python version, the (c)python 3 version should be used.
A lot of these are backports for older versions of openSUSE that either have outdated versions of the library or lack the library entirely. So yes, I agree if it really has non-Factory dependencies it probably shouldn't be there. But I think the bigger problem is that many of these are (or were) needed but are building for openSUSE versions where they aren't really needed, or where the version they were needed for is no longer supported. A cleanup of these is certainly warranted. They should also probably be renamed (package, not .spec file) to include the version of openSUSE they are needed for so we can clean them up more easily later.
Agreed.
This is partially true in the new macro set already.
As I said above, I think the exceptions should be fixed in the macros before we start using them. It will be much harder to change the macros later.
Agreed.
Probably pip-pp2.7 and pip-pp3.5. Jython would be pip-jp2.7 if we ever got around to supporting it. Some additional issues: 1. The layout of the repositories. With single spec files, does "devel:languages:python3" even make sense anymore? I think we should phase it out. 2. Package naming. Might this be a chance to switch from using python-foo to python2-foo? Either way, I would strongly suggest pypy packages using pypy2-foo and pypy3-foo from the beginning, rather than pypy-foo. 3. Version-specific packages. There are still a lot of packages that only support python 2, and an increasing number that only support python 3. Should these use the macros or be kept simple? 4. Repository cleanup. There are packages that haven't been updated in many years and are not under active development. Who knows what sort of security or compatibility issues they might have. It might be worthwhile to move them to a separate repository (perhaps d:l:p:archive or d:l:p:legacy) with lower build priority, or delete them outright, to free up resources for packages that are better-maintained. 5. Disabled packages. Packages that are completely disabled should almost certainly be deleted. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/28/2016 05:05 AM, Todd Rme wrote:
There is almost always a better repo that they could be in which is less confusing to users who care about what applications do rather then which language they live in. For example variety which is a cross desktop wallpaper fetcher and changer written in python lives in the X11:Utilites repo because its a Utility for X11 apps, similarly if the purpose of the application is to monitor servers we have a nice Server:Monitoring repo I believe. The reason its probably worth being part of this discussion is for almost all libraries should be built for python2 + python3 (Unless python2 is not supported upstream anymore). Where as applications should only be built for one or the other. As for applications requiring dependencies only in d:l:py the simple policy should be anything added to d:l:py should also then be forwarded to openSUSE:Factory where it will hopefully get picked up for the next Leap release as well. If this is enforced the required deps will always be available for all repos and it won't be an issue. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Thu, Oct 27, 2016 at 5:07 PM, Simon Lees <sflees@suse.de> wrote:
The problem is that the distinction between "application" and "library" is not very clear-cut with most python packages. In the rare case where something is purely an application that happens to be written in Python, then yes I suppose you are right. But more often the "application" contains both an executable and a python module that python scripts can make use of. What should we do in that situation? For the pure applications, a bunch of those are python shells, python IDEs, debuggers, or other development tools that are connected in some way to the python version they were built with and thus need to have both versions available. Do you have any specific examples of python-based packages in d:l:p or d:l:p3 that you think shouldn't be there?
Are you talking about all packages or only non-python packages? Because there are a lot of niche python tools in d:l:p that I don't think belong in openSUSE:Factory. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/28/2016 08:01 AM, Todd Rme wrote:
These are probably both valid examples of things that are not libraries that could stay.
Do you have any specific examples of python-based packages in d:l:p or d:l:p3 that you think shouldn't be there?
No I personally have not looked I was simply agreeing with the above quoted statement "One, d:l:py is collecting applications that happen to be *written in* Python, but have nothing to do with python development" and agreeing that they should be moved to more appropriate locations
All packages, unless they are of no interest to anyone even those doing python development. Users shouldn't need to add development repo's to there systems especially ones like d:l:p where there is a high chance that they can completely break there system by updating with it enabled. So if its useful to any openSUSE users it should go in the distro if its not there is probably a question of does it actually need to be there still? Richard has been pushing this view far more then I but its a view I agree with (you can watch his openSUSE conference talk). -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Thu, Oct 27, 2016 at 7:35 PM, Simon Lees <sflees@suse.de> wrote:
That is almost all, if not all, of the "applications" in d:l:p.
My point is that this tends to be a very blurry line with python. We need some more specific rules to actually implement this sort of policy.
There are certainly people who hold this view, but I have seen nothing to suggest it is remotely close to a consensus view. I personally cannot emphasize how strongly I disagree with this view, and how much I think it would harm projects like d:l:p. If it becomes official openSUSE policy, then certainly d:l:p will have to follow it. But I am strongly opposed d:l:p implementing this policy on its own. Especially since it would involve eliminating hundreds of useful packages that our volunteers have been willing to package but aren't willing to go through the often months-long process of getting accepted into openSUSE:Factory. We simply don't have the manpower to get every useful package into openSUSE:Factory, even if they all belonged there (and I think many of them don't belong there). And I don't think there is "a high chance that they can completely break there system by updating with it enabled", as long as we fix the issue with the linked libraries that was already brought up. System-critical python-based packages are not handled in that repo (if they are they will be moved under the new policy), nor is the packaging of python itself (that happens in an isolated sub-project). -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/28/2016 01:55 PM, Todd Rme wrote:
While the legal review can take up to a month, although from personal experience its gotten better of late (Even with the haskell development project adding 1500 packages at once), It's not like its a month of hard effort required. If something is packaged reasonably well to start with and according to openSUSE's guidelines the amount of work required to get it included is pretty minimal and any issues should be resolved within 1 or 2 submit requests. If people would like help with this I and others are willing to spend time helping them fix there issues. Personally I think that packages in openSUSE development repo's should be following openSUSE guidelines and packages that are not willing to do that should be in some other sub repository like d:l:p:Playground but i'm guessing others would disagree.
That still doesn't rule out some users hitting some weird breakages at times (I've been helping with support on irc for at least 5 years and have seen some pretty weird breakages due to extra repo's in my time). Having said that i'm not doing most of the work, although i'm happy to review packages people push to factory before they do so what I say doesn't count that much. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Fri, Oct 28, 2016 at 12:22 AM, Simon Lees <sflees@suse.de> wrote:
I disagree with most of what you said, but this can be discussed further in the Factory mailing list. What you are suggesting is a fundamental change to the most basic aspects of how OSC is organized. I am strongly against using d:l:p as a testing ground for such a change. Even if I liked the policy, the repo is undergoing too many other changes right now. We just don't have the manpower to additionally sift through probably somewhere around 1000 packages, figure out which one should go into factory, clean them all up, wade through the submission process, do whatever it is you propose we do with those that don't make the cut. It is probably going to take months to just get the scripts implemented, and that process is mostly automated.
That is very different from "a high chance that they can completely break there system by updating with it enabled". -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/28/2016 10:59 PM, Todd Rme wrote:
Given that most of the Desktop environments including (KDE/Gnome/Enlightenment) and other repo's such as haskell and probably others that I don't know about are already doing this so d:l:p would hardly be the first or a testing ground.
Again i'm not suggesting that 1000 packages should be fixed right now at once, but if you were to adopt this policy now would be a good time to consider it given that presumably you need to go through each package to add these new macro's anyway, while doing that you could decide 1. This spec file looks in pretty good shape I may as well put in a request to add it to factory 2. This package is a long way from meeting openSUSE packaging guidelines I probably should document that in the description so users are aware and someone can come back to do the work to get it up to standard later if they desire if not its labeled and we may choose to move it to another repo later. 3. This package is old and not maintained upstream its not worth migrating it to the new macro's i'll disable the build in factory and we will slowly phase it out. None of that is much extra effort on top of what already needs to be done. I have a list of packages I use from pip that I will add to openSUSE:Factory at some point, I haven't even got to checking if there in d:l:p yet because for me using pip is far easier and less complicated then trying to enable another repo for 1-2 packages. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Fri, Oct 28, 2016 at 9:31 PM, Simon Lees <sflees@suse.de> wrote:
Again, discussing major changes to the basic structure of OSC is off-topic for this thread. This is just about the changes needed for python packages, not for a complete reorganization of OSC. I would strongly prefer not to further derail this thread with this discussion. There are many reasons I don't like this policy, but they are way off topic, and getting into a lengthy discussion about them distracts from all the work that has to get done related to the actual topic of discussion. So if you wish to discuss this, I ask you again to please create a new thread. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/29/2016 12:15 PM, Todd Rme wrote:
This is not anything to do with a complete reorganization of osc it is only discussing organising d:l:p (which is getting reshuffles anyway) with a aim to getting more of its contents into the "openSUSE Distribution" as a whole given this is the openSUSE packaging list discussing packaging of the openSUSE Distro it seems appropriate.
Well then lets agree to disagree, I don't think I have anything more to add on this context anyway. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On Thu, Oct 27, 2016 at 5:07 PM, Simon Lees <sflees@suse.de> wrote:
I have numerous packages (20?) I maintain that are c libraries at their core, but have python binding sub-packages. I'm slowly making them have both python- and python3- sub-packages. They currently live in security:forensics (and factory). Will they be able to leverage the proposed new macros? Or do they have to be moved to d:l:py? Is it current policy they should be there already? I also have 4 or 5 dual python app / python library packages (python-dfVFS, etc.) and one pure python app (python-plaso). My preference is to leave all of the above in security:forensics. Greg -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Fri, Oct 28, 2016 at 9:30 AM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
That is one of the questions we are trying to answer. Ideally I would prefer the macros would apply to all of OSC, and Jan has said this as well, but AFAIK we haven't had anyone with any authority on the matter weigh in on the issue.
Is it current policy they should be there already?
I do not think so. From what I have seen non-python packages that have Python bindings follow the same rules as any other non-python package: they are only there (or at least should only be there) to allow python packages for older distros to build. The fact that they happen to provide python bindings is irrelevant.
python-plaso still provides usable python modules, and according to the documentation these modules are intended for direct use [1], so I would say it is also a dual python app / library. I can't speak for anyone else, but I would think they should stay where they are, but you could (but don't need to) have them linked in d:l:p as long as they don't have any dependency issues and you think they might be interesting to people outside of your project. From my perspective, something like plaso that is used to extract file information can be useful in a wide variety of situations besides forensics (such as sorting directories or searching), so I would recommend linking it. But the decision is up to you, I am not aware of any policy requiring that all python packages always have to live in d:l:p. [1] https://github.com/log2timeline/plaso/wiki/Users-Guide#the-tools -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/29/2016 12:00 AM, Greg Freemyer wrote:
I had a similar situation with the efl (Enlightenment Foundation Libraries) bindings which are much better left in the enlightenment devel repo, I was also planning on using the macro's but given that every known application that uses these bindings also uses python3 in the end I just swapped them out directly. So I agree that it would be ideal if the macro's become available everywhere. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
Hello, Am Donnerstag, 27. Oktober 2016, 14:35:11 CEST schrieb Todd Rme:
On Thu, Oct 27, 2016 at 12:51 PM, jan matejek wrote:
An (IMHO) easier way to handle this is: - disable building those packages for factory, but keep the build for released versions (42.x, 13.x) enabled - if you know that for example 42.1 doesn't need this package from d:l:py, feel free to also disable the build for 42.1 - whenever a release goes EOL, no longer build for this release - when a package has no remaining build targets (see the "Monitor" page), drop it I'm using this method in my home repo (for backporting some packages that are missing in older releases) and it works fine. I never tested this in a repo with a large number of packages, but I'd expect it to be useable there also (it's guaranteed to work on the technical side, the only question is if it is maintainer-friendly).
Agreed.
2. Package naming. Might this be a chance to switch from using python-foo to python2-foo?
That sounds like lots of (IMHO superfluous) package renames ;-)
No objections. Regards, Christian Boltz -- Opensuse-Factory is mainly for systemd infights and KDE3 legacy maintainence questions nowadays. I guess you chose the right list ;-) [Ralf Lang in opensuse-programming] -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thu, Oct 27, 2016 at 6:24 PM, Christian Boltz <opensuse@cboltz.de> wrote:
The idea would be that the spec file would be named "python-foo.spec", while the python2 subpackage created by the macros would be automatically named "python2-foo" instead of "python-foo" as they now. So the macros would handle the naming, and that is the case no matter what naming scheme we choose. Only packages that aren't compatible with the macros would need any additional work. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 10/27/2016 10:51 PM, Todd Rme wrote:
I don't think there will be lots of renaming. The names of the binary packages should be handled by the macros, and the number of Python-2 only packages should be shrinking, more below.
Agreed the macros should produce a python2-foo binary package. But also we need to consider that there are probably still some packages that are Python-2 only, although the number is getting smaller. In any event I think we need to account for the following: - packages that only build for Python-2 - packages that only build for Python-3 So my proposal for package names, meaning the names seen when looking at d:l:p in the web UI: - python2-foo -> a package that produces only a python2-foo binary package - python-foo -> a package that may produce a python-foo and python2-foo binary package or a python-foo package only Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Public Cloud Architect LINUX rjschwei@suse.com IRC: robjo
On 10/29/2016 05:31 AM, Robert Schweikert wrote:
And this is what I get for responding to later messages first, that I contradict myself. So let me revise this proposal and expand on it just a little bit. So my proposal for package names, meaning the names seen when looking at d:l:p in the web UI: - python2-foo -> a package that produces only a python2-foo binary package - python-foo -> a package that may produce a python3-foo and python2-foo binary package or a python3-foo package only And we would drop python-foo as binary package names. We could then add Provides: python-foo to the spec files, via macro. As long as Python-2 is the default interpreter the python2-foo packages provide python-foo and when Python-3 becomes the default interpreter than the python3-foo packages will provide python-foo. This was it is all the same for the end user: zypper in python-foo will provide the package that matches the default interpreter for the distribution. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Public Cloud Architect LINUX rjschwei@suse.com IRC: robjo
On 10/29/2016 12:04 PM, Robert Schweikert wrote:
And if there's no python3 version of the package, then python-foo is provided by python3-foo or not? Sebastian -- python programming - mail server - photo - video - https://sebix.at cryptographic key at https://sebix.at/DC9B463B.asc and on public keyservers
On 10/27/2016 02:35 PM, Todd Rme wrote:
I think there is another question hidden in here. If we make the version number explicit in the macros, I think that is a good idea, then maybe we should consider to do the same for the binary packages, i.e. maybe we should drop python-foo packages and only have python2-foo and python3-foo packages. Of course that could potentially create all kinds of other problems but most of those could probably be addressed with a few scripts. Making the package names explicit with respect to the version would get people used to a disassociation of the command line name and the associated packages. Today, with the Python-2 interpreter being the default one uses "python" on the command line and installs modules with binary packages called "python-foo". Thus there is a nice match in names. However, at some point in the future the "python" command will start the Python-3 interpreter and then all of a sudden one has to install "python3-foo" to use the module. So my thought is, if we break this association now, while Python-2 is still the default, then the change when Python-3 becomes the default is not going to be as traumatic as people will already be used to not having the "python" command and the binary package names be associated by name.
I agree I don't think we should start to evict packages from d:l:p based on some nebulous arbitrary policy decision. If an application is written in Python it should be able to live in d:l:p. Chances are the application provides some modules that are useful for other purposes and the maintenance burden is reduced on the maintainer. If we evict the application from d:l:p and make it live in d:myapp then the maintainer of myapp has a good chance of having to back link a rather large number of packages from d:l:p in addition to having to deal with prjconfig, repo setup etc. which are things that are currently handled by the d:l:p project maintainers. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Public Cloud Architect LINUX rjschwei@suse.com IRC: robjo
On 29.10.2016 11:57, Robert Schweikert wrote:
At this point it seems realistic to aim to switch the default python to Python 3 in SLE 13. For that, we would need to rename "python" to "python2", and keep "python" just as a provided symbol. This might also be an opportunity to move to "python2-foo" + "python3-foo" and make "python-foo" just a provided symbol too. This way the association is never broken. (Please note that at this point there is no concrete plan for the switch. It's just what seems realistic to me personally.)
OTOH, if you want to add a repository for, say, all geodesics packages, you shouldn't need to add Application:Geo ... AND d:l:py and d:l:ruby and d:l:haskell and d:l:javascript and d:l:intercal, just because some of your geodesics apps happen to be written in a particular language. Also, adding the package to, say, Application:Geo, the maintenance burden is moved from d:l:py maintainers to App:Geo maintainers. This even makes more sense in that App:Geo maintainers probably care more about keeping a "core" package up to date, as opposed to d:l:py maintainers for whom a random app is not considered core. The "provided modules" thing is also nebulous. For me, a good rule of thumb is this: if the application does not have an API documentation (or if the API is merely a way to invoke the utility without using system() ), it is probably not useful as a library. And conversely, if there *is* an extensive and documented API, it's probably a good Python library that happens to have an app bundled, and I am happy about having it in d:l:py ;)
That's why these packages should be submitted to Factory in the first place.
which also should not be necessary in general. If it is, the solution should be to make sure you can build a package anywhere without hassle, not to collect it in a project that did the work. Of course, the latter solution is more practical. But while we're talking about big changes, I'm aiming for the clean approach. regards m.
Later, Robert
On 10/31/2016 11:51 AM, jan matejek wrote:
Well I am not necessarily certain that SLE should be a determining factor here. We should consider when there is a realistic expectation to switch :Factory to Python 3 as the default. We are talking about a development project for openSUSE after all and SLE branches from Tumbleweed.
So we agree :) , but for me this implies that our binary packages should start being named as python2-foo and python3-foo rather sooner than later. If that cannot be handled via macro magic, although I am not certain why it couldn't, then yes we'd end up having to do a renaming exercise, so be it. But I could imagine that we can come up with something like: Name: %{python_version_name}-foo and %ifdef %{is_python_default_version} Provides: python-foo %endif Then python_version_name expands to python2 and python3 respectively on two consecutive builds, thus creating binary packages of python2-foo and python3-foo. As long a Python-2 is the default the python2-foo build creates the Provides: entry in the package and when we switch to Python-3 then the python3 package build will get the provides. So this means we still have to touch every spec file in d:l:p but we do not have to rename the packages per-se. And we will end up having to touch all the .spec files in d:l:p anyway when we move to a new macro set. For packages that are only Python-2 they'd get Name: python2-bar and %ifdef %{is_python_default_version} Provides: python-foo %endif Packages that are only Python-3 get Name: python3-bar and %ifdef %{is_python_default_version} Provides: python-foo %endif If we do want to think about Python-4 then those should also be macros but I am not certain we need to go there.
I don't think this is what anyone is saying. If there is Application:Geo and people want to stick their Python based app there that is great. However they still end up having to link some of d:l:p as it is unlikely that myGeoApp only depends on stuff that is part of Python. Alternately the packager may decide that myGeoApp could/should live in d:l:p to avoid the linking stuff all together. In any event I think this is a topic that is only tangentially related to how we handle integrated builds for python2 and python3 from a single spec file. As we go and touch all the spec files we might run into cases where we might decide that something may be better off having a new home and we can then engage that maintainer, but I am very hesitant to a.) intermingle the two topics (integrated python build and "what should live in d:l:p") and b.) continue to speak in the abstract without having concrete examples where and how these new proposed rules would/could apply. From my perspective we should focus on the prize, which is an integrated build. Everything else is a distraction and we cannot boil the ocean. Co-mingling of topics makes things bigger and may bring us to the point where once again nothing gets done. Thus rather than going down a rat hole I'd rather find consensus on the key topics, which appears to me that we are not really all that far away, and then actually get something done, rather then discuss tangentially related items. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Public Cloud Architect LINUX rjschwei@suse.com IRC: robjo
hello, On 27.10.2016 20:35, Todd Rme wrote:
I would very much prefer to NOT support pypy 2 at all, now that pypy3 is making significant progress. Given that we don't really support pypy2 at this point, I don't see any benefits in starting. That's why my proposal uses unversioned "pypy". That is not going to be future-proof if we ever get pypy 4, but, well, I say we take that risk ;)
I have always considered d:l:py to be a generic place for Python modules and tools -- that is, for software where Python is a fundamental property, as opposed to implementation detail. From your conversation with Simon, it doesn't seem like you disagree on this, but you're concerned about the "edge case" packages -- where it's unclear whether the "library" or the "utility" part is more important. Let me clarify that I don't propose to throw these out summarily. I'm more concerned about the actually-clear-cut apps. I don't know how many of those are in d:l:py now, but I have rejected several in the last year, and I'd like to make it clearer that these don't belong. (Also, being more careful about the edge cases would not hurt, IMHO ;) )
to this I would add, "and if the executable doesn't natively support multiple python versions". For instance, if pylint3 could work on python 2 by itself, I would include it in the rule and say that we will only distribute pylint3. This could very well be the case for IDEs and similar.
It doesn't make sense to me to build a binding for, say, 13.2, for a library that is not present or too old in 13.2. In such case, I think the right solution is to disable build for the older SUSE version -- and optionally link the binding to a project that provides the newer library for the older distribution.
Definitely. That is one of the goals.
Probably not, for technical reasons. The problem is that the macroset can't modify what's already explicitly written in the spec file. So if the spec is for python-foo, and has a %files section, then we MUST build package python-foo that includes the files listed. We could of course rename the package and spec, but, as Christian pointed out, that is a lot of renames for no clear benefit. What makes more sense is to gradually rename packages to python3-foo, and generate python2-foo subpackages (with Provides: python-foo, which could in time move to the python3 subpackage). The macroset is designed with this possibility in mind, although it's probably buggy at the moment. (as for pypy, as I wrote before, i would prefer to never even use versioned names for it)
python2-only, I would consider legacy and keep simple, until such a time when they move to python 3 python3-only, I'd use the macros and "%define have_python2 0". After all, these could work on pypy or other flavors that might be created in the future.
good ideas regards m.
On 10/31/2016 11:28 AM, jan matejek wrote:
Whether or not we support pypy2 is probably not necessarily the issue here. It would just be nice to take the guess work and ambiguity out of the system and stick with versioned names, whether there will be a Python 4 at some distant future or not is really immaterial. You can look at it this way, currently we have stuff with "python" and "python3" and we all know python really means python2. But at some point in the future python3 will become the default, does that mean that python3 turns into python and what is now python turns into python2. If that is the case the confusion and ripples will be big and forever after it is inevitable that people will hit a point where they will have to figure out from scratch by looking at a wiki or other source of information when the transition occured and what the "best" course of action is. All of this can be avoided by simply adding the version number now when we go through a transition anyway and make anything that is not versioned go away. I doubt our fingers will wear off by having to type an extra character all the time ;) Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Public Cloud Architect LINUX rjschwei@suse.com IRC: robjo
On Tuesday 2016-11-01 14:51, Robert Schweikert wrote:
The Python people recognized with the python3 rollout that you cannot have huge incompatible generations anymore without harming the project. So I would like to believe in my vision that it won't be necessary to have multiple concurrent python versions in the future -- just like perl5 has demonstrated for some 10+ years in (open)SUSE. [That's something the Ruby devs should look into as well, hint hint.] -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tue, Nov 1, 2016 at 10:45 AM, Jan Engelhardt <jengelh@inai.de> wrote:
You are still talking about renaming all the python3-foo packages to python4-foo, probably in 4 or 5 years. That is what I am hoping to avoid. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Mon, Oct 31, 2016 at 11:28 AM, jan matejek <jmatejek@suse.com> wrote:
Fair enough, but I still think keeping things consistent would be preferable. So even if there is no "pypy2" commands, I think we should still call iot "pypy3" to maintain the same pattern.
That's why my proposal uses unversioned "pypy". That is not going to be future-proof if we ever get pypy 4, but, well, I say we take that risk ;)
We are getting about 1 python minor version a year, with 3.6 coming out any time now. Guido has said he that python 4 will just be python 3.10 because he doesn't like double-digit versions. This means we are probably about 4 years away from Python 4. So I would include the python 4 transition in any plans we make, it isn't far off at all. In fact it will probably happen about the same time as 2.7 is EOL.
Agreed. My point is that we need to be very clear about this policy. The current python application naming policy is pretty fuzzy and I don't think it handles these "edge" cases well (scare quotes because they are by far the most common case).
Agreed. Although IDEs are a problem because the python shell they support is usually the same as they python version they were built against.
What about, say, a general-purpose numerical library that needs some C library to do some task? What about an update to a key existing program that gets a new indirect C dependency?
Can't you have the macros based off, say "%files -n python%{default_py_ver}-foo" or something like that and have no %files section? I would very much prefer no version number in the .spec file. With python 4 likely very close, I think it would be better to be prepared for this now rather than having to do this all over again before too much longer.
Sound reasonable. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Tue, Nov 1, 2016 at 5:19 PM, Todd Rme <toddrme2178@gmail.com> wrote:
I don't think a blanket ban on non-python packages will work. I think we need to look at how the packages are used and what needs them on a case-by-case basis. I think there are a variety of scenarios that need different approaches. The first is that the package is unused by anything, or where the version currently available from the openSUSE version is sufficient. Of course this should be deleted outright (after a one-week warning period if they aren't linked to somewhere else). The second is that a package is so fundamental that using newer versions than the openSUSE version was built against has a high chance of breakage. These packages should never be allowed. The third is that a package is niche, but the python package depends on an executable. These packages should alo never be allowed. The fourth are niche packages used by only bindings for a niche python package, or are extremely subject-specific. These should be moved to the appropriate repository along with their python bindings. If the library is in Factory the bindings can be linked, but only where newer versions of the library is not needed. The fifth packages with a low chance of conflicts whose library is required by very important python packages. In cases where a newer version is needed, these should have a minimal version copied (not linked) into d:l:p, have the -devel package be renamed from "foo-devel" to "foo-sonum-devel" (where "sonum" is the so number), and have the OSC package name renamed from "foo" to "foo-sonom-osver" (where osver is the openSUSE version). The last are libraries that are essentially only used by python packages that are not part of the original package. These are very rare. They should be owned by d:l:p. Here is the list of non-python packages along with my assessment of whether they should stay or go. I won't distinguish between d:l:p and d:l:p3 because it won't matter soon. These are all highly debatable (except probably the unused ones, although I may have made mistakes with them), this is just my own opinion. Also, note that I am not addressing python-based applications here, that is a separate issue. apparently unused (or a dependency of an apparently unused package) gmsh mongodb openblas petsc tetgen cgns - apparently used only by gmsh blacs - apparently used only by petsc hypre - apparently used only by petsc scalapack - apparently used only by petsc newer version not useful hdf netcdf-cxx qwt fundamental packages with a high chance of breakage: GeoIP agg cmake doxygen freeimage lapack libgit2 libsmi lz4 mercurial opencv-qt5 openpgm snappy vtk vtkdata xrootd zeromq swig3 - this has be renamed to just "swig", packages shouldn't use it anymore low-risk packages that can be copied if needed cfitsio fftw3 geos hdf5 hdf5-1_8 libcryptopp libsodium netcdf netcdf-cxx4 parallel-netcdf portmidi qhull qscintilla qscintilla-qt5 suitesparse used only by python packages, should be owned by d:l:p blosc DSDP should by be in a subject-specific repo (moved in necessary) gccxml liblouis libsvm meep pythonocc rrdtool sundials python/3-PyAVM python/3-pyMinuit2 mhash - used by meep harminv - used by meep libctl - used by meep oce - used by pythonocc geom - used by pythonocc smesh - used by pythonocc erfa - used by python/3-PyAVM wcslib - used by python/3-PyAVM python/3-meep - bindings for meep python/3-py-rrdtool - bindings for rrdtool python/3-pygccxml - bindings for gccxml python/3-pysundials - bindings for sundials -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 8.11.2016 16:40, Todd Rme wrote:
Thank you, this (and the list of packages) is wonderful work. I'm about to review and delete and/or file deleterequests for most packages on your list in the first three sections
This doesn't look like a good idea. It is going to introduce dependencies on "non-Factory" symbols, or complicate spec files (%if suse_version == 1320; Requires: foo-1320-devel; %else; Requires: foo-devel; %endif) Renaming also does not help if there are file conflicts between the renamed and the distro-provided version. Instead of renaming, I'd keep the same name and only enable build for distributions that need them. (as discussed earlier) And, of course, never enable for Factory, instead push the required version.
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Mon, Oct 31, 2016 at 11:28 AM, jan matejek <jmatejek@suse.com> wrote:
Another issue: might this be a good time to drop support for SLES 11.4? It is the only version that doesn't support python 3, and many packages require some significant workaround to keep building. I think it is probably past due to drop support for it, or at least to disable all builds and freeze the repo. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 11/03/2016 06:38 AM, Todd Rme wrote:
As general info SLED is out of support and SLES will have General support until 31 Mar 2019 and LTSS support until 31 Mar 2022 which is still quite some time. I don't have a strong opinion here but dropping it seems reasonable to me unless people indicate they are using it and are willing to do the work. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adeliade Australia, UTC+9:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
On 10/27/2016 12:51 PM, jan matejek wrote:
Agreed, anything that is prjconfig related should apply to the Build Service as a whole. Python code is really spread all over and having to tell all project maintainers where ther might be a snippet of Python that the prjconfig has to change and then making sure that any new project that springs up gets the same treatment is too error prone. Later, Robert -- Robert Schweikert MAY THE SOURCE BE WITH YOU Public Cloud Architect LINUX rjschwei@suse.com IRC: robjo
On Thu, Oct 27, 2016 at 12:51 PM, jan matejek <jmatejek@suse.com> wrote:
Another thing that occurred to me is that I think it would be really handy to have a pypi service that would automatically download the latest version of the package from pypi. Of course the service would need to be manually triggered and we would need to check whether the build succeeds, but I think it would be handy for the repo. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (8)
-
Christian Boltz -
Greg Freemyer -
Jan Engelhardt -
jan matejek -
Robert Schweikert -
Sebastian -
Simon Lees -
Todd Rme