Todd Rme píše v St 03. 04. 2019 v 11:57 -0400:

The naming policy from a long time now is that all jupyter-related packages, which pretty much by definition includes ipython-related packages since ipython is part of the jupyter software ecoystem, have used the jupyter_ prefix. This was discussed on the mailing list [1] in 2015 and using this approach was the consensus. Further, since the split it has been the policy that all such packages go into devel:languages:python:jupyter.

Recently there was a move to rename one such package, python-jupyter_ipdb, to just python-ipdb [2], on the grounds that we "we should not rename upstream stuff", and further the new version was pushed into devel:languages:python instead of devel:languages:python:jupyter.

Yeah I should've sent it to :jupyter, my bad there, but easy&quick to amend.

This is a massive change in both python package naming policy and repository polices, which if implemented consistently would require renaming and moving dozens of packages. It would also lead to a lot of ambiguity since whether a package is really an ipython package or a jupyter package is ambiguous and can change over time, which is one reason we don't try to draw the distinction right now.

Considering how many packages this would affect, I would request we discuss it on this mailing list beforehand rather than unilaterally moving and renaming individual packages. I personally think the policies works well, and similar policies are used effectively for a number of other python package families (like django).

[1] https://lists.opensuse.org/opensuse-packaging/2015-08/msg00065.html [2] https://build.opensuse.org/request/show/690081

I actually didn't notice this thread before and I can understand the motivation behind prefixing all the packages as jupyter- for nice namespace, but it has serious drawbacks as I said in the other email. 1) differs from upstream 2) differs from other distributions The above two make it hard for any newcomer to figure out what is the proper name to require, heck I added the ipdb because I was not able to find it. If we would want to have the prefix like with django, there it is all in upstream names so we should coordinate with upstream developers to consider renaming if it is so important for us. 3) tooling can't decipher it This is fixable, yet as of now all the other package receive nice update notifications and difference warnings. Another thing for IPython/Jupyter we could provide is additional pattern that would install all the namespace packages that are deemed good idea to have to avoid having the jupyter prefix as a helper to determine what the thing is about. Cheers Tom

On 2019-04-04, 09:15 GMT, you wrote:

The above two make it hard for any newcomer to figure out what is the proper name to require, heck I added the ipdb because I was not able to find it.

Besides we have very pleasant convention than one can use %{python_module PyPI-name} whenever we need new BuildRequirement.

Another thing for IPython/Jupyter we could provide is additional pattern that would install all the namespace packages that are deemed good idea to have to avoid having the jupyter prefix as a helper to determine what the thing is about.

Well, do we need patterns here? Just installing python3-jupyter_notebook or python3-jupyter_ipython and all necessary packages should be Required or Suggested/Recommended, right? Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl@ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 If we rise from prayer better persons, our prayers have been answered. -- a Jewish prayer book -- To unsubscribe, e-mail: opensuse-python+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-python+owner@opensuse.org

Todd Rme píše v Čt 04. 04. 2019 v 13:49 -0400:

On Thu, Apr 4, 2019 at 5:15 AM Tomas Chvatal wrote:

...

Todd Rme píše v St 03. 04. 2019 v 11:57 -0400:

...

The naming policy from a long time now is that all jupyter- related packages, which pretty much by definition includes ipython- related packages since ipython is part of the jupyter software ecoystem, have used the jupyter_ prefix. This was discussed on the mailing list [1] in 2015 and using this approach was the consensus. Further, since the split it has been the policy that all such packages go into devel:languages:python:jupyter.

Recently there was a move to rename one such package, python-jupyter_ipdb, to just python-ipdb [2], on the grounds that we "we should not rename upstream stuff", and further the new version was pushed into devel:languages:python instead of devel:languages:python:jupyter.

Yeah I should've sent it to :jupyter, my bad there, but easy&quick to amend.

...

This is a massive change in both python package naming policy and repository polices, which if implemented consistently would require renaming and moving dozens of packages. It would also lead to a lot of ambiguity since whether a package is really an ipython package or a jupyter package is ambiguous and can change over time, which is one reason we don't try to draw the distinction right now.

Considering how many packages this would affect, I would request we discuss it on this mailing list beforehand rather than unilaterally moving and renaming individual packages. I personally think the policies works well, and similar policies are used effectively for a number of other python package families (like django).

[1] https://lists.opensuse.org/opensuse-packaging/2015-08/msg00065.html [2] https://build.opensuse.org/request/show/690081

I actually didn't notice this thread before and I can understand the motivation behind prefixing all the packages as jupyter- for nice namespace, but it has serious drawbacks as I said in the other email.

First, this seems like the sort of thing that should be discussed on the openSUSE python mailing list, or at the very least with the jupyter project maintainer, before changing it. The first I heard that anyone had a problem with the current policy was when I got a delete request for a package. If the consensus is to change it, that is fine, but is the sort of thing that should go through the proper process first, or at the very least be brought to the attention of the person in charge of those packages.

Well I didn't notice the package at all until I got conflict reply from staging bots where I simply deleted the spec that does not adhere to the naming policy. Again you happen to have discussion on packaging ML in 2015 which was never converted to any documentation on the wiki, so unless I would scrape the mailinglist I couldn't have known and simply followed the guidelines at https://en.opensuse.org/openSUSE:Packaging_Python#Naming_policy (note this was not written by me during the singlespec conversion, this policy is there for god knows how long and it does not say anything about the jupyter).

...

1) differs from upstream

To be pedantic, we do this with most python packages already by adding a python- prefix. We very rarely use the upstream package name verbatim, unlike something like Anaconda.

Well the documentation is on my side to be anal about this. We say all python modules should be python-<PYPI>.

I think thinking about why we do this is worthwhile. At least from my way of thinking about it, it is because these packages are not useful outside of a python context. If you install numpy or django, you aren't going to be able to use it outside of a python interpreter. Python is not just a dependency, these packages will not do anything without being run inside a python interpreter. Putting the python- prefix clearly shows that.

The same, however, is true of jupyter packages. Jupyter packages are not useful outside of a jupyter context. If you try to run them in a normal python interpreter they won't work. In many cases, the fact that they are even written in python is just an implementation detail (just like the fact that the "python" package is written in C is an implementation detail). This is different than essentially every other "python-foo" package. You can run a "python-django-foo" package in a normal python interpeter. django is just a dependency. So if we are talking about renames, I would be more included to drop the "python" prefix than the "jupyter" one. Given how the ecosystem has evolved, calling them "python" packages can be outright wrong.

Let's take "rise" as an example. Its purpose is to make presentations using Jupyter notebooks. It is useless in a normal python interpreter. It has no user-facing python interface. It has 5 times more javascript code than python code, and the python code it does have is just for installation and loading the javascript code. So I think calling it "python-rise" would be downright misleading, since it really has nothing to do with python.

Similarly, "octave_kernel" allows you to run octave code in a jupyter notebook or interpeter. It is written in a mix of python and octave, but that it uses python is an implementation detail. You can't use it from a normal python interpreter. So I think calling it "python-octave_kernel" would be misleading since it has nothing to do with python.

After thinking this through I am personally coming down pretty strongly in favor of dropping the "python-" part, and adding a "jupyter-" part with rules otherwise the same as the python rules. I think the "python-" part has become misleading over the past 4 years.

I think the python- prefixing still applies sadly, as for some of our pypi upstreams the names are too generic without it, packages like daemon, configuration, etc would not go really well as a regular names. With the jupyter you can indeed point out parts where it does not make much sense to call them with the python prefix (like the additional js files for slides) otoh the package that made you unhappy is IPDB which is ipython debugger that works directly from python where i can just do import ipdb and do work with it === import ipdb ipdb.set_trace() ipdb.run(<PYTHONCODE>) === In general I guess none will care how the packages in jupyter are called as long as they can't be imported in python, but those that are importable and can/might be used by other depending packages it will not be easy to find the package given the non-standard name and people will try to introduce them again.

...

2) differs from other distributions

Again, to be pedantic, distributions all handle python packages differently. Some use the pypi name, some use the egg name, some use the module name, some use python-, others use python2-, etc. openSUSE itself has varied in this regard over time. I am more concerned with making things work well for users.

Yet all of them when searching for the jupyter_ stuff do not prefix this stuff and call it python-<PYPI name> for both Fedora and Debian, gentoo just goes with dev-python/<PYPI name>. Thus no guide ever written fro those two will be applicable on openSUSE and we would have to be treated as special snowflake.

...

The above two make it hard for any newcomer to figure out what is the proper name to require, heck I added the ipdb because I was not able to find it.

In my defense, my expectation at the time was that all of these issues were going to be solved in the near future by having automatic python package provides. At the time it was supposed to be just around the corner, but as far as I am aware it never really panned out.

Yea without this in place in my view the wiki policy still wins usability wise. The least the jupyter stuff should do is provide the pypi names as symbols for others to seach and install and the wiki should be updated to mention the rationale around this different naming. (Also all the tooling needs to be adapted to take this into account, which I wonder if it is time spent well for just ~60 pkgs). Cheers Tom

On Fri, Apr 5, 2019 at 8:11 AM Matěj Cepl wrote:

On 2019-04-05, 09:19 GMT, you wrote:

...

...

The same, however, is true of jupyter packages. Jupyter packages are not useful outside of a jupyter context. If you try to run them in a normal python interpreter they won't work. In many cases, the fact that they are even written in python is just an implementation detail (just like the fact that the "python" package is written in C is an implementation detail). This is different than essentially every other "python-foo" package. You can run a "python-django-foo" package in a normal python interpeter. django is just a dependency. So if we are talking about renames, I would be more included to drop the "python" prefix than the "jupyter" one. Given how the ecosystem has evolved, calling them "python" packages can be outright wrong.

a) I don't think deep arcane philosophical distictions on The True Nature™ of packages matter. It is more important that all those packages are Python enough to use python-rpm-macros macros, and use similar technology (BTW, if there are some Jupyter-specific macros which you would like to standardize upon, feel free to suggest them or add them to python-rpm-macros).

It isn't about philosophy, it is about broadcasting to users what the package is all about. There are lots of packages that use python-rpm-macros but don't use the "python-" prefix because those packages are meant to be used from the command line or as GUI applications rather than being used from the Python interpreter. We have a specific exception for such packages in the python naming rules. At the very least, I think jupyter packages that aren't meant to be used from the python interpreter should be fall under the same exception, for the same reason. The reason being, that calling that "python-" packages is confusing and misleading.

b) The line between Jypyter and non-Jupyter packages is really murky. https://pypi.org/project/ipdb/ (and related packages, https://pypi.org/search/?q=ipdb ) is one clear example of package which is widely used outside of the Jupyter universe.

If that is the case, then the exception wouldn't apply. We would have to make a judgement call. The line is also murky with the command-line tool exception since a lot of "python-" packages also provide command-line tools. In these situations we also have to make judgement calls.

However, when going through d:l:p:jupyter packages I can see other ones which are at least suspiciously non-Jypter-specific (and they even break your own naming policy) python-testpath, python-traitlets, python-traittypes, and perhaps some others.

Note that none of those have the "jupyter_" prefix. They are part of the devel:languages:python:jupyter project because they are created by the ipython/jupyter maintainers for the primary purpose of being used by ipython/jupyter and are developed under one of the ipython/jupyter github projects. But they are normal python packages so they don't have the prefix.

...

...

After thinking this through I am personally coming down pretty strongly in favor of dropping the "python-" part, and adding a "jupyter-" part with rules otherwise the same as the python rules. I think the "python-" part has become misleading over the past 4 years.

That's another possible solution, to move all those packages to let's say science:jupyter project or something. However, I don't think it is wise. It really doesn't resolve problem with packages which are truly Python packages and which should still follow Python packaging policy (including names).

I would probably keep everything in the devel:languages:python:jupyter project to simplify the packaging workflow. Again comparing to command-line tools without the "python-" prefix, we still have a bunch of those in the various devel:languages:python projects.

...

In general I guess none will care how the packages in jupyter are called as long as they can't be imported in python, but those that are importable and can/might be used by other depending packages it will not be easy to find the package given the non-standard name and people will try to introduce them again.

This.

...

Yea without this in place in my view the wiki policy still wins usability wise. The least the jupyter stuff should do is provide the pypi names as symbols for others to seach and install and the wiki should be updated to mention the rationale around this different naming. (Also all the tooling needs to be adapted to take this into account, which I wonder if it is time spent well for just ~60 pkgs).

I agree. So,

1. I don't think there is any problem when non-Python (and not python- named, e.g., https://github.com/google/evcxr/tree/master/evcxr_jupyter or https://irkernel.github.io/; BTW, it is a pitty we don't have the latter packaged, or https://github.com/JuliaLang/IJulia.jl) packages are in d:l:p:jupyter (or if the whole thing moves to science:jupyter), it doesn't matter that much. But python packages (i.e., using %python macros and especially importable by other python packages) should still follow Python packaging policy.

I would use the same standard we currently use for command-line tools. That is, if they "are not designed and planned to be used as [python] libraries", then they should be called with the jupyter- prefix. If they are, they should be called with the python- prefix. Note that this is different than right now. As you both pointed out, there are some packages that are more properly python packages. Those would be changed to follow normal python naming rules.

2. All this should be documented. I don't care if you add a paragraph or two to https://en.opensuse.org/openSUSE:Packaging_Python or you make your own https://en.opensuse.org/openSUSE:Packaging_Jupyter (referred by the former), but it should be documented somewhere.

Fair enough. I would add the following paragraph at the end of the naming policy section: Similarly, this policy doesn't apply to jupyter kernels, extensions, and similar packages that are not designed and planned to be used as libraries. If they are meant to be used exclusively within the confines of a jupyter environment, they should be called "jupyter-modulename" instead of "python-modulename", where "modulename" is still the the name of this module on the Python Package Index. Such packages must also provide the "python-modulename" package if they contain python code.

3. Tooling must be adapted, and I really don't think doing it for 60+ packages makes sense. I would rather help with renaming all those python-jupyter_ packages to python- ones.

In terms of python rpm macros, this shouldn't be a huge issue since jupyter-related packages should really be python3-only. If it is python interpreter-dependent in normal usage then it would be used as a python library, which means it should be packaged as a "python-" package. And the packages would follow the same "{prefix}-{pypi name}" pattern as normal python packages. So what would need to change is that the pypi update detection script would need to strip both the "python-" and "jupyter-" prefixes. Out of curiosity, how does the update detection script deal with command-line applications, or does it? As for the number, pypi lists 427 packages that fall under either the jupyter or ipython classifier, and 1,942 packages just from searching for "jupyter". And these numbers are growing fast. And as you pointed out, there are more elsewhere that aren't distributed through pypi. So although not all of these would fall under my proposed policy change, we are still talking about a potentially large chunk of packages. That we haven't had a chance to package a huge number of them yet is just due to a lack of manpower. And I do think a "jupyter" pattern would be worthwhile. There are a lot of packages that either are meant primarily to be used in jupyter while still being usable in python, and lots of packages that provide additional functionality when used in a jupyter environment. -- To unsubscribe, e-mail: opensuse-python+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-python+owner@opensuse.org

On Wed, Apr 10, 2019 at 8:09 AM Matej Cepl wrote:

On 2019-04-08, 16:41 GMT, you wrote:

...

At the very least, I think jupyter packages that aren't meant to be used from the python interpreter should be fall under the same exception, for the same reason. The reason being, that calling that "python-" packages is confusing and misleading.

Hmm, there is a problem of consistency: package python-jupyter_calysto comes from the PyPI package calysto, but python-jupyter_highlight_selected_word from the PyPI package jupyter_highlight_selected_word.

I think it would be really better to be consistent, get rid of the superfluous jupyters from the package names and enjoy the compatibility with the rest of packages and ability to use standard packaging tools. I think there is sufficient signal to developers (users are not that important as they will just consume package as dependencies from the main Jupyter package), there is a sufficient signal to developers that package belong to the Jupyter universe by being located in d:l:p:jupyter project.

Yes, as I said later, the consistency issues would need to be fixed.

...

Fair enough. I would add the following paragraph at the end of the naming policy section:

Similarly, this policy doesn't apply to jupyter kernels, extensions, and similar packages that are not designed and planned to be used as libraries. If they are meant to be used exclusively within the confines of a jupyter environment, they should be called "jupyter-modulename" instead of "python-modulename", where "modulename" is still the the name of this module on the Python Package Index. Such packages must also provide the "python-modulename" package if they contain python code.

Looks good.

...

In terms of python rpm macros, this shouldn't be a huge issue since jupyter-related packages should really be python3-only.

Python macros are useful not only because of Py2k/Py3k dual packaging, I use them even for packages which have %define skip_python2 1 and are thus python3-only.

My point is that there wouldn't need to be any changes to the macros to support this.

...

So what would need to change is that the pypi update detection script would need to strip both the "python-" and "jupyter-" prefixes.

That would break jupyter_highlight_selected_word.

That would be packaged as "jupyter-jupyter_highlight_selected_word", so it shouldn't break.

...

Out of curiosity, how does the update detection script deal with command-line applications, or does it?

Badly (patches to https://gitlab.com/mcepl/dlp_check_version_PyPI are more than welcome), we just strip python- part and then have a list of exceptions where we know we should skip the package. Not good.

Overall, I think I spent on the issue jupyter packages more than I want, so bascially do whatever you think is right (and document it).

Okay, fair enough. -- To unsubscribe, e-mail: opensuse-python+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-python+owner@opensuse.org