[opensuse-packaging] Automatic Python Provides
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This apparently originally came from Mandriva and Mageia, according to the mailing list discussion [2]. They are also adding macros to make these easier to use, so you can use the macros to depend on the right provides automatically. This seems like something that would make it much easier to handle python packages in a distro-agnostic manner. The downside is it requires rebuilding all python packages to work, which may limit its applicably to previous openSUSE/SLE releases. What does everything think about implementing this on our end as well? I am not exactly clear from the discussion what exactly it would entail. [1] https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Pac... [2] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje... -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
hello, On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories in site-packages directory. It would be beneficial to have both: the canonical name from pypi, what fedora is doing now, and also provides for imports -- even though the latter would never be perfect. Some RPM person please speak up? m.
[1] https://fedoraproject.org/wiki/Changes/Automatic_Provides_for_Python_RPM_Pac... [2] https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproje...
jan matejek <jmatejek@suse.com> wrote:
hello,
On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
+1 ! We already do this for Ruby: https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_aut... (Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has changed at least once, and I am not sure if that doc was correctly updated at the same time. But hopefully it's good enough to give you an idea how it works.)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories
Yes, this was newly completed in py2pack by Tom: https://github.com/openSUSE/py2pack/pull/45 The next step would be to make this happen automatically at build-time via autoreqprov and macros, instead of at .spec-editing-time.
in site-packages directory.
I don't think site-packages is the right source for this metadata. It should come from the setup.py infrastructure (which may or may not source the dependencies from somewhere like requires.txt).
It would be beneficial to have both: the canonical name from pypi, what fedora is doing now, and also provides for imports -- even though the latter would never be perfect.
I guess the most important part is having the correct dependency names, which means using the pypi names. I'm not sure whether providing virtual packages for import namespace names is a good idea.
Some RPM person please speak up?
Hope the above info helps. I'm sure that any contributions to improve this will be very welcome. Maybe it could be done by extending py2pack somehow so that it gets called in a special mode at build-time to extract the metadata - Tom, what do you think? BTW there is a new github team: https://github.com/orgs/openSUSE/teams/python-packaging Anyone who works on this should probably be added to that team. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
jan matejek <jmatejek@suse.com> wrote:
hello,
On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
+1 !
We already do this for Ruby:
https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_aut...
(Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has changed at least once, and I am not sure if that doc was correctly updated at the same time. But hopefully it's good enough to give you an idea how it works.)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories
Where can I find the autoreqprov code?
Yes, this was newly completed in py2pack by Tom:
https://github.com/openSUSE/py2pack/pull/45
The next step would be to make this happen automatically at build-time via autoreqprov and macros, instead of at .spec-editing-time.
in site-packages directory.
I don't think site-packages is the right source for this metadata. It should come from the setup.py infrastructure (which may or may not source the dependencies from somewhere like requires.txt).
It would be beneficial to have both: the canonical name from pypi, what fedora is doing now, and also provides for imports -- even though the latter would never be perfect.
I guess the most important part is having the correct dependency names, which means using the pypi names. I'm not sure whether providing virtual packages for import namespace names is a good idea.
Some RPM person please speak up?
Hope the above info helps. I'm sure that any contributions to improve this will be very welcome. Maybe it could be done by extending py2pack somehow so that it gets called in a special mode at build-time to extract the metadata - Tom, what do you think?
I would like to have something like gem2rpm for python. This could be py2pack. It would generate a .spec file from a given pypi tarball. The process to generate the spec could be tweaked via a config file (similar to gem2prm). Is that what you have in mind? Best, Tom -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Thomas Bechtold <tbechtold@suse.com> wrote:
On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
jan matejek <jmatejek@suse.com> wrote:
On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
+1 !
We already do this for Ruby:
https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_aut...
(Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has changed at least once, and I am not sure if that doc was correctly updated at the same time. But hopefully it's good enough to give you an idea how it works.)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories
Where can I find the autoreqprov code?
Please see the above link :)
Yes, this was newly completed in py2pack by Tom:
https://github.com/openSUSE/py2pack/pull/45
The next step would be to make this happen automatically at build-time via autoreqprov and macros, instead of at .spec-editing-time.
in site-packages directory.
I don't think site-packages is the right source for this metadata. It should come from the setup.py infrastructure (which may or may not source the dependencies from somewhere like requires.txt).
It would be beneficial to have both: the canonical name from pypi, what fedora is doing now, and also provides for imports -- even though the latter would never be perfect.
I guess the most important part is having the correct dependency names, which means using the pypi names. I'm not sure whether providing virtual packages for import namespace names is a good idea.
Some RPM person please speak up?
Hope the above info helps. I'm sure that any contributions to improve this will be very welcome. Maybe it could be done by extending py2pack somehow so that it gets called in a special mode at build-time to extract the metadata - Tom, what do you think?
I would like to have something like gem2rpm for python. This could be py2pack.
I think py2pack already *is* the gem2rpm equivalent for Python.
It would generate a .spec file from a given pypi tarball.
Uhh ... isn't that what it does already? Or am I going senile?
The process to generate the spec could be tweaked via a config file (similar to gem2prm). Is that what you have in mind?
Well, there's certainly no harm in stealing ideas for features from gem2rpm. But I thought the main point of this thread was to calculate the Requires and Provides at build-time, so that py2pack doesn't need to put them in the .spec file at all? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Mon, Aug 22, 2016 at 02:13:14PM +0100, Adam Spiers wrote:
Thomas Bechtold <tbechtold@suse.com> wrote:
On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
jan matejek <jmatejek@suse.com> wrote:
On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
+1 !
We already do this for Ruby:
https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_aut...
(Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has changed at least once, and I am not sure if that doc was correctly updated at the same time. But hopefully it's good enough to give you an idea how it works.)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories
Where can I find the autoreqprov code?
Please see the above link :)
Yes, this was newly completed in py2pack by Tom:
https://github.com/openSUSE/py2pack/pull/45
The next step would be to make this happen automatically at build-time via autoreqprov and macros, instead of at .spec-editing-time.
in site-packages directory.
I don't think site-packages is the right source for this metadata. It should come from the setup.py infrastructure (which may or may not source the dependencies from somewhere like requires.txt).
It would be beneficial to have both: the canonical name from pypi, what fedora is doing now, and also provides for imports -- even though the latter would never be perfect.
I guess the most important part is having the correct dependency names, which means using the pypi names. I'm not sure whether providing virtual packages for import namespace names is a good idea.
Some RPM person please speak up?
Hope the above info helps. I'm sure that any contributions to improve this will be very welcome. Maybe it could be done by extending py2pack somehow so that it gets called in a special mode at build-time to extract the metadata - Tom, what do you think?
I would like to have something like gem2rpm for python. This could be py2pack.
I think py2pack already *is* the gem2rpm equivalent for Python.
Except that you usually need to adjust the generated spec file by hand (i.e. update-alternative handling)
It would generate a .spec file from a given pypi tarball.
Uhh ... isn't that what it does already? Or am I going senile?
Yes. But as I said, the spec needs to be adjusted by hand usually. You can't just call py2pack for a new upstream release. It's currently only useful for the initial convertion from pypi to a spec.
The process to generate the spec could be tweaked via a config file (similar to gem2prm). Is that what you have in mind?
Well, there's certainly no harm in stealing ideas for features from gem2rpm. But I thought the main point of this thread was to calculate the Requires and Provides at build-time, so that py2pack doesn't need to put them in the .spec file at all?
That's the question I guess. Do we want to have the Requires/BuildRequires explicit in the spec (added by a py2pack rerun whenever you update the spec) or implicit via a macro (which may also run py2pack to extract the Requires/BuildRequires). Or am I missunderstanding something? Best, Tom -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Thomas Bechtold <tbechtold@suse.com> wrote:
On Mon, Aug 22, 2016 at 02:13:14PM +0100, Adam Spiers wrote:
Thomas Bechtold <tbechtold@suse.com> wrote:
I would like to have something like gem2rpm for python. This could be py2pack.
I think py2pack already *is* the gem2rpm equivalent for Python.
Except that you usually need to adjust the generated spec file by hand (i.e. update-alternative handling)
Last time I looked, you did for gem2rpm too. Maybe that's changed more recently?
It would generate a .spec file from a given pypi tarball.
Uhh ... isn't that what it does already? Or am I going senile?
Yes. But as I said, the spec needs to be adjusted by hand usually. You can't just call py2pack for a new upstream release. It's currently only useful for the initial convertion from pypi to a spec.
That's certainly also how gem2rpm started out, at least.
The process to generate the spec could be tweaked via a config file (similar to gem2prm). Is that what you have in mind?
Well, there's certainly no harm in stealing ideas for features from gem2rpm. But I thought the main point of this thread was to calculate the Requires and Provides at build-time, so that py2pack doesn't need to put them in the .spec file at all?
That's the question I guess. Do we want to have the Requires/BuildRequires explicit in the spec (added by a py2pack rerun whenever you update the spec) or implicit via a macro (which may also run py2pack to extract the Requires/BuildRequires).
Implicit via a macro! Why have extra mess in the .spec if we can avoid it? As per the OP, Fedora is doing it via macros, and we already do it via macros for Ruby, so I can't think why we wouldn't want to do this. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
jan matejek <jmatejek@suse.com> wrote:
hello,
On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
+1 !
We already do this for Ruby:
https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_aut...
(Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has changed at least once, and I am not sure if that doc was correctly updated at the same time. But hopefully it's good enough to give you an idea how it works.)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories
Yes, this was newly completed in py2pack by Tom:
I plan to remove this from py2pack again because I split the code to a new tool called metaextract (https://pypi.python.org/pypi/metaextract) I plan to use that also in another project (OpenStack packaging related)
The next step would be to make this happen automatically at build-time via autoreqprov and macros, instead of at .spec-editing-time.
in site-packages directory.
I don't think site-packages is the right source for this metadata. It should come from the setup.py infrastructure (which may or may not source the dependencies from somewhere like requires.txt).
It would be beneficial to have both: the canonical name from pypi, what fedora is doing now, and also provides for imports -- even though the latter would never be perfect.
I guess the most important part is having the correct dependency names, which means using the pypi names. I'm not sure whether providing virtual packages for import namespace names is a good idea.
Some RPM person please speak up?
Hope the above info helps. I'm sure that any contributions to improve this will be very welcome. Maybe it could be done by extending py2pack somehow so that it gets called in a special mode at build-time to extract the metadata - Tom, what do you think?
BTW there is a new github team:
https://github.com/orgs/openSUSE/teams/python-packaging
Anyone who works on this should probably be added to that team. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Thomas Bechtold <tbechtold@suse.com> wrote:
On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
jan matejek <jmatejek@suse.com> wrote:
hello,
On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
+1 !
We already do this for Ruby:
https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_aut...
(Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has changed at least once, and I am not sure if that doc was correctly updated at the same time. But hopefully it's good enough to give you an idea how it works.)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories
Yes, this was newly completed in py2pack by Tom:
I plan to remove this from py2pack again because I split the code to a new tool called metaextract (https://pypi.python.org/pypi/metaextract) I plan to use that also in another project (OpenStack packaging related)
Splitting it out sounds good. But just to be clear - you aren't removing the *functionality* from py2pack, right? I mean, py2pack will still use metaextract to retain the metadata extraction functionality it already has, rather than regressing to the unreliable mechanism of parsing setup.py? -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Fri, Aug 26, 2016 at 11:20:35AM +0100, Adam Spiers wrote:
Thomas Bechtold <tbechtold@suse.com> wrote:
On Mon, Aug 22, 2016 at 10:51:01AM +0100, Adam Spiers wrote:
jan matejek <jmatejek@suse.com> wrote:
hello,
On 17.8.2016 18:09, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This (...) What does everything think about implementing this on our end as well?
yes, let's please do this. :)
+1 !
We already do this for Ruby:
https://en.opensuse.org/openSUSE:Packaging_Ruby#How_gem_dependencies_are_aut...
(Caveat: since I wrote that doc, the Ruby autoreqprov mechanism has changed at least once, and I am not sure if that doc was correctly updated at the same time. But hopefully it's good enough to give you an idea how it works.)
I am not exactly clear from the discussion what exactly it would entail.
We'd need to extend the autoreqprov generator in our RPM. There is already some python-related code, IIRC, something that can generate provides from files/directories
Yes, this was newly completed in py2pack by Tom:
I plan to remove this from py2pack again because I split the code to a new tool called metaextract (https://pypi.python.org/pypi/metaextract) I plan to use that also in another project (OpenStack packaging related)
Splitting it out sounds good. But just to be clear - you aren't removing the *functionality* from py2pack, right? I mean, py2pack will still use metaextract to retain the metadata extraction functionality it already has, rather than regressing to the unreliable mechanism of parsing setup.py?
Yes. Just the code moved to an external module. Sorry for not being precise here :-) Best, Tom -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 08/17/2016 10:09 AM, Todd Rme wrote:
Fedora is implementing a system to automatically add reliable, distro-agnostic provides to their python packages [1]. This apparently originally came from Mandriva and Mageia, according to the mailing list discussion [2]. They are also adding macros to make these easier to use, so you can use the macros to depend on the right provides automatically.
This seems like something that would make it much easier to handle python packages in a distro-agnostic manner. The downside is it requires rebuilding all python packages to work, which may limit its applicably to previous openSUSE/SLE releases.
What does everything think about implementing this on our end as well? I am not exactly clear from the discussion what exactly it would entail.
I am supportive of the idea, but with a caveat. As I understand from the first link provided, they are using a "canonical name" in the "Provides" tag. This canonical name "can differ slightly from the name displayed, for example, on PyPI". If this is the case, we would be setting ourselves up for a situation where at some point (if not already) two packages made for different PyPi distributions end up having the same Provides tag. PyPi is the de facto standard for Python distributions, and beyond that PEP 426 specifically defines what a metadata "name" for a distribution is. To go against PyPi and PEP 426 would be, IMO incorrect. The PEP 426 name can only be letters, digits, underscores, hyphens and periods and must start and end with a letter or digit. Can the Provides tag not include such a token? -- Jason Craig -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (5)
-
Adam Spiers -
jan matejek -
Jason Craig -
Thomas Bechtold -
Todd Rme