Hi,
I would like to raise attention to the recent refactoring of python-python-subunit. the pypi name of that is still "python-subunit", and there are a number of things depending on the pypi name to be correct, so now again all of our python stack is completely unresolveable.
you can check that this is correct here:
grep name= subunit-1.3.0/setup.py
name='python-subunit',
this broke just recently with the drop of python-python-subunit from Factory, which however still existed in devel:languages:python, so we had a weird mixmatch of things.
so, I would like to understand why we were renaming the package away from a standard name to a one-off naming, and not even provide a compatibility provide to at least fix all the rpm unresolvables introduced by this. Is there some magic new policy that I'm not aware of? I wasn't able to spot a reason for taht in the changes file.
Also, since this broke all of the OpenStack CI and rendered about 700 packages completely unbuildable, I would have really liked to see this act of renaming discussed before it was simply done. Could we somehow try to coordinate such major things without pulling the rug under everyone else while they're travelling from/to susecon?
Thanks, Dirk
Dirk Müller píše v So 06. 04. 2019 v 11:41 -0500:
Hi,
I would like to raise attention to the recent refactoring of python- python-subunit. the pypi name of that is still "python-subunit", and there are a number of things depending on the pypi name to be correct, so now again all of our python stack is completely unresolveable.
We were matching up what other distros do to track it from the main subunit package as they needed to be all shipped at once: https://tracker.debian.org/pkg/subunit https://src.fedoraproject.org/cgit/rpms/subunit.git/tree/
As otherwise the only workable version was 1.2.
you can check that this is correct here:
grep name= subunit-1.3.0/setup.py
name='python-subunit',this broke just recently with the drop of python-python-subunit from Factory, which however still existed in devel:languages:python, so we had a weird mixmatch of things.
so, I would like to understand why we were renaming the package away from a standard name to a one-off naming, and not even provide a compatibility provide to at least fix all the rpm unresolvables introduced by this. Is there some magic new policy that I'm not aware of? I wasn't able to spot a reason for taht in the changes file.
There are still the provides for python2-bla and python3-bla it seems we accidentally missed to provide also python-bla to keep it working on Leaps.
Also, since this broke all of the OpenStack CI and rendered about 700 packages completely unbuildable, I would have really liked to see this act of renaming discussed before it was simply done. Could we somehow try to coordinate such major things without pulling the rug under everyone else while they're travelling from/to susecon?
This was being processed for 5 months in staging projects for Tumbleweed. It is kinda weird no cloud integration checks spotted it.
Also I see you took upon yourself 'fixing' it over the weekend in the develprj where you messed it even more when you removed the patches and didn't add unittest2 dependency now.
Also the python3-iso8601 is not broken, they just absolutely changed api between version 0.11 and 0.12 where no distribution apart from us ever shipped the 0.12 so you can fix it by downgrading:
https://build.opensuse.org/package/rdiff/devel:languages:python/python-iso86...
Cheers
Tom
On Monday, April 8, 2019 10:27:05 AM CEST Tomas Chvatal wrote:
Hi Tomas,
are a number of things depending on the pypi name to be correct, so now again all of our python stack is completely unresolveable.
We were matching up what other distros do to track it from the main subunit package as they needed to be all shipped at once:
I'm okay with making "subunit" the main package and providing the proper tools, that was indeed a long standing problem, so it is good to get it addressed.
grep name= subunit-1.3.0/setup.py
name='python-subunit',so, I would like to understand why we were renaming the package away from a standard name to a one-off naming
There are still the provides for python2-bla and python3-bla it seems we accidentally missed to provide also python-bla to keep it working on Leaps.
So the explanation is that we switched to a one-off naming because we match up what other distros were doing? Is this the new naming policy?
I'm specifically interested why we are not able to name the subunit subpackage based on the standard naming, which would have prevented the situation in the first place.
This was being processed for 5 months in staging projects for Tumbleweed. It is kinda weird no cloud integration checks spotted it.
Because in the staging projects python-python-subunit still existed, and no problem was ever occurring from happening. The rename to python-subunit was the one that caused the issue.
Also, the Cloud stagings are working against Leap, not tumbleweed.
Also I see you took upon yourself 'fixing' it over the weekend in the develprj where you messed it even more when you removed the patches and didn't add unittest2 dependency now.
its a build dependency only, and currently pulled in via testtools. if you need it to be listed explictely, that can be done.
Thanks for the quotes around 'fixing' btw, I highly appreciate that ;)
Also the python3-iso8601 is not broken, they just absolutely changed api between version 0.11 and 0.12 where no distribution apart from us ever shipped the 0.12 so you can fix it by downgrading:
0.12 is required by openstack, so I can't just downgrade it.
https://build.opensuse.org/package/rdiff/devel:languages:python/python-iso86 01?linkrev=base&rev=25
Yes, thanks for this other breakage, I wanted to keep the mail to one problem at a time though.
What exactly is the problem with just using "subunit.iso8601" as upstream does ? if there is a problem, can we just fix it upstream first? what happened to the "not modify stuff downstream" policy?
Greetings, Dirk
Dirk Mueller píše v Po 08. 04. 2019 v 11:30 +0200:
On Monday, April 8, 2019 10:27:05 AM CEST Tomas Chvatal wrote:
Hi Tomas,
are a number of things depending on the pypi name to be correct, so now again all of our python stack is completely unresolveable.
We were matching up what other distros do to track it from the main subunit package as they needed to be all shipped at once:
I'm okay with making "subunit" the main package and providing the proper tools, that was indeed a long standing problem, so it is good to get it addressed.
grep name= subunit-1.3.0/setup.py
name='python-subunit',so, I would like to understand why we were renaming the package away from a standard name to a one-off naming
There are still the provides for python2-bla and python3-bla it seems we accidentally missed to provide also python-bla to keep it working on Leaps.
So the explanation is that we switched to a one-off naming because we match up what other distros were doing? Is this the new naming policy?
I'm specifically interested why we are not able to name the subunit subpackage based on the standard naming, which would have prevented the situation in the first place.
Well that is for Matej to answer. I didn't control how the packages are named I didn't even notice the pypi name is python-subunit and not just subunit.
This was being processed for 5 months in staging projects for Tumbleweed. It is kinda weird no cloud integration checks spotted it.
Because in the staging projects python-python-subunit still existed, and no problem was ever occurring from happening. The rename to python- subunit was the one that caused the issue.
Also, the Cloud stagings are working against Leap, not tumbleweed.
Also I see you took upon yourself 'fixing' it over the weekend in the develprj where you messed it even more when you removed the patches and didn't add unittest2 dependency now.
its a build dependency only, and currently pulled in via testtools. if you need it to be listed explictely, that can be done.
Thanks for the quotes around 'fixing' btw, I highly appreciate that ;)
Well your submission is still screwed up, so it is "fixing" rather than really fixing the issue is it not? https://build.opensuse.org/request/show/692028
Also the python3-iso8601 is not broken, they just absolutely changed api between version 0.11 and 0.12 where no distribution apart from us ever shipped the 0.12 so you can fix it by downgrading:
0.12 is required by openstack, so I can't just downgrade it.
How do the others actually pull the openstack in if they do ship only the 0.11? Also note that the 0.12 behaves differently between py2 and py3 so I wonder how that works too if you check the code.
Yes, thanks for this other breakage, I wanted to keep the mail to one problem at a time though.
What exactly is the problem with just using "subunit.iso8601" as upstream does ? if there is a problem, can we just fix it upstream first? what happened to the "not modify stuff downstream" policy?
Well there also is a policy of not using bundled libraries if it can be avoided. Also here the upstream was not interested in using the system iso8601 if you check the PRs. Sadly otoh there are changes in this module by the subunit team: https://github.com/testing-cabal/subunit/commit/4ea90f04f1b71d01eed6721c36f9... https://github.com/testing-cabal/subunit/commit/86b85e3f83f4030a63026386ffc3...
Note that I only cared about this stuff so my other changes actually do get to Tumbleweed. So I am perfectly fine if you prune all the patches and make it bundled for me it was just faster to revert the iso8601 to older version when I got told by DimStar that my staging won't get in unless this gets fixed.
Cheers
Tom
On Monday, April 8, 2019 12:11:55 PM CEST Tomas Chvatal wrote:
Hi Tomas,
I'm specifically interested why we are not able to name the subunit subpackage based on the standard naming, which would have prevented the situation in the first place.
Well that is for Matej to answer. I didn't control how the packages are named I didn't even notice the pypi name is python-subunit and not just subunit.
Ok, assuming there is no reply then I guess we're good with going back to following standard practices and treat this as a bug. I'll be happy to "fix" that then, just wanted to avoid misunderstanding policies and then being educated about it after I did the work.
Well your submission is still screwed up, so it is "fixing" rather than really fixing the issue is it not? https://build.opensuse.org/request/show/692028
you're referring to the "provides"? yeah, I noticed that this was just the tip of the iceberg. I meanwhile added a fixed package as an overlay to the CI.
0.12 is required by openstack, so I can't just downgrade it.
How do the others actually pull the openstack in if they do ship only the 0.11? Also note that the 0.12 behaves differently between py2 and py3 so I wonder how that works too if you check the code.
"others" typically have a openstack specific repository of packages that is in itself consistent and matching what openstack expects.
What exactly is the problem with just using "subunit.iso8601" as upstream does ? if there is a problem, can we just fix it upstream first? what happened to the "not modify stuff downstream" policy?
Well there also is a policy of not using bundled libraries if it can be avoided.
This is not a bundled library, its a vendored library. I mean with that that it is installed as subunit.iso8601. I totally agree with you that installing the iso8601 module from subunit would be wrong (this would be "bundled"). If it is vendored inside a module, we can still try to untangle it (as we tried with requests for several releases). I would hope that we do learn from the requests mess though:
* almost every "version update of requests" introduced new severe regressions because the unvendoring was done downstream, and every downstream did it slightly differently, annoying the hell out of upstream because every downstream introduced a slightly different set of bugs * upstream eventually gave in and did the unvendoring themselves.
As such I would consider a vendored library an engagement with upstream to figure out a proper way of doing that rather than replacing stuff with symlinks in the packages (and then adding patches to fix it up (and then adding patches to drop the tests that no longer pass (and then add a patch that breaks it for everyone that didn't get noticed because the tests that would notice it got dropped))).
So I am perfectly fine if you prune all the patches and make it bundled for me it was just faster to revert the iso8601 to older version when I got told by DimStar that my staging won't get in unless this gets fixed.
Yeah, I don't really like how really really super central packages are just being version downgraded to "get my stuff in" without even doing high level cross checking with the maintainers of "osc whatdependson XXX" but that is for another rant some other time. For now I would really like to get the subunit package in a working state first.
Greetings, Dirk
On 2019-04-08, 10:11 GMT, you wrote:
Well that is for Matej to answer. I didn't control how the packages are named I didn't even notice the pypi name is python-subunit and not just subunit.
No, there is no renaming whatsoever. python-subunit project on PyPI (what was the upstream for our package python-python-subunit) was terminated as a project. So, we have completely replaced this package with new (mostly C++) package subunit, which also includes bindings (providing API compatibility) for many scripting languages, Python among them. So, python-python-subunit package should be completely eliminated by a subpackage from the subunit package (with the new upstream).
Also the python3-iso8601 is not broken, they just absolutely changed api between version 0.11 and 0.12 where no distribution apart from us ever shipped the 0.12 so you can fix it by downgrading:
0.12 is required by openstack, so I can't just downgrade it.
Do we know how complicated API we are talking about? How complicated it would be to patch subunit to work with the new API?
Best,
Matěj
On Monday, April 8, 2019 1:24:40 PM CEST mcepl@mcepl.eu wrote:
packages are named I didn't even notice the pypi name is python-subunit and not just subunit.
No, there is no renaming whatsoever. python-subunit project on PyPI (what was the upstream for our package python-python-subunit) was terminated as a project. So, we have completely replaced this package with new (mostly C++) package subunit, which also includes bindings (providing API compatibility) for many scripting languages, Python among them.
Hi,
I understand this part. yet this "new" package has "python-subunit" as pypi name configured, so it should be pythonX-python-subunit. However, for reasons that are not understandable to me, the new subunit package uses "pythonX- subunit" as package naming. Given that I had a bout 700 packages depending on the old name, I would really like to understand why we're introducing this inconsistency.
I quoted the setup.py name= setting in the previous mail, please have a look there.
So, python-python-subunit package should be completely eliminated by a subpackage from the subunit package (with the new upstream).
Yes, I did that now from devel:languages:python after I noticed this conflict (it was there before, which caused this problem to be hidden for quite some time).
Do we know how complicated API we are talking about? How complicated it would be to patch subunit to work with the new API?
It is near trivial, but that isn't the point. the point is that we're checkin in packages that don't work, and that causes large fallout for our CI. I'm all for getting this fixed, but only if the result actually works.
Greetings, Dirk
On 2019-04-09, 09:08 GMT, you wrote:
Given that I had a bout 700 packages depending on the old name, I would really like to understand why we're introducing this inconsistency.
Provides: python2-python-subunit = %{version} Provides: python3-python-subunit = %{version}
All your packages should work just fine. OK, the reasons for simplification were mainly esthetic, but really all your packages should find their dependency. Don't they?
I quoted the setup.py name= setting in the previous mail, please have a look there.
OK, that I missed. However, do you have some compelling reason, why this uglification should happen?
Do we know how complicated API we are talking about? How complicated it would be to patch subunit to work with the new API?
It is near trivial, but that isn't the point.
Could I ask for that trivial patch, please? So we could stay with the latest packages and not hold anything back?
Thank you,
Matěj
Hi,
On 4/9/19 2:03 PM, mcepl@cepl.eu wrote:
On 2019-04-09, 09:08 GMT, you wrote:
Given that I had a bout 700 packages depending on the old name, I would really like to understand why we're introducing this inconsistency.
Provides: python2-python-subunit = %{version} Provides: python3-python-subunit = %{version}
All your packages should work just fine. OK, the reasons for simplification were mainly esthetic, but really all your packages should find their dependency. Don't they?
I quoted the setup.py name= setting in the previous mail, please have a look there.
OK, that I missed. However, do you have some compelling reason, why this uglification should happen?
We can automatically translate the pypi-name (or the name set in setup.py) to a rpm package name (by prefixing it with pythonX-). That's very useful if you need to translate install_requires/extra_requires from python to Requires/BuildRequires/Recommends from RPM. That will no longer work.
See also https://en.opensuse.org/openSUSE:Packaging_Python#Naming_policy
So please keep the correct name which is pythonX-python-subunit.
TIA,
Tom
On Tuesday, April 9, 2019 2:03:55 PM CEST mcepl@cepl.eu wrote:
Hi Matej,
could you please fix your email address? its slightly annoying.. (I can't send mail to it).
Provides: python2-python-subunit = %{version} Provides: python3-python-subunit = %{version} All your packages should work just fine.
They don't. first of all obsoletes is missing, and 2nd the package is actually called python-* in older releases.
OK, the reasons for simplification were mainly esthetic, but really all your packages should find their dependency. Don't they?
they don't.
I quoted the setup.py name= setting in the previous mail, please have a look there.
OK, that I missed. However, do you have some compelling reason, why this uglification should happen?
because we have a clear naming policy and we should stay consistent to that one? why did you rename all the ipython packages and were quoting the policy and here you're deviating for the policy because "its ugly" ?
ipython was a *clear* exception. We had dozens of packages properly consistently named, and now everything has been messed up to follow the policy (and the existing maintainers were not even involved in the discussion). And now we're introducing this breakage for a single package and argue it should stay that way because its "de-uglifying" it?
complicated it would be to patch subunit to work with the new API?
It is near trivial, but that isn't the point.
Could I ask for that trivial patch, please? So we could stay with the latest packages and not hold anything back?
Just by not patching the package and shipping it as it is provided by upstream everything works. If you want to improve things, then improve it upstream and make a patch there, get it merged, and make upstream rleease a new version.
Adding a pile of downstream only patches to the package that are completely breaking the way the package works for the rest of the distribution is not a good way forward.
Thanks, Dirk
On 2019-04-10, 08:58 GMT, you wrote:
So please keep the correct name which is pythonX-python-subunit.
Dirk has already fixed it. Thank you.
Best,
Matěj
-
Dirk Mueller -
Dirk Müller
-
Matej Cepl -
mcepl＠cepl.eu
-
mcepl＠mcepl.eu -
Thomas Bechtold -
Tomas Chvatal