python setuptools_scm and rpm versioning
The "duplicity" package uses python's setuptools_scm module to substitute the package version in its own sources. For development versions that aren't tagged, setuptools_scm uses the format "$NEXT_VERSION.dev$N", where NEXT_VERSION is the guessed subsequent/future version, and $N is the commit distance from the last released version. This scheme doesn't work with rpm. AFAIK, for rpm we must use $NEXT_VERSION~$N instead to achieve the desired ordering: $CURRENT_VERSION < $NEXT_VERSION~$N < $NEXT_VERSION~$((N+1)) < $NEXT_VERSION Unfortunately the ".dev" format is hard-coded in setuptools. Has anyone encountered this issue before? Is there a smart solution to modify setuptools' behavior, or some other trick? Regards Martin
Hi, Am 09.07.21 um 12:43 schrieb Martin Wilck:
The "duplicity" package uses python's setuptools_scm module to substitute the package version in its own sources.
For development versions that aren't tagged, setuptools_scm uses the format "$NEXT_VERSION.dev$N", where NEXT_VERSION is the guessed subsequent/future version, and $N is the commit distance from the last released version.
This scheme doesn't work with rpm. AFAIK, for rpm we must use $NEXT_VERSION~$N instead to achieve the desired ordering:
$CURRENT_VERSION < $NEXT_VERSION~$N < $NEXT_VERSION~$((N+1)) < $NEXT_VERSION
Unfortunately the ".dev" format is hard-coded in setuptools. Has anyone encountered this issue before? Is there a smart solution to modify setuptools' behavior, or some other trick?
Regards Martin
Please don't modify setuptool's behavior. This would lead to issues with importlib whenever a package requests a module and finds some unexpected version scheme in the dist-info/egg-info metadata. It's a common pattern to use $NEXT_VERSION~$suffix for pre-releases and $CURRENT_VERSION+$suffix for post releases and git commits after a release. The setuptools version used in the egg-info/dist-info metadata and the RPM version don't have to be exactly the same. As duplicity uses the obs_scm service, you have to adjust the "versionrewrite-pattern" and "versionrewrite-replacement" tags. Check these sections of PEP440: https://www.python.org/dev/peps/pep-0440/#normalization https://www.python.org/dev/peps/pep-0440/#summary-of-permitted-suffixes-and-... HTH, Ben
On Fr, 2021-07-09 at 13:40 +0200, Ben Greiner wrote:
It's a common pattern to use $NEXT_VERSION~$suffix for pre-releases and $CURRENT_VERSION+$suffix for post releases and git commits after a release. The setuptools version used in the egg-info/dist-info metadata and the RPM version don't have to be exactly the same.
Ok.
As duplicity uses the obs_scm service, you have to adjust the "versionrewrite-pattern" and "versionrewrite-replacement" tags.
I had made the change towards obs_scm last week, and I've reverted that back to download_files now (download from PyPi), as I found it too fragile to have to run "setup.py sdist" during the build procedure (e.g. it would use localtime as release time stamp, messing up reproducible builds). Details in boo#1188132. If we had a way to run "setup.py sdist" in a source service, things might be different. The timestamp problem would remain though. Thanks, Martin
Am 09.07.21 um 15:26 schrieb Martin Wilck:
On Fr, 2021-07-09 at 13:40 +0200, Ben Greiner wrote:
It's a common pattern to use $NEXT_VERSION~$suffix for pre-releases and $CURRENT_VERSION+$suffix for post releases and git commits after a release. The setuptools version used in the egg-info/dist-info metadata and the RPM version don't have to be exactly the same. Ok.
As duplicity uses the obs_scm service, you have to adjust the "versionrewrite-pattern" and "versionrewrite-replacement" tags. I had made the change towards obs_scm last week, and I've reverted that back to download_files now (download from PyPi), as I found it too fragile to have to run "setup.py sdist" during the build procedure (e.g. it would use localtime as release time stamp, messing up reproducible builds).
I see.
Details in boo#1188132.
I see no problem in a duplicity-0.8.21~dev6-1.1.x86_64.rpm containing %{python_sitearch}/duplicity-0.8.21.dev6.egg-info and reporting "duplicity 0.8.21.dev6" when doing `duplicity --version`.
If we had a way to run "setup.py sdist" in a source service, things might be different.
Well, there is obs-service-python_sdist [1, 2], although it is quite old and there is no package of it in Factory. What's the difference to a call of setup.py sdist in %build or %prep? Just mirror setuptools_scm's conversion of git tag to version string in "versionrewrite-pattern" and "versionrewrite-replacement".
The timestamp problem would remain though.
You could patch line 58 `Reldate = ...` in setup.py. The problem is not the source service itself, but the difference between the published PyPI sdist and the git repository. You could as well have `Source: https://gitlab.com/duplicity/duplicity/-/archive/rel.0.8.20/duplicity-rel.%{version}.tar.gz` and face the exact same problem. BTW, you don't need to specify the download_files service. `osc service runall download_files` works even without a _service file. Ben [1] https://github.com/openSUSE/obs-service-python_sdist [2] https://build.opensuse.org/package/show/openSUSE:Tools/obs-service-python_sd...
On Fr, 2021-07-09 at 16:35 +0200, Ben Greiner wrote:
I see no problem in a duplicity-0.8.21~dev6-1.1.x86_64.rpm containing %{python_sitearch}/duplicity-0.8.21.dev6.egg-info and reporting "duplicity 0.8.21.dev6" when doing `duplicity --version`.
I agree there's no fundamental issue with it, but I don't like it much, either. I'm sure some users would find it confusing.
If we had a way to run "setup.py sdist" in a source service, things might be different.
Well, there is obs-service-python_sdist [1, 2], although it is quite old and there is no package of it in Factory.
Thanks, that might be worth a look. To be really useful, it would need to be available as server-side service on OBS.
What's the difference to a call of setup.py sdist in %build or %prep?
I could do that, but as long as I stick to the released PyPi version (which I currently do), it seems overengineered to me.
Just mirror setuptools_scm's conversion of git tag to version string in "versionrewrite-pattern" and "versionrewrite-replacement".
IMO trying to mirror the behavior of one tool in another tool is fragile and not well suited for long-term maintenance. This would break when upstream changes the logic (it has happened before). I'd rather avoid it as long as I can.
The timestamp problem would remain though.
You could patch line 58 `Reldate = ...` in setup.py.
I sure could. But I'd rather avoid that additional complexity. Why should I re-build this archive during every OBS build if I can simply fetch it from PyPi in a source service? It seems safer, simpler, and less error-prone to do the latter. Moreover, AFAICS it would be impossible to reproduce the original PyPi timestamp, and thus the original PyPi package. It adds trust if users can verify our sources with the PyPi checksum. My initial post was motivated by the idea that at some time we might want to use a dev$X version. But as it's not currently necessary, and making it possible introduces complexity, I'd rather refrain from doing this now.
The problem is not the source service itself, but the difference between the published PyPI sdist and the git repository. You could as well have `Source: https://gitlab.com/duplicity/duplicity/-/archive/rel.0.8.20/duplicity-rel.%{version}.tar.gz` and face the exact same problem.
Right, that's what I'd figured after some trial-and-error. "setup.py install" in a virtualenv has exactly that effect, it also makes "duplicity --version" print "$version".
BTW, you don't need to specify the download_files service. `osc service runall download_files` works even without a _service file.
Thanks for the hint. Regards, Martin
On Fr, 2021-07-09 at 16:35 +0200, Ben Greiner wrote:
I see no problem in a duplicity-0.8.21~dev6-1.1.x86_64.rpm containing %{python_sitearch}/duplicity-0.8.21.dev6.egg-info and reporting "duplicity 0.8.21.dev6" when doing `duplicity --version`.
I agree there's no fundamental issue with it, but I don't like it much, either. I'm sure some users would find it confusing.
If we had a way to run "setup.py sdist" in a source service, things might be different.
Well, there is obs-service-python_sdist [1, 2], although it is quite old and there is no package of it in Factory.
Thanks, that might be worth a look. To be really useful, it would need to be available as server-side service on OBS.
What's the difference to a call of setup.py sdist in %build or %prep?
I could do that, but as long as I stick to the released PyPi version (which I currently do), it seems overengineered to me.
Just mirror setuptools_scm's conversion of git tag to version string in "versionrewrite-pattern" and "versionrewrite-replacement".
IMO trying to mirror the behavior of one tool in another tool is fragile and not well suited for long-term maintenance. This would break when upstream changes the logic (it has happened before). I'd rather avoid it as long as I can.
The timestamp problem would remain though.
You could patch line 58 `Reldate = ...` in setup.py.
I sure could. But I'd rather avoid that additional complexity. Why should I re-build this archive during every OBS build if I can simply fetch it from PyPi in a source service? It seems safer, simpler, and less error-prone to do the latter. Moreover, AFAICS it would be impossible to reproduce the original PyPi timestamp, and thus the original PyPi package. It adds trust if users can verify our sources with the PyPi checksum. My initial post was motivated by the idea that at some time we might want to use a dev$X version. But as it's not currently necessary, and making it possible introduces complexity, I'd rather refrain from doing this now.
The problem is not the source service itself, but the difference between the published PyPI sdist and the git repository. You could as well have `Source: https://gitlab.com/duplicity/duplicity/-/archive/rel.0.8.20/duplicity-rel.%{version}.tar.gz` and face the exact same problem.
Right, that's what I'd figured after some trial-and-error. "setup.py install" in a virtualenv has exactly that effect, it also makes "duplicity --version" print "$version".
BTW, you don't need to specify the download_files service. `osc service runall download_files` works even without a _service file.
Thanks for the hint. Regards, Martin
participants (3)
-
Ben Greiner -
Martin Wilck -
Martin Wilck