[opensuse-buildservice] Improvements for building .deb packages for projects with an in-tree debian/ (or debian.upstream/) directory
Hi, I'm using the OpenSUSE OBS to develop DEB and RPM packaging for my Barcode Writer in Pure PostScript project. My goal is to provide OBS with nothing but a single _service file with everything else extracted directly from the project's repository via extract_file, etc. I have this working but with a few rough edges which may be down to me not understanding something, overlooking some functionality or these might become requests for improvements to debtransform. 1. SUPERFLUOUS TAR FILE: I'm using tar_scm to create the tarball from the project's Git repository and to provide automatic versioning, however I seem to require a duplicate tarball just for DEB that has a static filename as referenced from the DEBTRANSFORM-TAR tag. It would be great to be able to simply use the versioned output of tar_scm (recompressed). 2. DEBIAN.* FILES: I have to separately map each of the files in my project's debian.upstream/ directory to debian.* files for OBS. It would be great to indicate to debtransforn that a particular directory in the tarball contains Debian packaging. Then you would only need to provide the .dsc file, not the debian.{rules,control} files, etc. [ 2a. For context, a year or two back I was successfully building DEBs on OBS by having the Debian packaging directory called debian/ in the project repo/tarball. Recent versions of OBS appear to fail with an in-tarball debian/ directory because the diff generated by debtransform from the OBS debian.{control,rules} files conflicts with the existing debian/{control,rules} files — the pre-existing files appear to get patched into rather than replaced or left as-is. Besides this, Debian guidance is for upstream projects to name any included DEB packaging something other than debian/ such as debian.upstream/ as I now do. So indicating that some otherwise-named directory provides the DEB packaging seems preferable. ] [ 2b. I'm aware that you can supply the components of a Debian source package (e.g. a project-debian.tar.gz) for OBS to build instead of the individual .dsc and debian.* files. I'm wanting to use OBS for Continuous Integration testing (perhaps just for tagged releases) so having to build a source package first elsewhere to supply to OBS by download_url or otherwise defeats my purpose somewhat.] 3. VERSIONING: A Debian non-native package must have a version that ends in "-N", which is incompatible with RPM versioning. I'm using tar_scm to provides automatic versioning using the versionformat parameter but am unable to simultaneously satisfy both RPM and DEB version formatting requirements. I've raised an issue for this here: https://github.com/openSUSE/obs-service-set_version/issues/25 Hopefully my annotated _service file should make this clear: https://build.opensuse.org/package/show/home:terryburton:postscriptbarcode/l... I'd be happy to be told that I've missed some functionality and that some of this can already be done. If not, I'm happy to put some work into smoothing out these issues but I first want to check that my use case ("in-project-tree Debian packaging") is something that the OBS project would approve of and support going forward so that I don't waste time needlessly optimising for my own needs. Many thanks, Terry -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Monday 2016-02-22 22:51, Terry Burton wrote:
1. SUPERFLUOUS TAR FILE: I'm using tar_scm to create the tarball from the project's Git repository and to provide automatic versioning, however I seem to require a duplicate tarball just for DEB that has a static filename as referenced from the DEBTRANSFORM-TAR tag. It would be great to be able to simply use the versioned output of tar_scm (recompressed).
Solution: Do not supply a DEBTRANSFORM-TAR line at all.
2. DEBIAN.* FILES: I have to separately map each of the files in my project's debian.upstream/ directory to debian.* files for OBS. It would be great to indicate to debtransforn that a particular directory in the tarball contains Debian packaging. [ 2a. [...] Debian guidance is for upstream projects to name any included DEB packaging something other than debian/ such as debian.upstream/ as I now do. So indicating that some otherwise-named directory provides the DEB packaging seems preferable. ]
You could just remove the debian/ directory from the source tarball. Many projects, so I believe, have done that over the last 15 years as they realized that distro-specific code should be with the distro instead, also because it has been a bitrot factor.
3. VERSIONING: A Debian non-native package must have a version that ends in "-N", which is incompatible with RPM versioning.
DEBTRANSFORM-RELEASE: 1 is necessary in the .dsc to get the automated "N" like it already happens for .spec builds. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 22 February 2016 at 22:27, Jan Engelhardt
On Monday 2016-02-22 22:51, Terry Burton wrote:
1. SUPERFLUOUS TAR FILE: I'm using tar_scm to create the tarball from the project's Git repository and to provide automatic versioning, however I seem to require a duplicate tarball just for DEB that has a static filename as referenced from the DEBTRANSFORM-TAR tag. It would be great to be able to simply use the versioned output of tar_scm (recompressed).
Solution: Do not supply a DEBTRANSFORM-TAR line at all.
Thanks. I seem to recall that when DEBTRANSFORM-TAR is omitted debtransform croaks because it sees both _service:tar_scm:postscriptbarcode-{VERSION}.tar and_service:recompress:tar_scm:postscriptbarcode-{VERSION}.tar.gz as candidates for the tarfile. I will test this tomorrow as it's getting late.
2. DEBIAN.* FILES: I have to separately map each of the files in my project's debian.upstream/ directory to debian.* files for OBS. It would be great to indicate to debtransforn that a particular directory in the tarball contains Debian packaging. [ 2a. [...] Debian guidance is for upstream projects to name any included DEB packaging something other than debian/ such as debian.upstream/ as I now do. So indicating that some otherwise-named directory provides the DEB packaging seems preferable. ]
You could just remove the debian/ directory from the source tarball. Many projects, so I believe, have done that over the last 15 years as they realized that distro-specific code should be with the distro instead, also because it has been a bitrot factor.
I agree that this is true for most standard packages especially if the authors do not care about packaging. BWIPP is an extensive resource programmed in an atypical programming language (PostScript) and uses custom compilation routines to prepare the resources in a number of formats. Beyond this it also has a basic C helper library with bindings for several languages. Experienced packagers have struggled to get the build procedure right because they are so unfamiliar with the PostScript environment and they tend to pick a subset of the bindings that they are most familiar with and package only those leading to the wrong kind of diversity. Providing some guideline packaging that sets a firm standard and works well helps to promote consistency across distributions, even if they choose to trim the fat or specialise things. They have a benchmark against which to compare their own packaging and a neutral place to contribute improvements that might be useful for other distributions (or at least I can cherry-pick them). Continuous Integration of the packaging routines using services such as OBS helps to prevent bit rot (assuming the authors care about failures).
3. VERSIONING: A Debian non-native package must have a version that ends in "-N", which is incompatible with RPM versioning.
DEBTRANSFORM-RELEASE: 1
is necessary in the .dsc to get the automated "N" like it already happens for .spec builds.
This works. Very helpful, thanks! -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday 2016-02-23 00:19, Terry Burton wrote:
Solution: Do not supply a DEBTRANSFORM-TAR line at all.
Thanks. I seem to recall that when DEBTRANSFORM-TAR is omitted debtransform croaks because it sees both _service:tar_scm:postscriptbarcode-{VERSION}.tar and_service:recompress:tar_scm:postscriptbarcode-{VERSION}.tar.gz as candidates for the tarfile. I will test this tomorrow as it's getting late.
The former will be deleted. At least that is what I can see from just looking at /srv/obs/sources/PKG.
You could just remove the debian/ directory from the source tarball. Many projects, so I believe, have done that over the last 15 years as they realized that distro-specific code should be with the distro instead, also because it has been a bitrot factor.
I agree that this is true for most standard packages especially if the authors do not care about packaging. BWIPP is an extensive resource programmed in an atypical programming language (PostScript) and uses custom compilation routines to prepare the resources in a number of formats.
What I gather from https://github.com/bwipp/postscriptbarcode/releases is that there appears to be no C source code, so the rest is data files which can be flung into an arbitrary directory that you just have to point ghostscript to. Iff there is something to actually build/compile, then, if a Makefile/configure/etc. thing is not sufficient generate it for the user, that Makefile (or lack thereof) should be revised. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 23 February 2016 at 09:58, Jan Engelhardt
On Tuesday 2016-02-23 00:19, Terry Burton wrote:
Solution: Do not supply a DEBTRANSFORM-TAR line at all.
Thanks. I seem to recall that when DEBTRANSFORM-TAR is omitted debtransform croaks because it sees both _service:tar_scm:postscriptbarcode-{VERSION}.tar and_service:recompress:tar_scm:postscriptbarcode-{VERSION}.tar.gz as candidates for the tarfile. I will test this tomorrow as it's getting late.
The former will be deleted. At least that is what I can see from just looking at /srv/obs/sources/PKG.
You're right! Apologies for the noise with this. Not sure why my early builds had problems. I might step through the revisions to find what I did wrong.
You could just remove the debian/ directory from the source tarball. Many projects, so I believe, have done that over the last 15 years as they realized that distro-specific code should be with the distro instead, also because it has been a bitrot factor.
I agree that this is true for most standard packages especially if the authors do not care about packaging. BWIPP is an extensive resource programmed in an atypical programming language (PostScript) and uses custom compilation routines to prepare the resources in a number of formats.
What I gather from https://github.com/bwipp/postscriptbarcode/releases is that there appears to be no C source code, so the rest is data files which can be flung into an arbitrary directory that you just have to point ghostscript to.
The API for the C library and bindings are not currently finalised so I don't ship it in the release builds yet. The development code is here: https://github.com/bwipp/postscriptbarcode/tree/master/libs I don't want to presume on your interest so forgive me if the following is more information that you care to digest and not directly related to OBS packaging: BWIPP traditionally shipped tarballs containing the PS resources in a number of formats that work best in certain scenarios: a monolithic file, a set of split files with a resource descriptor, each in either "raw-PS" or "packaged" into a compressed form. BWIPP is used by a large number of open source software applications (Scribus, LaTeX, KBarcode, ...), libraries (Ruby GS, bwip-js, python-Elaphe, ...) and many more commercial applications including being embedding in printer hardware. Each project has tended to bundle some variant of the BWIPP PS (akin to a bundled library) and interface with it in a custom way, often embedding a lot of metadata about the capabilities of each barcode encoder within their application code. Obvious problems ensue. My plan to remedy with the C helper library and bindings which when finalised will be packaged consistently for major distributions. This will allow projects to interface consistently against a common library and no longer bundle the PS, provide quirky interfaces or embed lots of unnecessary code. Hence my interest in packaging since I think this will be important to the long-term maintainability of all of the products in the BWIPP ecosystem.
Iff there is something to actually build/compile, then, if a Makefile/configure/etc. thing is not sufficient generate it for the user, that Makefile (or lack thereof) should be revised.
Sorry if I have been misleading. The build process for the PS resources and bindings can in fact be driven by make, but packaging a number of modules for different languages requires learning a number of different packaging conventions as can be seen from this spec file: https://raw.githubusercontent.com/bwipp/postscriptbarcode/master/open-build-... So my aim with providing upstream packaging files is to do the initial hard work which others can improve on. OBS just makes this just a little more difficult that seems necessary (separately enumerating each debian.* file) but not as awkward as I feared. Thanks again as you've been a great help. All the best, Terry -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, Feb 22, 2016 at 09:51:46PM +0000, Terry Burton wrote:
My goal is to provide OBS with nothing but a single _service file with everything else extracted directly from the project's repository via extract_file, etc.
I already have this working for a few years: https://build.opensuse.org/package/view_file/home:e9925248:grandorgue/grando...
1. SUPERFLUOUS TAR FILE: I'm using tar_scm to create the tarball from the project's Git repository and to provide automatic versioning, however I seem to require a duplicate tarball just for DEB that has a static filename as referenced from the DEBTRANSFORM-TAR tag. It would be great to be able to simply use the versioned output of tar_scm (recompressed).
The two tar files are not causing any issue [I don't use DEBTRANSFORM-TAR].
2. DEBIAN.* FILES: I have to separately map each of the files in my project's debian.upstream/ directory to debian.* files for OBS. It would be great to indicate to debtransforn that a particular directory in the tarball contains Debian packaging. Then you would only need to provide the .dsc file, not the debian.{rules,control} files, etc.
I have included debian/ in the tarball. I only extract a dummy file [debian.debtransform] to trigger a debtransform run as well as the changelog, so that debtransform can update it. I would still allow me to add/update file in the debian/ directory, if I just add futher debian.* files.
[ 2a. For context, a year or two back I was successfully building DEBs on OBS by having the Debian packaging directory called debian/ in the project repo/tarball. Recent versions of OBS appear to fail with an in-tarball debian/ directory because the diff generated by debtransform from the OBS debian.{control,rules} files conflicts with the existing debian/{control,rules} files ??? the pre-existing files appear to get patched into rather than replaced or left as-is. Besides this, Debian guidance is for upstream projects to name any included DEB packaging something other than debian/ such as debian.upstream/ as I now do. So indicating that some otherwise-named directory provides the DEB packaging seems preferable. ]
Included debian/ is still working. I contributed years ago some code to update/replace files in the orig tarball. Regards, Martin -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 23 February 2016 at 18:22, Martin Koegler
On Mon, Feb 22, 2016 at 09:51:46PM +0000, Terry Burton wrote: <...snip...>
2. DEBIAN.* FILES: I have to separately map each of the files in my project's debian.upstream/ directory to debian.* files for OBS. It would be great to indicate to debtransforn that a particular directory in the tarball contains Debian packaging. Then you would only need to provide the .dsc file, not the debian.{rules,control} files, etc.
I have included debian/ in the tarball. I only extract a dummy file [debian.debtransform] to trigger a debtransform run as well as the changelog, so that debtransform can update it. I would still allow me to add/update file in the debian/ directory, if I just add futher debian.* files.
Thanks Martin and Jan for your help and examples. The result is a vast improvement over what I started with. My working _service file: https://build.opensuse.org/package/view_file/home:terryburton:postscriptbarc... All the best, Terry -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Jan Engelhardt -
Martin Koegler -
Terry Burton