[opensuse-buildservice] Fun with source services
G'day, Second attempt at sending, this time with no HTML. In the 21st century. So I've been trying to use OBS source services for my project novaprova https://build.opensuse.org/project/show/home:gnb:novaprova to do a development tree (i.e. not a release) build on days when there are checkins. What I want is a process which does 1. get the latest source from the master branch in the git repo 2. pull the RPM and DEB package control files (.spec etc) out of a directory in there 3. figure out a version number 4. apply the version number to the .spec file 5. perform build 6. upload build results to repositories I read this document http://en.opensuse.org/openSUSE:Build_Service_Tips_and_Tricks#Perform_a_nigh... but I didn't want to do an osc commit for every build, or have a server sitting around whose only purpose was to run a cronjob. So I thought I could do some clever tricks with source services, resulting a single _service file which could be used unmodified for every nightly build. It almost worked. I found the following issues: 1. If the source service runs correctly, there is no visible log. The build log starts after that. 2. If the source service fails, the error messages only appear in a tooltip. 3. The build log page has a bug where it will repeat large chunks of a log. 4. Source services could be better documented. A lot better. For example I discovered @PARENT_TAG@ by reading the source. 5. The tar_scm service creates an uncompressed tarball, i.e. foo-git.1234567.tar. My .spec file was expecting a compressed tarball and failed. 6. The recompress service could be used to compress the tarball from tar_scm, but it takes a fixed incoming filename, without wildcards. This doesn't work when tar_scm is generating a tarball filename which has a version number encoded in it, which version number is not known at the time the _service file is written. 7. Likewise, the extract_files service could be used to extract a spec file from inside the tarball, but it requires a fixed incoming filename without wildcards. Given that in most projects there will be exactly one tarball, I would have thought that extract_files could have guessed correctly, or accepted a wildcard. 8. For the version number, I wanted to use a timestamp in ISO8601 basic format to day precision, i.e. "20131120" for 20 November 2013. The closest I can get is to use %ci as the versionformat in the tar_scm service, but this doesn't quite work. Git gives extended format to second precision withhttps://help.github.com/articles/post-receive-hooks timezone, e.g. 2013-11-20 10:06:35 -0800. The tar_scm script tries to clean this up, but the result still has useless information I don't want and spaces which break the other services. Also, what's a good way to kick off an OBS build from a GitHub webhook? https://help.github.com/articles/post-receive-hooks -- Greg. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Mittwoch, 20. November 2013, 10:24:33 schrieb Greg Banks:
G'day,
Second attempt at sending, this time with no HTML. In the 21st century.
So I've been trying to use OBS source services for my project novaprova
https://build.opensuse.org/project/show/home:gnb:novaprova
to do a development tree (i.e. not a release) build on days when there are checkins. What I want is a process which does
1. get the latest source from the master branch in the git repo 2. pull the RPM and DEB package control files (.spec etc) out of a directory in there 3. figure out a version number 4. apply the version number to the .spec file 5. perform build 6. upload build results to repositories
I read this document
http://en.opensuse.org/openSUSE:Build_Service_Tips_and_Tricks#Perform_a_nigh...
but I didn't want to do an osc commit for every build, or have a server sitting around whose only purpose was to run a cronjob. So I thought I could do some clever tricks with source services, resulting a single _service file which could be used unmodified for every nightly build. It almost worked. I found the following issues:
1. If the source service runs correctly, there is no visible log. The build log starts after that.
2. If the source service fails, the error messages only appear in a tooltip.
source service support in webui has been removed and yet be reimplemented unfortunatly.
3. The build log page has a bug where it will repeat large chunks of a log.
4. Source services could be better documented. A lot better. For example I discovered @PARENT_TAG@ by reading the source.
feel free to submit updates to the source services...
5. The tar_scm service creates an uncompressed tarball, i.e. foo-git.1234567.tar. My .spec file was expecting a compressed tarball and failed.
6. The recompress service could be used to compress the tarball from tar_scm, but it takes a fixed incoming filename, without wildcards. This doesn't work when tar_scm is generating a tarball filename which has a version number encoded in it, which version number is not known at the time the _service file is written.
No, it does not osc cat OBS:Server:Unstable/build/_service
7. Likewise, the extract_files service could be used to extract a spec file from inside the tarball, but it requires a fixed incoming filename without wildcards. Given that in most projects there will be exactly one tarball, I would have thought that extract_files could have guessed correctly, or accepted a wildcard.
yes, could be added .. patches are welcome :)
8. For the version number, I wanted to use a timestamp in ISO8601 basic format to day precision, i.e. "20131120" for 20 November 2013. The closest I can get is to use %ci as the versionformat in the tar_scm service, but this doesn't quite work. Git gives extended format to second precision withhttps://help.github.com/articles/post-receive-hooks timezone, e.g. 2013-11-20 10:06:35 -0800. The tar_scm script tries to clean this up, but the result still has useless information I don't want and spaces which break the other services.
tar_scm can be configured to use different version schemes via the versionformat parameter. Check the _service file above for an example.
Also, what's a good way to kick off an OBS build from a GitHub webhook?
using tokens and the OBS github plugin. I will document and blog about it these days. -- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
G'day, Thanks for responding. On 20 November 2013 12:38, Adrian Schröter <adrian@suse.de> wrote:
Am Mittwoch, 20. November 2013, 10:24:33 schrieb Greg Banks:
[...] 1. If the source service runs correctly, there is no visible log. The build log starts after that.
2. If the source service fails, the error messages only appear in a tooltip.
source service support in webui has been removed and yet be reimplemented unfortunatly.
Is there some kind of schedule for re-implementing it?
6. The recompress service could be used to compress the tarball from tar_scm, but it takes a fixed incoming filename, without wildcards. This doesn't work when tar_scm is generating a tarball filename which has a version number encoded in it, which version number is not known at the time the _service file is written.
No, it does not
osc cat OBS:Server:Unstable/build/_service
Ok, I tried that example and it worked! The difference seems to be that I was using <param name="file">novaprova*.tar</param> instead of <param name="file">*.tar</param> Thanks :)
8. For the version number, I wanted to use a timestamp in ISO8601 basic format to day precision, i.e. "20131120" for 20 November 2013. The closest I can get is to use %ci as the versionformat in the tar_scm service, but this doesn't quite work. Git gives extended format to second precision withhttps://help.github.com/articles/post-receive-hooks timezone, e.g. 2013-11-20 10:06:35 -0800. The tar_scm script tries to clean this up, but the result still has useless information I don't want and spaces which break the other services.
tar_scm can be configured to use different version schemes via the versionformat parameter. Check the _service file above for an example.
Yep, I know about versionformat, I was using it. What I didn't realise was that git log was being run with --date=short so that %ad and %cd would actually do exactly what I wanted. Thanks for the example!
Also, what's a good way to kick off an OBS build from a GitHub webhook?
using tokens and the OBS github plugin. I will document and blog about it these days.
I'm very much looking forward to that blog post. Thanks for all your help! -- Greg. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
G'day, On 20 November 2013 19:42, Greg Banks <gnb@fmeh.org> wrote:
6. The recompress service could be used to compress the tarball from tar_scm, but it takes a fixed incoming filename, without wildcards. This doesn't work when tar_scm is generating a tarball filename which has a version number encoded in it, which version number is not known at the time the _service file is written.
No, it does not
osc cat OBS:Server:Unstable/build/_service
Ok, I tried that example and it worked! The difference seems to be that I was using
<param name="file">novaprova*.tar</param>
instead of
<param name="file">*.tar</param>
I see now what I did wrong: the actual filename is _service:tar_scm:novaprova-9.git.20131101.tar not novaprova-9.git.20131101.tar.
Also, what's a good way to kick off an OBS build from a GitHub webhook?
using tokens and the OBS github plugin. I will document and blog about it these days.
Ok, so I poked around github setting and found the "Obs" service plugin there. The documentation says 2. Create an authentification token for your account using "osc" commandline tool: osc token --create but that doesn't work for me; the "token" command is neither documented nor present in the osc_0.141.0_all.deb package. -- Greg. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
G'day, On 20 November 2013 21:44, Greg Banks <gnb@fmeh.org> wrote:
osc token --create
but that doesn't work for me; the "token" command is neither documented nor present in the osc_0.141.0_all.deb package.
Ah, I see you've just released 0.142 which contains a working and documented "osc token" command. I've just added an OBS Service hook to github, using a new token, and it works just fine. Thanks! -- Greg. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Donnerstag, 21. November 2013, 12:29:11 schrieb Greg Banks:
G'day,
On 20 November 2013 21:44, Greg Banks <gnb@fmeh.org> wrote:
osc token --create
but that doesn't work for me; the "token" command is neither documented nor present in the osc_0.141.0_all.deb package.
Ah, I see you've just released 0.142 which contains a working and documented "osc token" command.
I've just added an OBS Service hook to github, using a new token, and it works just fine.
Nice that you found already. For the rest, please read this blog post http://openbuildservice.org/2013/11/22/Source-Update-Via_Token/ and/or this chapter in the book: http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.authori...
Thanks!
-- Adrian Schroeter email: adrian@suse.de SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Thanks Adrian, that's very useful. On 22 November 2013 04:03, Adrian Schröter <adrian@suse.de> wrote:
Am Donnerstag, 21. November 2013, 12:29:11 schrieb Greg Banks:
G'day,
On 20 November 2013 21:44, Greg Banks <gnb@fmeh.org> wrote:
osc token --create
but that doesn't work for me; the "token" command is neither documented nor present in the osc_0.141.0_all.deb package.
Ah, I see you've just released 0.142 which contains a working and documented "osc token" command.
I've just added an OBS Service hook to github, using a new token, and it works just fine.
Nice that you found already. For the rest, please read this blog post
http://openbuildservice.org/2013/11/22/Source-Update-Via_Token/
and/or this chapter in the book:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.authori...
Thanks!
--
Adrian Schroeter email: adrian@suse.de
SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
-- Greg. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Wednesday 20 November 2013 10:24:33 Greg Banks wrote:
G'day,
Second attempt at sending, this time with no HTML. In the 21st century.
So I've been trying to use OBS source services for my project novaprova
https://build.opensuse.org/project/show/home:gnb:novaprova
to do a development tree (i.e. not a release) build on days when there are checkins. What I want is a process which does
1. get the latest source from the master branch in the git repo 2. pull the RPM and DEB package control files (.spec etc) out of a directory in there 3. figure out a version number 4. apply the version number to the .spec file 5. perform build 6. upload build results to repositories
I read this document
http://en.opensuse.org/openSUSE:Build_Service_Tips_and_Tricks#Perform_a_nigh tly_build_from_a_VCS_checkout
but I didn't want to do an osc commit for every build, or have a server sitting around whose only purpose was to run a cronjob. So I thought I could do some clever tricks with source services, resulting a single _service file which could be used unmodified for every nightly build. It almost worked. I found the following issues:
1. If the source service runs correctly, there is no visible log. The build log starts after that.
2. If the source service fails, the error messages only appear in a tooltip.
3. The build log page has a bug where it will repeat large chunks of a log.
4. Source services could be better documented. A lot better. For example I discovered @PARENT_TAG@ by reading the source.
I guess I'm partly to blame here. The tar_scm source service documents that feature in it's parameter manifest (/usr/lib/obs/services/tar_scm.service). But I agree there is currently no easy way to introspect a service how to really use it. That would probably be an osc feature.
5. The tar_scm service creates an uncompressed tarball, i.e. foo-git.1234567.tar. My .spec file was expecting a compressed tarball and failed.
6. The recompress service could be used to compress the tarball from tar_scm, but it takes a fixed incoming filename, without wildcards. This doesn't work when tar_scm is generating a tarball filename which has a version number encoded in it, which version number is not known at the time the _service file is written.
7. Likewise, the extract_files service could be used to extract a spec file from inside the tarball, but it requires a fixed incoming filename without wildcards. Given that in most projects there will be exactly one tarball, I would have thought that extract_files could have guessed correctly, or accepted a wildcard.
8. For the version number, I wanted to use a timestamp in ISO8601 basic format to day precision, i.e. "20131120" for 20 November 2013. The closest I can get is to use %ci as the versionformat in the tar_scm service, but this doesn't quite work. Git gives extended format to second precision withhttps://help.github.com/articles/post-receive-hooks timezone, e.g. 2013-11-20 10:06:35 -0800. The tar_scm script tries to clean this up, but the result still has useless information I don't want and spaces which break the other services.
The "versionformat" parameter capabilities depend on the underlying SCM. So they look slightly different depending on that, here are some best-practice examples: mercurial: https://build.opensuse.org/package/view_file/devel:languages:go/go-freetype-... git: https://build.opensuse.org/package/view_file/devel:languages:go/go-assert/_s... In git's case, the tar_scm service uses --pretty-format=... A quick check of the man-page reveals: %ad: author date (format respects --date= option) For --date, stackoverflow has all the answers with examples (that are also found in the man-page): http://stackoverflow.com/questions/7853332/git-log-date-formats So everything is already there.
Also, what's a good way to kick off an OBS build from a GitHub webhook?
-- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
G'day, On 21 November 2013 00:42, Sascha Peilicke <speilicke@suse.com> wrote:
On Wednesday 20 November 2013 10:24:33 Greg Banks wrote:
4. Source services could be better documented. A lot better. For example I discovered @PARENT_TAG@ by reading the source.
I guess I'm partly to blame here. The tar_scm source service documents that feature in it's parameter manifest (/usr/lib/obs/services/tar_scm.service). But I agree there is currently no easy way to introspect a service how to really use it. That would probably be an osc feature.
If so, osc would have to pull down the .service file from the server. But I'd settle for online documentation in the WebUI, perhaps generated from the .service file.
The "versionformat" parameter capabilities depend on the underlying SCM. So they look slightly different depending on that, here are some best-practice examples:
mercurial: https://build.opensuse.org/package/view_file/devel:languages:go/go-freetype-... git: https://build.opensuse.org/package/view_file/devel:languages:go/go-assert/_s...
In git's case, the tar_scm service uses --pretty-format=... A quick check of the man-page reveals:
%ad: author date (format respects --date= option)
For --date, stackoverflow has all the answers with examples (that are also found in the man-page):
http://stackoverflow.com/questions/7853332/git-log-date-formats
So everything is already there.
Yes, I got what I want using %cd in versionformat. Unfortunately the description in the .service file implies - incorrectly - that the version string will be exactly what "git log --pretty=format:" would give, which is *not* true. Firstly the output is sanitised with a sed script. Secondly git is given the --date=short option which changes the output of %cd. So just from reading the documentation I would never have tried %cd. -- Greg. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Thursday 21 November 2013 12:38:13 Greg Banks wrote:
G'day,
On 21 November 2013 00:42, Sascha Peilicke <speilicke@suse.com> wrote:
On Wednesday 20 November 2013 10:24:33 Greg Banks wrote:
4. Source services could be better documented. A lot better. For example I discovered @PARENT_TAG@ by reading the source.
I guess I'm partly to blame here. The tar_scm source service documents that feature in it's parameter manifest (/usr/lib/obs/services/tar_scm.service). But I agree there is currently no easy way to introspect a service how to really use it. That would probably be an osc feature.
If so, osc would have to pull down the .service file from the server.
Not really, check your /usr/lib/obs/services directory :-) osc already demands some services, depending on which OBS projects you work. So you should have some of those installed. If you want local invocation or disabled mode, you need them anyway.
But I'd settle for online documentation in the WebUI, perhaps generated from the .service file.
The "versionformat" parameter capabilities depend on the underlying SCM. So they look slightly different depending on that, here are some best-practice
examples: mercurial: https://build.opensuse.org/package/view_file/devel:languages:go/go-freetyp e-go/_service?expand=1> git: https://build.opensuse.org/package/view_file/devel:languages:go/g o-assert/_service?expand=1> In git's case, the tar_scm service uses --pretty-format=... A quick check of> the man-page reveals: %ad: author date (format respects --date= option)
For --date, stackoverflow has all the answers with examples (that are also
found in the man-page): http://stackoverflow.com/questions/7853332/git-log-date-formats
So everything is already there.
Yes, I got what I want using %cd in versionformat.
Unfortunately the description in the .service file implies - incorrectly - that the version string will be exactly what "git log --pretty=format:" would give, which is *not* true. Firstly the output is sanitised with a sed script. Secondly git is given the --date=short option which changes the output of %cd. So just from reading the documentation I would never have tried %cd.
Well, the source service XML metadata files aren't Bible chapters, so they may contain mistakes. But we allow pull requests :-) -- With kind regards, Sascha Peilicke SUSE Linux GmbH, Maxfeldstr. 5, D-90409 Nuernberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer HRB 16746 (AG Nürnberg)
G'day, On 22 November 2013 00:23, Sascha Peilicke <speilicke@suse.com> wrote:
On Thursday 21 November 2013 12:38:13 Greg Banks wrote:
G'day,
On 21 November 2013 00:42, Sascha Peilicke <speilicke@suse.com> wrote:
But I agree there is currently no easy way to introspect a service how to really use it. That would probably be an osc feature.
If so, osc would have to pull down the .service file from the server.
Not really, check your /usr/lib/obs/services directory :-)
That directory doesn't exist on my machine. The only OBS packages I've installed is osc, because the introductory documentation said this was all I needed to set up a project on the hosted service.
Well, the source service XML metadata files aren't Bible chapters, so they may contain mistakes. But we allow pull requests :-)
https://github.com/openSUSE/obs-service-tar_scm/pull/24 -- Greg. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 11/20/2013 1:24 PM, Greg Banks wrote:
G'day,
Second attempt at sending, this time with no HTML. In the 21st century.
Why should anyone have read any further than this expression of contempt and disregard for their wishes? Also, by the 21st century, you somehow never noticed that many techie/admin/developer focused mail lists prefer or even flatly require text? Let alone why? By the 21st century, this should no longer be news. -- bkw -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hey, On 22.11.2013 10:17, Brian K. White wrote:
HTML mail flame
If you really must, discuss this off-list. Thanks in advance. Henne -- Henne Vogelsang, Mailinglist Admin http://www.opensuse.org -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (5)
-
Adrian Schröter -
Brian K. White -
Greg Banks -
Henne Vogelsang -
Sascha Peilicke