[opensuse-factory] Rabbitmq OBS build success event client - Easy package testing - rabbit_obs_client
Hi, I made up a tiny package: home:trenn:rabbit_obs_client/rabbit_obs_client https://build.opensuse.org/package/show/home:trenn:rabbit_obs_client/rabbit_... I looked around before, but couldn't find anything similar. I use this instead of a (IMO complex/overdosed) jenkins worker or whatever solution to trigger test runs in case of successful OBS builds. I paste the description of the package below. Please let me know if this one helps and where/how it could be improved. Or whether something similar exists already or whatever you think about this.. Thanks!: ---- Use this package for automated tests, which are triggered once the OBS server has performed a successful build. OpenSUSE Build Service rabbit client is a systemd service written in python. It listens to rabbitmq build_success events from an OBS server. If an event is captured that matches a package in the config file, the newly built package rpm is retrieved from the OBS server, the local system is updated with the package and a specified command which may start tests is executed and logged. You need an osc user configured on the system and then simply add to /etc/rabbit_obs.conf [test_pkg] package = test_pkg project = home:trenn:cobbler_test_build repo = SLE_15_SP1 pkg_arch = noarch trigger_cmd = "/bin/test_my_package" [other_test_pkg] ... It would be great to get some feedback if this package helps and what could be improved, etc... Thanks, Thomas PS: Until today only tested with one package monitored, but it should work with multiple. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, May 13, 2020 at 11:52 AM Thomas Renninger <trenn@suse.de> wrote:
Hi,
I made up a tiny package: home:trenn:rabbit_obs_client/rabbit_obs_client https://build.opensuse.org/package/show/home:trenn:rabbit_obs_client/rabbit_...
I looked around before, but couldn't find anything similar. I use this instead of a (IMO complex/overdosed) jenkins worker or whatever solution to trigger test runs in case of successful OBS builds. I paste the description of the package below. Please let me know if this one helps and where/how it could be improved. Or whether something similar exists already or whatever you think about this.. Thanks!:
I've actually got some similar in-progress work that I was doing for an internal tool that I'm working to release publicly, but this code is quite helpful for understanding how it works. :) Also, in case you might be interested, I have a repo where I'm developing a collection of simple scripts to automate workflows (there's only one right now): https://pagure.io/obs-packaging-scripts -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Neal Gompa <ngompa13@gmail.com> writes:
On Wed, May 13, 2020 at 11:52 AM Thomas Renninger <trenn@suse.de> wrote:
Hi,
I made up a tiny package: home:trenn:rabbit_obs_client/rabbit_obs_client https://build.opensuse.org/package/show/home:trenn:rabbit_obs_client/rabbit_...
I looked around before, but couldn't find anything similar. I use this instead of a (IMO complex/overdosed) jenkins worker or whatever solution to trigger test runs in case of successful OBS builds. I paste the description of the package below. Please let me know if this one helps and where/how it could be improved. Or whether something similar exists already or whatever you think about this.. Thanks!:
I've actually got some similar in-progress work that I was doing for an internal tool that I'm working to release publicly, but this code is quite helpful for understanding how it works. :)
Also, in case you might be interested, I have a repo where I'm developing a collection of simple scripts to automate workflows (there's only one right now): https://pagure.io/obs-packaging-scripts
Since we're showcasing other projects now ;-), I'd like to highlight a hackweek project of a colleague: https://github.com/msirringhaus/obs_chat_bot This is a bot for Matrix.org that will ping you when a package finishes building on OBS. It also supports listening to openQA events, in case you want to get a pinged when your test finishes. Cheers, Dan -- Dan Čermák <dcermak@suse.com> Software Engineer Development tools SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer
Am Mittwoch, 13. Mai 2020, 20:17:36 CEST schrieb Dan Čermák:
Neal Gompa <ngompa13@gmail.com> writes:
On Wed, May 13, 2020 at 11:52 AM Thomas Renninger <trenn@suse.de> wrote:
Also, in case you might be interested, I have a repo where I'm developing a collection of simple scripts to automate workflows (there's only one right now): https://pagure.io/obs-packaging-scripts
This is for importing sources from fedora servers via fedpkg source into obs? I never needed this, didn't even know fedpkg exists. Looks a bit like the osc client, but not as powerful? About automating workflows, you probably want to have a look at: https://kamarada.github.io/en/2019/03/19/integrating-the-open-build-service-... and: https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.source_serv... (especially from 7.4 ongoing) I have added webhook (with a previously generated obs token) to my github cobbler fork which triggers a rebuild of my OBS cobbler test_build package: https://build.opensuse.org/package/show/home:trenn:cobbler_test_build/cobble... This package retrieves possibly new cobbler sources from github at build time via _service file. -------------------------------------------------------------------------------------- This is the first part of automation. Looks like this is what you are doing with fedpkg source? But from fedora dist servers, instead of gitlab or github? Anyway, above might be convenient for you when it comes to github/gitlab repos. Fedora pkgs might be based on these as well? The second part is with rabbit_obs_client: -> Once my sources which I submitted into my test_build github branch automatically got built in OBS, it downloads the fresh rpm(s), installs them, execute some tests and stores the results e.g. in: home:trenn:cobbler_test_build_cobbler_SLE_15_SP1_13_04_2020_19_56_48.log (datestring in file should be YYYY_MM_DD, already is iirc). What is missing (could certainly extended by quite some more, depending on whatever you focus), is kind of notification system. This is what Dan's chat bot below tries to achieve..., right?
Since we're showcasing other projects now ;-), I'd like to highlight a hackweek project of a colleague: https://github.com/msirringhaus/obs_chat_bot
This is a bot for Matrix.org that will ping you when a package finishes building on OBS. It also supports listening to openQA events, in case you want to get a pinged when your test finishes.
Nice! I even got a matrix account, but this is somewhat overdosed and too (matrix) specific for my simple notifications. I'd prefer email notifications and filtering, but implementing this on the rabbit client is nasty (smtp must properly be set up). I just had the idea to go back through OBS. I checked osc commands and notification settings: https://build.opensuse.org/my/subscriptions You might want to have a look at: - Package has failed to build - Package source service has failed - New comment for package created Receive notifications of comments created on a package for which you are... Watching the project/package X email x web x rss I dig a bit into this. Maybe the solution is not that difficult to set up. I could do (one may say misuse the comments for test notifications?): osc comment PROJECT PACKAGE "Test succeeded in X mins - Created 5 whatever" and watch the packages I test or watch the "test project" 's comments. I'll give this a try the next days... Also Marcus (Huewe)'s hint might help a lot. Thanks a lot for all your feedback! Thomas -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, May 14, 2020 at 12:04 PM Thomas Renninger <trenn@suse.de> wrote:
Am Mittwoch, 13. Mai 2020, 20:17:36 CEST schrieb Dan Čermák:
Neal Gompa <ngompa13@gmail.com> writes:
On Wed, May 13, 2020 at 11:52 AM Thomas Renninger <trenn@suse.de> wrote:
Also, in case you might be interested, I have a repo where I'm developing a collection of simple scripts to automate workflows (there's only one right now): https://pagure.io/obs-packaging-scripts
This is for importing sources from fedora servers via fedpkg source into obs? I never needed this, didn't even know fedpkg exists. Looks a bit like the osc client, but not as powerful?
fedpkg is considerably more powerful than osc. It can do _quite_ a lot more, I just needed it to fetch source tarballs for pushing to OBS. You can see a bit of all it is capable of in its man page: https://www.mankier.com/1/fedpkg There's other scripts I have that I'm working on cleaning up, that's just the first of many. :)
About automating workflows, you probably want to have a look at: https://kamarada.github.io/en/2019/03/19/integrating-the-open-build-service-... and: https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.source_serv... (especially from 7.4 ongoing)
I have added webhook (with a previously generated obs token) to my github cobbler fork which triggers a rebuild of my OBS cobbler test_build package: https://build.opensuse.org/package/show/home:trenn:cobbler_test_build/cobble...
This package retrieves possibly new cobbler sources from github at build time via _service file.
The problem is that _service files are "blind" and I want to be able to push specific refs or revisions from CI or whatever. OBS' capabilities here are _extremely_ poor. While my script defaults to Fedora, it's trivial to extend it to other servers using Dist-Git technology.
--------------------------------------------------------------------------------------
This is the first part of automation. Looks like this is what you are doing with fedpkg source? But from fedora dist servers, instead of gitlab or github? Anyway, above might be convenient for you when it comes to github/gitlab repos. Fedora pkgs might be based on these as well?
Fedora's package servers use a forge solution called Pagure: https://pagure.io/pagure There was a talk at the openSUSE Conference last year about it: https://www.youtube.com/watch?v=wastUxOT6IQ -- 真実はいつも一つ！/ Always, there's only one truth! -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Thomas, Thomas Renninger <trenn@suse.de> writes:
Hi,
I made up a tiny package: home:trenn:rabbit_obs_client/rabbit_obs_client https://build.opensuse.org/package/show/home:trenn:rabbit_obs_client/rabbit_...
I looked around before, but couldn't find anything similar. I use this instead of a (IMO complex/overdosed) jenkins worker or whatever solution to trigger test runs in case of successful OBS builds.
Does your rabbit listener react on successful builds or publish events? Depending on the load on the publisher, there is a non-negligible time difference between those two events. In the worst case you'll test your previous build, because the new one hasn't hit the repos yet. Cheers, Dan -- Dan Čermák <dcermak@suse.com> Software Engineer Development tools SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer
On 13/05/2020 20.14, Dan Čermák wrote:
Does your rabbit listener react on successful builds or publish events? Depending on the load on the publisher, there is a non-negligible time difference between those two events. In the worst case you'll test your previous build, because the new one hasn't hit the repos yet.
https://github.com/watologo1/rabbit_obs_client/blob/master/rabbit_obs_client... says it listens on build-success, but then it does not need a publish, because it uses osc getbinaries so it will also work with repos that have publishing disabled -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Donnerstag, 14. Mai 2020, 06:18:08 CEST schrieb Bernhard M. Wiedemann:
On 13/05/2020 20.14, Dan Čermák wrote:
Does your rabbit listener react on successful builds or publish events? Depending on the load on the publisher, there is a non-negligible time difference between those two events. In the worst case you'll test your previous build, because the new one hasn't hit the repos yet.
https://github.com/watologo1/rabbit_obs_client/blob/master/rabbit_obs_client .py#L63 says it listens on build-success, but then it does not need a publish, because it uses osc getbinaries
Correct, very well reviewed, thanks!
so it will also work with repos that have publishing disabled
Yes, publishing took sometimes several minutes up to half an hour or longer. Therefore I went the extra way of: osc get binaries One big disadvantage is, that this is the only reason you need a user with osc credentials (~/.config/osc/oscrc) the rabbit_obs_client systemd daemon can sudo (or su?) into. On the other hand side, this allows a rather rapid build and test workflow. Thomas -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2020-05-14 09:10:33 +0200, Thomas Renninger wrote:
Am Donnerstag, 14. Mai 2020, 06:18:08 CEST schrieb Bernhard M. Wiedemann:
On 13/05/2020 20.14, Dan Čermák wrote:
Does your rabbit listener react on successful builds or publish events? Depending on the load on the publisher, there is a non-negligible time difference between those two events. In the worst case you'll test your previous build, because the new one hasn't hit the repos yet.
https://github.com/watologo1/rabbit_obs_client/blob/master/rabbit_obs_client .py#L63 says it listens on build-success, but then it does not need a publish, because it uses osc getbinaries
Correct, very well reviewed, thanks!
so it will also work with repos that have publishing disabled
Yes, publishing took sometimes several minutes up to half an hour or longer. Therefore I went the extra way of: osc get binaries One big disadvantage is, that this is the only reason you need a user with osc credentials (~/.config/osc/oscrc) the rabbit_obs_client systemd daemon can sudo (or su?) into.
It is also possible to fetch the binaries without an account via the (unofficial) public route. - retrieve the binarylist marcus@linux:~> curl https://api.opensuse.org/public/build/home:trenn:rabbit_obs_client/openSUSE_... <binarylist> <binary filename="_buildenv" size="20112" mtime="1587061857"/> <binary filename="_statistics" size="949" mtime="1587061857"/> <binary filename="rabbit_obs_client-1.0+git20200416.6c6ee7d-7.1.noarch.rpm" size="16960" mtime="1587061857"/> <binary filename="rabbit_obs_client-1.0+git20200416.6c6ee7d-7.1.src.rpm" size="25002" mtime="1587061857"/> <binary filename="rpmlint.log" size="58" mtime="1587061857"/> </binarylist> marcus@linux:~> (it is also possible to filter src.rpms etc.) - retrieve the binaries (several requests) curl https://api.opensuse.org/public/build/home%3Atrenn%3Arabbit_obs_client/openS... \ -o binary.rpm curl https://api.opensuse.org/public/build/home%3Atrenn%3Arabbit_obs_client/openS... \ -o source.rpm - retrieve the binaries (single request) curl 'https://api.opensuse.org/public/build/home%3Atrenn%3Arabbit_obs_client/openSUSE_Tumbleweed/x86_64/rabbit_obs_client?view=cpio&binary=rabbit_obs_client-1.0%2Bgit20200416.6c6ee7d-7.1.noarch.rpm&binary=rabbit_obs_client-1.0%2Bgit20200416.6c6ee7d-7.1.src.rpm' \ -o binaries.cpio Marcus -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 14.05.20 um 09:10 schrieb Thomas Renninger:
One big disadvantage is, that this is the only reason you need a user with osc credentials (~/.config/osc/oscrc) the rabbit_obs_client systemd daemon can sudo (or su?) into.
You could just curl the binaries from the API. If it is a public project, you do not need user auth to fetch the binaries, just go the /public route. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (6)
-
Bernhard M. Wiedemann -
Dan Čermák -
Marcus Hüwe -
Neal Gompa -
Stefan Seyfried -
Thomas Renninger