[opensuse-buildservice] how to restart obs services correctly?
Hi all, last weeks my instance (2.10) had strange problems, often everything was slow / sluggish but the OBS server was more or less idle (I have thrown enough hardware at the problem I guess, 16GB of RAM and 10 VCPUs on a not totally up-to-date intel xeon x5670). Even the webfrontend was not responding and "osc st" for example took many minutes to eventually finish. Investigating, I found lots of "rpc timeout" messages in all services log files. Just restarting some of them (src server, scheduler) seemed to fix the issue often, but then it reappeard a day or a week later. Finally I just rebooted the whole server, now everything is fine again. I also enabled $logtimeout =1 in BSRPC.pm, even before rebooting. It looked like communication with bs_srcserver was the one that timed out, and from my hopeless tries in stracing the issue I'd guess some lock around the lastevents file was the problem. I think that this was caused by restarting services in the wrong order, which was finally "fixed" by rebooting the server. Is there a recommended way to restart all services without rebooting the server? Or is a reboot the simplest way to achieve that ;-) Thanks, seife -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello Stefan, (Hopefully, this 3rd attempt of mine will reach the list!) On Thu, 2020-01-16 at 16:33 +0100, Stefan Seyfried wrote:
I think that this was caused by restarting services in the wrong order, which was finally "fixed" by rebooting the server.
Is there a recommended way to restart all services without rebooting the server? Or is a reboot the simplest way to achieve that ;-)
The Administrator Guide has this table: https://openbuildservice.org/help/manuals/obs-admin-guide/obs.cha.installati... Below the table, you will see this: The sequence in the table reflects the start sequence... Hope this helps, Srinidhi.N�����r��y隊Z)z{.���Wlz��qﮞ˛���m�)z{.��+�: �{Zr�az�'z��j)h���Ǜ�)]���Ǿ� ޮ�^�ˬz��
Hi Srinidhi, Am 05.02.20 um 05:34 schrieb Srinidhi B:
Hello Stefan,
(Hopefully, this 3rd attempt of mine will reach the list!)
Sucess!! ;-)
On Thu, 2020-01-16 at 16:33 +0100, Stefan Seyfried wrote:
I think that this was caused by restarting services in the wrong order, which was finally "fixed" by rebooting the server.
Is there a recommended way to restart all services without rebooting the server? Or is a reboot the simplest way to achieve that ;-)
The Administrator Guide has this table:
https://openbuildservice.org/help/manuals/obs-admin-guide/obs.cha.installati...
Below the table, you will see this:
The sequence in the table reflects the start sequence...
Hope this helps,
Yes, this is very helpful. I will take a note to look into the excellent documentation in the future before asking questions that are already answered there... ;-) Thanks again, -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am 06.02.20 um 14:53 schrieb Stefan Seyfried:
Hi Srinidhi,
Am 05.02.20 um 05:34 schrieb Srinidhi B:
Hello Stefan,
The Administrator Guide has this table:
https://openbuildservice.org/help/manuals/obs-admin-guide/obs.cha.installati...
Below the table, you will see this:
The sequence in the table reflects the start sequence...
Hope this helps,
Yes, this is very helpful.
I will take a note to look into the excellent documentation in the future before asking questions that are already answered there... ;-)
I looked a bit closer today (while writing a "obs-restart" script... ;-) and found some possible problems in that table. This is the doc, I'll comment inline systemctl start obssrcserver.service # => After=obsservice.service!! systemctl start obsrepserver.service systemctl start obsservice.service # will already be running because of obssrcserver systemctl start obsdodup.service systemctl start obsdeltastore.service systemctl start obsscheduler.service # => After=obsrepserver obssrcserver => all fine systemctl start obsdispatcher.service # => After=obsrepserver obssrcserver obsscheduler => ok systemctl start obspublisher.service systemctl start obssigner.service systemctl start obswarden.service systemctl start obsclouduploadworker.service systemctl start obsclouduploadserver.service So obssrcserver obviously should be between obsservice and obsdodup. Missing from the table is obsservicedispatch.service, which has "After=obsservice.service", so it should be also before obsdodup. Am I right that https://github.com/openSUSE/obs-docu is the correct place for a pull request? Thanks, seife -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Am Donnerstag, 16. Januar 2020, 16:33:04 CET schrieb Stefan Seyfried:
Is there a recommended way to restart all services without rebooting the server? Or is a reboot the simplest way to achieve that ;-)
It escapes me, why there's no obs-server.target, that does this job properly without so much fuzz. Cheers, Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hey, On 05.02.20 12:28, Hans-Peter Jansen wrote:
Am Donnerstag, 16. Januar 2020, 16:33:04 CET schrieb Stefan Seyfried:
Is there a recommended way to restart all services without rebooting the server? Or is a reboot the simplest way to achieve that ;-)
It escapes me, why there's no obs-server.target, that does this job properly without so much fuzz.
For the frontend there is obs-api-support.target. Patches for the backend welcome :) Henne -- Henne Vogelsang http://www.opensuse.org Everybody has a plan, until they get hit. - Mike Tyson -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 5. Februar 2020, 17:25:18 CET Henne Vogelsang wrote:
Hey,
On 05.02.20 12:28, Hans-Peter Jansen wrote:
Am Donnerstag, 16. Januar 2020, 16:33:04 CET schrieb Stefan Seyfried:
Is there a recommended way to restart all services without rebooting the server? Or is a reboot the simplest way to achieve that ;-)
It escapes me, why there's no obs-server.target, that does this job properly without so much fuzz.
For the frontend there is obs-api-support.target. Patches for the backend welcome :)
no, not really.
The backend works distributed via different backends and the design of the backend
is actually different. It should not really matter.
Actually same for the api part, I was just too lazy to get it right in all parts ...
--
Adrian Schroeter
Hello Stefan, (Hopefully, this 4th attempt of mine will reach the list!) On Thu, 2020-01-16 at 16:33 +0100, Stefan Seyfried wrote:
I think that this was caused by restarting services in the wrong order, which was finally "fixed" by rebooting the server.
Is there a recommended way to restart all services without rebooting the server? Or is a reboot the simplest way to achieve that ;-)
The Administrator Guide has this table: https://openbuildservice.org/help/manuals/obs-admin-guide/obs.cha.installati... Below the table, you will see this: The sequence in the table reflects the start sequence... Hope this helps, Srinidhi.N�����r��y隊Z)z{.���Wlz��qﮞ˛���m�)z{.��+�: �{Zr�az�'z��j)h���Ǜ�)]���Ǿ� ޮ�^�ˬz��
participants (5)
-
Adrian Schröter -
Hans-Peter Jansen -
Henne Vogelsang -
Srinidhi B -
Stefan Seyfried