Mailinglist Archive: opensuse-buildservice (88 mails)

< Previous Next >
[opensuse-buildservice] bs_srcserver seems stuck at a specific lastnotification number
Hi,

I was investigating why my OBS server has a considerable amount of CPU load all
the time.
It's bs_srcserver that occupies one CPU almost entirely.
Investigating, it looks like it is processing the
/srv/obs/events/lastnotifications over and over again.

obs:~ # grep lastnotifications /srv/obs/log/src_server.log|tail
2018-06-19 15:21:17: [2007] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:20: [2011] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:23: [2014] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:27: [2018] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:30: [2022] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:33: [2024] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:36: [2030] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:39: [2040] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:43: [2048] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-06-19 15:21:46: [2052] GET (127.0.0.1)
/lastnotifications?start=297709&block=1

Actually, the oldest copy I have of this logfile is from 2018-05-01, and it has
the same start= number:

obs:~ # xzgrep lastnotifications /srv/obs/log/src_server.log-20180501.xz|head -5
2018-04-30 08:45:09: [30828] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-04-30 08:45:11: [30833] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-04-30 08:45:14: [30837] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-04-30 08:45:17: [30841] GET (127.0.0.1)
/lastnotifications?start=297709&block=1
2018-04-30 08:45:19: [30843] GET (127.0.0.1)
/lastnotifications?start=297709&block=1

obs:~ # l /srv/obs/events/
total 26832
drwxr-xr-x 9 obsrun obsrun 4096 Jun 4 12:37 ./
drwxr-xr-x 23 obsrun obsrun 4096 Sep 28 2017 ../
drwxr-xr-x 2 obsrun obsrun 4096 Jun 12 08:21 deltastore/
drwxr-xr-x 2 obsrun obsrun 4096 Jun 19 11:07 dispatch/
drwxr-xr-x 2 obsrun obsrun 905216 Jun 19 11:16 i586/
-rw-r--r-- 1 obsrun obsrun 520080 Jun 19 15:11 lastevents
-rw-r--r-- 1 obsrun obsrun 25804967 Jun 19 14:22 lastnotifications
drwxr-xr-x 2 obsrun obsrun 4096 Jun 19 14:22 publish/
drwxr-xr-x 2 obsrun obsrun 4096 Jun 19 15:11 repository/
drwxr-xr-x 2 obsrun obsrun 8192 Jun 19 14:21 signer/
drwxr-xr-x 2 obsrun obsrun 90112 Jun 19 14:22 x86_64/

note that I have no i586 scheduler anymore (had it before, but finally got rid
of it maybe a year ago).
The i586 events directory is full of files named after projects

obs:~ # l /srv/obs/events/i586/|wc
1557 14006 141552

Most of these are old (older than 2017-08-24, maybe the date I got rid of the
i586 scheduler), but two were updated today:

vsa1001:~ # ls -l /srv/obs/events/i586/ -rt|tail -2
-rw-r--r-- 1 obsrun obsrun 122 Jun 19 10:46 relsync:Images:SaaS:BM::images::i586
-rw-r--r-- 1 obsrun obsrun 122 Jun 19 11:16 relsync:Images:SaaS:HV::images::i586

I have no real idea how this all plays together, and what I have to do to get
this into a sane state again.

Any hints?

Best regards, and thanks in advance,

Stefan
--
Stefan Seyfried

"For a successful technology, reality must take precedence over
public relations, for nature cannot be fooled." -- Richard Feynman
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
Follow Ups