[opensuse-buildservice] after latest update (1.6.99-114.1) load is very high
After updating to the latest obs* packages (1.6.99-114.1) load is very high. I see a load of between 8 and 10. Since I have two workers going, I'd expect to see a load of more like 2 or occasionally 3. The logs show *lots* "GET /workerstatus?" to the rep_server, and "GET /build/_workerstatus" to the src_server, and the delayed_job log is constantly noisy. Stopping and starting the obs software doesn't help. One thing I also changed today is I added a remote repo. -- Jon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag 05 Februar 2010 schrieb Jon Nelson:
After updating to the latest obs* packages (1.6.99-114.1) load is very high.
I see a load of between 8 and 10. Since I have two workers going, I'd expect to see a load of more like 2 or occasionally 3.
The logs show *lots* "GET /workerstatus?" to the rep_server, and "GET /build/_workerstatus" to the src_server, and the delayed_job log is constantly noisy.
You need to check the logs - very likely of the api. The cron job fetches the workerstatus every minute, so depending on what '*lots*' means to you, this could be a bug. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Mon, Feb 8, 2010 at 3:01 AM, Stephan Kulow
Am Freitag 05 Februar 2010 schrieb Jon Nelson:
After updating to the latest obs* packages (1.6.99-114.1) load is very high.
I see a load of between 8 and 10. Since I have two workers going, I'd expect to see a load of more like 2 or occasionally 3.
The logs show *lots* "GET /workerstatus?" to the rep_server, and "GET /build/_workerstatus" to the src_server, and the delayed_job log is constantly noisy.
Upgraded to 1.7.0 It turns out to be the delayed job code. Every second (as in 60 times a minute) it is requesting _workerstatus and inserting 5 rows into status_histories and DELETING rows from delayed_jobs. Also an UPDATE in there, too. Is there anything that will prune those tables? Is it really necessary to run once every second? Load does return to normal-ish after a while, 0.1 instead of 0.0. I'll see if it comes back. -- Jon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Donnerstag 11 Februar 2010 schrieb Jon Nelson:
On Mon, Feb 8, 2010 at 3:01 AM, Stephan Kulow
wrote: Am Freitag 05 Februar 2010 schrieb Jon Nelson:
After updating to the latest obs* packages (1.6.99-114.1) load is very high.
I see a load of between 8 and 10. Since I have two workers going, I'd expect to see a load of more like 2 or occasionally 3.
The logs show *lots* "GET /workerstatus?" to the rep_server, and "GET /build/_workerstatus" to the src_server, and the delayed_job log is constantly noisy.
Upgraded to 1.7.0
It turns out to be the delayed job code. Every second (as in 60 times a minute) it is requesting _workerstatus and inserting 5 rows into status_histories and DELETING rows from delayed_jobs. Also an UPDATE in there, too. Is there anything that will prune those tables? Is it really necessary to run once every second?
Delayed job wil run as often as rake jobs:workerstatus was called. And as the cron job only emits on job per minute, you very likely had the cron job running before you had the delayed job daemon running. And no, the status_histories table is not yet pruned, I'm still not sure if long time history can't be interesting and so I need to prune and replace with averages but for that I need a) more data and b) time to experiment. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Donnerstag, 11. Februar 2010 14:12:48 schrieb Jon Nelson:
On Mon, Feb 8, 2010 at 3:01 AM, Stephan Kulow
wrote: Am Freitag 05 Februar 2010 schrieb Jon Nelson:
After updating to the latest obs* packages (1.6.99-114.1) load is very high.
I see a load of between 8 and 10. Since I have two workers going, I'd expect to see a load of more like 2 or occasionally 3.
The logs show *lots* "GET /workerstatus?" to the rep_server, and "GET /build/_workerstatus" to the src_server, and the delayed_job log is constantly noisy.
Upgraded to 1.7.0
It turns out to be the delayed job code. Every second (as in 60 times a minute) it is requesting _workerstatus and inserting 5 rows into status_histories and DELETING rows from delayed_jobs. Also an UPDATE in there, too. Is there anything that will prune those tables? Is it really necessary to run once every second?
it doesn't usually. But you had most likely a backlog which it had to process.
Load does return to normal-ish after a while, 0.1 instead of 0.0. I'll see if it comes back.
-- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Feb 11, 2010 at 7:20 AM, Adrian Schröter
Am Donnerstag, 11. Februar 2010 14:12:48 schrieb Jon Nelson:
On Mon, Feb 8, 2010 at 3:01 AM, Stephan Kulow
wrote: Am Freitag 05 Februar 2010 schrieb Jon Nelson:
After updating to the latest obs* packages (1.6.99-114.1) load is very high.
I see a load of between 8 and 10. Since I have two workers going, I'd expect to see a load of more like 2 or occasionally 3.
The logs show *lots* "GET /workerstatus?" to the rep_server, and "GET /build/_workerstatus" to the src_server, and the delayed_job log is constantly noisy.
Upgraded to 1.7.0
It turns out to be the delayed job code. Every second (as in 60 times a minute) it is requesting _workerstatus and inserting 5 rows into status_histories and DELETING rows from delayed_jobs. Also an UPDATE in there, too. Is there anything that will prune those tables? Is it really necessary to run once every second?
it doesn't usually. But you had most likely a backlog which it had to process.
Aha! That does explain everything. -- Jon -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (3)
-
Adrian Schröter
-
Jon Nelson
-
Stephan Kulow