[Bug 859360] New: rc.status performance regression due to systemd integration
https://bugzilla.novell.com/show_bug.cgi?id=859360 https://bugzilla.novell.com/show_bug.cgi?id=859360#c0 Summary: rc.status performance regression due to systemd integration Classification: openSUSE Product: openSUSE 13.1 Version: Final Platform: i686 OS/Version: openSUSE 13.1 Status: NEW Severity: Major Priority: P5 - None Component: Basesystem AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: Stromeko@NexGo.DE QAContact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; U; noOS;) Gecko/20100101 Firefox/4.0 rc.status hands off to systemctl, which in turn can expend significantly more work than before. Specifically, systemctl status will search the log for the last ten lines by default and that takes more than two minutes on an old system with cold caches. Reproducible: Always Steps to Reproduce: 1. rcntp status 2. rcntp status 3. rcntp status Actual Results: The first invocation above takes more than two minutes on my system after the system has booted into KDE. The subsequent invocations are fast, but after some time of other disk activity repeating the exercise will lead to a slow first response again. The slowdown is caused by the log searching performed by systemctl (which rc.status hands the work off to). Expected Results: There is no reason to incur such large delays since the log output isn't used at all in scripts (it is only for human consumtion). Adding the option "-n0" to the hand-off in /etc/rc.status: exec /bin/systemctl -n0 $1 "${_rc_base}" resolves the problem and gets boot time back to normal. If the output is deemed important for interactive uses of the rc scripts, then the number of lines should be configurable with a variable (standard is ten lines) that is set to zero during the boot process. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c1
Michael Buchau
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c
Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c2
Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c3
--- Comment #3 from Ludwig Nussel
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c4
--- Comment #4 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c5
--- Comment #5 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c7
--- Comment #7 from Bernhard Wiedemann
Jounalctl told me I had almost a GB of logs. What I really had was 47MB of logs plus a few dozen of corrupt journal files (*.journal~), most of them from crashes due to GPU hangups. So journald/systemd switched to a new journal file, marked the old file as corrupted and still searched all of them. With all the corrupt journals out of the way log performance is actually OK at the moment.
IHMO it is OK to keep crashed journals for some days to find out why it crashed, but later it should either be rotated away or be purged or be ignored to not waste so much disk-IO on every "status" request -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c8
--- Comment #8 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c9
--- Comment #9 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c12
Marius Tomaschewski
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c15
--- Comment #15 from Ludwig Nussel
Created an attachment (id=575476) --> (http://bugzilla.novell.com/attachment.cgi?id=575476) [details] git diff HEAD^
this should do the work
That removes the journal display feature entirely. Would that -t 1 check work? If so I think it would be worthy to have. You have also added --ignore-dependencies, is there a reason/bug for that? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c16
--- Comment #16 from Dr. Werner Fink
You have also added --ignore-dependencies, is there a reason/bug for that?
Indeed ... the old rc<service> call had never expand any dependencies, that is user do not expect this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c17
--- Comment #17 from Bernhard Wiedemann
(In reply to comment #7)
I agree that this is somehow related to bnc#838475 and this is the reason for patch 1014-journald-with-journaling-FS.patch but this may trigger other problems compare with bnc#854884 and the last problem I can not debug as it is not reproducible here :(
I found a way to reproduce journald creating xxx.journal~ files using rcntp restart ; echo b > /proc/sysrq-trigger and also found that echo 1 > /proc/sys/vm/drop_caches ; time rcntp status becomes slower with every simulated crash (12-15 seconds after 5 crashes). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c18
Marius Tomaschewski
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c19
--- Comment #19 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c20
--- Comment #20 from Ludwig Nussel
(In reply to comment #15)
You have also added --ignore-dependencies, is there a reason/bug for that?
Indeed ... the old rc<service> call had never expand any dependencies, that is user do not expect this.
Yes but the old behavor sucked :) It's pretty neat to have rc<something> start mandatory services like rpcbind. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c21
--- Comment #21 from Dr. Werner Fink
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c23
--- Comment #23 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c25
--- Comment #25 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c27
--- Comment #27 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c28
Ludwig Nussel
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c29
--- Comment #29 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c30
--- Comment #30 from Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c31
--- Comment #31 from Swamp Workflow Management
https://bugzilla.novell.com/show_bug.cgi?id=859360
https://bugzilla.novell.com/show_bug.cgi?id=859360#c32
--- Comment #32 from Swamp Workflow Management
participants (1)
-
bugzilla_noreply@novell.com