[opensuse-factory] service ntp
Ever since systemd took over it waits for ntp to complete startup, which takes a long time on my machine. Is there any way I can make systemd start ntp, but not wait for it before it considers bringing up kdm? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-01-16 19:59, Achim Gratz wrote:
Ever since systemd took over it waits for ntp to complete startup, which takes a long time on my machine. Is there any way I can make systemd start ntp, but not wait for it before it considers bringing up kdm?
I think it is intentional to wait. When eventually ntp service syncs, the clock can jump ahead or backwards, hopefully by less than a second if the cmos clock is not off by much, but eventually by minutes or hours. Some of the Linux services that are started after ntp can react badly to clock changes. I know of at least one (dovecot) that bails out if the clock goes back. It simply crashes nicely, with a message as to the reason. Of course, if you have no network you should disable or remove ntp. Or find out why it starts slowly. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Carlos E. R. writes:
I think it is intentional to wait.
When eventually ntp service syncs, the clock can jump ahead or backwards, hopefully by less than a second if the cmos clock is not off by much, but eventually by minutes or hours.
That is already done by ntpdate so the ntpd should only slew.
Some of the Linux services that are started after ntp can react badly to clock changes. I know of at least one (dovecot) that bails out if the clock goes back. It simply crashes nicely, with a message as to the reason.
Of course, if you have no network you should disable or remove ntp.
I have.
Or find out why it starts slowly.
I found out why it does that and surprisingly it looks like systemd actually starts it too late. The standard config is to chroot ntpd and that produces a lot of disk activity that takes a long time, especially during boot when there are other processes competing for disk I/O. I think that with the old init system where it was started realtively early, there simply wasn't as much competition for disk and perhaps there also weren't as many things to copy over into the chroot. I've switched chrooting off as an experiment, lets see if the boot times go back to something halfway sane (it's an old machine, so that'll still be some time). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Achim Gratz writes:
I've switched chrooting off as an experiment, lets see if the boot times go back to something halfway sane (it's an old machine, so that'll still be some time).
That didn't change things. To make sure things are clean, I de- and reinstalled ntp The first two starts (i.e. the one while booting and the next one from a root shell) of service ntp will take over two minutes with lots of disk activity (like it's seeking through lots of directories) and from then on it's down to a few seconds cold cache and under two seconds warm. If I then thrash the cache again by runing a find somewhere else on the disk, the startup time goes back to around two minutes. So somehow starting ntp is triggering a lot of stats and directory traversals, but I don't quite see where. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Achim Gratz wrote:
Achim Gratz writes:
I've switched chrooting off as an experiment, lets see if the boot times go back to something halfway sane (it's an old machine, so that'll still be some time).
That didn't change things. To make sure things are clean, I de- and reinstalled ntp The first two starts (i.e. the one while booting and the next one from a root shell) of service ntp will take over two minutes with lots of disk activity (like it's seeking through lots of directories) and from then on it's down to a few seconds cold cache and under two seconds warm. If I then thrash the cache again by runing a find somewhere else on the disk, the startup time goes back to around two minutes. So somehow starting ntp is triggering a lot of stats and directory traversals, but I don't quite see where.
I thought about your prob... and thought it "shouldn't" be hard to tell ... going through.. strace and adding... then I realized... it might be a bit more complicated than some might like.. Attached is shell & perl script that should tell you what system calls are taking up the most time, with highest cumulative times at the top... By default it will show the top 20 lines. Make sure whatever you time doesn't go into background daemon mode and never exits -- or the command won't terminate (it does follow children). It will dump a trace file in /tmp so it can read it in and tally the results. If you don't want it when you are done, feel free to delete it (/tmp/timings.{txt,cmd}) 'wheretime' -- where is the time going? example:
wheretime "gvim -c q" generating timings for gvim -c q... 1.55sec 0.11usr 0.38sys (31.70% cpu) 0.235206 [pid 94759]access("/usr/share/fonts/TTF/ 0.121237 [pid 94759]open("/usr/share/locale/local 0.119617 [pid 94759]access("/usr/share/fonts/TTF/ 0.119050 [pid 94759]brk(0x17c2000) = 0x17c2000 0.099169 brk(0x230c000) = 0x230c000 0.075638 brk(0x1962000) = 0x1962000 0.028015 [pid 94759]recvfrom(5, ""..., 4096, 0, N 0.025954 [pid 94759]stat("/usr/share/fonts/baekmu 0.021227 [pid 94759]recvfrom(5, 0x1638df4, 4096, 0.018889 [pid 94759]poll([...], 1, 4294967295) =
----------- It's a rough tool but it might help you narrow down where the activity is taking place. The numbers in the column are in seconds / same call... I,e. any call w/same params is summed w/other calls of same params... children are listed as [pid xxx]and there commands... To change the number of lines displayed, put it as another param on the cmdline... (don't forget quotes around the input command)... (gvim -c q sees how fast gvim loads and where it spends time.) like: law> wheretime "gvim -c q" 30 generating timings for gvim -c q... 1.54sec 0.12usr 0.36sys (31.98% cpu) 0.235588 [pid 95675]access("/usr/share/fonts/TTF/ 0.121255 [pid 95675]open("/usr/share/locale/local 0.119125 [pid 95675]brk(0x28f6000) = 0x28f6000 0.119075 [pid 95675]access("/usr/share/fonts/TTF/ 0.098136 brk(0x3440000) = 0x3440000 0.072332 brk(0x2a96000) = 0x2a96000 0.026680 [pid 95675]stat("/usr/share/fonts/baekmu 0.023179 [pid 95675]recvfrom(5, ""..., 4096, 0, N 0.021721 [pid 95675]recvfrom(5, 0x276cdf4, 4096, 0.019983 [pid 95675]poll([...], 1, 4294967295) = 0.018202 [pid 95675]close(3) = 0 0.015783 [pid 95675]stat("/usr/share/fonts/new", 0.013872 exit_group(0) = ? 0.013137 [pid 95675]stat("/usr/share/fonts/OTF/co 0.013011 read(3, "", 262144) = 0 0.011248 [pid 95675]stat("/usr/share/fonts/TTF/ka 0.010954 close(3) = 0 0.010867 [pid 95675]poll([...], 2, 0) = 0 (Timeou 0.009741 [pid 95675]read(11, ""..., 8192) = 8192 0.008310 [pid 95675]munmap(0x7f598060e000, 398521 0.007350 openat(AT_FDCWD, "/home/law/.vim/compile 0.007157 [pid 95675]brk(0x2938000) = 0x2938000 0.006779 read(6, "", 262144) = 0 0.006668 [pid 95675]brk(0x2958000) = 0x2958000 0.006654 [pid 95675]fstatfs(3, {f_type=0x58465342 0.006460 [pid 95675]read(8, "", 8192) = 0 0.005972 read(3, ""..., 832) = 832 0.005547 [pid 95675]brk(0x2978000) = 0x2978000 0.005107 recvfrom(5, 0x276cdf4, 4096, 0, 0, 0) = 0.004956 [pid 95675]brk(0x2917000) = 0x2917000 (hope it works for you! ;-)) #!/bin/bash # by Linda A. Walsh (c) 2014 -- free for use to nice people! # (not guaranteed to do anything useful)... p=$0 cmd="/bin/sleep 1.5" lines=10 if [[ $# -lt 1 ]]; then echo "wheretime \"cmd [flags]\" [opt#lines]" echo " - w/cmd+flags in quotes -- to time cmd & flags" echo "using default test \"$cmd\" $lines" else cmd="$1" shift fi if [[ $# ]]; then if [[ $1 =~ ^[0-9]+$ ]]; then lines=$1; fi fi #testing #if [[ $p -nt /tmp/timings.txt || # /tmp/timings.cmd -nt /tmp/timings.txt ]]; then rm -f /tmp/timings.{cmd,txt} #fi read last < <(if [[ -e /tmp/timings.cmd ]]; then cat /tmp/timings.cmd; else echo ""; fi) TIMEFORMAT="%2Rsec %2Uusr %2Ssys (%P%% cpu)" if [[ $last != $cmd ]]; then echo "generating timings for $cmd..." echo "$cmd" >/tmp/timings.cmd time strace -s 0 -r -qq -f $cmd >& /tmp/timings.txt fi prog=' ##!/usr/bin/perl;use warnings;use strict;use P; my ($val, $calls)=(0,{}); my $lines=15; if ($#ARGV>=0) { $lines=$ARGV[0] if $ARGV[0] =~ /^\d+$/; shift; } while(<>) { chomp; if (m{((?:\[pid\s\d+\])?)\s*([\d\.]+)\s(\S+.*)$}){ my ($pid,$time,$call)=($1, $2, $3); $call=($pid||"").$call; if (ref $calls->{$call}) { $calls->{$call}[0]=$calls->{$call}[0]+$time; #P "cc1=%s", $calls->{$call} } else { $calls->{$call}=[$time, $call]; } } } my @scalls = sort { my ($ca, $cb) = @{$calls}{$a,$b}; $cb->[0] <=> $ca->[0] || $ca->[1] cmp $cb->[1] } keys %$calls; my $i; for ($i=0; $i<$lines; ++$i) { my $sc=$scalls[$i]; printf " %11.6f %.40s\n", @{$calls->{$sc}}[0,1]; } # vim: ts=2 sw=2 ' perl -e "$prog" "$lines" </tmp/timings.txt # vim: ts=2 sw=2
On 2014-01-18 00:19, Linda Walsh wrote:
Attached is shell & perl script that should tell you what system calls are taking up the most time, with highest cumulative times at the top...
Interesting! Thanks. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Linda Walsh writes:
I thought about your prob... and thought it "shouldn't" be hard to tell ... going through.. strace and adding... then I realized... it might be a bit more complicated than some might like..
I'll have to isolate the part of the init script that causes the disk activity before I can further trace it.
Attached is shell & perl script that should tell you what system calls are taking up the most time, with highest cumulative times at the top...
Thanks, I think it'll come in handy once I know what program actually tries to deep-dive on the disk. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Achim Gratz wrote:
Thanks, I think it'll come in handy once I know what program actually tries to deep-dive on the disk.
I thought you said it was when the ntp start up script ran? and that you tried both with and without the "chroot" with same result. With that, the only thing the startup script should be doing is running the ntp daemon. I.e. here's the systemV init code: case "$1" in start) #<linda>## so pretend this is set to "no", initially (il if [ "$NTPD_FORCE_SYNC_ON_STARTUP" = "yes" ]; then # get the initial date from the timeservers configured in ntp.conf ntpd_is_running || $0 ntptimeset fi echo -n "Starting network time protocol daemon (NTPD)" # do we need a refclock symlink? parse_symlink #<linda># this is for the chroot case # do we run chrooted? test "${NTPD_RUN_CHROOTED}" = "yes" && prepare_chroot #<linda>#just break this line down for your test startproc $NTPD_BIN -p ${NTPD_PID#${CHROOT_PREFIX}} $NTPD_OPTIONS -c $NTP_CONF if [ -n "${NTP_KEYS}" ]; then runtime_configuration #<linda> ignore this for 1 test.. fi rc_status -v ---------- That's the startup case. -- just use the values after NTPD_BIN..like NTPD_BIN="/usr/sbin/ntpd" NTPD_OPTIONS="-g -u ntp:ntp" If that is a the area of your problem, add "-d" to NTPD_OPTIONS /usr/sbin/ntpd -g -u ntp:ntp -n [-q] [-A] Use -q to have it exit after the first time set - that would be a minimal case, I would think. Maybe -A to have it run w/o auth calls (if they would be the problem). -------------- This type of script is what is missing in "systemd" -- that makes it hard to debug problems like these, BTW, so going back to an earlier suse version to pull out the start scripts is a good start-point for seeing what systemd "should" be doing. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Linda Walsh writes:
I thought you said it was when the ntp start up script ran?
Exactly.
and that you tried both with and without the "chroot" with same result.
Yes.
With that, the only thing the startup script should be doing is running the ntp daemon.
You'd think, but it's not that. I've ran the actual invocation by hand and never observed anything unusual.
This type of script is what is missing in "systemd" -- that makes it hard to debug problems like these, BTW, so going back to an earlier suse version to pull out the start scripts is a good start-point for seeing what systemd "should" be doing.
Since it "hangs" more than two minutes on boot, I was able to log in on the console and see what was running and what caught my eye was systemctl status ntp So I added "set -x" to the init script and ran it again after KDE started (this is where it takes again two minutes) and it hung at the exact same command, more specifically when it was trying to list the log related to ntp (long after it had already determined the status of the daemon). So I wasn't seeing ghosts after all and this is indeed caused by systemd and it's "improved" way of dealing with system logging. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 19.01.2014 10:03, schrieb Achim Gratz:
So I wasn't seeing ghosts after all and this is indeed caused by systemd and it's "improved" way of dealing with system logging.
journald is totally broken wrt. its logs not living on a SSD. Either update your system to SSD or disable persistant journal ("rm -rf /var/log/journal && reboot" should probably achieve this). This is such utter crap, I still cannot believe that this was made default... :-( Sorry for the rant, I tried for a long time to find nice words but these were the nicest I could find :-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
В Sun, 19 Jan 2014 12:39:14 +0100 Stefan Seyfried <stefan.seyfried@googlemail.com> пишет:
Am 19.01.2014 10:03, schrieb Achim Gratz:
So I wasn't seeing ghosts after all and this is indeed caused by systemd and it's "improved" way of dealing with system logging.
journald is totally broken wrt. its logs not living on a SSD.
While I do not argue with that, all those "$0 status" calls in /etc/init.d/ntp are totally redundant in case of systemd.
Either update your system to SSD or disable persistant journal ("rm -rf /var/log/journal && reboot" should probably achieve this).
Unfortunately that comes at the expense of losing historical logs. I wonder if limiting "systemctl status" output to messages since last boot would make it faster.
This is such utter crap, I still cannot believe that this was made default... :-(
It is not default for all I can tell. On all installations I did I had to enable persistent journal manually.
Sorry for the rant, I tried for a long time to find nice words but these were the nicest I could find :-)
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi Andrey, Am 19.01.2014 13:05, schrieb Andrey Borzenkov:
В Sun, 19 Jan 2014 12:39:14 +0100 Stefan Seyfried <stefan.seyfried@googlemail.com> пишет: Unfortunately that comes at the expense of losing historical logs. I wonder if limiting "systemctl status" output to messages since last boot would make it faster.
If it does not do that already, I'd consider this another indication of "total broken piece of ..." :-)
This is such utter crap, I still cannot believe that this was made default... :-(
It is not default for all I can tell. On all installations I did I had to enable persistent journal manually.
Correct, just checked that with a fresh 13.1 install. So apparently lots of people got tricked into creating /var/log/journal manually by early journal propaganda :-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Sun, 19 Jan 2014 13:25:35 +0100 schrieb Stefan Seyfried:
It is not default for all I can tell. On all installations I did I had to enable persistent journal manually.
Correct, just checked that with a fresh 13.1 install. So apparently lots of people got tricked into creating /var/log/journal manually by early journal propaganda :-)
I have several systems which were updated either with DVD or zypper dup from pre-systemd versions and they all have the /var/log/journal directory. I am quite sure I did not create it on any of these systems. Dieter -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-01-19 13:47, dieter wrote:
I have several systems which were updated either with DVD or zypper dup from pre-systemd versions and they all have the /var/log/journal directory. I am quite sure I did not create it on any of these systems.
Not here. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Am Sonntag, 19. Januar 2014, 13:25:35 schrieb Stefan Seyfried:
[...] Correct, just checked that with a fresh 13.1 install. So apparently lots of people got tricked into creating /var/log/journal manually by early journal propaganda :-)
Or: A lot of people do not read release notes when upgrading: https://www.suse.com/releasenotes/x86_64/openSUSE/12.3/#idm47462148845248 Gruß Jan -- You cannot strengthen the weak by weakening the strong. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Jan Ritzerfeld wrote:
Am Sonntag, 19. Januar 2014, 13:25:35 schrieb Stefan Seyfried:
[...] Correct, just checked that with a fresh 13.1 install. So apparently lots of people got tricked into creating /var/log/journal manually by early journal propaganda :-)
Or: A lot of people do not read release notes when upgrading: https://www.suse.com/releasenotes/x86_64/openSUSE/12.3/#idm47462148845248
You mean right under where it says: ---- 5.2. systemd: Cleaning Directories (/tmp and /var/tmp) By default, systemd cleans tmp directories daily as configured in /usr/lib/tmpfiles.d/tmp.conf. Users can change it by copying /usr/lib/tmpfiles.d/tmp.conf to /etc/tmpfiles.d/tmp.conf and modifying the copied file. It will override /usr/lib/tmpfiles.d/tmp.conf. Note: systemd [does not support compatibility] with sysconfig variables in /etc/sysconfig/cron such as TMP_DIRS_TO_CLEAR. --- Given how accurate the release notes are about cleaning tmp dirs, do you really find it surprising that alot of people don't read the release notes? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Samstag, 25. Januar 2014, 20:04:19 schrieb Linda Walsh:
[...] You mean right under where it says: ---- 5.2. systemd: Cleaning Directories (/tmp and /var/tmp)
By default, systemd cleans tmp directories daily as configured in /usr/lib/tmpfiles.d/tmp.conf. Users can change it by copying /usr/lib/tmpfiles.d/tmp.conf to /etc/tmpfiles.d/tmp.conf and modifying the copied file. It will override /usr/lib/tmpfiles.d/tmp.conf.
Note: systemd [does not support compatibility] with sysconfig variables in /etc/sysconfig/cron such as TMP_DIRS_TO_CLEAR. --- Given how accurate the release notes are about cleaning tmp dirs, do you really find it surprising that alot of people don't read the release notes?
The release note you quoted is correct and even tells you how and where you should configure the age used to decide what files to delete when cleaning. Additionally, it warns you that the old sysconfig variables are not honored anymore. However, there were some bugs regarding private temps. Gruß Jan -- There may be times when we are powerless to prevent injustice, but there must never be a time when we fail to protest it. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
В Sun, 26 Jan 2014 14:47:09 +0100 Jan Ritzerfeld <suse@mailinglists.jan.ritzerfeld.org> пишет:
Am Samstag, 25. Januar 2014, 20:04:19 schrieb Linda Walsh:
[...] You mean right under where it says: ---- 5.2. systemd: Cleaning Directories (/tmp and /var/tmp)
By default, systemd cleans tmp directories daily as configured in /usr/lib/tmpfiles.d/tmp.conf. Users can change it by copying /usr/lib/tmpfiles.d/tmp.conf to /etc/tmpfiles.d/tmp.conf and modifying the copied file. It will override /usr/lib/tmpfiles.d/tmp.conf.
Note: systemd [does not support compatibility] with sysconfig variables in /etc/sysconfig/cron such as TMP_DIRS_TO_CLEAR. --- Given how accurate the release notes are about cleaning tmp dirs, do you really find it surprising that alot of people don't read the release notes?
The release note you quoted is correct
Quoting /usr/lib/tmpfiles.d/tmp.conf: # Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories This does not sound like "daily by default". -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am Sonntag, 26. Januar 2014, 17:50:39 schrieb Andrey Borzenkov:
В Sun, 26 Jan 2014 14:47:09 +0100
Jan Ritzerfeld <suse@mailinglists.jan.ritzerfeld.org> пишет:
Am Samstag, 25. Januar 2014, 20:04:19 schrieb Linda Walsh: [...]
5.2. systemd: Cleaning Directories (/tmp and /var/tmp) [...] The release note you quoted is correct
Quoting /usr/lib/tmpfiles.d/tmp.conf:
# Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories
This does not sound like "daily by default".
Well, it does exactly what is said in the whole(!) sentence: "By default, systemd cleans tmp directories daily as configured in /usr/lib/tmpfiles.d/tmp.conf." And, as you correctly stated, /usr/lib/tmpfiles.d/tmp.conf tells systemd to never delete anything in the tmp directories. Not deleting has been the default behavior of openSUSE for years IIRC. Gruß Jan -- Computer hackers do it all night long. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-01-26 14:50, Andrey Borzenkov wrote:
В Sun, 26 Jan 2014 14:47:09 +0100 Jan Ritzerfeld <> пишет:
The release note you quoted is correct
Quoting /usr/lib/tmpfiles.d/tmp.conf:
# Clear tmp directories separately, to make them easier to override # SUSE policy: we don't clean those directories
This does not sound like "daily by default".
So, the situation is that systemd does not clean /tmp, and the old cron scripts are disabled, so nothing clear them. Right? I do not know about other tmp directories. Some are tmpfs, so cleared on boot; but many machines are not rebooted in long time. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Am Sonntag, 26. Januar 2014, 15:08:14 schrieb Carlos E. R.:
[...] So, the situation is that systemd does not clean /tmp, and the old cron scripts are disabled, so nothing clear them. Right? [...]
systemd deletes old items in /tmp and /var/tmp if you tell systemd to do so "by copying /usr/lib/tmpfiles.d/tmp.conf to /etc/tmpfiles.d/tmp.conf and modifying the copied file. " and /usr/lib/tmpfiles.d/tmp.conf says "See tmpfiles.d(5) for details". That is, you simply have to specify the desired age and systemd deletes old items every day and 15 minutes after boot. However, there were some bugs regarding private temps. Gruß Jan -- The future isn't what it used to be. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-01-26 15:48, Jan Ritzerfeld wrote:
Am Sonntag, 26. Januar 2014, 15:08:14 schrieb Carlos E. R.:
[...] So, the situation is that systemd does not clean /tmp, and the old cron scripts are disabled, so nothing clear them. Right? [...]
systemd deletes old items in /tmp and /var/tmp if you tell systemd to do so "by copying /usr/lib/tmpfiles.d/tmp.conf to /etc/tmpfiles.d/tmp.conf and modifying the copied file. " and /usr/lib/tmpfiles.d/tmp.conf says "See tmpfiles.d(5) for details". That is, you simply have to specify the desired age and systemd deletes old items every day and 15 minutes after boot. However, there were some bugs regarding private temps.
That is more complicated than the old sysconfig method in YaST. No manual reading was required. It even seems less configurable (users to skip). I have seen several posts referring people to documentation about how to configure tmp dir clearing, the old sysconfig way - which of course, does not work. The variables in "/etc/sysconfig/cron" still exist, and there is no mention in the config files that they do not work. So people try them. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Am Sonntag, 26. Januar 2014, 16:06:04 schrieb Carlos E. R.:
[...] I have seen several posts referring people to documentation about how to configure tmp dir clearing, the old sysconfig way - which of course, does not work. The variables in "/etc/sysconfig/cron" still exist, and there is no mention in the config files that they do not work. So people try them.
Yes. That is indeed a problem/bug: https://bugzilla.novell.com/show_bug.cgi?id=860058 Gruß Jan -- You have not converted a man because you have silenced him. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 26.01.2014 16:59, Jan Ritzerfeld wrote:
Am Sonntag, 26. Januar 2014, 16:06:04 schrieb Carlos E. R.:
[...] I have seen several posts referring people to documentation about how to configure tmp dir clearing, the old sysconfig way - which of course, does not work. The variables in "/etc/sysconfig/cron" still exist, and there is no mention in the config files that they do not work. So people try them.
Yes. That is indeed a problem/bug: https://bugzilla.novell.com/show_bug.cgi?id=860058
Gruß Jan
Now, what's desirable? Shouldn't the yast component be fixed instead of removed? If there is any. I always set this by hand in the past. Anyway, the topic should be changed. -- Ralf Lang Linux Consultant / Developer Tel.: +49-170-6381563 Mail: lang@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg / http://www.b1-systems.de GF: Ralph Dehner / Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Sunday 19 January 2014, Stefan Seyfried wrote:
Hi Andrey,
Am 19.01.2014 13:05, schrieb Andrey Borzenkov:
В Sun, 19 Jan 2014 12:39:14 +0100 Stefan Seyfried <stefan.seyfried@googlemail.com> пишет: Unfortunately that comes at the expense of losing historical logs. I wonder if limiting "systemctl status" output to messages since last boot would make it faster.
If it does not do that already, I'd consider this another indication of "total broken piece of ..." :-)
This is such utter crap, I still cannot believe that this was made default... :-(
It is not default for all I can tell. On all installations I did I had to enable persistent journal manually.
Correct, just checked that with a fresh 13.1 install. So apparently lots of people got tricked into creating /var/log/journal manually by early journal propaganda :-)
Was this intended? Just a while back we've had a real syslog. Then we were told that journal is the better one but per default we get nothing? cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
* Andrey Borzenkov <arvidjaar@gmail.com> [2014-01-19 13:06]:
Unfortunately that comes at the expense of losing historical logs. I wonder if limiting "systemctl status" output to messages since last boot would make it faster.
Fortunately you can export it before switching to rsyslog.
This is such utter crap, I still cannot believe that this was made default... :-(
It is not default for all I can tell. On all installations I did I had to enable persistent journal manually.
In Factory it will be: https://github.com/openSUSE/patterns/commit/e5d89dc9b6f5c5e02ae4bd4170d28606... One more of these major changes that were sneaked in without any discussion and despite its major problems. It should also be noted that it provides zero advantages over rsyslog which has evolved significantly and provides a superset of systemd funtionality including log signing and structured logging mechanisms (see e.g. http://de.slideshare.net/rainergerhards1/rsyslog-vsjournal) So the switch makes no sense at all. -- Guido Berhoerster -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Le dimanche 19 janvier 2014 à 16:05 +0400, Andrey Borzenkov a écrit :
В Sun, 19 Jan 2014 12:39:14 +0100 Stefan Seyfried <stefan.seyfried@googlemail.com> пишет:
Am 19.01.2014 10:03, schrieb Achim Gratz:
So I wasn't seeing ghosts after all and this is indeed caused by systemd and it's "improved" way of dealing with system logging.
journald is totally broken wrt. its logs not living on a SSD.
While I do not argue with that, all those "$0 status" calls in /etc/init.d/ntp are totally redundant in case of systemd.
Either update your system to SSD or disable persistant journal ("rm -rf /var/log/journal && reboot" should probably achieve this).
Unfortunately that comes at the expense of losing historical logs. I wonder if limiting "systemctl status" output to messages since last boot would make it faster.
This is already the case in git HEAD (not sure about 13.1). And it wouldn't change anything regarding the issue with btrfs not playing well with journal persistent storage. -- Frederic Crozat <fcrozat@suse.com> SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2014 09:58, schrieb Frederic Crozat:
This is already the case in git HEAD (not sure about 13.1). And it
Good.
wouldn't change anything regarding the issue with btrfs not playing well with journal persistent storage.
It is not only btrfs. It is every FS I tested (ext3, ext4, XFS). The journal data file format apparently is just grossly inefficient wrt. disk access patterns. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Stefan Seyfried writes:
journald is totally broken wrt. its logs not living on a SSD.
That isn't the only way in which it is broken, apparently.
Either update your system to SSD or disable persistant journal ("rm -rf /var/log/journal && reboot" should probably achieve this).
Good call. I've never installed systemd-logger (I originally had rsyslog, then due to some 100% CPU-load bug switched to syslog-ng). Jounalctl told me I had almost a GB of logs. What I really had was 47MB of logs plus a few dozen of corrupt journal files (*.journal~), most of them from crashes due to GPU hangups. So journald/systemd switched to a new journal file, marked the old file as corrupted and still searched all of them. With all the corrupt journals out of the way log performance is actually OK at the moment. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sun, Jan 19, 2014 at 11:54 PM, Achim Gratz <Stromeko@nexgo.de> wrote:
Jounalctl told me I had almost a GB of logs. What I really had was 47MB of logs plus a few dozen of corrupt journal files (*.journal~), most of them from crashes due to GPU hangups. So journald/systemd switched to a new journal file, marked the old file as corrupted and still searched all of them. With all the corrupt journals out of the way log performance is actually OK at the moment.
Did you save journal state that exhibited this broken behavior? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Andrey Borzenkov writes:
Did you save journal state that exhibited this broken behavior?
No I didn't. But this is quite likely another bug: the "backup" journal files left when systemd decides the old journal is corrupted and it creates a new one are searched together with "normal" journals. How this is a good idea when the journal is corrupted I can't say, but so far this is even documented. However, I should never had so many log files in the first place since they should have been winged (the ext4 fs they're on would permit for about 300MByte of logs before log truncation should start, again according to the manpage). Apparently the "backup" logs however never figure into that number and additionally there is no external cleanup script to remove them. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-01-18 23:17, Linda Walsh wrote:
This type of script is what is missing in "systemd" -- that makes it hard to debug problems like these, BTW, so going back to an earlier suse version to pull out the start scripts is a good start-point for seeing what systemd "should" be doing.
But Linda, ntp is still started by an old style systemv init script, /etc/init.d/ntp. Telcontar:~ # systemctl status ntp.service ntp.service - LSB: Network time protocol daemon (ntpd) Loaded: loaded (/etc/init.d/ntp) <<======== Active: active (running) since Tue, 2014-01-07 14:58:29 CET; 1 weeks and 4 days ago CGroup: name=systemd:/system/ntp.service └ 3728 /usr/sbin/ntpd -p /var/run/ntp/ntpd.pid -g -u ntp:ntp -i /var/lib/ntp -c /etc/ntp.conf Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. Telcontar:~ # If you search for ntp.service, it does not exist, so it is created dynamically. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On Sun, 2014-01-19 at 14:16 +0100, Carlos E. R. wrote:
On 2014-01-18 23:17, Linda Walsh wrote:
This type of script is what is missing in "systemd" -- that makes it hard to debug problems like these, BTW, so going back to an earlier suse version to pull out the start scripts is a good start-point for seeing what systemd "should" be doing.
But Linda, ntp is still started by an old style systemv init script, /etc/init.d/ntp.
Correct for Factory.. there seems some reluctance in porting ntp to a native systemd service (which would have its merits by the way). I myself am running my 'branched' package of ntp (see home:dimstar:branches:network:time/ntp) which comes with a systemd service and integrates fully into systemd, so that also timedatectl (part of the system toolset) work as expected and know about the time daemon. I did submit that package in the past but after the SR was lingering for too long I figured there was no interest on large scale... so I keep on running it for myself (given: it does not do all the mess the sysV script does.. I'd consider this startup script a very good example on what not to do: either the daemon is smart of not.. working around a dumb daemon with smart startup scripts sounds like investing in the wrong corner). Cheers, Dominique PS: If anybody want to test above ntp package, please go ahead.. -- Dimstar / Dominique Leuenberger <dimstar@opensuse.org> -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 01/19/14 15:52, Dimstar / Dominique Leuenberger wrote:
I did submit that package in the past but after the SR was lingering for too long I figured there was no interest on large scale... so I keep on running it for myself (given: it does not do all the mess the sysV script does.. I'd consider this startup script a very good example on what not to do: either the daemon is smart of not.. working around a dumb daemon with smart startup scripts sounds like investing in the wrong corner).
Translated: The startup script did sensible things that I don't think are needed. I don't know which one, but I'm quite sure. All this complexity, I'm sure it was introduced for ??? I don't need about this; and for sure, I don't need to look around this issue. Great. Nice example to refer to, why systemd proponents destroy user experiences. It worked before, so it should not work afterwards.(TM) Great way to enhance the Linux experience, guys. Hmm, back compatibility, a demand that a guy, not interested in Linux at all, not interesing at all, thinks is important. Well, he's not important, Linus or so it's name is. In HIS open source projects, changing the API is forbidden. In openSUSE, as it would be with Linus, it would not just be, but ridiculed if it's accepted. Customers of mine go to Windows Server steadily, owing to the strong-headness of Linux plumbers, not caring for their operational demands. I can't even rate them negatively, owing to more self-inflected issues than ever. Sigh, Joachim -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod, Roedermark, Germany Email: jschrod@acm.org -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
В Thu, 16 Jan 2014 19:59:49 +0100 Achim Gratz <Stromeko@nexgo.de> пишет:
Ever since systemd took over it waits for ntp to complete startup, which takes a long time on my machine.
What "it" above means? sysvinit would wait for ntp service to start as well. Could you please explain in more details what you mean?
Is there any way I can make systemd start ntp, but not wait for it before it considers bringing up kdm?
How is it related to systemd at all? bor@opensuse:~> grep Should-Start /etc/init.d/xdm # Should-Start: ypbind $syslog gpm firstboot kbd resmgr earlyxdm winbind acpid ntp systemd-user-sessions This is classical sysvinit xdm startup script which waits for classical sysvinit ntp startup script. It would have waited under sysvinit as well. If you want to change it - delete "ntp" from list in /etc/init.d/xdm. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Andrey Borzenkov writes:
This is classical sysvinit xdm startup script which waits for classical sysvinit ntp startup script. It would have waited under sysvinit as well.
Systemd may be a red herring, but ntp taking this long to start coincided with its introduction.
If you want to change it - delete "ntp" from list in /etc/init.d/xdm.
I want it to start as fast as it did before, taking a few seconds at maximum. From playing around with starting and stopping it manually the problem seems to be that during start it seems to traverse a lot of directories and I still haven't got an idea why it does that or how to debug it. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-01-17 20:59, Achim Gratz wrote:
I want it to start as fast as it did before, taking a few seconds at maximum. From playing around with starting and stopping it manually the problem seems to be that during start it seems to traverse a lot of directories and I still haven't got an idea why it does that or how to debug it.
But once systemd tells ntpd to start, if it takes long it does not have to do with systemd at all. Typical delay cause would be no network, or configured peers not responding fast. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Carlos E. R. wrote:
On 2014-01-17 20:59, Achim Gratz wrote:
I want it to start as fast as it did before, taking a few seconds at maximum. From playing around with starting and stopping it manually the problem seems to be that during start it seems to traverse a lot of directories and I still haven't got an idea why it does that or how to debug it.
But once systemd tells ntpd to start, if it takes long it does not have to do with systemd at all.
Typical delay cause would be no network, or configured peers not responding fast.
---- Dunno if it would help in the OP's case, but I keep an old 900MHz Celeron running linux almost exclusively to keep track of time when my mainserver goes down. It used to do dns resolves, but some suse change killed that off, and I haven't bothered to fix it...(sigh).. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sat, 18 Jan 2014 00:24, Linda Walsh <suse@...> wrote:
Carlos E. R. wrote:
On 2014-01-17 20:59, Achim Gratz wrote:
I want it to start as fast as it did before, taking a few seconds at maximum. From playing around with starting and stopping it manually the problem seems to be that during start it seems to traverse a lot of directories and I still haven't got an idea why it does that or how to debug it.
But once systemd tells ntpd to start, if it takes long it does not have to do with systemd at all.
Typical delay cause would be no network, or configured peers not responding fast.
Dunno if it would help in the OP's case, but I keep an old 900MHz Celeron running linux almost exclusively to keep track of time when my mainserver goes down.
It used to do dns resolves, but some suse change killed that off, and I haven't bothered to fix it...(sigh)..
Similar setup here. Mini-box with ntp / dnsmasq / snmp dnsmasq was the solution for my dns needs, small, fast, near stable config (just added features) since more than 5 years. Back to OP.: With a harshly limiting config you can drop chroot in most cases. My hint to solving some of your troubles would be: - Do a timing for all your upwards-peers. - Exchange some upwards-peers if needed. Use: http://support.ntp.org/bin/view/Servers/WebHome as a starting point to locate valid timeservers, then take a look at them with "traceroute" to get the ones with the least number of network-hops. For normal servers and desktops, stratum 3 / pool is good enough. Be aware that some 'local' reference clocks (gps-receivers, terristic-broadcast-clocks, etc) can take some time to get started and give valid data. - Yamaban. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Carlos E. R. writes:
But once systemd tells ntpd to start, if it takes long it does not have to do with systemd at all.
The relation to systemd is probably (I can't test this anymore) that the old init system started xdm before firing off ntp. Systemd does it the other way around, thus annoyingly exposing the delay.
Typical delay cause would be no network, or configured peers not responding fast.
As I said before, the wait is ultimately caused by still unexplained disk activity and more specifically uncached metadata. Warming the cache by restarting ntp once is also slow, but subsequent restarts of ntp take almost no time. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2014-01-18 10:58, Achim Gratz wrote:
Carlos E. R. writes:
But once systemd tells ntpd to start, if it takes long it does not have to do with systemd at all.
The relation to systemd is probably (I can't test this anymore) that the old init system started xdm before firing off ntp. Systemd does it the other way around, thus annoyingly exposing the delay.
I don't know if I can still test this. I have old virtual machines, but they never use ntpd. However, xdm is precisely one of the services that can cough if time jumps backwards, so it does no sense to start xdm in advance.
Typical delay cause would be no network, or configured peers not responding fast.
As I said before, the wait is ultimately caused by still unexplained disk activity and more specifically uncached metadata. Warming the cache by restarting ntp once is also slow, but subsequent restarts of ntp take almost no time.
But this makes no sense; ntp is not a huge program such that loading it from disk would be that slow. -- Cheers / Saludos, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Achim Gratz writes:
Ever since systemd took over it waits for ntp to complete startup, which takes a long time on my machine. Is there any way I can make systemd start ntp, but not wait for it before it considers bringing up kdm?
This delay is related to "systemctl status" searching the log (which isn't even necessary since the output is discarded anyway), so I've added the "-n0" option to the systemctl handoff in /etc/rc.status and my boot time has gone from about four minutes to xdm password prompt to just a hair under a minute (as it was before systemd with the old init system). Bug #859360 Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Achim Gratz writes:
Bug #859360
Thanks to everybody who has been helping to implement and release a proper fix, much appreciated. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (14)
-
Achim Gratz -
Andrey Borzenkov -
Carlos E. R. -
dieter -
Dimstar / Dominique Leuenberger -
Frederic Crozat -
Guido Berhoerster -
Jan Ritzerfeld -
Joachim Schrod -
Linda Walsh -
Ralf Lang -
Ruediger Meier -
Stefan Seyfried -
Yamaban