[opensuse-factory] [TW-20150328] "Started D-Bus System Message Bus." - then it hangs (sort of)
Hi there, I first observed $SUBJECT when I upgraded to 20150327, but another upgrade to snapshot 20150328 did not help either. My system uses one SSD for the root fs, four disks in soft-mdraid10 mode for various LVs. When the system hangs (sort of), having to Ctrl-Alt-SysRq "s", followed by "u" followed by "b" is not really fun, because the mirror will then resync afterwards (for appr. 6 hours). I saw some dependency cycles with AppArmor and disabled apparmor.service, hoping this would help (AppArmor is not installed, only the stuff which is actually really required), but it didn't. What's really surprising me is, the system boots properly every second time, but hangs after the "Started D-Bus System Message Bus." during the other boot attempts. Switching to console 10 is still possible, so is the usage of the Ctrl-Alt-SysRq's... hmm, maybe I should look into the current task list to see which process is making trouble... (followed by another resync for 6 hours...). Anyway, this problem cannot be seen on a laptop with only one disk. I'm suspecting the upgrade to udev-219, as a lot of "waiting for ..." messages now appear on the multi-disk system, while nothing like that could be seen with the udev version before that (-210?). Is anybody else seeing similar strange behaviour? TIA, cheers. l8er manfred
On Monday 2015-03-30 18:46, Manfred Hollstein wrote:
My system uses one SSD for the root fs, four disks in soft-mdraid10 mode for various LVs. When the system hangs (sort of), having to Ctrl-Alt-SysRq "s", followed by "u" followed by "b" is not really fun, because the mirror will then resync afterwards (for appr. 6 hours).
RAIDs created with yast should have a write-intent bitmap attached so that resyncs go by much faster. If you manually created them, they may be lacking that if you did not specify one. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, 30 Mar 2015, 18:51:00 +0200, Jan Engelhardt wrote:
On Monday 2015-03-30 18:46, Manfred Hollstein wrote:
My system uses one SSD for the root fs, four disks in soft-mdraid10 mode for various LVs. When the system hangs (sort of), having to Ctrl-Alt-SysRq "s", followed by "u" followed by "b" is not really fun, because the mirror will then resync afterwards (for appr. 6 hours).
RAIDs created with yast should have a write-intent bitmap attached so that resyncs go by much faster. If you manually created them, they may be lacking that if you did not specify one.
hmm, I thought I had added the bitmap, but apparently not :( I've now added one using mdadm --grow --bitmap=internal /dev/md0 Will see how that changes the behaviour once the current resync has finished... Cheers. l8er manfred
В Mon, 30 Mar 2015 18:46:16 +0200 Manfred Hollstein <mhollstein@t-online.de> пишет:
Hi there,
I first observed $SUBJECT when I upgraded to 20150327, but another upgrade to snapshot 20150328 did not help either.
My system uses one SSD for the root fs, four disks in soft-mdraid10 mode for various LVs. When the system hangs (sort of), having to Ctrl-Alt-SysRq "s", followed by "u" followed by "b" is not really fun, because the mirror will then resync afterwards (for appr. 6 hours).
I saw some dependency cycles with AppArmor and disabled apparmor.service, hoping this would help (AppArmor is not installed, only the stuff which is actually really required), but it didn't. What's really surprising me is, the system boots properly every second time, but hangs after the "Started D-Bus System Message Bus."
How long did you wait? Default timeout for starting services is 90 seconds. Please try without "quiet" on kernel command line (and plymouth disabled) - systemd should display animation when service takes too much time.
during the other boot attempts. Switching to console 10 is still possible, so is the usage of the Ctrl-Alt-SysRq's... hmm, maybe I should look into the current task list to see which process is making trouble... (followed by another resync for 6 hours...).
Anyway, this problem cannot be seen on a laptop with only one disk. I'm suspecting the upgrade to udev-219, as a lot of "waiting for ..." messages now appear on the multi-disk system, while nothing like that could be seen with the udev version before that (-210?).
Is anybody else seeing similar strange behaviour?
TIA, cheers.
l8er manfred
On Mon, 30 Mar 2015, 19:09:32 +0200, Andrei Borzenkov wrote:
В Mon, 30 Mar 2015 18:46:16 +0200 Manfred Hollstein <mhollstein@t-online.de> пишет:
Hi there,
I first observed $SUBJECT when I upgraded to 20150327, but another upgrade to snapshot 20150328 did not help either.
My system uses one SSD for the root fs, four disks in soft-mdraid10 mode for various LVs. When the system hangs (sort of), having to Ctrl-Alt-SysRq "s", followed by "u" followed by "b" is not really fun, because the mirror will then resync afterwards (for appr. 6 hours).
I saw some dependency cycles with AppArmor and disabled apparmor.service, hoping this would help (AppArmor is not installed, only the stuff which is actually really required), but it didn't. What's really surprising me is, the system boots properly every second time, but hangs after the "Started D-Bus System Message Bus."
How long did you wait? Default timeout for starting services is 90 seconds.
appr. 10-15 minutes.
Please try without "quiet" on kernel command line (and plymouth disabled) - systemd should display animation when service takes too much time.
I already have plymouth using "Theme=text", but I'll try without "quiet" and plymouth completely disabled next time. Will report back then. Cheers. l8er manfred
Am 30.03.2015 um 18:46 schrieb Manfred Hollstein:
Hi there,
I first observed $SUBJECT when I upgraded to 20150327, but another upgrade to snapshot 20150328 did not help either.
[...]
Is anybody else seeing similar strange behaviour?
Me :( On a desktop box with 1 ssd only. Yesterday everything worked fine, and today my system is b0rken. It takes a long time until the login screen appears, and I cannot log in. I enter my password, and I am thrown back to the login screen some 30 seconds later. No text in any log. Root can login, however. I use LDAP login, via sssd. The problem seems to be related to the network connection. Until this morning I used a bridge br0, because I frequently run VMs on my box. This morning, after spending an hour with boot problems, trying a live CD and so on, I reverted from br0 to enp0s25, but this did not help, installing wicked from plain 13.2 did not help either. Working with yast at the console is annoying, because ncurses uses some console font that does not harmonize with the font the console is really using and the screen is about unusable. Only tty1 (reachable with Ctrl-Alt-F1) is available, tty2-6 are not present. Logging in via console instead of the graphical screen works, but takes ages. The screensaver works in a very final mode, it never wakes up again. "ip ad sh" shows currently 2 IPv4 addresses for my box, but the number of addresses increases by 1 every minute or so, this software is so crappy... NetworkManager is running as well as wicked. Why? I can neither disable NW manager nor restart wicked, timeout in dbus, g-dbus-error-quark. Shutting down the system leads to a long hanging phase. Powering off is the choice now. Where can I find a TW version from before the last snapshot to get my working system back? Regards, Werner
On Tue, 2015-03-31 at 12:35 +0200, Werner Flamme wrote:
Where can I find a TW version from before the last snapshot to get my working system back
Did you try to revert to the last snapper snapshot? -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 31.03.2015 um 12:41 schrieb Dominique Leuenberger / DimStar:
On Tue, 2015-03-31 at 12:35 +0200, Werner Flamme wrote:
Where can I find a TW version from before the last snapshot to get my working system back
Did you try to revert to the last snapper snapshot?
No, I did not, because Snapper is not active on my box. Does it work with ext4? I only know it from btrfs. --
On Tue, 31 Mar 2015, 12:35:18 +0200, Werner Flamme wrote:
Am 30.03.2015 um 18:46 schrieb Manfred Hollstein:
Hi there,
I first observed $SUBJECT when I upgraded to 20150327, but another upgrade to snapshot 20150328 did not help either.
[...]
Is anybody else seeing similar strange behaviour?
Me :( On a desktop box with 1 ssd only.
welcome to the club... :(
Yesterday everything worked fine, and today my system is b0rken.
It takes a long time until the login screen appears, and I cannot log in. I enter my password, and I am thrown back to the login screen some 30 seconds later. No text in any log. Root can login, however. I use LDAP login, via sssd.
Do you use the splash screen? I.e. not being able to see any relevant stuff? If so, please press the Esc key to see the messages during bootup. When you're logged in, are there any *strange* dependency related messages (cat /var/log/boot.log), like I wrote for AppArmor in my case? FWIW, something during the update to 20150327 must have really messed up the dependencies, like which services were enabled and remained so; Cups for instance got deactivated on one of my three systems while is was activated before the upgrade... Please check your enabled/disabled services pretty carefully! FWIW2: "systemctl disable apparmor.service" helped *a lot* on my laptop, but apparmor wasn't even activated before the upgrade... Hmm.
The problem seems to be related to the network connection. Until this morning I used a bridge br0, because I frequently run VMs on my box. This morning, after spending an hour with boot problems, trying a live CD and so on, I reverted from br0 to enp0s25, but this did not help, installing wicked from plain 13.2 did not help either.
This sounds as it might be related to the udev-219 upgrade, as I mentioned before. To me udev and systemd look like a very tightly coupled agenda, and now that systemd controls dependencies between devices, services and whatnot, I actually hope that the systemd-219 update will show up pretty soon (even though I don't like it).
Working with yast at the console is annoying, because ncurses uses some console font that does not harmonize with the font the console is really using and the screen is about unusable.
Yeah, that exists for almost 5-6 years now, unbelievable.
Only tty1 (reachable with Ctrl-Alt-F1) is available, tty2-6 are not present. Logging in via console instead of the graphical screen works, but takes ages. The screensaver works in a very final mode, it never wakes up again.
"ip ad sh" shows currently 2 IPv4 addresses for my box, but the number of addresses increases by 1 every minute or so, this software is so crappy... NetworkManager is running as well as wicked. Why? I can neither disable NW manager nor restart wicked, timeout in dbus, g-dbus-error-quark.
Hmm, dbus again...
Shutting down the system leads to a long hanging phase. Powering off is the choice now.
Where can I find a TW version from before the last snapshot to get my working system back?
Do we still have a version of udev before 2.19 around somewhere?
Regards, Werner
Cheers. l8er manfred
On Tue, 2015-03-31 at 14:09 +0200, Manfred Hollstein wrote:
This sounds as it might be related to the udev-219 upgrade, as I mentioned before. To me udev and systemd look like a very tightly coupled agenda, and now that systemd controls dependencies between devices, services and whatnot, I actually hope that the systemd-219 update will show up pretty soon (even though I don't like it).
They are as tightly bound as that udev is part of the systemd source tree for a while already (see https://lwn.net/Articles/490413/ from 2012) systemd 219 IS in the repositories (since the same day udev 219 landed there: zypper se -s systemd: i | systemd | package | 219-3.1 | x86_64 | openSUSE-Tumbleweed-Oss v | systemd | package | 219-3.1 | i586 | openSUSE-Tumbleweed-Oss Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, 31 Mar 2015, 14:20:58 +0200, Dominique Leuenberger / DimStar wrote:
On Tue, 2015-03-31 at 14:09 +0200, Manfred Hollstein wrote:
This sounds as it might be related to the udev-219 upgrade, as I mentioned before. To me udev and systemd look like a very tightly coupled agenda, and now that systemd controls dependencies between devices, services and whatnot, I actually hope that the systemd-219 update will show up pretty soon (even though I don't like it).
They are as tightly bound as that udev is part of the systemd source tree for a while already (see https://lwn.net/Articles/490413/ from 2012)
systemd 219 IS in the repositories (since the same day udev 219 landed there:
zypper se -s systemd: i | systemd | package | 219-3.1 | x86_64 | openSUSE-Tumbleweed-Oss v | systemd | package | 219-3.1 | i586 | openSUSE-Tumbleweed-Oss
hmm, I'm only installing from snapshots, which I mirror to my local environment. Apparently, udev-219 made it into 20150327, but not yet systemd-219. Even the latest announced snapshot 20150329 does not contain it. Looks like the snapshot scripts need to add another rule to keep systemd and udev in sync...
Cheers, Dominique
Cheers. l8er manfred
On Tue, 2015-03-31 at 14:25 +0200, Manfred Hollstein wrote:
hmm, I'm only installing from snapshots, which I mirror to my local environment. Apparently, udev-219 made it into 20150327, but not yet systemd-219. Even the latest announced snapshot 20150329 does not contain it. Looks like the snapshot scripts need to add another rule to keep systemd and udev in sync...
http://download.opensuse.org/tumbleweed/repo/oss/suse/x86_64/?P=*-219-3.1* It surely IS in the tree of Tumbleweed - Check your repository setup - and check if you have any locks preventing you from updating systemd to version 219. http://download.opensuse.org/tumbleweed/iso/Changes.20150327.txt lists systemd as being updated to version 219: ==== libgudev-1_0-0 ==== Version update (210 -> 219) Subpackages: libgudev-1_0-devel libudev-devel libudev1 libudev1-32bit systemd systemd-32bit systemd-bash-completion systemd-logger systemd- sysvinit typelib-1_0-GUdev-1_0 udev it is one unified group of packages - all built off the same source tree. Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Tue, 31 Mar 2015, 14:33:51 +0200, Dominique Leuenberger / DimStar wrote:
On Tue, 2015-03-31 at 14:25 +0200, Manfred Hollstein wrote:
hmm, I'm only installing from snapshots, which I mirror to my local environment. Apparently, udev-219 made it into 20150327, but not yet systemd-219. Even the latest announced snapshot 20150329 does not contain it. Looks like the snapshot scripts need to add another rule to keep systemd and udev in sync...
http://download.opensuse.org/tumbleweed/repo/oss/suse/x86_64/?P=*-219-3.1*
It surely IS in the tree of Tumbleweed - Check your repository setup - and check if you have any locks preventing you from updating systemd to version 219.
http://download.opensuse.org/tumbleweed/iso/Changes.20150327.txt lists systemd as being updated to version 219:
==== libgudev-1_0-0 ==== Version update (210 -> 219) Subpackages: libgudev-1_0-devel libudev-devel libudev1 libudev1-32bit systemd systemd-32bit systemd-bash-completion systemd-logger systemd- sysvinit typelib-1_0-GUdev-1_0 udev
it is one unified group of packages - all built off the same source tree.
shame on me, really, I just search for "^==== " in Ludwig's great announcement e-mails and did not see systemd there. Both, systemd-219 and udev-219 are actually in my local mirror, created using: rsync -avOJ --chown=manfred:users --exclude=debuginfo --exclude=i586 --exclude=i686 --exclude=ppc --exclude=ppc64 --exclude=src --exclude='kernel-debug-*.rpm' --exclude='kiwi-image-livecd*.rpm' --exclude='texlive*.rpm' --exclude='*.delta.rpm' --exclude='*.patch.rpm' --delete --delete-excluded ftp.halifax.rwth-aachen.de::opensuse/factory/repo/oss repo/ rsync -avOJ --chown=manfred:users --exclude=debuginfo --exclude=i586 --exclude=i686 --exclude=ppc --exclude=ppc64 --exclude=src --exclude='kernel-debug-*.rpm' --exclude='kiwi-image-livecd*.rpm' --exclude='texlive*.rpm' --exclude='*.delta.rpm' --exclude='*.patch.rpm' --delete --delete-excluded ftp.halifax.rwth-aachen.de::opensuse/factory/repo/non-oss repo/ Now I find it unbelievable, though, that my system runs very well with udev-210 and systemd-219 installed, while I found all the trouble with both in sync... Strange.
Cheers, Dominique
Cheers. l8er manfred
On Tue, 31 Mar 2015, 14:09:40 +0200, Manfred Hollstein wrote: [...]
Do we still have a version of udev before 2.19 around somewhere?
I just downgraded to the latest udev version from 13.2 and got my system back to work. As long as systemd-219 is not included in a snapshot, we should really retract udev-219 and related packages! Shall I put this in bugzilla? Cheers. l8er manfred
Hi there, first of all, apologies to everyone for the noise I created. I have to rephrase the statements I made earlier wrt/ the upgrade to systemd-219 (which I obviously completely overlooked - *it's there in all the snapshots I wrote about*) and udev-219. My systems now work as they did before, but only after _somehow_ disabling "apparmor.service". Due to systemd-219 now having a builtin unit for apparmor, my systems got into dependency trouble at runtime as I don't even have apparmor installed (besides the two required packages "apparmor-parser" and "libapparmor1"), and up until before Tumbleweed snapshot 20150327 it wasn't necessary to explicitly disable boot.apparmor, it just didn't harm. When I wrote _somehow_ above, it's because it wasn't really guranteed to successfully boot to a TW login prompt, while it was enabled; so I had to find ways to disable without systemctl. If anybody else has a similar environment (only minimal/required apparmor, but not actually using it) and sees all sorts of trouble when booting the more recent TW snapshots, you should try to mount TW's root file system and remove "./etc/init.d/boot.d/*boot.apparmor". FWIW, I got the idea from another thread in this list "Apparmor fails to start on boot in Tumbleweed". @Werner: I'm not sure if this will help with your problems, but it looks like you might want to try this out. HTH, cheers. l8er manfred
Am 31.03.2015 um 15:36 schrieb Manfred Hollstein: [...]
you should try to mount TW's root file system and remove "./etc/init.d/boot.d/*boot.apparmor". FWIW, I got the idea from another thread in this list "Apparmor fails to start on boot in Tumbleweed".
@Werner: I'm not sure if this will help with your problems, but it looks like you might want to try this out.
Thank you, I already disabled AA via commandline. My system now works, too. Regards, Werner --
On Tue, Mar 31, 2015 at 10:36 AM, Manfred Hollstein <mhollstein@t-online.de> wrote:
Hi there,
first of all, apologies to everyone for the noise I created. I have to rephrase the statements I made earlier wrt/ the upgrade to systemd-219 (which I obviously completely overlooked - *it's there in all the snapshots I wrote about*) and udev-219.
My systems now work as they did before, but only after _somehow_ disabling "apparmor.service". Due to systemd-219 now having a builtin unit for apparmor
huh ? systemd's apparmor support currently consists in a single directive ApparmorProfile to change the default profile of the service if desired, it has no effect unless the system administrator changes it. it does not do the basic initialization, that has to be done with an early boot unit..unfortunately appamor packages currently only include an init script. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 31.03.2015 um 14:09 schrieb Manfred Hollstein:
On Tue, 31 Mar 2015, 12:35:18 +0200, Werner Flamme wrote:
Am 30.03.2015 um 18:46 schrieb Manfred Hollstein:
Hi there,
I first observed $SUBJECT when I upgraded to 20150327, but another upgrade to snapshot 20150328 did not help either.
[...]
Is anybody else seeing similar strange behaviour?
Me :( On a desktop box with 1 ssd only.
welcome to the club... :(
Yesterday everything worked fine, and today my system is b0rken.
It takes a long time until the login screen appears, and I cannot log in. I enter my password, and I am thrown back to the login screen some 30 seconds later. No text in any log. Root can login, however. I use LDAP login, via sssd.
Do you use the splash screen? I.e. not being able to see any relevant stuff? If so, please press the Esc key to see the messages during bootup.
I know, thanks. I try to disable the splash screen wherever and whenever I can, but... at leastm, I replace all "splash=silent" with "splash=native" in /etc/default/grub.
When you're logged in, are there any *strange* dependency related messages (cat /var/log/boot.log), like I wrote for AppArmor in my case? FWIW, something during the update to 20150327 must have really messed up the dependencies, like which services were enabled and remained so; Cups for instance got deactivated on one of my three systems while is was activated before the upgrade...
Apparmor is mentioned in a dependency cycle, but that is not my problem, I guess.
Please check your enabled/disabled services pretty carefully! FWIW2: "systemctl disable apparmor.service" helped *a lot* on my laptop, but apparmor wasn't even activated before the upgrade... Hmm.
Apparmor was active on my box, but in complain mode.
The problem seems to be related to the network connection. Until this morning I used a bridge br0, because I frequently run VMs on my box. This morning, after spending an hour with boot problems, trying a live CD and so on, I reverted from br0 to enp0s25, but this did not help, installing wicked from plain 13.2 did not help either.
This sounds as it might be related to the udev-219 upgrade, as I mentioned before. To me udev and systemd look like a very tightly coupled agenda, and now that systemd controls dependencies between devices, services and whatnot, I actually hope that the systemd-219 update will show up pretty soon (even though I don't like it).
I do not want to start the next war about systemd, so I keep my fingers away from the keyboard now :P. I did a "zypper up" and hope that something will work better on next boot. A new lvm2 was delivered, and device-mapper also. And yes, my box is back again! YO!
Working with yast at the console is annoying, because ncurses uses some console font that does not harmonize with the font the console is really using and the screen is about unusable.
Yeah, that exists for almost 5-6 years now, unbelievable.
It never was so bad before, AFAIR.
g-dbus-error-quark.
Hmm, dbus again...
Yes, I prefer taxi ;)
Do we still have a version of udev before 2.19 around somewhere?
Nope, but a newer one did the trick for me. Regards, Werner --
participants (6)
-
Andrei Borzenkov -
Cristian Rodríguez -
Dominique Leuenberger / DimStar -
Jan Engelhardt -
Manfred Hollstein -
Werner Flamme