http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c1 --- Comment #1 from Franck Bui --- Here are my findings so far: - kdm is started on either tty7 or tty8 - kdm package is carrying a openSUSE patch which is responsible to choose a free tty considering if plymouth is running and on which tty - this doesn't seem to be specific to systemd v232. I can reproduce the same behavior with systemd v228. To reproduce the issue: - install TW in server text mode - install xdm, kdm and enable kdm through /etc/sysconfig/displaymanager - install xorg-x11-driver-input if it's missing Then restart a couple of time kdm with: $ systemctl isolate multi-user.target ; \ systemctl isolate graphical.target ; \ sleep 2 # check the VT used by Xorg $ ps aux | grep Xorg root 2241 3.6 3.7 365488 37792 tty7 Ssl+ 17:56 0:00 /usr/bin/Xorg -br :0 vt7 ... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c3 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fabian@ritter-vogt.de --- Comment #3 from Fabian Vogt --- I did some investigation on this (based solely on log files though, haven't tried to reproduce it locally yet). Before 232: Dec 30 05:26:27 linux-oikq display-manager[652]: Starting service kdm..done Dec 30 05:26:27 linux-oikq kdm[695]: plymouth is running Dec 30 05:26:27 linux-oikq kdm[695]: plymouth is active on VT 7, reusing for :0 Dec 30 05:26:27 linux-oikq kdm[695]: plymouth should quit after server startup With 232: Jan 17 03:25:43 linux-oikq display-manager[1412]: Starting service kdm..done Jan 17 03:25:43 linux-oikq kdm[1507]: plymouth is running Jan 17 03:25:43 linux-oikq kdm[1507]: plymouth is running Jan 17 03:25:43 linux-oikq systemd[1]: Received SIGRTMIN+21 from PID 270 (plymouthd). Jan 17 03:25:43 linux-oikq kdm[1507]: plymouth is NOT running The SIGRTMIN+21 is a signal from plymouth to systemd, only emitted during quit. However, kdm didn't tell plymouth to quit yet, which is only the case if it has not an active vt. And this is easily confirmed by looking at the video: Plymouth is missing! As systemd-vconsole-setup takes a whopping 0.5s suddenly (close to 0 before) it might be related. One testcase does not fit into this picture though: cryptlvm. As the password prompt is displayed successfully by plymouth, it's working fine here (in the initrd, at least). However, the journal tells us that systemd-vconsole-setup actually failed here: Jan 17 00:03:20 linux-qoht systemd[1]: Started Plymouth switch root service. Jan 17 00:03:20 linux-qoht systemd-vconsole-setup[991]: KD_FONT_OP_GET failed while trying to get the font metadata: Invalid argument Jan 17 00:03:20 linux-qoht systemd-vconsole-setup[991]: Fonts will not be copied to remaining consoles Jan 17 00:03:20 linux-qoht systemd[1]: Started Setup Virtual Console. This error does not appear on failing test runs so that seems to indicate that plymouth only runs successfully iff systemd-vconsole-setup failed. This is true on all two openQA TW snapshot runs so probably not a conincidence. Race condition? Conclusion: Not KDMs fault, as far as I can see. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c5 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fvogt@suse.com --- Comment #5 from Fabian Vogt --- My guess (only a guess so far, I'll install systemd 232 in a VM now) is that kdm chooses vt8 because vt7 is unavailable, for the same reason plymouth did not work. This reason is probably systemd-vconsole-setup taking longer than it should (+ improper unit ordering?) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c8 --- Comment #8 from Franck Bui --- Created attachment 710667 --> http://bugzilla.opensuse.org/attachment.cgi?id=710667&action=edit logs showing plymouth crashing before switching to new rootfs and kdm selecting tty8 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c10 --- Comment #10 from Fabian Vogt --- (In reply to Franck Bui from comment #7)

Ok, here's my best bet:

Since systemd v232, systemd-vconsole-setup.service has no more the RemainAfterExit=true property.

Since plymouth-start.service has Wants=systemd-vconsole-setup.service, the later is now started several time within the initramfs: it's pulled in by initrd.target and by initrd-switch-root.target. And during the second time vconsole-setup is configuring tty7, which was probably already activated by plymouth-start.

This explains why cryptlvm succeeds, as systemd-vconsole-setup fails there and thus it does not confuse plymouth.

This has the bad effect to "confuse" plymouth as vconsole-setup is now configuring tty7 after plymouth started using it. This makes actually plymouth crashes.

The crashes may happen either right before switching to the rootfs or after. If the former then systemd will start it again after switching to the new rootfs otherwise it won't.

Now regarding KDM, not sure if it's expected but it selects:

- tty7 if plymouth is not running (IOW if it crashed after switching to new rootfs)

- tty8 if plymouth is running (IOW if plymouth crashed before switching to the new rootfs)

So far I haven't seen plymouthd crash during tests. Plymouth is running in *both* cases ("plymouth is running" is printed with systemd 228 and 232) and kdm selects tty8 because tty7 is blocked by plymouth but not active (as it got confused).

From that, there're a couple issues:

- why does plymouth-start.service wants systemd-vconsole-setup.service: I don't see the point as vconsole-setup will automatically configure any tty detected by udev ?

Probably due to ordering. While plymouth is running, the active ttys can't be configured. However, every tty has to be configured, so this needs to happen *before* plymouth starts.

- systemd-vconsole-setup.service is embedded in initramfs and this doesn't seem to be needed.

It is because of above. There are various bug reports about wrong keyboard layout if systemd-vconsole-setup fails during the initrd phase.

- why does plymouth crash in this case (might be fixed in plymouth git repo as Fabian reported a better behavior on IRC) ?

I will try that again with everything set to the default and plymouth from git master.

- Why does KDM select either tty7 or tty8 ? in both case tty7 seems to have been activated by plymouth ?

kdm does not get tty7 (or does not consider it as valid answer) when asking plymouth for its active vt. So it chooses to try the next one instead. This is good as using tty7 would probably fail here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c12 --- Comment #12 from Franck Bui --- (In reply to Fabian Vogt from comment #11)

The root cause is that systemd-vconsole-setup touches the same consoles more than once.

Not exactly. It seems to first configure tty1 and then tty7 however plymouth is already on it at this time. This is due to plymouth-start.service which has Wants=systemd-vconsole-setup.service *and* systemd-vconsole-setup.service having RemainsAfterExit=no (since v232). A possible fix might be: - remove Wants=systemd-vconsole-setup.service from plymouth-start.service. I'm not sure to understand the point to do so actually. If it's done by plymouth in order to configure the console which it's going to use (tty7) then that's probably wrong as the active console when vconsole-setup will be running will be a different console (tty1). - include in the initramfs /usr/lib/udev/rules.d/90-vconsole.rules This seems to be enough for running vconsole-setup on tty7 when plymouth activates (or switches to) it but before it actually uses it, so no crash happens in this case. But someone with knowledge in plymouth and in the console/tty stuff should double check. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c14 --- Comment #14 from Fabian Vogt --- (In reply to Franck Bui from comment #7)

Ok, here's my best bet:

Since systemd v232, systemd-vconsole-setup.service has no more the RemainAfterExit=true property.

Confirmed. After changing that back it works fine. This proves that systemd-vconsole-setup configures consoles more than once, doesn't it?

- include in the initramfs /usr/lib/udev/rules.d/90-vconsole.rules This seems to be enough for running vconsole-setup on tty7 when plymouth activates (or switches to) it but before it actually uses it, so no crash happens in this case.

This is already in the initrd. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c16 --- Comment #16 from Franck Bui --- Ok let's try to make some progress... systemd-setup-console.service is started several times during the early boot. This was actually already the case with v228 where the service was started 3 times I think: - 1 time started by plymouth-start.service due to a dependency Wants=systemd-setup-console.service used in the service file. - 2 times started by 90-vconsole.rules. I don't really understand why the rule is called a second time here but it seems that one of the vtconsole device is removed (after being added the first time) and then added back again right after. This happens during udev coldplug all devices (systemd-udev-trigger.service). Now with v232, systemd-setup-console.service has RemainsAfterExit=false. I don't know if this change is correct or not but it has the downside to start one more time systemd-setup-console.service lately during the early boot. And this new console configuration seems to confuse plymouth for some reason and makes it crash. Even if configuring the console several times is not nice, I don't think plymouth is supposed to crash in anyways (BTW I think there is already a bug open for the crash of plymouth) and this bug should be fixed. @Fabian, in comment #11, you reported that plymouth git master (and also the version in Base:System) has a fix for preventing plymouth from crahsing. So my suggestion here is to fix plymouth by either upgrading Base:System to git master or by identifying the fix and backport it to Factory. In the meantime I think we could also drop the Wants=systemd-setup-console.service from plymouth-start.service because setup-console is supposed to be done via a udev rule (which shouldhappen before plymouth is started). @Fabian, WDYT ? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c18 --- Comment #18 from Franck Bui --- (In reply to Fabian Vogt from comment #17)

(In reply to Franck Bui from comment #16)

...

Now with v232, systemd-setup-console.service has RemainsAfterExit=false. I don't know if this change is correct or not but it has the downside to start one more time systemd-setup-console.service lately during the early boot.

This is incorrect and needs to be changed back (or the service must not be started at all) AFAICS.

The service is *only* started because plymouth is pulling it.

Once a console got configured, it stays configured.

This assertion is false in both case (v228 and v232). The console is configured several times by udev because it receives a "add/remove/add" sequence for one console. And most important any application is not supposed to *crash*.

...

And this new console configuration seems to confuse plymouth for some reason and makes it crash.

Even if configuring the console several times is not nice, I don't think plymouth is supposed to crash in anyways (BTW I think there is already a bug open for the crash of plymouth) and this bug should be fixed.

@Fabian, in comment #11, you reported that plymouth git master (and also the version in Base:System) has a fix for preventing plymouth from crahsing.

So my suggestion here is to fix plymouth by either upgrading Base:System to git master or by identifying the fix and backport it to Factory.

I already requested that, zaitor had an update already prepared and now submitted it as sr#451329.

Good.

...

In the meantime I think we could also drop the Wants=systemd-setup-console.service from plymouth-start.service because setup-console is supposed to be done via a udev rule (which shouldhappen before plymouth is started).

@Fabian, WDYT ?

udev alone is not enough, otherwise the .service wouldn't be needed at all.

udev should be enough. The .service is mostly useless now (I think no other service is requiring it but plymouth).

When the service fails, bugs like 927250 and its three dups happen. (Or the udev behaviour is buggy, in which case the .service should be removed entirely, I guess)

I'd proceed this way for now:

- Change back RemainAfterExit in systemd

I'm not sure about this change at all and this should be discussed upstream first. But IMHO this not the (root) issue here it just exposes the shortcoming in plymouth.

- Update plymouth to sr#451329

It seems to me the real fix here: plymouth won't be crashing anymore. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c20 --- Comment #20 from Franck Bui --- (In reply to Fabian Vogt from comment #19)

(In reply to Franck Bui from comment #18)

...

...

When the service fails, bugs like 927250 and its three dups happen. (Or the udev behaviour is buggy, in which case the .service should be removed entirely, I guess)

Can you comment on this^^^^?

Sorry but I can't parse other bugs currently, my plate is full ;)

...

...

I'd proceed this way for now:

- Change back RemainAfterExit in systemd

I'm not sure about this change at all and this should be discussed upstream first. But IMHO this not the (root) issue here it just exposes the shortcoming in plymouth.

AFAICS it's still wrong. Either systemd-vconsole-setup.service needs to go completely or set RemainAfterExit=true.

The only reason I can see for the existence of systemd-vconsole-setup.service is that others (3rd party) services are mentioning it. And plymouth is one of those... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c30 --- Comment #30 from Franck Bui --- (In reply to Fabian Vogt from comment #29)

So, I'm back after some further analysis on this.

Thanks a lot for investigating deeper in this.

The second one is 0001-let-it-become-a-real-daemon.patch which totally breaks console locking. With broken console locking you cannot guarantee for anything anymore and systemd-vconsole-setup breaks the console that plymouth uses, which ultimately led to the crash in plymouth's terminal keyboard driver.

Is the crash in plymouth's driver fixable ? With the broken patch removed, will systemd-vconsole-setup fail when opening the vtconsole if it has been locked by plymouth ?

The patch is AFAICS wrong as the reason it got introduced was a misconfiguration of systemd services (bsc#892526). Removal of this patch means however that systemd-vconsole-setup cannot configure the console while plymouth is running (which it never did with RemainAfterExit=true before v232) so we likely need the Wants= and After= for systemd-vconsole-setup.service in plymouth-start.service to not bring back console font/keymap issues.

Well I don't see any need to keep systemd-vconsole-setup.service but the vtconsole stuff is an obscure area to me so I may miss some useful use cases. If we agree on the fact that this service is unneeded I can open an issue upstream and ask to remove the service completely. If upstream doesn't agree then we could at least ask for the correctness of setting RemainAfterExists=no.

Franck's idea of removing the Wants= from plymouth-start.service worked because systemd-vconsole-setup.service didn't get pulled in from anything else and so never ran at all. However, issue #1 prevented it from working altogether in openQA.

Yeah I realized that this morning.

Now with 0001-let-it-become-a-real-daemon.patch removed, everything will work just fine *if* we make sure that plymouth gets stopped when needed (e.g. by YaST firstboot, X and other display servers).

Result of this is in https://build.opensuse.org/package/show/home:favogt:ply-hell/plymouth Please review and test!

I will give it a test. Thanks ! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1020327 http://bugzilla.opensuse.org/show_bug.cgi?id=1020327#c32 --- Comment #32 from Franck Bui --- (In reply to Fabian Vogt from comment #31)

Make sure to get at least rev 17, I did some changes in the patch to avoid DRM initialization delays.

So I gave it a test and Xorg was always started on tty7, however I could still see plymouth crashed: Jan 26 11:15:35 localhost systemd[1]: plymouth-start.service: Main process exited, code=dumped, status=6/ABRT I thought that should have been fixed by your changes but apparently this doesn't work as expected or I'm miss understood your changes. -- You are receiving this mail because: You are on the CC list for the bug.