systemd-vconsole-setup.service failure
Yesterday I updated several TW machines and also TW vms from 20231120 to 20231228. After rebooting I ran systemctl --failed which shows UNIT LOAD ACTIVE SUB DESCRIPTION ? systemd-vconsole-setup.service loaded failed failed Virtual Console Setup systemctl status systemd-vconsole-setup.service provides the details × systemd-vconsole-setup.service - Virtual Console Setup Loaded: loaded (/usr/lib/systemd/system/systemd-vconsole-setup.service; static) Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27 EST; 23h ago Duration: 5h 2.219s Docs: man:systemd-vconsole-setup.service(8) man:vconsole.conf(5) Main PID: 1105 (code=exited, status=1/FAILURE) CPU: 5ms systemd[1]: Starting Virtual Console Setup... systemd-vconsole-setup[1105]: No usable source console found: Device or resource busy systemd[1]: systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: systemd-vconsole-setup.service: Failed with result 'exit-code'. systemd[1]: Failed to start Virtual Console Setup. Not sure what is causing the "No usable source console found: Device or resource busy" error but it happens on multiple TW machines and also on my TW vms. systemctl restart systemd-vconsole-setup.service Restarts the service with no errors or problems so wondering if something timewise is not ready during the boot process ? Is this a known issue ? -- Regards, Joe
I saw this too. I haven't had time to debug it, but my hunch is a missing dependency in it's unit file. That would cause a race condition where vconsole-setup tries to start before its dependencies are running. Hope this helps. -- Tony On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to 20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB DESCRIPTION ? systemd-vconsole-setup.service loaded failed failed Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup Loaded: loaded (/usr/lib/systemd/system/systemd-vconsole-setup.service; static) Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27 EST; 23h ago Duration: 5h 2.219s Docs: man:systemd-vconsole-setup.service(8) man:vconsole.conf(5) Main PID: 1105 (code=exited, status=1/FAILURE) CPU: 5ms
systemd[1]: Starting Virtual Console Setup... systemd-vconsole-setup[1105]: No usable source console found: Device or resource busy systemd[1]: systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: systemd-vconsole-setup.service: Failed with result 'exit-code'. systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or resource busy" error but it happens on multiple TW machines and also on my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if something timewise is not ready during the boot process ?
Is this a known issue ?
-- Regards,
Joe
I finally found some time to debug and find a solution... The problem seems to be a race-condition caused by plymouth not terminating quickly enough. To make the error with vconsole-setuo stop, I modified /usr/lib/systemd/system/plymouth-start.service as follows: 1) change "KillMode" from none to "mixed", 2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and 3) run "dracut -f". I rebooted a few times and the errors were not generated. Let me know if this works for you and I will submit a bug and patch. -- Tony Walker <tony.walker.iu@gmail.com> PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/ 9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C On Fri, Jan 5, 2024 at 6:58 PM Tony Walker <tony.walker.iu@gmail.com> wrote:
I saw this too. I haven't had time to debug it, but my hunch is a missing dependency in it's unit file. That would cause a race condition where vconsole-setup tries to start before its dependencies are running.
Hope this helps.
-- Tony
On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to 20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB DESCRIPTION ? systemd-vconsole-setup.service loaded failed failed Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup Loaded: loaded (/usr/lib/systemd/system/systemd-vconsole-setup.service; static) Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27 EST; 23h ago Duration: 5h 2.219s Docs: man:systemd-vconsole-setup.service(8) man:vconsole.conf(5) Main PID: 1105 (code=exited, status=1/FAILURE) CPU: 5ms
systemd[1]: Starting Virtual Console Setup... systemd-vconsole-setup[1105]: No usable source console found: Device or resource busy systemd[1]: systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: systemd-vconsole-setup.service: Failed with result 'exit-code'. systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or resource busy" error but it happens on multiple TW machines and also on my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if something timewise is not ready during the boot process ?
Is this a known issue ?
-- Regards,
Joe
On Monday, 8 January 2024 04:05:15 ACDT Tony Walker wrote:
I finally found some time to debug and find a solution...
The problem seems to be a race-condition caused by plymouth not terminating quickly enough. To make the error with vconsole-setuo stop, I modified /usr/lib/systemd/system/plymouth-start.service as follows:
1) change "KillMode" from none to "mixed", 2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and 3) run "dracut -f".
The correct way to do this is to run systemctl edit <unit> - this will load an override file in your configured editor. In the section clearly marked, add the additional lines you need - e.g.: [Unit] IgnoreOnIsolate=true Then save and quit the editor and run systemctl daemon-reload (followed by dracut -f if needed). This ensures that local overrides won't be overwritten by system updates replacing the default unit file. You can see that the overrides have been applied using systemctl status <unit>, as seen here: plymouth-start.service - Show Plymouth Boot Screen Loaded: loaded (/usr/lib/systemd/system/plymouth-start.service; static) Drop-In: /etc/systemd/system/plymouth-start.service.d └─override.conf Active: inactive (dead) Note the Drop-In line - that indicates that a local override is in place. Regards, -- ============================================================================== ==================================== Rodney Baker rodney.baker@iinet.net.au ============================================================================== ====================================
I rebooted a few times and the errors were not generated.
Let me know if this works for you and I will submit a bug and patch.
-- Tony Walker <tony.walker.iu@gmail.com> PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/ 9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C
On Fri, Jan 5, 2024 at 6:58 PM Tony Walker <tony.walker.iu@gmail.com> wrote:
I saw this too. I haven't had time to debug it, but my hunch is a missing dependency in it's unit file. That would cause a race condition where vconsole-setup tries to start before its dependencies are running.
Hope this helps.
-- Tony
On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to 20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB DESCRIPTION
? systemd-vconsole-setup.service loaded failed failed
Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup
Loaded: loaded
(/usr/lib/systemd/system/systemd-vconsole-setup.service; static)
Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27
EST; 23h ago
Duration: 5h 2.219s
Docs: man:systemd-vconsole-setup.service(8)
man:vconsole.conf(5)
Main PID: 1105 (code=exited, status=1/FAILURE)
CPU: 5ms
systemd[1]: Starting Virtual Console Setup... systemd-vconsole-setup[1105]: No usable source console found: Device or resource busy systemd[1]: systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: systemd-vconsole-setup.service: Failed with result 'exit-code'. systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or resource busy" error but it happens on multiple TW machines and also on my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if something timewise is not ready during the boot process ?
Is this a known issue ?
-- Regards,
Joe
Thank you! You taught an old dog a new trick! I need to spend some quality time with systend. -- Tony On Mon, Jan 8, 2024, 8:50 PM Rodney Baker <rodney.baker@iinet.net.au> wrote:
On Monday, 8 January 2024 04:05:15 ACDT Tony Walker wrote:
I finally found some time to debug and find a solution...
The problem seems to be a race-condition caused by plymouth not
terminating quickly enough. To make the error with vconsole-setuo
stop, I modified /usr/lib/systemd/system/plymouth-start.service as
follows:
1) change "KillMode" from none to "mixed",
2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and
3) run "dracut -f".
The correct way to do this is to run systemctl edit <unit> - this will load an override file in your configured editor. In the section clearly marked, add the additional lines you need - e.g.:
[Unit]
IgnoreOnIsolate=true
Then save and quit the editor and run systemctl daemon-reload (followed by dracut -f if needed).
This ensures that local overrides won't be overwritten by system updates replacing the default unit file.
You can see that the overrides have been applied using systemctl status <unit>, as seen here:
plymouth-start.service - Show Plymouth Boot Screen Loaded: loaded (/usr/lib/systemd/system/plymouth-start.service; static) Drop-In: /etc/systemd/system/plymouth-start.service.d └─override.conf Active: inactive (dead)
Note the Drop-In line - that indicates that a local override is in place.
Regards,
--
==================================================================================================================
Rodney Baker
rodney.baker@iinet.net.au
==================================================================================================================
I rebooted a few times and the errors were not generated.
Let me know if this works for you and I will submit a bug and patch.
--
Tony Walker <tony.walker.iu@gmail.com>
PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/
9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C
On Fri, Jan 5, 2024 at 6:58 PM Tony Walker <tony.walker.iu@gmail.com> wrote:
I saw this too. I haven't had time to debug it, but my hunch is a missing
dependency in it's unit file. That would cause a race condition where
vconsole-setup tries to start before its dependencies are running.
Hope this helps.
-- Tony
On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to
20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB
DESCRIPTION
? systemd-vconsole-setup.service loaded failed failed
Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup
Loaded: loaded
(/usr/lib/systemd/system/systemd-vconsole-setup.service; static)
Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27
EST; 23h ago
Duration: 5h 2.219s
Docs: man:systemd-vconsole-setup.service(8)
man:vconsole.conf(5)
Main PID: 1105 (code=exited, status=1/FAILURE)
CPU: 5ms
systemd[1]: Starting Virtual Console Setup...
systemd-vconsole-setup[1105]: No usable source console found: Device or
resource busy
systemd[1]: systemd-vconsole-setup.service: Main process exited,
code=exited, status=1/FAILURE
systemd[1]: systemd-vconsole-setup.service: Failed with result
'exit-code'.
systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or
resource busy" error but it happens on multiple TW machines and also on
my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if
something timewise is not ready during the boot process ?
Is this a known issue ?
--
Regards,
Joe
Hi Rodney, That's what I actually did ( systemctl edit plymouth-start.service ) but this was only to test out the solution Tony suggested as he plans to submit the fix so that no override is needed once that change hits the TW repos. Prior to that I was just restarted the systemd-vconsole.setup.service after rebooting which also temporarily resolved it and I don't reboot that often. Joe On 1/8/24 02:04, Rodney Baker wrote:
On Monday, 8 January 2024 04:05:15 ACDT Tony Walker wrote:
I finally found some time to debug and find a solution...
The problem seems to be a race-condition caused by plymouth not
terminating quickly enough. To make the error with vconsole-setuo
stop, I modified /usr/lib/systemd/system/plymouth-start.service as
follows:
1) change "KillMode" from none to "mixed",
2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and
3) run "dracut -f".
The correct way to do this is to run systemctl edit <unit> - this will load an override file in your configured editor. In the section clearly marked, add the additional lines you need - e.g.:
[Unit]
IgnoreOnIsolate=true
Then save and quit the editor and run systemctl daemon-reload (followed by dracut -f if needed).
This ensures that local overrides won't be overwritten by system updates replacing the default unit file.
You can see that the overrides have been applied using systemctl status <unit>, as seen here:
plymouth-start.service - Show Plymouth Boot Screen Loaded: loaded (/usr/lib/systemd/system/plymouth-start.service; static) Drop-In: /etc/systemd/system/plymouth-start.service.d └─override.conf Active: inactive (dead)
Note the Drop-In line - that indicates that a local override is in place.
Regards,
--
==================================================================================================================
Rodney Baker
rodney.baker@iinet.net.au
==================================================================================================================
I rebooted a few times and the errors were not generated.
Let me know if this works for you and I will submit a bug and patch.
--
Tony Walker <tony.walker.iu@gmail.com>
PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/
9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C
On Fri, Jan 5, 2024 at 6:58 PM Tony Walker <tony.walker.iu@gmail.com> wrote:
I saw this too. I haven't had time to debug it, but my hunch is a missing
dependency in it's unit file. That would cause a race condition where
vconsole-setup tries to start before its dependencies are running.
Hope this helps.
-- Tony
On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to
20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB
DESCRIPTION
? systemd-vconsole-setup.service loaded failed failed
Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup
Loaded: loaded
(/usr/lib/systemd/system/systemd-vconsole-setup.service; static)
Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27
EST; 23h ago
Duration: 5h 2.219s
Docs: man:systemd-vconsole-setup.service(8)
man:vconsole.conf(5)
Main PID: 1105 (code=exited, status=1/FAILURE)
CPU: 5ms
systemd[1]: Starting Virtual Console Setup...
systemd-vconsole-setup[1105]: No usable source console found: Device or
resource busy
systemd[1]: systemd-vconsole-setup.service: Main process exited,
code=exited, status=1/FAILURE
systemd[1]: systemd-vconsole-setup.service: Failed with result
'exit-code'.
systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or
resource busy" error but it happens on multiple TW machines and also on
my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if
something timewise is not ready during the boot process ?
Is this a known issue ?
--
Regards,
Joe
-- Regards, Joe
Hi Tony, Thanks for debugging this, I was pretty sure it was timing related too because after booting you could just restart the service and it was fine. I tested your change and rebooted multiple times and your changes definitely fixed the problem. THANK YOU! Joe On 1/7/24 12:35, Tony Walker wrote:
I finally found some time to debug and find a solution...
The problem seems to be a race-condition caused by plymouth not terminating quickly enough. To make the error with vconsole-setuo stop, I modified /usr/lib/systemd/system/plymouth-start.service as follows:
1) change "KillMode" from none to "mixed", 2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and 3) run "dracut -f".
I rebooted a few times and the errors were not generated.
Let me know if this works for you and I will submit a bug and patch.
-- Tony Walker <tony.walker.iu@gmail.com> PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/ 9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C
On Fri, Jan 5, 2024 at 6:58 PM Tony Walker <tony.walker.iu@gmail.com> wrote:
I saw this too. I haven't had time to debug it, but my hunch is a missing dependency in it's unit file. That would cause a race condition where vconsole-setup tries to start before its dependencies are running.
Hope this helps.
-- Tony
On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to 20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB DESCRIPTION ? systemd-vconsole-setup.service loaded failed failed Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup Loaded: loaded (/usr/lib/systemd/system/systemd-vconsole-setup.service; static) Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27 EST; 23h ago Duration: 5h 2.219s Docs: man:systemd-vconsole-setup.service(8) man:vconsole.conf(5) Main PID: 1105 (code=exited, status=1/FAILURE) CPU: 5ms
systemd[1]: Starting Virtual Console Setup... systemd-vconsole-setup[1105]: No usable source console found: Device or resource busy systemd[1]: systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: systemd-vconsole-setup.service: Failed with result 'exit-code'. systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or resource busy" error but it happens on multiple TW machines and also on my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if something timewise is not ready during the boot process ?
Is this a known issue ?
-- Regards,
Joe
-- Regards, Joe
On Sun, 2024-01-07 at 12:35 -0500, Tony Walker wrote:
I finally found some time to debug and find a solution...
The problem seems to be a race-condition caused by plymouth not terminating quickly enough.
Thanks for debugging this and finding a workaround. It has been bothering me, too, but I never took the time to take deeper look. However, I am not quite certain whether your solution is correct. At least we should try to understand why it works.
To make the error with vconsole-setuo stop, I modified /usr/lib/systemd/system/plymouth-start.service as follows:
1) change "KillMode" from none to "mixed",
plymouth gets special treatment from systemd. Rather than stopping "plymouth-start.service", systemd runs "plymouth-quit.service". The reason is AFAIR that the plymouth splash screen is supposed to keep being displayed while the system is switching root. For this reason, plymouthd uses "KillMode=none". I suppose if you replace this with "mixed", plymouthd will be stopped when the initramfs shuts down. Do you see any difference in the visual appearance of the boot screen after applying your changes?
2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and
The effect of this directive is unclear to me for the case at hand. The man page says "If true, this unit will not be stopped when isolating another unit". But this unit hasn't been stopped before your changes, because it had "KillMode=none" set. Is this really needed to solve the issue?
3) run "dracut -f".
I rebooted a few times and the errors were not generated.
Let me know if this works for you and I will submit a bug and patch.
On my system, systemd-vconsole-setup.service runs successfully very early during initramfs processing. It prints the "device or resource busy" error when it's started again later, after switching root, probably because the plymouthd instance from the initrd is still running ("KillMode=none"). It's probably unnecessary to start systemd- vconsole-setup again after switching root, because the first invocation from the initramfs should have already set the fonts, keymaps, etc. Actually, looking at my boot log, it is weird to see that systemd- vconsole.setup is started and stopped more than 10 times during a single boot, while it should really be sufficient to do this once and for all very early at system startup. This looks like a systemd issue to me. I suggest you open a bug so that it can be properly analyzed. Regards, Martin
-- Tony Walker <tony.walker.iu@gmail.com> PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/ 9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C
On Fri, Jan 5, 2024 at 6:58 PM Tony Walker <tony.walker.iu@gmail.com> wrote:
I saw this too. I haven't had time to debug it, but my hunch is a missing dependency in it's unit file. That would cause a race condition where vconsole-setup tries to start before its dependencies are running.
Hope this helps.
-- Tony
On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to 20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB DESCRIPTION ? systemd-vconsole-setup.service loaded failed failed Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup Loaded: loaded (/usr/lib/systemd/system/systemd-vconsole-setup.service; static) Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27 EST; 23h ago Duration: 5h 2.219s Docs: man:systemd-vconsole-setup.service(8) man:vconsole.conf(5) Main PID: 1105 (code=exited, status=1/FAILURE) CPU: 5ms
systemd[1]: Starting Virtual Console Setup... systemd-vconsole-setup[1105]: No usable source console found: Device or resource busy systemd[1]: systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: systemd-vconsole-setup.service: Failed with result 'exit-code'. systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or resource busy" error but it happens on multiple TW machines and also on my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if something timewise is not ready during the boot process ?
Is this a known issue ?
-- Regards,
Joe
Thanks and good points, Martin! TL;DR I submitted a bug report (https://bugzilla.opensuse.org/show_bug.cgi?id=1218623), but missed an existing one (https://bugzilla.opensuse.org/show_bug.cgi?id=1218618). Not surprisingly, the maintainers were ahead of me and my understanding .I assumed that Plymouth was holding onto the tty and preventing systemd-vconsole-setup from resetting the tty. A simple view of my fix is that it kills Plymouth if it takes too long. However, it appears that Plymouth may be leaving the tty in graphics mode. Longer story...
However, I am not quite certain whether your solution is correct. At least we should try to understand why it works.
I am not sure either. ;-) Systemd and the boot process are outside my area of expertise. I started using Linux in the mid 90's and learned all about rc scripts. When Debian moved to systemd, I just went along for the ride and became a dinosaur who knew just enough to start and stop daemons. This seemed like a good excuse to learn, so please correct me.
plymouth gets special treatment from systemd. Rather than stopping "plymouth-start.service", systemd runs "plymouth-quit.service". The reason is AFAIR that the plymouth splash screen is supposed to keep being displayed while the system is switching root. For this reason, plymouthd uses "KillMode=none". I suppose if you replace this with "mixed", plymouthd will be stopped when the initramfs shuts down. Do you see any difference in the visual appearance of the boot screen after applying your changes?
Yes, there is a brief (~1-2s) view of the console before and after Plymouth. That is, this is a crude fix. My thinking was that there are other ways to fix that. Part of my reasoning is that Fedora uses the exact settings I proposed, so if I could find the time to dig into Fedora, I might find some other changes elsewhere. Of course, because Fedora sticks its head in a toilet, we don't have to. Still, I thought the Fedora folks would probably have a good implementation given their proximity to its developers and it might be good to see what they do. Crudely killing Plymouth when initramfs is done is exactly what I intended. My understanding is that systemd-vconsole-setup is called by systemd-localed.service, localectl, etc. My thinking was that Plymouth was holding the tty while the next boot step was trying to configure it. A classic race condition (I'm a programmer not an admin, so I always go for synchronization first).
2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and
The effect of this directive is unclear to me for the case at hand. The man page says "If true, this unit will not be stopped when isolating another unit". But this unit hasn't been stopped before your changes, because it had "KillMode=none" set. Is this really needed to solve the issue?
I honestly don't know. My testing says no. However, I saw this and thought it might fix a corner case that I never saw but thought might happen eventually. Basically, I am completely ignorant of what Plymouth does. I thought Plymouth might fork some other processes and, because I am ignorant, IgnoreOnIsolate=false might cause systemd to kill Plymouth due to timing. That is, I wanted to kill Plymouth but not too quickly. When I looked at the Fedora (rawhide) package, I saw that they do use IgnoreOnIsolate=true. Again, I don't know and we don't need to do what Fedora does. However, it did make me feel better.
On my system, systemd-vconsole-setup.service runs successfully very early during initramfs processing. It prints the "device or resource busy" error when it's started again later, after switching root, probably because the plymouthd instance from the initrd is still running ("KillMode=none"). It's probably unnecessary to start systemd- vconsole-setup again after switching root, because the first invocation from the initramfs should have already set the fonts, keymaps, etc.
Good point. I saw that too and did not bother to look at the logs from my last Debian system to see if the same thing happens there. However, I suspect so. As I said above, my understanding is that systemd-vconsole-setup is called by systemd-localed.service, localectl, etc. As such, I would expect to see repeated service starts during boot while the tty is reconfigured for different stages. Again, this is outside of my expertise, so I may be wrong.
Actually, looking at my boot log, it is weird to see that systemd- vconsole.setup is started and stopped more than 10 times during a single boot, while it should really be sufficient to do this once and for all very early at system startup.
As I said above, my assumption is it will be restarted several times. I am very likely wrong.
This looks like a systemd issue to me. I suggest you open a bug so that it can be properly analyzed.
I searched for the bug and opened one: https://bugzilla.opensuse.org/show_bug.cgi?id=1218623. Unfortunately, I missed an existing one: https://bugzilla.opensuse.org/show_bug.cgi?id=1218618. Sorry! I did find similar reports: https://bugzilla.opensuse.org/show_bug.cgi?id=1215282 and https://bugzilla.opensuse.org/show_bug.cgi?id=943312. In those bug reports, it appears that Plymouth leaves the tty in graphics mode. My thinking was that killing Plymouth would do some "garbage collection" and reset the terminal if this is the root of my problem. Either way, it seemed I had a partial solution. Feel free to teach me or ask me questions. -- Tony Walker <tony.walker.iu@gmail.com> PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/ 9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C On Tue, Jan 9, 2024 at 4:46 AM Martin Wilck <mwilck@suse.com> wrote:
On Sun, 2024-01-07 at 12:35 -0500, Tony Walker wrote:
I finally found some time to debug and find a solution...
The problem seems to be a race-condition caused by plymouth not terminating quickly enough.
Thanks for debugging this and finding a workaround. It has been bothering me, too, but I never took the time to take deeper look.
However, I am not quite certain whether your solution is correct. At least we should try to understand why it works.
To make the error with vconsole-setuo stop, I modified /usr/lib/systemd/system/plymouth-start.service as follows:
1) change "KillMode" from none to "mixed",
plymouth gets special treatment from systemd. Rather than stopping "plymouth-start.service", systemd runs "plymouth-quit.service". The reason is AFAIR that the plymouth splash screen is supposed to keep being displayed while the system is switching root. For this reason, plymouthd uses "KillMode=none". I suppose if you replace this with "mixed", plymouthd will be stopped when the initramfs shuts down. Do you see any difference in the visual appearance of the boot screen after applying your changes?
2) add "IgnoreOnIsolate=true" to the "[Unit]" section, and
The effect of this directive is unclear to me for the case at hand. The man page says "If true, this unit will not be stopped when isolating another unit". But this unit hasn't been stopped before your changes, because it had "KillMode=none" set. Is this really needed to solve the issue?
3) run "dracut -f".
I rebooted a few times and the errors were not generated.
Let me know if this works for you and I will submit a bug and patch.
On my system, systemd-vconsole-setup.service runs successfully very early during initramfs processing. It prints the "device or resource busy" error when it's started again later, after switching root, probably because the plymouthd instance from the initrd is still running ("KillMode=none"). It's probably unnecessary to start systemd- vconsole-setup again after switching root, because the first invocation from the initramfs should have already set the fonts, keymaps, etc.
Actually, looking at my boot log, it is weird to see that systemd- vconsole.setup is started and stopped more than 10 times during a single boot, while it should really be sufficient to do this once and for all very early at system startup.
This looks like a systemd issue to me. I suggest you open a bug so that it can be properly analyzed.
Regards, Martin
-- Tony Walker <tony.walker.iu@gmail.com> PGP Key @ https://tonywalker1.github.io/ or https://keys.openpgp.org/ 9F46 D66D FF6C 182D A5AC 11E1 8559 98D1 7543 319C
On Fri, Jan 5, 2024 at 6:58 PM Tony Walker <tony.walker.iu@gmail.com> wrote:
I saw this too. I haven't had time to debug it, but my hunch is a missing dependency in it's unit file. That would cause a race condition where vconsole-setup tries to start before its dependencies are running.
Hope this helps.
-- Tony
On Fri, Jan 5, 2024, 6:24 PM Joe Salmeri <jmscdba@gmail.com> wrote:
Yesterday I updated several TW machines and also TW vms from 20231120 to 20231228.
After rebooting I ran systemctl --failed which shows
UNIT LOAD ACTIVE SUB DESCRIPTION ? systemd-vconsole-setup.service loaded failed failed Virtual Console Setup
systemctl status systemd-vconsole-setup.service provides the details
× systemd-vconsole-setup.service - Virtual Console Setup Loaded: loaded (/usr/lib/systemd/system/systemd-vconsole-setup.service; static) Active: failed (Result: exit-code) since Thu 2024-01-04 17:10:27 EST; 23h ago Duration: 5h 2.219s Docs: man:systemd-vconsole-setup.service(8) man:vconsole.conf(5) Main PID: 1105 (code=exited, status=1/FAILURE) CPU: 5ms
systemd[1]: Starting Virtual Console Setup... systemd-vconsole-setup[1105]: No usable source console found: Device or resource busy systemd[1]: systemd-vconsole-setup.service: Main process exited, code=exited, status=1/FAILURE systemd[1]: systemd-vconsole-setup.service: Failed with result 'exit-code'. systemd[1]: Failed to start Virtual Console Setup.
Not sure what is causing the "No usable source console found: Device or resource busy" error but it happens on multiple TW machines and also on my TW vms.
systemctl restart systemd-vconsole-setup.service
Restarts the service with no errors or problems so wondering if something timewise is not ready during the boot process ?
Is this a known issue ?
-- Regards,
Joe
participants (4)
-
Joe Salmeri -
Martin Wilck -
Rodney Baker -
Tony Walker