[opensuse-kubic] Failure bringing up kubic VMs with terraform/libvirt: timed out waiting for device /dev/disk/by-label/ignition
Hi, I am trying to set up a Kubic cluster using terraform-provider-libvirt [1]. I have started with the terraform scripts from the 'official' kubic project [2]. The first problem was that I needed to downgrade to terraform 0.12, as TW does not have 0.13. I won't send the full diff, but it was just about 1) adjusting the required_version and removing the required_providers and 2) shortening the network name. After that was done, all VMs fail to boot since apparently they can't find the ignition boot config. Connecting through virt-manager I see the following output (manually copied, may contain typos): [ TIME ] Timed out waiting for device /dev/disk/by-label/ignition [DEPEND] Dependency failed for Combustion [DEPEND] Dependency failed for Initrd Default Target [FAILED] Failed to start Ingnition (fetch-offline) (snip) I am dropped into an emergency console, but for some reason I am unable to actually type commands in that console, and `virsh console` does not work either, so I don't know how to gather more information. What I can tell is that qemu is started with the correct flag for igntition: -fw_cfg name=opt/com.coreos/config,file=/var/libvirt-pool/kubic- ignition-0 I tried to blindly fix some things, and I noticed that the locally installed ignition-validate complains about the generated ignition config, so I switched the version from 3.1.0-experimental to 3.1.0 . This changes things somewhat, in the sense that the following messages cycle indefinitely [ TIME ] Timed out waiting for device /dev/disk/by-label/ignition [DEPEN D] Dependency failed for Combustion [DEPEND] Dependency failed for Initrd Default Target At this point, I have no idea how to advance. Can anyone please point me in the right direction with setting up a set of Kubic VMs with terraform/libvirt? Thanks, Robert [1]: https://github.com/dmacvicar/terraform-provider-libvirt [2]: https://github.com/kubic-project/kubic-terraform-kvm -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Hi, Am Dienstag, 15. September 2020, 10:48:33 CEST schrieb Robert Munteanu:
Hi,
I am trying to set up a Kubic cluster using terraform-provider-libvirt [1]. I have started with the terraform scripts from the 'official' kubic project [2].
The first problem was that I needed to downgrade to terraform 0.12, as TW does not have 0.13. I won't send the full diff, but it was just about 1) adjusting the required_version and removing the required_providers and 2) shortening the network name.
After that was done, all VMs fail to boot since apparently they can't find the ignition boot config. Connecting through virt-manager I see the following output (manually copied, may contain typos):
[ TIME ] Timed out waiting for device /dev/disk/by-label/ignition [DEPEND] Dependency failed for Combustion [DEPEND] Dependency failed for Initrd Default Target [FAILED] Failed to start Ingnition (fetch-offline)
The failed ignition-fetch-offline.service was most likely because of the outdated config with "-experimental". The initrd default target failure is fixed already in the next snapshot, which should be released soon. As a workaround, you can use a config drive instead of fw_cfg. Cheers, Fabian
(snip)
I am dropped into an emergency console, but for some reason I am unable to actually type commands in that console, and `virsh console` does not work either, so I don't know how to gather more information.
What I can tell is that qemu is started with the correct flag for igntition:
-fw_cfg name=opt/com.coreos/config,file=/var/libvirt-pool/kubic- ignition-0
I tried to blindly fix some things, and I noticed that the locally installed ignition-validate complains about the generated ignition config, so I switched the version from 3.1.0-experimental to 3.1.0 . This changes things somewhat, in the sense that the following messages cycle indefinitely
[ TIME ] Timed out waiting for device /dev/disk/by-label/ignition [DEPEN D] Dependency failed for Combustion [DEPEND] Dependency failed for Initrd Default Target
At this point, I have no idea how to advance. Can anyone please point me in the right direction with setting up a set of Kubic VMs with terraform/libvirt?
Thanks, Robert
[1]: https://github.com/dmacvicar/terraform-provider-libvirt [2]: https://github.com/kubic-project/kubic-terraform-kvm
-- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Hi Fabian, On Tue, 2020-09-15 at 11:00 +0200, Fabian Vogt wrote:
Hi,
Am Dienstag, 15. September 2020, 10:48:33 CEST schrieb Robert Munteanu:
Hi,
I am trying to set up a Kubic cluster using terraform-provider- libvirt [1]. I have started with the terraform scripts from the 'official' kubic project [2].
The first problem was that I needed to downgrade to terraform 0.12, as TW does not have 0.13. I won't send the full diff, but it was just about 1) adjusting the required_version and removing the required_providers and 2) shortening the network name.
After that was done, all VMs fail to boot since apparently they can't find the ignition boot config. Connecting through virt-manager I see the following output (manually copied, may contain typos):
[ TIME ] Timed out waiting for device /dev/disk/by-label/ignition [DEPEND] Dependency failed for Combustion [DEPEND] Dependency failed for Initrd Default Target [FAILED] Failed to start Ingnition (fetch-offline)
The failed ignition-fetch-offline.service was most likely because of the outdated config with "-experimental". The initrd default target failure is fixed already in the next snapshot, which should be released soon.
As a workaround, you can use a config drive instead of fw_cfg.
Thanks for the context. I think I'll just wait for the next snapshot and come back if it still does not work. Thanks, Roebrt -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Hello again, On Tue, 2020-09-15 at 11:08 +0200, Robert Munteanu wrote:
Hi Fabian,
On Tue, 2020-09-15 at 11:00 +0200, Fabian Vogt wrote:
Hi,
Am Dienstag, 15. September 2020, 10:48:33 CEST schrieb Robert Munteanu:
Hi,
I am trying to set up a Kubic cluster using terraform-provider- libvirt [1]. I have started with the terraform scripts from the 'official' kubic project [2].
The first problem was that I needed to downgrade to terraform 0.12, as TW does not have 0.13. I won't send the full diff, but it was just about 1) adjusting the required_version and removing the required_providers and 2) shortening the network name.
After that was done, all VMs fail to boot since apparently they can't find the ignition boot config. Connecting through virt-manager I see the following output (manually copied, may contain typos):
[ TIME ] Timed out waiting for device /dev/disk/by-label/ignition [DEPEND] Dependency failed for Combustion [DEPEND] Dependency failed for Initrd Default Target [FAILED] Failed to start Ingnition (fetch-offline)
The failed ignition-fetch-offline.service was most likely because of the outdated config with "-experimental". The initrd default target failure is fixed already in the next snapshot, which should be released soon.
As a workaround, you can use a config drive instead of fw_cfg.
Thanks for the context. I think I'll just wait for the next snapshot and come back if it still does not work.
For the sake of completeness: - pulling a new Kubic image today has fixed the VM creation process - the following message still appears [ TIME ] Timed out waiting for device /dev/disk/by-label/ignition - ignition has successfully been executed though, judging by the hostname settings So for now my problem is solved. Thanks, Robert -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Robert, just wanted to share that I too am having issues with Ignition
after firstboot.
I'm on a Raspberry Pi 4, but even though
`/boot/writable/firstboot_happened` exists it's still looking for that
disk and delaying my startup.
The next thing I'm going to try is doing a `transactional-update
initrd` to see if there's a config file being changed but not updated
in the initramfs.
Jim
---------------------------------------------------
James Heald
Production Engineer
Facebook
james.r.heald@gmail.com
On Wed, Sep 16, 2020 at 1:34 AM Robert Munteanu
Hello again,
On Tue, 2020-09-15 at 11:08 +0200, Robert Munteanu wrote:
Hi Fabian,
On Tue, 2020-09-15 at 11:00 +0200, Fabian Vogt wrote:
Hi,
Am Dienstag, 15. September 2020, 10:48:33 CEST schrieb Robert Munteanu:
Hi,
I am trying to set up a Kubic cluster using terraform-provider- libvirt [1]. I have started with the terraform scripts from the 'official' kubic project [2].
The first problem was that I needed to downgrade to terraform 0.12, as TW does not have 0.13. I won't send the full diff, but it was just about 1) adjusting the required_version and removing the required_providers and 2) shortening the network name.
After that was done, all VMs fail to boot since apparently they can't find the ignition boot config. Connecting through virt-manager I see the following output (manually copied, may contain typos):
[ TIME ] Timed out waiting for device /dev/disk/by-label/ignition [DEPEND] Dependency failed for Combustion [DEPEND] Dependency failed for Initrd Default Target [FAILED] Failed to start Ingnition (fetch-offline)
The failed ignition-fetch-offline.service was most likely because of the outdated config with "-experimental". The initrd default target failure is fixed already in the next snapshot, which should be released soon.
As a workaround, you can use a config drive instead of fw_cfg.
Thanks for the context. I think I'll just wait for the next snapshot and come back if it still does not work.
For the sake of completeness:
- pulling a new Kubic image today has fixed the VM creation process - the following message still appears
[ TIME ] Timed out waiting for device /dev/disk/by-label/ignition
- ignition has successfully been executed though, judging by the hostname settings
So for now my problem is solved.
Thanks, Robert -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
-- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Am 16.09.20 um 16:53 Uhr schrieb Jim Heald:
Robert, just wanted to share that I too am having issues with Ignition after firstboot.
I'm on a Raspberry Pi 4, but even though `/boot/writable/firstboot_happened` exists it's still looking for that disk and delaying my startup.
The next thing I'm going to try is doing a `transactional-update initrd` to see if there's a config file being changed but not updated in the initramfs.
This seems to be a bug in Combustion's systemd handling - we are working on it, no action required from your side. Cheers, Ignaz -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
On Thu, 2020-09-17 at 09:39 +0200, Ignaz Forster wrote:
Am 16.09.20 um 16:53 Uhr schrieb Jim Heald:
Robert, just wanted to share that I too am having issues with Ignition after firstboot.
I'm on a Raspberry Pi 4, but even though `/boot/writable/firstboot_happened` exists it's still looking for that disk and delaying my startup.
The next thing I'm going to try is doing a `transactional-update initrd` to see if there's a config file being changed but not updated in the initramfs.
This seems to be a bug in Combustion's systemd handling - we are working on it, no action required from your side.
FWIW I filed https://bugzilla.opensuse.org/show_bug.cgi?id=1176566 about the startup behaviour; I was suprised to see the system becoming unbootable after just installing ignition locally. Thanks, Robert -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
participants (4)
-
Fabian Vogt -
Ignaz Forster -
Jim Heald -
Robert Munteanu