[Bug 1214406] New: [kubeadm1.27] Multiple Issues before `kubeadm init` runs successfully
https://bugzilla.suse.com/show_bug.cgi?id=1214406 Bug ID: 1214406 Summary: [kubeadm1.27] Multiple Issues before `kubeadm init` runs successfully Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: All OS: openSUSE Tumbleweed Status: NEW Severity: Normal Priority: P5 - None Component: Containers Assignee: priyanka.saggu@suse.com Reporter: eich@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: Development Blocker: --- 1. The Kubernetes package on TW seems to require cri-o as runtime dependency, however, it requires containerd. Thus, - cri-o should not be installed as a dependency - containerd had to be installed enabled and started (containerd doesn't have an rccontainerd link) 2. kubelet.service does not start: It complains about the option `--container-runtime` not being known. This option is pulled in from /etc/sysconfig/kubelet. It needs to be deleted from this file. It is not clear who has created this file and which package owns it. The rpm database does not know it. Since it is pulled in by ..../systemd/system/kubelet.service.d/10-kubeadm.conf I suspect it comes from kubernetes1.27-kubeadem but it's hard to tell where it is created - at least not in a post-install script. It would be good if the file was '%ghost' in the %files section of the package it originates from. The same is true for the environment file /var/lib/kubelet/kubeadm-flags.env which is written by `kubeadm init`. 3. 'kubeadm init' checks if the kubelet service is enabled. 4. Moreover, it requires a reboot to start all dependent services and load all required kernel modules: - br_netfilter needs to be loaded - containerd needs to be started This was fine for kubic these packages were originally meant for as this had transactional updates and required a reboot. On a 'regular' system where this is not the case, these modules should probably be loaded and services be started from a 'postinstall' script. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c1 --- Comment #1 from Egbert Eich <eich@suse.com> --- I was partly wrong with my assessment in item 1. in above description: It is perfectly fine to use crio runtime - however, it needs to be enabled and running at the time `kubeadm init` runs. Thus, this should be set in a postinstall script. NOTE: before doing so it should be checked if another container runtime (containerd) is already enabled (or even running). It it is running, nothing should be done (ie crio should not be touched). If it is just enabled, it should be started. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c2 --- Comment #2 from Priyanka Saggu <priyanka.saggu@suse.com> --- Hey Egbert, no updates just yet -- just writing to acknowledge the ticket. I'm discussing re-packaging kubernetes, will use the issues raised in the bugzilla as feedback, for the same! Once I've any implementation updates, will post here. Thanks again! -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c3 Dan Čermák <dcermak@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dcermak@suse.com, | |eich@suse.com Flags| |needinfo?(eich@suse.com) --- Comment #3 from Dan Čermák <dcermak@suse.com> --- (In reply to Egbert Eich from comment #0)
2. kubelet.service does not start: It complains about the option `--container-runtime` not being known. This option is pulled in from /etc/sysconfig/kubelet. It needs to be deleted from this file.
Could you please paste the contents of that file here? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c4 Egbert Eich <eich@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(eich@suse.com) | --- Comment #4 from Egbert Eich <eich@suse.com> --- (In reply to Dan Čermák from comment #3)
(In reply to Egbert Eich from comment #0)
2. kubelet.service does not start: It complains about the option `--container-runtime` not being known. This option is pulled in from /etc/sysconfig/kubelet. It needs to be deleted from this file.
Could you please paste the contents of that file here?
Do you mean before or after I've manually edited and fixed it? Since I don't have a backup copy of the 'before' version, I'd have to go thru the entire exercise of installing k8s and initializing a controller again. I'd suggest you folks try this yourself. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c5 --- Comment #5 from Egbert Eich <eich@suse.com> --- Actually, since the sysconfig files are backed up by 'fillup-templates', here is the content of /usr/share/fillup-templates/sysconfig.kubelet: KUBELET_EXTRA_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=15m --cgroup-driver=systemd -v=2" This is the content of /etc/sysconfig/kubelet - after I've manually fixed it: KUBELET_VER=1.27 KUBELET_EXTRA_ARGS="--container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=14m --cgroup-driver=systemd -v=2" -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c6 --- Comment #6 from Egbert Eich <eich@suse.com> --- To somewhat lift the mystery around /etc/sysconfig/kubelet: 1. The 'fillup template' is from cri-o-kubeadm-criconfig-1.24.3 2. cri-o-kubeadm-criconfig-1.24.3 provides 'kubernetes-kubeadm-criconfig' - unversioned. 3. It seems like cri-o is released in tandem with kubernetes (the versions are in sync). Therefore, 1. cri-o should be updated in tandem with kubernetes. 2. Dependencies between cri-o and kubernetes should be set so that only matching versions are installed. I've used patterns-kubernetes-kubeadm to install all of the goodness required by kubeadm, this only has versioned dependencies. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c7 --- Comment #7 from Egbert Eich <eich@suse.com> --- (In reply to Egbert Eich from comment #6)
I've used patterns-kubernetes-kubeadm to install all of the goodness required by kubeadm, this only has versioned dependencies.
I meant to say 'unversioned' dependencies. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c8 --- Comment #8 from Priyanka Saggu <priyanka.saggu@suse.com> --- Hey Egbert, I've raised the following SRs: - cri-o ~ https://build.opensuse.org/request/show/1108439 - kubernetes1.27 ~ https://build.opensuse.org/request/show/1108445 - containerd ~ https://build.opensuse.org/request/show/1108423 With these changes, I was able to have a successful `kubeadm init` run! # Although - before `kubeadm init`, I had to manually load the kernel modules (doing that in %post scriptlet didn't work, it complained for modprobe not found. Nor did adding it as `ExecStartPre=` in 10-kubeadm.conf work. I'll explore more options)
sudo modprobe overlay && sudo modprobe br_netfilter
# with cri-o as CRI sudo kubeadm init --kubernetes-version 1.27.4 --cri-socket=unix:///var/run/crio/crio.sock
# with containerd as CRI sudo kubeadm init --kubernetes-version 1.27.4 --cri-socket=unix:///var/run/containerd/containerd.sock
---
3. It seems like cri-o is released in tandem with kubernetes (the versions are in sync). Therefore, 1. cri-o should be updated in tandem with kubernetes. 2. Dependencies between cri-o and kubernetes should be set so that only matching versions are installed.
I agree ^. I'll fix the crio/kubernetes version dependency part. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c9 --- Comment #9 from Priyanka Saggu <priyanka.saggu@suse.com> --- Also, I'm yet to fix - "updating the `KUBELET_VER=x.yz` in `/etc/sysconfig/kubelet` file" on a new kubernetes version installation (after an existing kubernetes installation). --- Also, this just fixes the kubernetes 1.27 install, the previous versions still need fixing. There's drift between kubeadm and kubelet/kubelet-common versions
$ sudo zypper in kubernetes1.26-kubeadm
... Resolving package dependencies...
The following 6 NEW packages are going to be installed: cri-o cri-o-kubeadm-criconfig kubernetes1.25-kubelet kubernetes1.26-kubeadm kubernetes1.27-kubelet kubernetes1.27-kubelet-common -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c10 --- Comment #10 from Egbert Eich <eich@suse.com> --- (In reply to Priyanka Saggu from comment #8)
Hey Egbert, I've raised the following SRs:
- cri-o ~ https://build.opensuse.org/request/show/1108439 - kubernetes1.27 ~ https://build.opensuse.org/request/show/1108445 - containerd ~ https://build.opensuse.org/request/show/1108423
With these changes, I was able to have a successful `kubeadm init` run!
Cool! We should actually get the OpenQA tests going once there is a version which works in manual testing.
# Although - before `kubeadm init`, I had to manually load the kernel modules (doing that in %post scriptlet didn't work, it complained for modprobe not found. Nor did adding it as `ExecStartPre=` in 10-kubeadm.conf work. I'll explore more options)
Ok, did you specify the full path to modprobe (ie /sbin/modprobe)? Usually, %pre/%post install scripts do so. The same is true with `ExecStartPre=`: grepping through /usr/lib/systemd/system I find multiple examples with: `ExecStartPre=-/sbin/modprobe ...` so it should be expected to work. (In reply to Priyanka Saggu from comment #9)
Also, I'm yet to fix - "updating the `KUBELET_VER=x.yz` in `/etc/sysconfig/kubelet` file" on a new kubernetes version installation (after an existing kubernetes installation).
Looks like the `fillup` semantics should support this: `/bin/fillup -x /etc/sysconfig/kubelet /usr/share/fillup-templates/sysconfig.kubelet-kubernetes1.27` Unfortunately, there is no convenience macro for this - like the SUSE-bespoke %fillup_only macro. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c12 --- Comment #12 from Maintenance Automation <maint-coord+maintenance-robot@suse.de> --- SUSE-RU-2023:4341-1: An update that contains one feature and has two fixes can now be installed. Category: recommended (moderate) Bug References: 1213829, 1214406 Jira References: PED-5839 Sources used: openSUSE Leap 15.4 (src): kubernetes1.27-1.27.6-150400.9.3.2 openSUSE Leap 15.5 (src): kubernetes1.27-1.27.6-150400.9.3.2 Containers Module 15-SP4 (src): kubernetes1.27-1.27.6-150400.9.3.2 Containers Module 15-SP5 (src): kubernetes1.27-1.27.6-150400.9.3.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c13 --- Comment #13 from Maintenance Automation <maint-coord+maintenance-robot@suse.de> --- SUSE-RU-2023:4366-1: An update that contains one feature and has two fixes can now be installed. Category: recommended (moderate) Bug References: 1213829, 1214406 Jira References: PED-5839 Sources used: openSUSE Leap 15.4 (src): kubernetes1.26-1.26.9-150400.9.3.2 openSUSE Leap 15.5 (src): kubernetes1.26-1.26.9-150400.9.3.2 Containers Module 15-SP4 (src): kubernetes1.26-1.26.9-150400.9.3.2 Containers Module 15-SP5 (src): kubernetes1.26-1.26.9-150400.9.3.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c14 --- Comment #14 from Maintenance Automation <maint-coord+maintenance-robot@suse.de> --- SUSE-RU-2023:4365-1: An update that contains one feature and has one fix can now be installed. Category: recommended (moderate) Bug References: 1214406 Jira References: PED-5839 Sources used: Containers Module 15-SP4 (src): kubernetes1.25-1.25.14-150400.9.3.2 Containers Module 15-SP5 (src): kubernetes1.25-1.25.14-150400.9.3.2 openSUSE Leap 15.4 (src): kubernetes1.25-1.25.14-150400.9.3.2 openSUSE Leap 15.5 (src): kubernetes1.25-1.25.14-150400.9.3.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c15 --- Comment #15 from Maintenance Automation <maint-coord+maintenance-robot@suse.de> --- SUSE-RU-2023:4498-1: An update that has one fix can now be installed. Category: recommended (moderate) Bug References: 1214406 Sources used: openSUSE Leap 15.4 (src): kubernetes1.24-1.24.17-150400.9.11.2 Containers Module 15-SP4 (src): kubernetes1.24-1.24.17-150400.9.11.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c16 --- Comment #16 from Maintenance Automation <maint-coord+maintenance-robot@suse.de> --- SUSE-RU-2023:4497-1: An update that has one fix can now be installed. Category: recommended (moderate) Bug References: 1214406 Sources used: openSUSE Leap 15.5 (src): kubernetes1.24-1.24.17-150500.3.10.2 Containers Module 15-SP5 (src): kubernetes1.24-1.24.17-150500.3.10.2 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c19 --- Comment #19 from Maintenance Automation <maint-coord+maintenance-robot@suse.de> --- SUSE-SU-2024:3343-1: An update that solves nine vulnerabilities and has six security fixes can now be installed. URL: https://www.suse.com/support/update/announcement/2024/suse-su-20243343-1 Category: security (important) Bug References: 1062303, 1194400, 1211630, 1211631, 1214406, 1216109, 1216123, 1219964, 1221400, 1222539, 1226136, 1229858, 1229867, 1229869, 1230323 CVE References: CVE-2021-25743, CVE-2023-2727, CVE-2023-2728, CVE-2023-39325, CVE-2023-44487, CVE-2023-45288, CVE-2024-0793, CVE-2024-24786, CVE-2024-3177 Maintenance Incident: [SUSE:Maintenance:35690](https://smelt.suse.de/incident/35690/) Sources used: openSUSE Leap 15.3 (src): kubernetes1.24-1.24.17-150300.7.6.1 SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src): kubernetes1.24-1.24.17-150300.7.6.1 SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src): kubernetes1.24-1.24.17-150300.7.6.1 SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src): kubernetes1.24-1.24.17-150300.7.6.1 SUSE Enterprise Storage 7.1 (src): kubernetes1.24-1.24.17-150300.7.6.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1214406 https://bugzilla.suse.com/show_bug.cgi?id=1214406#c20 --- Comment #20 from Maintenance Automation <maint-coord+maintenance-robot@suse.de> --- SUSE-SU-2024:3341-1: An update that solves nine vulnerabilities and has six security fixes can now be installed. URL: https://www.suse.com/support/update/announcement/2024/suse-su-20243341-1 Category: security (important) Bug References: 1062303, 1194400, 1211630, 1211631, 1214406, 1216109, 1216123, 1219964, 1221400, 1222539, 1226136, 1229858, 1229867, 1229869, 1230323 CVE References: CVE-2021-25743, CVE-2023-2727, CVE-2023-2728, CVE-2023-39325, CVE-2023-44487, CVE-2023-45288, CVE-2024-0793, CVE-2024-24786, CVE-2024-3177 Maintenance Incident: [SUSE:Maintenance:32855](https://smelt.suse.de/incident/32855/) Sources used: openSUSE Leap 15.4 (src): kubernetes1.24-1.24.17-150400.9.16.1 SUSE Linux Enterprise High Performance Computing ESPOS 15 SP4 (src): kubernetes1.24-1.24.17-150400.9.16.1 SUSE Linux Enterprise High Performance Computing LTSS 15 SP4 (src): kubernetes1.24-1.24.17-150400.9.16.1 SUSE Linux Enterprise Server 15 SP4 LTSS 15-SP4 (src): kubernetes1.24-1.24.17-150400.9.16.1 SUSE Linux Enterprise Server for SAP Applications 15 SP4 (src): kubernetes1.24-1.24.17-150400.9.16.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com