[Bug 1218694] New: [k8s,kube-flannel] YAML File to set up Flannel Network Add-On is outdated
https://bugzilla.suse.com/show_bug.cgi?id=1218694 Bug ID: 1218694 Summary: [k8s,kube-flannel] YAML File to set up Flannel Network Add-On is outdated Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: All OS: openSUSE Tumbleweed Status: NEW Severity: Normal Priority: P5 - None Component: Containers Assignee: containers-bugowner@suse.de Reporter: eich@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: Development Blocker: --- After successfully setting up k8s (using `kubeadm init`), the coredns pod remains in a 'Pending' state: NAMESPACE NAME READY STATUS RESTARTS AG kube-system coredns-86ccd44ff8-24h9w 0/1 Pending 0 12 kube-system coredns-86ccd44ff8-rz5vq 0/1 Pending 0 12 kube-system etcd-k8s-head 1/1 Running 1 12 kube-system kube-apiserver-k8s-head 1/1 Running 1 12 kube-system kube-controller-manager-k8s-head 1/1 Running 1 12 kube-system kube-proxy-qprnj 1/1 Running 0 12 kube-system kube-scheduler-k8s-head 1/1 Running 1 13 This is due to a not yet installed network plugin. When installing `kubeadm` thru 'patterns-kubernetes-kubeadm', the package 'flannel-k8s-yaml' is installed alongside which contains a yaml file to set up Flannel as network plugin. Running `kubectl apply -f /usr/share/k8s-yaml/flannel/kube-flannel.yaml` using the supplied file fails, however after: | clusterrole.rbac.authorization.k8s.io/flannel created | clusterrolebinding.rbac.authorization.k8s.io/flannel created | serviceaccount/flannel created | configmap/kube-flannel-cfg created | daemonset.apps/kube-flannel-ds created with the error: | error: resource mapping not found for name: "psp.flannel.unprivileged" namespace: "" from "/usr/share/k8s-yaml/flannel/kube-flannel.yaml": no matches for kind "PodSecurityPolicy" in version "policy/v1beta1" ensure CRDs are installed first When using the yaml file from https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel... (wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/ku...) it works, however, the flannel pod is still cycling thru a CrashLoopBackOff - it fails with (from log): | E0110 11:46:48.988799 1 main.go:332] Error registering network: failed to acquire lease: node "k8s-head" pod cidr not assigned -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 https://bugzilla.suse.com/show_bug.cgi?id=1218694#c1 Priyanka Saggu <priyanka.saggu@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |priyanka.saggu@suse.com --- Comment #1 from Priyanka Saggu <priyanka.saggu@suse.com> --- Thanks for the ticket, @Egbert Eich! For the crashing flannel pods: flannel, by default, expects the podCIDR to be set to "10.244.0.0/16"[2] at the time of "kubeadm init":
sudo kubeadm init --pod-network-cidr 10.244.0.0/16
(Or in case of custom podCIDR, same needs to be updated in the kube-flannel.yaml[1] file as well: ``` net-conf.json: | { "Network": "<customCIDR>", "Backend": { "Type": "vxlan" } } ``` ) [1] https://github.com/flannel-io/flannel/tree/master?tab=readme-ov-file#deployi... [2] https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.... ----- For the coreDNS pods, they get stuck in "pending/containerCreating" state due to missing CNI plugins in /opt/cni/bin path. ```
k describe pod coredns-86ccd44ff8-hz2lj -n kube-system
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 14s default-scheduler Successfully assigned kube-system/coredns-86ccd44ff8-hz2lj to 5 Warning FailedCreatePodSandBox 13s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "de201303e9a5802dbae988d668bed2640315da55b4654c2cdda3e5cffeec8ab4": plugin type="loopback" failed (add): failed to find plugin "loopback" in path [/opt/cni/bin] Normal SandboxChanged 13s kubelet Pod sandbox changed, it will be killed and re-created. ``` Flannel doesn't add these CNI binaries into the /opt/cni/bin path[3] Adding the binaries manually in the required path, fixes it:
mkdir -p /opt/cni/bin curl -O -L https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-... sudo tar -C /opt/cni/bin -xzf cni-plugins-linux-amd64-v1.4.0.tgz
[3]https://github.com/flannel-io/flannel/issues/1715 I'll send an update to kube-flannel package for above. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 https://bugzilla.suse.com/show_bug.cgi?id=1218694#c2 --- Comment #2 from Egbert Eich <eich@suse.com> --- The flaw of k8s is that it is extremely complex and this seems to be getting worse every time I look at it. The solution above may be acceptable for Tumbleweed, but it is not great: letting people download binaries from some place on the internet is not why people use distributions - it is definitely a security issue. If these plugins are required, they should be packaged and the package should be required by patterns-kubernetes-kubeadm. Pre-installing these plugins removes some flexibility, but the user who wants this can do things the hard way. k3s and RKE2 also make this choice for the user - and people love it. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 https://bugzilla.suse.com/show_bug.cgi?id=1218694#c3 Danish Prakash <danish.prakash@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |danish.prakash@suse.com --- Comment #3 from Danish Prakash <danish.prakash@suse.com> --- cni-plugins[1] is already packaged and is being used, jfyi [1] - https://build.opensuse.org/package/show/openSUSE:Factory/cni-plugins -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 https://bugzilla.suse.com/show_bug.cgi?id=1218694#c4 --- Comment #4 from Egbert Eich <eich@suse.com> --- (In reply to Danish Prakash from comment #3)
cni-plugins[1] is already packaged and is being used, jfyi
[1] - https://build.opensuse.org/package/show/openSUSE:Factory/cni-plugins
Indeed! This installs the plugins into /usr/lib(exec)/cni. I'm not sure if they are found there, though, as kubelet seems to expect them in /opt/cni/bin/. Otherwise, the 'cni-plugins' package should probably be added to the dependencies somewhere. I'll probably give this a test tonight. @Pryanka, Danish, thank you for all the hints! -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 https://bugzilla.suse.com/show_bug.cgi?id=1218694#c5 --- Comment #5 from Priyanka Saggu <priyanka.saggu@suse.com> --- Thanks for pointing to the "cni-plugins" package, @Danish Prakash.
Indeed! This installs the plugins into /usr/lib(exec)/cni. I'm not sure if they are found there, though, as kubelet seems to expect them in /opt/cni/bin/. Otherwise, the 'cni-plugins' package should probably be added to the dependencies somewhere.
@Egbert Eich, changing "bin_dir" value in "/etc/containerd/config.toml", from "/opt/cni/bin" -> "/use/libexec/cni" will change the search path for cni plugins. ``` [plugins."io.containerd.grpc.v1.cri".cni] bin_dir = "/usr/libexec/cni" ``` But then the default upstream "kube-flannel.yml" will also need updates in the init-containers[1] that install "flannel" binary to /opt/cni/bin [1] https://github.com/flannel-io/flannel/blob/master/Documentation/kube-flannel... Will patching the "kube-flannel.yml" file for ^ in the flannel package help? Or something else will be more ideal? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 https://bugzilla.suse.com/show_bug.cgi?id=1218694#c6 --- Comment #6 from Priyanka Saggu <priyanka.saggu@suse.com> --- Factory flannel package is up-to-date with upstream now - https://build.opensuse.org/request/show/1143027 -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 Alexandre Vicenzi <alexandre.vicenzi@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(priyanka.saggu@su | |se.com) CC| |alexandre.vicenzi@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1218694 Priyanka Saggu <priyanka.saggu@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(priyanka.saggu@su | |se.com) | Status|NEW |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com