http://bugzilla.opensuse.org/show_bug.cgi?id=1198946 Bug ID: 1198946 Summary: kured and coredns pods unable to start due to possible weave-net issues[?] on upgraded nodes Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Critical Priority: P5 - None Component: Kubic Assignee: kubic-bugs@opensuse.org Reporter: linuxdairy@gmail.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Build Identifier: kubelet, kubic-cl01-admin03 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-786c7c77c5-hd5s5_kube-system_f4bf99e9-de85-4ccd-b47a-11b4e09defc3_0(9a9735cfe115e606f3a239770ca167e1cf9ec03dd4aee9e4f0389a35f86d2182): error adding pod kube-system_coredns-786c7c77c5-hd5s5 to CNI network "weave": plugin type="portmap" failed (add): failed to parse config: could not parse prevResult: could not parse prevResult: result type supports [0.3.0 0.3.1 0.4.0] but unmarshalled CNIVersion is "0.1.0" This looks related to... https://github.com/weaveworks/weave/issues/3936 Where it's possible the CNI version if no longer being returned, and an inappropriate version is being guessed. Reproducible: Always Steps to Reproduce: 1. transactional-update dup on admin node, reboot, kubeadm update apply v1.23.4 2. kubectl get pods -n kube-system # kured and coredns pods will no longer start on updated node 3. Actual Results: repeated on second admin node...now I have 2/3 unhappy nodes for these pods. removed kured temporarily, updated coredns deployment to 4 replicas to borrow into non-upgraded worker nodes kubectl describe on either pod shows these events and never starts... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> Successfully assigned kube-system/coredns-786c7c77c5-hd5s5 to kubic-cl01-admin03 Warning FailedCreatePodSandBox 70s kubelet, kubic-cl01-admin03 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-786c7c77c5-hd5s5_kube-system_f4bf99e9-de85-4ccd-b47a-11b4e09defc3_0(67b41291a8e187a02c5081ed3274e46fa76291b02ffa04b1a91cda14509a7080): error adding pod kube-system_coredns-786c7c77c5-hd5s5 to CNI network "weave": plugin type="portmap" failed (add): failed to parse config: could not parse prevResult: could not parse prevResult: result type supports [0.3.0 0.3.1 0.4.0] but unmarshalled CNIVersion is "0.1.0" Warning FailedCreatePodSandBox 57s kubelet, kubic-cl01-admin03 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-786c7c77c5-hd5s5_kube-system_f4bf99e9-de85-4ccd-b47a-11b4e09defc3_0(b7af2bdf70a0fd94907393632db5872dd6cec24b776bb720589163d6acc2368e): error adding pod kube-system_coredns-786c7c77c5-hd5s5 to CNI network "weave": plugin type="portmap" failed (add): failed to parse config: could not parse prevResult: could not parse prevResult: result type supports [0.3.0 0.3.1 0.4.0] but unmarshalled CNIVersion is "0.1.0" Warning FailedCreatePodSandBox 44s kubelet, kubic-cl01-admin03 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-786c7c77c5-hd5s5_kube-system_f4bf99e9-de85-4ccd-b47a-11b4e09defc3_0(7b3896770118a07abc72f5d2f968b4a92e23b331904da3ccbc14f5470a278453): error adding pod kube-system_coredns-786c7c77c5-hd5s5 to CNI network "weave": plugin type="portmap" failed (add): failed to parse config: could not parse prevResult: could not parse prevResult: result type supports [0.3.0 0.3.1 0.4.0] but unmarshalled CNIVersion is "0.1.0" Warning FailedCreatePodSandBox 33s kubelet, kubic-cl01-admin03 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-786c7c77c5-hd5s5_kube-system_f4bf99e9-de85-4ccd-b47a-11b4e09defc3_0(835af599d7573e79039af5d4dd7a7f3352fa1129f997de2072d4c2530ac16d27): error adding pod kube-system_coredns-786c7c77c5-hd5s5 to CNI network "weave": plugin type="portmap" failed (add): failed to parse config: could not parse prevResult: could not parse prevResult: result type supports [0.3.0 0.3.1 0.4.0] but unmarshalled CNIVersion is "0.1.0" Warning FailedCreatePodSandBox 18s kubelet, kubic-cl01-admin03 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-786c7c77c5-hd5s5_kube-system_f4bf99e9-de85-4ccd-b47a-11b4e09defc3_0(83f6b1a6b896001b5e255ee1b7c2d66953d11611ae3265f0cd5d6a1bf614d576): error adding pod kube-system_coredns-786c7c77c5-hd5s5 to CNI network "weave": plugin type="portmap" failed (add): failed to parse config: could not parse prevResult: could not parse prevResult: result type supports [0.3.0 0.3.1 0.4.0] but unmarshalled CNIVersion is "0.1.0" Warning FailedCreatePodSandBox 6s kubelet, kubic-cl01-admin03 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-786c7c77c5-hd5s5_kube-system_f4bf99e9-de85-4ccd-b47a-11b4e09defc3_0(9a9735cfe115e606f3a239770ca167e1cf9ec03dd4aee9e4f0389a35f86d2182): error adding pod kube-system_coredns-786c7c77c5-hd5s5 to CNI network "weave": plugin type="portmap" failed (add): failed to parse config: could not parse prevResult: could not parse prevResult: result type supports [0.3.0 0.3.1 0.4.0] but unmarshalled CNIVersion is "0.1.0" Expected Results: coredns and kured pods should start. coredns is a little important to a kubernetes cluster. kured is nice to have. I went with critical for severity. No coredns will be bad...and my upgrade of 13 nodes is stuck in the middle. I could possibly reboot and go back, except the 1.23.0 images for x86_64 of kube-apiserver, etc. are no longer accessible on registry.opensuse.org leading to image pull issues. cni-1.0.1-1.1.x86_64 cni-plugins-1.1.1-1.1.x86_64 -- You are receiving this mail because: You are the assignee for the bug.