[Bug 1197490] New: Kubic 20220322 fails to set up CNI with weave
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 Bug ID: 1197490 Summary: Kubic 20220322 fails to set up CNI with weave Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: x86-64 OS: openSUSE Tumbleweed Status: NEW Severity: Critical Priority: P5 - None Component: Kubic Assignee: kubic-bugs@opensuse.org Reporter: rombert@apache.org QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- I have set up a set of libvirt VMs via https://github.com/kubic-project/kubic-terraform-kvm. I have initialised the control plane via kubeadm following the guide from https://en.opensuse.org/Kubic:kubeadm: kubeadm init --kubernetes-version=v1.23.4 # due to bug #1197489 mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config kubectl apply -f /usr/share/k8s-yaml/weave/weave.yaml At this point the pod network should be available, but instead I get $ kubectl get pod -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-fc8b57f45-kfsv6 0/1 CrashLoopBackOff 6 (88s ago) 5m37s <none> kubic-0 <none> <none> kube-system coredns-fc8b57f45-nv4x9 0/1 CrashLoopBackOff 6 (81s ago) 5m37s <none> kubic-0 <none> <none> kube-system etcd-kubic-0 1/1 Running 0 5m34s 10.16.0.170 kubic-0 <none> <none> kube-system kube-apiserver-kubic-0 1/1 Running 0 5m34s 10.16.0.170 kubic-0 <none> <none> kube-system kube-controller-manager-kubic-0 1/1 Running 0 5m34s 10.16.0.170 kubic-0 <none> <none> kube-system kube-proxy-hx8mx 1/1 Running 0 5m38s 10.16.0.170 kubic-0 <none> <none> kube-system kube-scheduler-kubic-0 1/1 Running 0 5m34s 10.16.0.170 kubic-0 <none> <none> kube-system weave-net-2t4lz 2/2 Running 1 (4m51s ago) 4m57s 10.16.0.170 kubic-0 <none> <none> This is presumably due to an error loading the modules $ kubectl -n kube-system logs weave-net-2t4lz -c weave-init modprobe: can't load module nfnetlink (kernel/net/netfilter/nfnetlink.ko.zst): invalid module format Ignore the error if "xt_set" is built-in in the kernel I have loaded the modules required by weave and bumped the pods $ modprobe nfnetlink $ modprobe ip_set $ kubectl -n kube-system rollout restart daemonset weave-net daemonset.apps/weave-net restarted This time there are no more logs in weave-init $ kubectl -n kube-system logs weave-net-jks7b -c weave-init The weave status seems ok $ kubectl -n kube-system exec weave-net-jks7b -c weave -- /home/weave/weave --local status Version: 2.8.1 (failed to check latest version - see logs; next check at 2022/03/24 23:46:26) Service: router Protocol: weave 1..2 Name: be:f7:8c:d7:49:12(kubic-0) Encryption: disabled PeerDiscovery: enabled Targets: 0 Connections: 0 Peers: 1 TrustedSubnets: none Service: ipam Status: ready Range: 10.32.0.0/12 DefaultSubnet: 10.32.0.0/12 I restarted the CoreDNS pods to make sure that they get a chance to be restarted and allocated IPs. $ kubectl -n kube-system rollout restart deployment coredns deployment.apps/coredns restarted After the restart they enter a CrashLoopBackOff rather quickly kube-system coredns-7d9d9d5f9f-7cz4w 0/1 CrashLoopBackOff 2 (2s ago) 21s kube-system coredns-7d9d9d5f9f-2cgwd 0/1 CrashLoopBackOff 2 (1s ago) 22s -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c1 --- Comment #1 from Robert Munteanu <rombert@apache.org> --- This issue is visible in openQA but not checked against. The last time this was confirmed working properly was 20220317 - https://openqa.opensuse.org/tests/2249915#step/kubeadm/17 . 20220318 already shows the containers as restarting multiple times - https://openqa.opensuse.org/tests/2253770#step/kubeadm/17 . Looking at the packages changed for that snapshot, both kubernetes and cri-o seem potential candidates for breaking things: https://lists.opensuse.org/archives/list/kubic@lists.opensuse.org/thread/JJO... Packages changed: busybox-links cri-o (1.22.0 -> 1.23.2) cri-tools (1.22.0 -> 1.23.0) grep kubernetes (1.23.0 -> 1.23.4) kubernetes1.22 (1.22.4 -> 1.22.7) kubernetes1.23 (1.23.0 -> 1.23.4) p11-kit (0.23.22 -> 0.24.1) -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c5 Dirk Datzert <dirk.datzert@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dirk.datzert@gmx.de --- Comment #5 from Dirk Datzert <dirk.datzert@gmx.de> --- I ran against the same issue last week at my company. I tried the alternate flannel cni plugin today. That works. -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c10 --- Comment #10 from Dirk Datzert <dirk.datzert@gmx.de> --- I did a kubeadm reset on my test-machine. Cleared the image cache from cri-o Cleared the artefakt and repository-cache from my harbor-instance since I'm behind a company proxy and firewall Cleared the whole proxy cache which the harbor-instance is attached to. And got the following output: kube01:~/kubic # get-pods NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-fc8b57f45-dx7px 0/1 CrashLoopBackOff 2 (2s ago) 40s <none> kube01 <none> <none> kube-system coredns-fc8b57f45-ww488 0/1 CrashLoopBackOff 2 (7s ago) 40s <none> kube01 <none> <none> kube-system etcd-kube01 1/1 Running 38 39s 10.9.80.80 kube01 <none> <none> kube-system kube-apiserver-kube01 1/1 Running 1 39s 10.9.80.80 kube01 <none> <none> kube-system kube-controller-manager-kube01 1/1 Running 1 45s 10.9.80.80 kube01 <none> <none> kube-system kube-proxy-spwqw 1/1 Running 0 40s 10.9.80.80 kube01 <none> <none> kube-system kube-scheduler-kube01 1/1 Running 1 39s 10.9.80.80 kube01 <none> <none> kube-system weave-net-pc5qg 2/2 Running 0 40s 10.9.80.80 kube01 <none> <none> kube01:~/kubic # describe -nkube-system pod weave-net-pc5qg | grep Image Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:4613c44921c60bbabda2831a55bca9c4327b288c2027443f55df1f7e31d59365 Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:4613c44921c60bbabda2831a55bca9c4327b288c2027443f55df1f7e31d59365 Image: registry.opensuse.org/kubic/weave-npc:2.8.1 Image ID: registry.opensuse.org/kubic/weave-npc@sha256:68274b1a376e094b0fd4c639b9d54f075df2c445d209ff1e7fe1e111b34c118a kube01:~/kubic # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c11 --- Comment #11 from Robert Munteanu <rombert@apache.org> --- (In reply to Dirk Datzert from comment #10) You are definitely pulling a different image. Your SHA is 4613c44921c60bbabda2831a55bca9c4327b288c2027443f55df1f7e31d59365, mine is 658af2bac5bced83b09437b21144afc53ca5acb7abd6356e066d46f3079181e4 . Not sure which one is the right one though :-) -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c12 --- Comment #12 from Dirk Datzert <dirk.datzert@gmx.de> --- I redone: kubeadm reset. Decoupled my harbor-cache for registry.opensuse.org and downloaded direct via out main proxy I get an other image and weave-net self crashed kube01:~/kubic # get-pods NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-fc8b57f45-8l2dw 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system coredns-fc8b57f45-brcg7 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system etcd-kube01 1/1 Running 40 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-apiserver-kube01 1/1 Running 3 6m17s 10.9.80.80 kube01 <none> <none> kube-system kube-controller-manager-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-proxy-vmk9p 1/1 Running 0 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-scheduler-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system weave-net-sfwhb 1/2 CrashLoopBackOff 5 (2m50s ago) 6m10s 10.9.80.80 kube01 <none> <none> kube01:~/kubic # describe -nkube-system pod weave-net-sfwhb | grep Image Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/kubic/weave-npc:2.8.1 Image ID: registry.opensuse.org/kubic/weave-npc@sha256:197897f4fab0b3b1004f6d5854995b014cd09350f22c5bd809f79a7f925e5c41 kube01:~/kubic # --- Comment #13 from Dirk Datzert <dirk.datzert@gmx.de> --- I redone: kubeadm reset. Decoupled my harbor-cache for registry.opensuse.org and downloaded direct via out main proxy I get an other image and weave-net self crashed kube01:~/kubic # get-pods NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-fc8b57f45-8l2dw 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system coredns-fc8b57f45-brcg7 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system etcd-kube01 1/1 Running 40 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-apiserver-kube01 1/1 Running 3 6m17s 10.9.80.80 kube01 <none> <none> kube-system kube-controller-manager-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-proxy-vmk9p 1/1 Running 0 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-scheduler-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system weave-net-sfwhb 1/2 CrashLoopBackOff 5 (2m50s ago) 6m10s 10.9.80.80 kube01 <none> <none> kube01:~/kubic # describe -nkube-system pod weave-net-sfwhb | grep Image Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/kubic/weave-npc:2.8.1 Image ID: registry.opensuse.org/kubic/weave-npc@sha256:197897f4fab0b3b1004f6d5854995b014cd09350f22c5bd809f79a7f925e5c41 kube01:~/kubic # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c12 --- Comment #12 from Dirk Datzert <dirk.datzert@gmx.de> --- I redone: kubeadm reset. Decoupled my harbor-cache for registry.opensuse.org and downloaded direct via out main proxy I get an other image and weave-net self crashed kube01:~/kubic # get-pods NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-fc8b57f45-8l2dw 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system coredns-fc8b57f45-brcg7 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system etcd-kube01 1/1 Running 40 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-apiserver-kube01 1/1 Running 3 6m17s 10.9.80.80 kube01 <none> <none> kube-system kube-controller-manager-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-proxy-vmk9p 1/1 Running 0 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-scheduler-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system weave-net-sfwhb 1/2 CrashLoopBackOff 5 (2m50s ago) 6m10s 10.9.80.80 kube01 <none> <none> kube01:~/kubic # describe -nkube-system pod weave-net-sfwhb | grep Image Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/kubic/weave-npc:2.8.1 Image ID: registry.opensuse.org/kubic/weave-npc@sha256:197897f4fab0b3b1004f6d5854995b014cd09350f22c5bd809f79a7f925e5c41 kube01:~/kubic # --- Comment #13 from Dirk Datzert <dirk.datzert@gmx.de> --- I redone: kubeadm reset. Decoupled my harbor-cache for registry.opensuse.org and downloaded direct via out main proxy I get an other image and weave-net self crashed kube01:~/kubic # get-pods NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-fc8b57f45-8l2dw 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system coredns-fc8b57f45-brcg7 0/1 ContainerCreating 0 6m9s <none> kube01 <none> <none> kube-system etcd-kube01 1/1 Running 40 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-apiserver-kube01 1/1 Running 3 6m17s 10.9.80.80 kube01 <none> <none> kube-system kube-controller-manager-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-proxy-vmk9p 1/1 Running 0 6m10s 10.9.80.80 kube01 <none> <none> kube-system kube-scheduler-kube01 1/1 Running 3 6m10s 10.9.80.80 kube01 <none> <none> kube-system weave-net-sfwhb 1/2 CrashLoopBackOff 5 (2m50s ago) 6m10s 10.9.80.80 kube01 <none> <none> kube01:~/kubic # describe -nkube-system pod weave-net-sfwhb | grep Image Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/kubic/weave-npc:2.8.1 Image ID: registry.opensuse.org/kubic/weave-npc@sha256:197897f4fab0b3b1004f6d5854995b014cd09350f22c5bd809f79a7f925e5c41 kube01:~/kubic # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c14 --- Comment #14 from Dirk Datzert <dirk.datzert@gmx.de> --- (In reply to Dirk Datzert from comment #13)
kube01:~/kubic # describe -nkube-system pod weave-net-sfwhb | grep Image Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave- kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image ID: registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave- kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image: registry.opensuse.org/kubic/weave-npc:2.8.1 Image ID: registry.opensuse.org/kubic/weave-npc@sha256: 197897f4fab0b3b1004f6d5854995b014cd09350f22c5bd809f79a7f925e5c41 kube01:~/kubic #
kube01:~/kubic # crictl pull registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1 Image is up to date for registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c kube01:~/kubic # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c15 --- Comment #15 from Dirk Datzert <dirk.datzert@gmx.de> --- I also tried to download you image: kube01:~/kubic # crictl pull registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:658af2bac5bced83b09437b21144afc53ca5acb7abd6356e066d46f3079181e4 FATA[0000] pulling image: rpc error: code = Unknown desc = reading manifest sha256:658af2bac5bced83b09437b21144afc53ca5acb7abd6356e066d46f3079181e4 in registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube: manifest unknown kube01:~/kubic # crictl pull registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c Image is up to date for registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube@sha256:1ae8f66fcefdfbc876c934ebd6c1ac14b9898564dc12fb345d3be1a331d2662c kube01:~/kubic # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c17 --- Comment #17 from Dirk Datzert <dirk.datzert@gmx.de> --- (In reply to Fabian Vogt from comment #16)
(In reply to Robert Munteanu from comment #11)
(In reply to Dirk Datzert from comment #10)
The current one should be fine again, so registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8. 1-21.11 and up.
Image 2.8.1-21.11 is loaded. But CoreDNS still crashing, getting no IP.
If that's still broken, what's the output of
kubectl -n kube-system logs weave-net-jks7b -c weave
kube01:~/kubic # kubectl -n kube-system logs weave-net-x4z46 -c weave DEBU: 2022/03/30 15:21:19.262044 [kube-peers] Checking peer "a2:da:56:4f:6a:af" against list &{[]} Peer not in list; removing persisted data INFO: 2022/03/30 15:21:19.374282 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true http-addr:127.0.0.1:6784 ipalloc-init:consensus=0 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:a2:da:56:4f:6a:af nickname:kube01 no-dns:true no-masq-local:true port:6783] INFO: 2022/03/30 15:21:19.374407 weave 2.8.1 INFO: 2022/03/30 15:21:20.121390 Bridge type is bridged_fastdp INFO: 2022/03/30 15:21:20.121408 Communication between peers is unencrypted. INFO: 2022/03/30 15:21:20.128480 Our name is a2:da:56:4f:6a:af(kube01) INFO: 2022/03/30 15:21:20.128510 Launch detected - using supplied peer list: [] INFO: 2022/03/30 15:21:20.128532 Using "no-masq-local" LocalRangeTracker INFO: 2022/03/30 15:21:20.128537 Checking for pre-existing addresses on weave bridge INFO: 2022/03/30 15:21:20.130593 [allocator a2:da:56:4f:6a:af] No valid persisted data INFO: 2022/03/30 15:21:20.140454 [allocator a2:da:56:4f:6a:af] Initialising via deferred consensus INFO: 2022/03/30 15:21:20.140515 Sniffing traffic on datapath (via ODP) INFO: 2022/03/30 15:21:20.144627 Listening for HTTP control messages on 127.0.0.1:6784 INFO: 2022/03/30 15:21:20.144652 Listening for metrics requests on 0.0.0.0:6782 INFO: 2022/03/30 15:21:20.183392 Error checking version: Get "https://checkpoint-api.weave.works/v1/check/weave-net?arch=amd64&flag_docker-version=none&flag_kernel-version=5.16.15-1-default&os=linux&signature=MU5KnTIEPIQuQWuWaIiXuRkUcNKV7KY6wvwmuRxgfJU%3D&version=2.8.1": dial tcp: lookup checkpoint-api.weave.works on 10.96.0.10:53: write udp 10.9.80.80:55167->10.96.0.10:53: write: operation not permitted INFO: 2022/03/30 15:21:20.409937 [kube-peers] Added myself to peer list &{[{a2:da:56:4f:6a:af kube01}]} DEBU: 2022/03/30 15:21:20.415769 [kube-peers] Nodes that have disappeared: map[] INFO: 2022/03/30 15:21:20.433849 Assuming quorum size of 1 INFO: 2022/03/30 15:21:20.433928 adding entry 10.32.0.0/12 to weaver-no-masq-local of 0 INFO: 2022/03/30 15:21:20.433948 added entry 10.32.0.0/12 to weaver-no-masq-local of 0 10.32.0.1 DEBU: 2022/03/30 15:21:20.547652 registering for updates for node delete events kube01:~/kubic # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c19 --- Comment #19 from Dirk Datzert <dirk.datzert@gmx.de> --- kube01:~/kubic # diff -uN weave.yaml.orig weave.yaml --- weave.yaml.orig 2022-03-26 19:31:49.997369526 +0000 +++ weave.yaml 2022-03-31 05:01:16.005476081 +0000 @@ -120,7 +120,7 @@ spec: initContainers: - name: weave-init - image: 'registry.opensuse.org/kubic/weave-kube:2.8.1' + image: 'registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1-21.11' command: - /home/weave/init.sh env: @@ -143,6 +143,8 @@ command: - /home/weave/launch.sh env: + - name: CHECKPOINT_DISABLE + value: "1" - name: INIT_CONTAINER value: "true" - name: HOSTNAME @@ -150,7 +152,7 @@ fieldRef: apiVersion: v1 fieldPath: spec.nodeName - image: 'registry.opensuse.org/kubic/weave-kube:2.8.1' + image: 'registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1-21.11' readinessProbe: httpGet: host: 127.0.0.1 The check of version has gone. But the CoreDNS still in CrashLoops: kube01:~/kubic # get-pods NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-fc8b57f45-l2c4c 0/1 CrashLoopBackOff 6 (8s ago) 4m52s <none> kube01 <none> <none> kube-system coredns-fc8b57f45-wkvgn 0/1 CrashLoopBackOff 5 (75s ago) 4m52s <none> kube01 <none> <none> kube-system etcd-kube01 1/1 Running 43 4m51s 10.9.80.80 kube01 <none> <none> kube-system kube-apiserver-kube01 1/1 Running 6 4m58s 10.9.80.80 kube01 <none> <none> kube-system kube-controller-manager-kube01 1/1 Running 6 4m57s 10.9.80.80 kube01 <none> <none> kube-system kube-proxy-84b99 1/1 Running 0 4m53s 10.9.80.80 kube01 <none> <none> kube-system kube-scheduler-kube01 1/1 Running 6 4m51s 10.9.80.80 kube01 <none> <none> kube-system weave-net-swrjl 2/2 Running 0 3m34s 10.9.80.80 kube01 <none> <none> kube01:~/kubic # kube01:~/kubic # kubectl -n kube-system logs weave-net-swrjl -c weave DEBU: 2022/03/31 05:01:22.114273 [kube-peers] Checking peer "a2:da:56:4f:6a:af" against list &{[]} Peer not in list; removing persisted data INFO: 2022/03/31 05:01:22.261773 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true http-addr:127.0.0.1:6784 ipalloc-init:consensus=0 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:a2:da:56:4f:6a:af nickname:kube01 no-dns:true no-masq-local:true port:6783] INFO: 2022/03/31 05:01:22.261815 weave 2.8.1 INFO: 2022/03/31 05:01:23.001814 Bridge type is bridged_fastdp INFO: 2022/03/31 05:01:23.001828 Communication between peers is unencrypted. INFO: 2022/03/31 05:01:23.017158 Our name is a2:da:56:4f:6a:af(kube01) INFO: 2022/03/31 05:01:23.017189 Launch detected - using supplied peer list: [] INFO: 2022/03/31 05:01:23.017209 Using "no-masq-local" LocalRangeTracker INFO: 2022/03/31 05:01:23.017213 Checking for pre-existing addresses on weave bridge INFO: 2022/03/31 05:01:23.018016 [allocator a2:da:56:4f:6a:af] No valid persisted data INFO: 2022/03/31 05:01:23.056922 [allocator a2:da:56:4f:6a:af] Initialising via deferred consensus INFO: 2022/03/31 05:01:23.056969 Sniffing traffic on datapath (via ODP) INFO: 2022/03/31 05:01:23.068441 Listening for HTTP control messages on 127.0.0.1:6784 INFO: 2022/03/31 05:01:23.068463 Listening for metrics requests on 0.0.0.0:6782 INFO: 2022/03/31 05:01:23.403839 [kube-peers] Added myself to peer list &{[{a2:da:56:4f:6a:af kube01}]} DEBU: 2022/03/31 05:01:23.410292 [kube-peers] Nodes that have disappeared: map[] INFO: 2022/03/31 05:01:23.425704 Assuming quorum size of 1 INFO: 2022/03/31 05:01:23.425773 adding entry 10.32.0.0/12 to weaver-no-masq-local of 0 INFO: 2022/03/31 05:01:23.425785 added entry 10.32.0.0/12 to weaver-no-masq-local of 0 10.32.0.1 DEBU: 2022/03/31 05:01:23.513649 registering for updates for node delete events kube01:~/kubic # Here the Events from kubectl -n kube-system describe pods weave-net Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 6m42s default-scheduler Successfully assigned kube-system/weave-net-swrjl to kube01 Normal Pulled 6m42s kubelet Container image "registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1-21.11" already present on machine Normal Created 6m41s kubelet Created container weave-init Normal Started 6m41s kubelet Started container weave-init Normal Pulled 6m39s kubelet Container image "registry.opensuse.org/home/favogt/boo1197490/container/kubic/weave-kube:2.8.1-21.11" already present on machine Normal Created 6m39s kubelet Created container weave Normal Started 6m39s kubelet Started container weave Normal Pulled 6m39s kubelet Container image "registry.opensuse.org/kubic/weave-npc:2.8.1" already present on machine Normal Created 6m39s kubelet Created container weave-npc Normal Started 6m39s kubelet Started container weave-npc Warning Unhealthy 6m38s kubelet Readiness probe failed: Get "http://127.0.0.1:6784/status": dial tcp 127.0.0.1:6784: connect: connection refused -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c20 --- Comment #20 from Dirk Datzert <dirk.datzert@gmx.de> --- Here the Events from kubectl -n kube-system describe pods coredns-****-*** Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 13m default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate. Normal Scheduled 13m default-scheduler Successfully assigned kube-system/coredns-fc8b57f45-l2c4c to kube01 Warning FailedMount 13m kubelet MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(66348984c61f9b05077874a606172899b24f9972b732be4f22bfe96de296b606): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/66348984c61f9b05077874a606172899b24f9972b732be4f22b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b5eafc2416f7af): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869a4d97218be78d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5a4b38a0c86d80): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c545aa2a74cab9): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f37e7bb8b47b6d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 11m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8cc406156bad36): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Killing 11m (x2 over 11m) kubelet Stopping container coredns Warning Unhealthy 11m (x3 over 11m) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal Created 11m (x3 over 11m) kubelet Created container coredns Normal Pulled 11m (x3 over 11m) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Started 11m (x3 over 11m) kubelet Started container coredns Normal SandboxChanged 3m16s (x431 over 11m) kubelet Pod sandbox changed, it will be killed and re-created. Warning FailedCreatePodSandBox 2m2s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad298f9e079e0a7b5): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad29...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Pulled 96s (x3 over 109s) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Created 96s (x3 over 109s) kubelet Created container coredns Normal Started 96s (x3 over 109s) kubelet Started container coredns Normal Killing 96s (x3 over 108s) kubelet Stopping container coredns Warning Unhealthy 94s (x5 over 108s) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal SandboxChanged 89s (x5 over 108s) kubelet Pod sandbox changed, it will be killed and re-created. Warning BackOff 89s (x2 over 90s) kubelet Back-off restarting failed container kube01:~ # --- Comment #21 from Dirk Datzert <dirk.datzert@gmx.de> --- Here the Events from kubectl -n kube-system describe pods coredns-****-*** Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 13m default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate. Normal Scheduled 13m default-scheduler Successfully assigned kube-system/coredns-fc8b57f45-l2c4c to kube01 Warning FailedMount 13m kubelet MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(66348984c61f9b05077874a606172899b24f9972b732be4f22bfe96de296b606): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/66348984c61f9b05077874a606172899b24f9972b732be4f22b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b5eafc2416f7af): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869a4d97218be78d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5a4b38a0c86d80): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c545aa2a74cab9): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f37e7bb8b47b6d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 11m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8cc406156bad36): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Killing 11m (x2 over 11m) kubelet Stopping container coredns Warning Unhealthy 11m (x3 over 11m) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal Created 11m (x3 over 11m) kubelet Created container coredns Normal Pulled 11m (x3 over 11m) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Started 11m (x3 over 11m) kubelet Started container coredns Normal SandboxChanged 3m16s (x431 over 11m) kubelet Pod sandbox changed, it will be killed and re-created. Warning FailedCreatePodSandBox 2m2s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad298f9e079e0a7b5): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad29...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Pulled 96s (x3 over 109s) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Created 96s (x3 over 109s) kubelet Created container coredns Normal Started 96s (x3 over 109s) kubelet Started container coredns Normal Killing 96s (x3 over 108s) kubelet Stopping container coredns Warning Unhealthy 94s (x5 over 108s) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal SandboxChanged 89s (x5 over 108s) kubelet Pod sandbox changed, it will be killed and re-created. Warning BackOff 89s (x2 over 90s) kubelet Back-off restarting failed container kube01:~ # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c20 --- Comment #20 from Dirk Datzert <dirk.datzert@gmx.de> --- Here the Events from kubectl -n kube-system describe pods coredns-****-*** Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 13m default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate. Normal Scheduled 13m default-scheduler Successfully assigned kube-system/coredns-fc8b57f45-l2c4c to kube01 Warning FailedMount 13m kubelet MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(66348984c61f9b05077874a606172899b24f9972b732be4f22bfe96de296b606): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/66348984c61f9b05077874a606172899b24f9972b732be4f22b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b5eafc2416f7af): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869a4d97218be78d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5a4b38a0c86d80): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c545aa2a74cab9): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f37e7bb8b47b6d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 11m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8cc406156bad36): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Killing 11m (x2 over 11m) kubelet Stopping container coredns Warning Unhealthy 11m (x3 over 11m) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal Created 11m (x3 over 11m) kubelet Created container coredns Normal Pulled 11m (x3 over 11m) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Started 11m (x3 over 11m) kubelet Started container coredns Normal SandboxChanged 3m16s (x431 over 11m) kubelet Pod sandbox changed, it will be killed and re-created. Warning FailedCreatePodSandBox 2m2s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad298f9e079e0a7b5): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad29...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Pulled 96s (x3 over 109s) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Created 96s (x3 over 109s) kubelet Created container coredns Normal Started 96s (x3 over 109s) kubelet Started container coredns Normal Killing 96s (x3 over 108s) kubelet Stopping container coredns Warning Unhealthy 94s (x5 over 108s) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal SandboxChanged 89s (x5 over 108s) kubelet Pod sandbox changed, it will be killed and re-created. Warning BackOff 89s (x2 over 90s) kubelet Back-off restarting failed container kube01:~ # --- Comment #21 from Dirk Datzert <dirk.datzert@gmx.de> --- Here the Events from kubectl -n kube-system describe pods coredns-****-*** Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 13m default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate. Normal Scheduled 13m default-scheduler Successfully assigned kube-system/coredns-fc8b57f45-l2c4c to kube01 Warning FailedMount 13m kubelet MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(66348984c61f9b05077874a606172899b24f9972b732be4f22bfe96de296b606): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/66348984c61f9b05077874a606172899b24f9972b732be4f22b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 13m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b5eafc2416f7af): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/6fab0014a39e8a1a563e9cda8415e3ca4e5152ea56f4a69795b...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869a4d97218be78d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/4f4531fd3e433a6ece14b12c656a662ab69e28548b70acd0869...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5a4b38a0c86d80): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/37547cc86e8ea0c1e7d7f675f2e58acf3dd23350c7bee85f2f5...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c545aa2a74cab9): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/9c79d44187f0ec786052672364cdd397cf797d1944a4ac0735c...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f37e7bb8b47b6d): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/dc61de70de32b3daced45e5482be29903fceee14ee2d104dd6f...": dial tcp 127.0.0.1:6784: connect: connection refused Warning FailedCreatePodSandBox 11m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8cc406156bad36): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/457d089ea58926ef5a4c81e62625b59b8c7f97655f231d9a9e8...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Killing 11m (x2 over 11m) kubelet Stopping container coredns Warning Unhealthy 11m (x3 over 11m) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal Created 11m (x3 over 11m) kubelet Created container coredns Normal Pulled 11m (x3 over 11m) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Started 11m (x3 over 11m) kubelet Started container coredns Normal SandboxChanged 3m16s (x431 over 11m) kubelet Pod sandbox changed, it will be killed and re-created. Warning FailedCreatePodSandBox 2m2s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-fc8b57f45-l2c4c_kube-system_e3b66af2-7266-4f1c-a42f-a8f232986a53_0(d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad298f9e079e0a7b5): error adding pod kube-system_coredns-fc8b57f45-l2c4c to CNI network "weave": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/d90ce69a672d0f341902f1e4347446fe28f9fe3e3fa9da1ad29...": dial tcp 127.0.0.1:6784: connect: connection refused Normal Pulled 96s (x3 over 109s) kubelet Container image "registry.opensuse.org/kubic/coredns:v1.8.6" already present on machine Normal Created 96s (x3 over 109s) kubelet Created container coredns Normal Started 96s (x3 over 109s) kubelet Started container coredns Normal Killing 96s (x3 over 108s) kubelet Stopping container coredns Warning Unhealthy 94s (x5 over 108s) kubelet Readiness probe failed: Get "http://:8181/ready": dial tcp :8181: connect: connection refused Normal SandboxChanged 89s (x5 over 108s) kubelet Pod sandbox changed, it will be killed and re-created. Warning BackOff 89s (x2 over 90s) kubelet Back-off restarting failed container kube01:~ # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c22 --- Comment #22 from Dirk Datzert <dirk.datzert@gmx.de> --- On the Kube-Host the Firewall DROP by this rule: kube01:~ # iptables -n --list | grep :6784 DROP tcp -- 0.0.0.0/0 127.0.0.1 tcp dpt:6784 ADDRTYPE match src-type !LOCAL ! ctstate RELATED,ESTABLISHED /* Block non-local access to Weave Net control port */ kube01:~ # As far as I understood this rule blocks access from external and should be fine. -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c23 --- Comment #23 from Dirk Datzert <dirk.datzert@gmx.de> --- Firewall-rules for kube-system kube01:~ # get-services -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 38m k8s-app=kube-dns kube01:~ # iptables -n --list | grep kube-system REJECT udp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns has no endpoints */ udp dpt:53 reject-with icmp-port-unreachable REJECT tcp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp has no endpoints */ tcp dpt:53 reject-with icmp-port-unreachable REJECT tcp -- 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:metrics has no endpoints */ tcp dpt:9153 reject-with icmp-port-unreachable ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 match-set weave-P.B|!ZhkAr5q=XZ?3}tMBA+0 dst /* DefaultAllow ingress isolation for namespace: kube-system */ WEAVE-NPC-EGRESS-ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src /* DefaultAllow egress isolation for namespace: kube-system */ RETURN all -- 0.0.0.0/0 0.0.0.0/0 match-set weave-E1ney4o[ojNrLk.6rOHi;7MPE src /* DefaultAllow egress isolation for namespace: kube-system */ kube01:~ # -- You are receiving this mail because: You are the assignee for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1197490 http://bugzilla.opensuse.org/show_bug.cgi?id=1197490#c24 --- Comment #24 from Dirk Datzert <dirk.datzert@gmx.de> --- I watched local interface on port 6784 and there is real http traffic with POST /ip/** and DELETE /ip/*** with HTTP-response code 200 / 204 I would say that weave is responding. -- You are receiving this mail because: You are the assignee for the bug.
participants (1)
-
bugzilla_noreply＠suse.com