On Wed, 2022-03-23 at 11:10 +0100, rbrown wrote:
On 2022-03-23 10:51, Robert Munteanu wrote:
On Wed, 2022-03-23 at 09:03 +0000, Attila Pinter wrote:
------- Original Message -------
On Wednesday, March 23rd, 2022 at 2:55 PM, Attila Pinter
wrote: Finished reinstalling my test cluster - from the latest Kubic iso - for Kubic latest (1.23.4) and see the same CrashLoopBackOff as before: https://paste.opensuse.org/92716101.
--
Br,
A.
Some additional logs from the weave, weave-init, and coredns pods: https://paste.opensuse.org/87628794. Seems to me that issue is here, can be wrong tho.
I have the same problem with kubic cluster:
$ kubectl -n kube-system logs weave-net-8mmmc -c weave-init modprobe: can't load module nfnetlink (kernel/net/netfilter/nfnetlink.ko.zst): invalid module format Ignore the error if "xt_set" is built-in in the kernel
If anyone needs more info I'd be glad to provide it.
Thanks, Robert
Hi Robert,
Very interesting! This might be the clue I've been missing - if the kernel updated in a way that broke networking, then the problem isn't anything to do with the recent kubernetes package updates but with the kernel..
That would explain why I couldn't find fault in what I'd done recently ;)
The last kernel update was in snapshot 0319..can people roll their Kubic hosts back to snapshots older than that and tell me if the problems go away?
Hi Richard, For the record, that caught my attention since it was also in Attila's paste output. I rolled back all my kubic VMs to a snapshot taken around "2022-03-18 01:43:57" and things are getting back to normal. FWIW, this is the node 'wide' output for a node after I rolled back kubic-worker-1 Ready <none> 484d v1.23.0 10.25.0.43 <none> openSUSE MicroOS 5.16.14-1-default cri- o://1.22.0 and this is the one for a new that was not rolled back (ignore the NotReady status, it was just rebooted). kubic-worker-2 NotReady <none> 484d v1.23.4 10.25.0.40 <none> openSUSE MicroOS 5.16.15-1-default cri- o://1.23.2 Now I have to wait a bit since I reached the DockerHub pull limits but things are stabilising. Thanks, Robert
Would be a huge help if we can narrow down the problem to that kernel update.
Regards, Richard