Comment # 9 on bug 1171770 from Richard Brown

Hi all - I've been looking at this all day, here is the current status:

I can confirm it happens with both kubicctl and kubeadm clusters made from the
current snapshots.

We know this doesn't occur on kubicctl clusters with multi-masters, which
suggests haproxy somehow works around the issue.

Recent snapshots have had the following recent changes which I suspect could be
related (listed in invasiveness according to my opinion):

- busybox package reworking
- kernel update from 5.6.11 to 5.6.12
- minor runc patch
- kured

There is also the possibility that the cause is something else, I'm at a loss
to be honest and trying to debug this just by going on what few clues we have
here - any more data points and examples from people would be greatly
appreciated.

Given the bug report shows the problem occurs with earlier kured versions and
with services other than kured, I think it's safe to rule out that update.

Given the busybox package fundamentally changed every image we use for
kubernetes, that was my first suspicion, so today I've built all of the images
based on the busybox-free Tumbleweed base image (which is much larger, but
obviously more likely to have everything each k8s component requires)

You can get these images from
registry.opensuse.org/home/rbrownsuse/branches/devel/kubic/containers/container/kubic

However, as anyone who wishes to help can see, a cluster created with "kubeadm
init --image-repository
registry.opensuse.org/home/rbrownsuse/branches/devel/kubic/containers/container/kubic"
still demonstrates this bug with a vengeance.

I've even tried using the heavyweight base containers with the weave image, to
no difference.

So I'm pretty convinced our images/busybox are not at fault.

This now leads me to wonder if the kernel or runc updates are to blame, which I
will look at tomorrow, unless someone beats me to it first.

Sorry that this doesn't look like it will be a quick fix. Anyone got any other
info that might help?