Hi, I have taken kubic for a spin with a local QEMU/KVM setup - 1 master node and 3 worker nodes. The cluster is up and running (ingress, load balancer, storage, monitoring ). It is basically stale - except the management pods nothing is deployed. I'm using the qcow2 files from [1], and launching the machines with qemu-system-x86_64 \ -machine type=pc,accel=kvm \ -cpu host \ -smp $cpus \ -drive file=$disk,if=virtio,l2-cache-size=3145728 -cdrom $cdrom \ -m 1024 \ -netdev bridge,id=hostnet0 \ -device e1000,netdev=hostnet0,mac=$mac \ -nographic $cpus is 2 for the master node and 1 for the others. I have one recurring problem with the master becoming unresponsive after some time. Looking at the grafana charts I can see that there is steadily increasing Disk I/O for the master nodee. At about ~30 minutes after launching all 4 nodes I see the load average is 6 on the master node, with disk read I/O increasing steadily. The 5 minute average as collected by prometheus is at about 350 MB/s. The situation seems to be "fixed" when the kube-scheduler-master and kube-controller-manager-master crash and are not scheduled. Looking at those pods' logs does not show match, they seem to exit because of a failed leader election, e.g. for kube-controller-manager-master I0326 15:59:39.521403 1 leaderelection.go:249] failed to renew lease kube-system/kube-controller-manager: failed to tryAcquireOrRenew context deadline exceeded for kube-scheduler-master I0326 15:59:39.589619 1 leaderelection.go:249] failed to renew lease kube-system/kube-scheduler: failed to tryAcquireOrRenew context deadline exceeded And looking at the I/O grafs it seems that there is a cycle - cluster is ok - IO increases - kube-*-master pods fail - IO decreases - kube-*-master pods recover Right now I'm at a loss about where to look next. Does anyone have any ideas about what might cause this or how to better debug this kind of situation? Thanks, Robert [1]: https://download.opensuse.org/repositories/devel:/kubic:/images/openSUSE_Tum... -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org