Comment # 1
on bug 1012374
from Miquel Sabate Sola
Hi Razvan. Sorry for the delay... I'd say that in general this is kind of
expected behavior. Let's dive into details.
First of all, the "Completed" status is fine. You can find where it's set in
the different implementations: rkt [1], docker [2] and dockershim [3]. Probably
dockershim is more clear on what's going on: the container has exited with exit
code 0. The reasons for that can be found in the logs (more on that later). The
fact that pods are still there even after they've finished is expected behavior
as well, see the documentation on garbage collection [4]. The bottom line is
that you don't want Kubernetes to remove all pods always because you'd like to
inspect dead containers, fetch their logs, etc.
Now, in this particular case, this happened to me in the past when the minions
are overwhelmed by the load. If you are creating 1000 pods, some of them will
be marked as "Pending", and some of them will be scheduled and eventually
failed. There's not much that Kubernetes can do here. At least that's what
'Error syncing pod, skipping: failed to "StartContainer" for "POD" with
RunContainerError: "runContainer: operation timeout: context deadline
exceeded"' is suggesting: it took too long to create a new container. That
might also be related to why you had to restart the kubelet process.
One thing I'd suggest is to upgrade your cluster to use Kubernetes 1.6 (it can
be found in obs://Virtualization:containers) just to discard a bug that has
already been fixed, and then report back with the full logs (kubectl logs of
everything, journalctl of the kubelet process, possibly also
kube-controller-manager and kube-scheduler in the master, etc.).
Thanks!
[1]
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/rkt/rkt.go#L2270
[2]
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockertools/docker_manager.go#L465
[3]
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_container.go#L384
[4]
https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/