http://bugzilla.suse.com/show_bug.cgi?id=1012374 Bug ID: 1012374 Summary: [Kubectl] Pod in state Completed Classification: openSUSE Product: openSUSE Distribution Version: Leap 42.1 Hardware: Other OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Virtualization:Other Assignee: virt-bugs@suse.de Reporter: rgherlea@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- We are conducting distributed load testing on a kubernetes with Locust (https://github.com/GoogleCloudPlatform/distributed-load-testing-using-kubern...) on cloud.suse.de, running 3 x minion's and 1 x kube-master. I deployed before the weekend a running cluster application of 1000 pods (running a container each). After the weekend, the kubernetes cluster was unresponsive, having the kubelet and kube-proxy services down on the minion nodes and the kubernetes* services down on the master node. I have restarted them as there is an open bug for this (1010441), and facing now a different issue, the locust-master pod has a state Completed having the following error message when describing the pod: razvan-kube-master:~ # kubectl describe pods locust-master-udl8f Name: locust-master-udl8f Namespace: default Node: razvan-kube-minion2.openstack.local/44.11.1.25 Start Time: Fri, 25 Nov 2016 13:21:51 +0000 Labels: name=locust role=master Status: Running IP: Controllers: ReplicationController/locust-master Containers: locust: Container ID: docker://68154aee0413f862883bcead0dfd3c78da989e55a74a571d4bed4bf45cc56647 Image: gcr.io/cloud-solutions-images/locust-tasks:latest Image ID: docker://sha256:26fa022c41367b15005c7d0e982e4b78691cb049725b72af9686fe6997a2212b Ports: 8089/TCP, 5557/TCP, 5558/TCP State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 25 Nov 2016 13:22:54 +0000 Finished: Mon, 28 Nov 2016 10:49:58 +0000 Ready: False Restart Count: 0 Environment Variables: LOCUST_MODE: master TARGET_HOST: http://workload-simulation-webapp.appspot.com Conditions: Type Status Initialized True Ready False PodScheduled True No volumes. QoS Tier: BestEffort Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 9m 9m 1 {kubelet razvan-kube-minion2.openstack.local} Warning MissingClusterDNS kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to DNSDefault policy. 7m 7m 1 {kubelet razvan-kube-minion2.openstack.local} Warning FailedSync Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: operation timeout: context deadline exceeded" It looks like the kube-master cannot locate the container that resides on the minion razvan-kube-minion2.openstack.local. It is not creating a new container. Cluster status: razvan-kube-master:~ # kubectl get pods|wc -l 1004 razvan-kube-master:~ # kubectl get pods|grep -v Running|wc -l 812 razvan-kube-master:~ # kubectl get pods|grep master locust-master-udl8f 0/1 Completed 0 2d razvan-kube-master:~ # kubectl get pods -o wide|grep master locust-master-udl8f 0/1 Completed 0 2d <none> razvan-kube-minion2.openstack.local razvan-kube-master:~ # Please let me know if you need any other information. Another information. Before the pod moved in the Completed status, the locust-master* pod had the status running: kubectl logs locust-master still shows the following logs, even after the status changed to completed: 8e2050787e8a' reported as ready. Currently 311 clients ready to swarm. [2016-11-25 20:25:56,960] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-7smh7_3a09ea67a077ad4b12436b8b06b61272' reported as ready. Currently 312 clients ready to swarm. [2016-11-25 20:26:26,688] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-7c4fn_0004dc82885fe71c83d79bd8c39f66a5' reported as ready. Currently 313 clients ready to swarm. [2016-11-25 20:27:46,044] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-7ruf6_177fbc96b9c35fdd7130055c356bcdb3' reported as ready. Currently 314 clients ready to swarm. [2016-11-25 20:33:37,754] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-nzwl7_d39c847b0b563bece3b29ae8054e8d26' reported as ready. Currently 315 clients ready to swarm. [2016-11-25 20:34:54,617] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-fprc0_a2278c4174159f88cb976e789b7394a7' reported as ready. Currently 316 clients ready to swarm. [2016-11-25 20:35:45,329] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-7d3r4_50457f871b63538b517ec90dcfe0b887' reported as ready. Currently 317 clients ready to swarm. [2016-11-25 20:41:15,367] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-638d3_c017284e7ea2b2942cebc388f9006eed' reported as ready. Currently 318 clients ready to swarm. [2016-11-25 20:41:16,236] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-0eseg_d1754f2648d2a6b7c8d0fa0ce779e4c1' reported as ready. Currently 319 clients ready to swarm. [2016-11-25 20:46:40,405] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-uelcp_f6d0193b4703e184fd21f4da50f21f8d' reported as ready. Currently 320 clients ready to swarm. [2016-11-25 20:47:41,856] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-9qjzk_f1dc55c3e4f4b05909fd118054e39bfe' reported as ready. Currently 321 clients ready to swarm. [2016-11-25 20:49:14,613] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-sonmm_a875cb0f4a06be1eac25e99f35c9dfbd' reported as ready. Currently 322 clients ready to swarm. [2016-11-25 20:50:25,881] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-qq5kk_73b2eba37c062c5008fabfd9ebfabbcd' reported as ready. Currently 323 clients ready to swarm. [2016-11-25 20:52:45,506] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-m4x6g_23f48f2c1f398ef8828db21c8965c224' reported as ready. Currently 324 clients ready to swarm. [2016-11-25 20:53:30,194] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-whgnm_78d3e26ec56da8c4a5625df15b226dfe' reported as ready. Currently 325 clients ready to swarm. [2016-11-25 21:01:19,799] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-rv5i0_6088c29a52d78ae79f777c6f85faff22' reported as ready. Currently 326 clients ready to swarm. [2016-11-25 21:01:26,268] locust-master-udl8f/INFO/locust.runners: Client 'locust-2195719602-xn59v_70cbb2ddb35e49025ce1201a594192a1' reported as ready. Currently 327 clients ready to swarm. [2016-11-28 08:25:06,169] locust-master-udl8f/INFO/locust.runners: Sending hatch jobs to 327 ready clients [2016-11-28 08:26:31,693] locust-master-udl8f/INFO/locust.runners: Sending hatch jobs to 327 ready clients Is this a how it should be or a bug ? -- You are receiving this mail because: You are on the CC list for the bug.