kube-proxy ImagePullBackOff: kube-proxy:v1.19.4 image not present
Hi, I noticed that one of the nodes in my kubic cluster is limping, with the root cause being a failing kube-proxy pod. kube-system kube-proxy-p4ljp 0/1 ImagePullBackOff 0 97d 10.25.0.43 kubic- worker-1 <none> <none> The root cause seems to be Normal BackOff 19m (x11214 over 42h) kubelet Back-off pulling image "registry.opensuse.org/kubic/kube-proxy:v1.19.4" The other nodes seem to work fine, and I assume they have the images in the kubelet cache. All nodes are at version 1.19.7 $ kubectl get node NAME STATUS ROLES AGE VERSION kubic-master-1 Ready master 97d v1.19.7 kubic-worker-1 Ready <none> 97d v1.19.7 kubic-worker-2 Ready <none> 97d v1.19.7 I could edit the DaemonSet manually, but I like the kubic unattended updates too much and would like to understand the contract related to minor version updates. Should I file a bug or is this expected behaviour? Thanks, Robert
Hi, On Mon, Mar 01, Robert Munteanu wrote:
I could edit the DaemonSet manually, but I like the kubic unattended updates too much and would like to understand the contract related to minor version updates.
Should I file a bug or is this expected behaviour?
If you never upgrade your kubernetes cluster yourself then this is expected behaviour: we don't automatically upgrade deployed kubernetes clusters. No idea how you deployed kubernetes, but you need to upgrade with that tool. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)
On Mon, 2021-03-01 at 06:46 +0100, Thorsten Kukuk wrote:
Hi,
On Mon, Mar 01, Robert Munteanu wrote:
I could edit the DaemonSet manually, but I like the kubic unattended updates too much and would like to understand the contract related to minor version updates.
Should I file a bug or is this expected behaviour?
If you never upgrade your kubernetes cluster yourself then this is expected behaviour: we don't automatically upgrade deployed kubernetes clusters.
Sorry, I was incorrect in the description. I am talking about patch version updates, which kubic performs in an unattended manner. One of these updates broke one of the nodes in my cluster, and I want to understand if this is expected or not.
No idea how you deployed kubernetes, but you need to upgrade with that tool.
I used kubeadm as documented, and for minor updates I will do the same (e.g. 1.19.x to 1.20.x ). But this was a patch version update which should (IMO) be handled automatically by kubic. Thanks, Robert
On Mon, 2021-03-01 at 12:27 +0100, Robert Munteanu wrote:
On Mon, 2021-03-01 at 06:46 +0100, Thorsten Kukuk wrote:
Hi,
On Mon, Mar 01, Robert Munteanu wrote:
I could edit the DaemonSet manually, but I like the kubic unattended updates too much and would like to understand the contract related to minor version updates.
Should I file a bug or is this expected behaviour?
If you never upgrade your kubernetes cluster yourself then this is expected behaviour: we don't automatically upgrade deployed kubernetes clusters.
Sorry, I was incorrect in the description. I am talking about patch version updates, which kubic performs in an unattended manner. One of these updates broke one of the nodes in my cluster, and I want to understand if this is expected or not.
No idea how you deployed kubernetes, but you need to upgrade with that tool.
I used kubeadm as documented, and for minor updates I will do the same (e.g. 1.19.x to 1.20.x ). But this was a patch version update which should (IMO) be handled automatically by kubic.
Thanks, Robert
There is no mechanism in Kubernetes for magically changing your deployed containers to match your deployed binaries. Users need to update their deployed containers themselves aligned with their deployed binaries.
On Mon, 2021-03-01 at 13:01 +0100, Richard Brown wrote:
On Mon, 2021-03-01 at 12:27 +0100, Robert Munteanu wrote:
On Mon, 2021-03-01 at 06:46 +0100, Thorsten Kukuk wrote:
Hi,
On Mon, Mar 01, Robert Munteanu wrote:
I could edit the DaemonSet manually, but I like the kubic unattended updates too much and would like to understand the contract related to minor version updates.
Should I file a bug or is this expected behaviour?
If you never upgrade your kubernetes cluster yourself then this is expected behaviour: we don't automatically upgrade deployed kubernetes clusters.
Sorry, I was incorrect in the description. I am talking about patch version updates, which kubic performs in an unattended manner. One of these updates broke one of the nodes in my cluster, and I want to understand if this is expected or not.
No idea how you deployed kubernetes, but you need to upgrade with that tool.
I used kubeadm as documented, and for minor updates I will do the same (e.g. 1.19.x to 1.20.x ). But this was a patch version update which should (IMO) be handled automatically by kubic.
Thanks, Robert
There is no mechanism in Kubernetes for magically changing your deployed containers to match your deployed binaries.
Users need to update their deployed containers themselves aligned with their deployed binaries.
Then I would argue that the container images should be kept for a longer time on https://registry.opensuse.org/ . 'Forever' would be ideal, but if that's not possible then perhaps as long as that version is supported on kubic. Thanks, Robert
On Mon, 2021-03-01 at 13:04 +0100, Robert Munteanu wrote:
Then I would argue that the container images should be kept for a longer time on https://registry.opensuse.org/ . 'Forever' would be ideal, but if that's not possible then perhaps as long as that version is supported on kubic.
That would need to be discussed and agreed with the OBS Admins. I don't have any levers I can pull to make that happen, the Kubic team doesn't control registry.opensuse.org I wouldn't be surprised if OBS doesn't have resources to support that requirement Besides, you could argue that registry.opensuse.org is following your proposal. Those old versions are not 'supported' on kubic - you cannot install 1.19.4 packages for quite some time, and those containers were around for quite some time after 1.19.4 packages were removed...
On Mon, 2021-03-01 at 13:07 +0100, Richard Brown wrote:
On Mon, 2021-03-01 at 13:04 +0100, Robert Munteanu wrote:
Then I would argue that the container images should be kept for a longer time on https://registry.opensuse.org/ . 'Forever' would be ideal, but if that's not possible then perhaps as long as that version is supported on kubic.
That would need to be discussed and agreed with the OBS Admins. I don't have any levers I can pull to make that happen, the Kubic team doesn't control registry.opensuse.org
I wouldn't be surprised if OBS doesn't have resources to support that requirement
Alright, I will contact the OBS admins.
Besides, you could argue that registry.opensuse.org is following your proposal.
Those old versions are not 'supported' on kubic - you cannot install 1.19.4 packages for quite some time, and those containers were around for quite some time after 1.19.4 packages were removed...
I think we'll end up disagreeing here as I think dropping support for older patch versions is drastic. But let's focus on something else ... How would you handle the tasks that kubic does not perform out of the box? a. Disable automatic upgrades b. Script something that notifies that the k8s version and various containers are out of sync and I should run kubeadm upgrade. c. Set up a container image repository proxy and make sure images don't expire I don't particularly like any of them. Thoughts? Thanks, Robert
Hi Robert, Richard, On 01.03.21 at 14:37 Robert Munteanu wrote:
How would you handle the tasks that kubic does not perform out of the box?
Is there any specific reason why kubic does not handle these miniscule updates? Other than no one having time to implement it this far? In other words, what would be needed to actually automatically upgrade your containers from 1.19.4 to 1.19.7 once the underlying packages do this version step? Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On Mon, 2021-03-01 at 14:37 +0100, Robert Munteanu wrote:
On Mon, 2021-03-01 at 13:07 +0100, Richard Brown wrote:
On Mon, 2021-03-01 at 13:04 +0100, Robert Munteanu wrote:
Then I would argue that the container images should be kept for a longer time on https://registry.opensuse.org/ . 'Forever' would be ideal, but if that's not possible then perhaps as long as that version is supported on kubic.
That would need to be discussed and agreed with the OBS Admins. I don't have any levers I can pull to make that happen, the Kubic team doesn't control registry.opensuse.org
I wouldn't be surprised if OBS doesn't have resources to support that requirement
Alright, I will contact the OBS admins.
https://lists.opensuse.org/archives/list/buildservice@lists.opensuse.org/thr... Thanks, Robert
Hi, On Wed, Mar 03, Johannes Kastl wrote:
Hi Robert, Richard,
On 01.03.21 at 14:37 Robert Munteanu wrote:
How would you handle the tasks that kubic does not perform out of the box?
Is there any specific reason why kubic does not handle these miniscule updates? Other than no one having time to implement it this far?
Exactly this, nobody had time to look at how this could be done and implement it.
In other words, what would be needed to actually automatically upgrade your containers from 1.19.4 to 1.19.7 once the underlying packages do this version step?
This would be the first step on the way to an implementation ;) But this is not as simple as it sounds like, since there are different ways to deploy and configure kubernetes on Kubic. Anyways, the upgrade of the containers from 1.19.4 to 1.19.7 has to be done with "kubeadm upgrade". Even if it is only a minor update. Question is how to orchestrate this calls. If you use kubic-control, it would be a simple call and salt would take care of the rest. If the deployment was done manually with kubeadm, you need to login manually on every node and upgrade if I don't miss another way. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)
participants (4)
-
Johannes Kastl -
Richard Brown -
Robert Munteanu -
Thorsten Kukuk