Error during "kubeadm init": failed to pull image registry.opensuse.org/kubic/xxx: output: Error response from daemon: manifest unknown
Hi all, sorry for asking here, but I guess there are more kubeadm experts than on other lists. I was just preparing a test setup of Kubernetes on openSUSE Leap 15.2 and noticed some errors: 1. Install patterns-containers-kubeadm 2. reboot 3. enable kubelet.service 4. run "kubeadm init ..." and get an error as no runtime is found 5. enable and start either docker.service or crio.service 6. re-reun "kubeadm init..." 7. Failure (see below) According to this issue at docker/hub it is a problem with tags missing on the images/registry. Any idea where to start? How to modify which containers are used? I started grepping through the system but even if I found the lines to what versions should I change them? Kind Regards, Johannes **docker**: ```
# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=81.163.192.187 I0227 19:03:51.388635 1950 version.go:252] remote version is much newer: 1.20.2; falling back to: stable-1.18 W0227 19:03:56.443331 1950 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.14 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-apiserver:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-controller-manager:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-scheduler:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-proxy:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/etcd:3.4.3-0: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/coredns:1.6.7: output: Error response from daemon: manifest unknown , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
``` **crio**: ```
# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=81.163.192.187 I0227 19:24:42.148453 2041 version.go:252] remote version is much newer: 1.20.2; falling back to: stable-1.18 W0227 19:24:42.202269 2041 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.14 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-apiserver:v1.18.14: output: time="2021-02-27T19:24:44Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-apiserver: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-controller-manager:v1.18.14: output: time="2021-02-27T19:24:45Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-controller-manager: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-scheduler:v1.18.14: output: time="2021-02-27T19:24:46Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-scheduler: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-proxy:v1.18.14: output: time="2021-02-27T19:24:47Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-proxy: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/etcd:3.4.3-0: output: time="2021-02-27T19:24:50Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest 3.4.3-0 in registry.opensuse.org/kubic/etcd: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/coredns:1.6.7: output: time="2021-02-27T19:24:51Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest 1.6.7 in registry.opensuse.org/kubic/coredns: manifest unknown" , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
``` -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi, On Sat, Feb 27, Johannes Kastl wrote:
Hi all,
sorry for asking here, but I guess there are more kubeadm experts than on other lists.
I was just preparing a test setup of Kubernetes on openSUSE Leap 15.2 and noticed some errors:
openSUSE Leap is not openSUSE Kubic ;) I don't think that anybody takes care of kubernetes on openSUSE Leap. To use the kubernetes container of openSUSE Kubic on Leap, you need kubernetes version 1.19.7 or 1.20.2 Thorsten
1. Install patterns-containers-kubeadm 2. reboot 3. enable kubelet.service 4. run "kubeadm init ..." and get an error as no runtime is found 5. enable and start either docker.service or crio.service 6. re-reun "kubeadm init..." 7. Failure (see below)
According to this issue at docker/hub it is a problem with tags missing on the images/registry.
Any idea where to start? How to modify which containers are used? I started grepping through the system but even if I found the lines to what versions should I change them?
Kind Regards, Johannes
**docker**: ```
# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=81.163.192.187 I0227 19:03:51.388635 1950 version.go:252] remote version is much newer: 1.20.2; falling back to: stable-1.18 W0227 19:03:56.443331 1950 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.14 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-apiserver:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-controller-manager:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-scheduler:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-proxy:v1.18.14: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/etcd:3.4.3-0: output: Error response from daemon: manifest unknown , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/coredns:1.6.7: output: Error response from daemon: manifest unknown , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
```
**crio**: ```
# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=81.163.192.187 I0227 19:24:42.148453 2041 version.go:252] remote version is much newer: 1.20.2; falling back to: stable-1.18 W0227 19:24:42.202269 2041 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.18.14 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-apiserver:v1.18.14: output: time="2021-02-27T19:24:44Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-apiserver: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-controller-manager:v1.18.14: output: time="2021-02-27T19:24:45Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-controller-manager: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-scheduler:v1.18.14: output: time="2021-02-27T19:24:46Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-scheduler: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-proxy:v1.18.14: output: time="2021-02-27T19:24:47Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest v1.18.14 in registry.opensuse.org/kubic/kube-proxy: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/etcd:3.4.3-0: output: time="2021-02-27T19:24:50Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest 3.4.3-0 in registry.opensuse.org/kubic/etcd: manifest unknown" , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/coredns:1.6.7: output: time="2021-02-27T19:24:51Z" level=fatal msg="pulling image failed: rpc error: code = Unknown desc = Error reading manifest 1.6.7 in registry.opensuse.org/kubic/coredns: manifest unknown" , error: exit status 1 [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...` To see the stack trace of this error execute with --v=5 or higher
``` -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de
B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
-- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)
Hi Thorsten, On 27.02.21 at 21:02 Thorsten Kukuk wrote:
On Sat, Feb 27, Johannes Kastl wrote:
sorry for asking here, but I guess there are more kubeadm experts than on other lists.
I was just preparing a test setup of Kubernetes on openSUSE Leap 15.2 and noticed some errors:
openSUSE Leap is not openSUSE Kubic ;)
I know, but my guess was that the kubernetes packages in Leap are the same ones that were used in kubic and maybe someone from kubic was involved in their packaging.
I don't think that anybody takes care of kubernetes on openSUSE Leap.
Unfortunately it looks like this, yes.
To use the kubernetes container of openSUSE Kubic on Leap, you need kubernetes version 1.19.7 or 1.20.2
I tried to install them from the devel:kubic repos (in Leap 15.2) but the installation fails:
# zypper in kubernetes-kubeadm kubernetes-kubelet kubernetes-client Loading repository data... Reading installed packages... Resolving package dependencies... 3 Problems: Problem: nothing provides kubernetes1.20-client = 1.20.2 needed by kubernetes-client-1.20.2-lp152.20.1.x86_64 Problem: nothing provides kubernetes1.20-kubelet needed by kubernetes-kubeadm-1.20.2-lp152.20.1.x86_64 Problem: nothing provides kubernetes1.20-kubelet = 1.20.2 needed by kubernetes-kubelet-1.20.2-lp152.20.1.x86_64
Problem: nothing provides kubernetes1.20-client = 1.20.2 needed by kubernetes-client-1.20.2-lp152.20.1.x86_64 Solution 1: do not install kubernetes-client-1.20.2-lp152.20.1.x86_64 Solution 2: break kubernetes-client-1.20.2-lp152.20.1.x86_64 by ignoring some of its dependencies
Choose from above solutions by number or skip, retry or cancel [1/2/s/r/c/d/?] (c):
Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 03.03.21 at 08:02 Johannes Kastl wrote:
Problem: nothing provides kubernetes1.20-client = 1.20.2 needed by kubernetes-client-1.20.2-lp152.20.1.x86_64 Solution 1: do not install kubernetes-client-1.20.2-lp152.20.1.x86_64 Solution 2: break kubernetes-client-1.20.2-lp152.20.1.x86_64 by ignoring some of its dependencies
This is caused by the kubernetes "meta-package" not having the same BuildRequires as the actual kubernetes1.20 package, so the pseudo packages like kubernetes-client-1.20.2-lp152.20.1.x86_64 are being built for Leap 15.2, but are not installable. First fix would be to no longer build those packages, as they are useless without the actual packages: https://build.opensuse.org/request/show/876602 But actually I would really like to see someone (with the permissions) submit a go1.15 package into that repository and build it for Leap15.2 only. Then both sets of packages could be built for Leap 15.2. I am not sure if I can request that via a SR, though... Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 03.03.21 at 19:51 Johannes Kastl wrote:
But actually I would really like to see someone (with the permissions) submit a go1.15 package into that repository and build it for Leap15.2 only. Then both sets of packages could be built for Leap 15.2.
I rebuilt the packages for Leap 15.2 and could successfully install kubeadm, kubelet and cri-o. First master and worker are created successfully!
kastl-kubernetes-masters-01:~ # kubectl get nodes NAME STATUS ROLES AGE VERSION kastl-kubernetes-masters-01 Ready control-plane,master 22h v1.20.2 kastl-kubernetes-workers-01 Ready <none> 22h v1.20.2 kastl-kubernetes-masters-01:~ # cat /etc/os-release NAME="openSUSE Leap" VERSION="15.2" ID="opensuse-leap" ID_LIKE="suse opensuse" VERSION_ID="15.2" PRETTY_NAME="openSUSE Leap 15.2" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:opensuse:leap:15.2" BUG_REPORT_URL="https://bugs.opensuse.org" HOME_URL="https://www.opensuse.org/" kastl-kubernetes-masters-01:~ #
However, joining two more masters (aka control-plane) did not work out. I'll dig deeper. Is this a known issue with 1.20?
[control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [check-etcd] Checking that the etcd cluster is healthy [hangs here until the timeout is reached...]
Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
Hi all, On 04.03.21 at 20:31 Johannes Kastl wrote:
However, joining two more masters (aka control-plane) did not work out. I'll dig deeper. Is this a known issue with 1.20?
Solved! Sometimes the solution is really easy: I had a typo in the ports allowed, so etcd 2379 was blocked... :-) Kind Regards, Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
On 27.02.21 at 20:30 Johannes Kastl wrote:
According to this issue at docker/hub it is a problem with tags missing on the images/registry.
Any idea where to start?
Seems like the issue is caused by the fact that there are simply no images tagged with 1.18 available in the openSUSE registry. I tried to install a newer kubernetes version from the devel:kubic repos, but failed due to dependency issues in the packages..
# zypper in kubernetes-kubeadm kubernetes-kubelet kubernetes-client Loading repository data... Reading installed packages... Resolving package dependencies... 3 Problems: Problem: nothing provides kubernetes1.20-client = 1.20.2 needed by kubernetes-client-1.20.2-lp152.20.1.x86_64 Problem: nothing provides kubernetes1.20-kubelet needed by kubernetes-kubeadm-1.20.2-lp152.20.1.x86_64 Problem: nothing provides kubernetes1.20-kubelet = 1.20.2 needed by kubernetes-kubelet-1.20.2-lp152.20.1.x86_64
Problem: nothing provides kubernetes1.20-client = 1.20.2 needed by kubernetes-client-1.20.2-lp152.20.1.x86_64 Solution 1: do not install kubernetes-client-1.20.2-lp152.20.1.x86_64 Solution 2: break kubernetes-client-1.20.2-lp152.20.1.x86_64 by ignoring some of its dependencies
Choose from above solutions by number or skip, retry or cancel [1/2/s/r/c/d/?] (c):
Installing a kubernetes-kubeadm in version 1.18 succeeds, but I guess there will be the same errors due to missing containers... Investigation further.. Johannes -- Johannes Kastl Linux Consultant & Trainer Tel.: +49 (0) 151 2372 5802 Mail: kastl@b1-systems.de B1 Systems GmbH Osterfeldstraße 7 / 85088 Vohburg http://www.b1-systems.de GF: Ralph Dehner Unternehmenssitz: Vohburg / AG: Ingolstadt,HRB 3537
participants (2)
-
Johannes Kastl -
Thorsten Kukuk