https://bugzilla.suse.com/show_bug.cgi?id=1213786 Bug ID: 1213786 Summary: [kubeadm] openSUSE kubeReleaseBucket Server outdated Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.5 Hardware: All OS: openSUSE Leap 15.5 Status: NEW Severity: Normal Priority: P5 - None Component: Containers Assignee: containers-bugowner@suse.de Reporter: eich@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: Development Blocker: --- Running 'kubeadm init' from kubernetes1.24-kubeadm fails with the error message: [ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.24.15" Control plane version: "1.23.4" Reason: kubeadm queries a release version server which resides at https://dl.k8s.io/release while for upstream but the Leap/TW kubeadm package has patched to https://kubic.opensuse.org/release. This responds to a requrest for https://kubic.opensuse.org/release/stable-1.txt with 1.23.4 although SUSE already provides version 1.27.3. Since this appears to announce that the containers provided by openSUSE are older than the kubeadm version, it refuses to work. Since the above URL is hard coded and cannot be overridden, the only way to overcome this obstacle was to rebuild the package with this patch removed. This made kubeadm query the upstream server which returned v1.27.4. Since here the minor number is higher than the one of kubeadm, the fallback mechanism kicked in querying for https://dl.k8s.io/release/stable-1.24.txt - which returned v1.24.16. Next problem: After dropping the fix,`init` proceeded attempting to download container images. This again failed with the error messages: error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-apiserver:v1.24.16: output: time="2023-07-30T19:14:10+02:00" level=fatal msg="pulling image failed: rpc error: code , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-controller-manager:v1.24.16: output: time="2023-07-30T19:14:12+02:00" level=fatal msg="pulling image failed: rpc err , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-scheduler:v1.24.16: output: time="2023-07-30T19:14:13+02:00" level=fatal msg="pulling image failed: rpc error: code , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-proxy:v1.24.16: output: time="2023-07-30T19:14:14+02:00" level=fatal msg="pulling image failed: rpc error: code = Un , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/coredns:v1.8.6: output: time="2023-07-30T19:14:28+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown , error: exit status 1 Obviously, it tried to download containers of the version advertised by upstream, however, these versions are not (yet) available on registry.opensuse.org - as can be checked using `podman search`: $ podman search --list-tags registry.opensuse.org/kubic/kube-apiserver NAME TAG registry.opensuse.org/kubic/kube-apiserver latest [..] registry.opensuse.org/kubic/kube-apiserver v1.24.3 registry.opensuse.org/kubic/kube-apiserver v1.24.3-3.1532 registry.opensuse.org/kubic/kube-apiserver v1.26.6 [..] registry.opensuse.org/kubic/kube-apiserver v1.27.3-5.91 registry.opensuse.org/kubic/kube-apiserver v1.27.3-5.93 The latest available version of v1.24 here is v1.24.3 - this should have been served querying https://kubic.opensuse.org/release/stable-1.24.txt, however, this URL returns 404. The last major.minor version combo served is: $ curl -L https://kubic.opensuse.org/release/stable-1.23.txt 1.23.4 i.e. the latest kubernetes-1.23 version available on registry.opensuse.org. It appears as if a script triggered by the build of new versions to update or newly create stable-1<N>.txt files is no longer running. Since the URL cannot be overridden it renders `kubernetes1.<X>-kubeadm` packages for <X> > 23 useless at least when running `kubeadm init`. -- You are receiving this mail because: You are on the CC list for the bug.