[Bug 1213786] New: [kubeadm] openSUSE kubeReleaseBucket Server outdated
https://bugzilla.suse.com/show_bug.cgi?id=1213786 Bug ID: 1213786 Summary: [kubeadm] openSUSE kubeReleaseBucket Server outdated Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.5 Hardware: All OS: openSUSE Leap 15.5 Status: NEW Severity: Normal Priority: P5 - None Component: Containers Assignee: containers-bugowner@suse.de Reporter: eich@suse.com QA Contact: qa-bugs@suse.de Target Milestone: --- Found By: Development Blocker: --- Running 'kubeadm init' from kubernetes1.24-kubeadm fails with the error message: [ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.24.15" Control plane version: "1.23.4" Reason: kubeadm queries a release version server which resides at https://dl.k8s.io/release while for upstream but the Leap/TW kubeadm package has patched to https://kubic.opensuse.org/release. This responds to a requrest for https://kubic.opensuse.org/release/stable-1.txt with 1.23.4 although SUSE already provides version 1.27.3. Since this appears to announce that the containers provided by openSUSE are older than the kubeadm version, it refuses to work. Since the above URL is hard coded and cannot be overridden, the only way to overcome this obstacle was to rebuild the package with this patch removed. This made kubeadm query the upstream server which returned v1.27.4. Since here the minor number is higher than the one of kubeadm, the fallback mechanism kicked in querying for https://dl.k8s.io/release/stable-1.24.txt - which returned v1.24.16. Next problem: After dropping the fix,`init` proceeded attempting to download container images. This again failed with the error messages: error execution phase preflight: [preflight] Some fatal errors occurred: [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-apiserver:v1.24.16: output: time="2023-07-30T19:14:10+02:00" level=fatal msg="pulling image failed: rpc error: code , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-controller-manager:v1.24.16: output: time="2023-07-30T19:14:12+02:00" level=fatal msg="pulling image failed: rpc err , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-scheduler:v1.24.16: output: time="2023-07-30T19:14:13+02:00" level=fatal msg="pulling image failed: rpc error: code , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/kube-proxy:v1.24.16: output: time="2023-07-30T19:14:14+02:00" level=fatal msg="pulling image failed: rpc error: code = Un , error: exit status 1 [ERROR ImagePull]: failed to pull image registry.opensuse.org/kubic/coredns:v1.8.6: output: time="2023-07-30T19:14:28+02:00" level=fatal msg="pulling image failed: rpc error: code = Unknown , error: exit status 1 Obviously, it tried to download containers of the version advertised by upstream, however, these versions are not (yet) available on registry.opensuse.org - as can be checked using `podman search`: $ podman search --list-tags registry.opensuse.org/kubic/kube-apiserver NAME TAG registry.opensuse.org/kubic/kube-apiserver latest [..] registry.opensuse.org/kubic/kube-apiserver v1.24.3 registry.opensuse.org/kubic/kube-apiserver v1.24.3-3.1532 registry.opensuse.org/kubic/kube-apiserver v1.26.6 [..] registry.opensuse.org/kubic/kube-apiserver v1.27.3-5.91 registry.opensuse.org/kubic/kube-apiserver v1.27.3-5.93 The latest available version of v1.24 here is v1.24.3 - this should have been served querying https://kubic.opensuse.org/release/stable-1.24.txt, however, this URL returns 404. The last major.minor version combo served is: $ curl -L https://kubic.opensuse.org/release/stable-1.23.txt 1.23.4 i.e. the latest kubernetes-1.23 version available on registry.opensuse.org. It appears as if a script triggered by the build of new versions to update or newly create stable-1<N>.txt files is no longer running. Since the URL cannot be overridden it renders `kubernetes1.<X>-kubeadm` packages for <X> > 23 useless at least when running `kubeadm init`. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1213786 https://bugzilla.suse.com/show_bug.cgi?id=1213786#c1 --- Comment #1 from Egbert Eich <eich@suse.com> --- Forgot to mention - this problem is biting TW as well, of course. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1213786 https://bugzilla.suse.com/show_bug.cgi?id=1213786#c3 --- Comment #3 from Egbert Eich <eich@suse.com> --- @Priyanka - thanks for looking into this so swiftly :) I've seen that you've removed the leading 'v' from the version to make it more compatible with upstream. I do wonder if this cannot be automated - i.e. generated by a script using registry data and triggered by OBS when a new container for kubic/* gets published. One more note: the 'upstream' version of this (from https://dl.k8s.io/release) does not have a trailing newline. I did not see any issues with it, though. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1213786 https://bugzilla.suse.com/show_bug.cgi?id=1213786#c4 --- Comment #4 from Egbert Eich <eich@suse.com> --- @Priyanka - since Richard has responded - after a gentle prod - I was able to test it. I'm afraid we are not quite there, yet: * Issues on Leap 15.5: Since we advertise version 1.24.16, we should have the containers for it. Yet, registry.opensuse.org/kubic/kube-apiserver:v1.24.16 registry.opensuse.org/kubic/kube-controller-manager:v1.24.16 registry.opensuse.org/kubic/kube-scheduler:v1.24.16 registry.opensuse.org/kubic/kube-proxy:v1.24.16 are not available on the registry, yet. * Issues on TW (with latest kubernetes 1.27.4: Here the following containers are missing from the registry: registry.opensuse.org/kubic/pause:3.9 registry.opensuse.org/kubic/etcd:3.5.7-0 There's also an issue with /usr/bin/kubelet (-> separate ticket). Thanks! -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1213786 https://bugzilla.suse.com/show_bug.cgi?id=1213786#c6 --- Comment #6 from Egbert Eich <eich@suse.com> --- (In reply to Priyanka Saggu from comment #5)
eich@suse.com, thanks a lot for testing and listing all the issues - really helpful!
I'll look into how to build/push the new missing container image versions on the registry (including pause & etcd).
That's much appreciated, thank you!
There's also an issue with /usr/bin/kubelet (-> separate ticket).
Ack. Will take a look. Thanks again!
I had some time left so I've pushed a fix for this, maybe you can have a look. Thank you! -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1213786 https://bugzilla.suse.com/show_bug.cgi?id=1213786#c7 --- Comment #7 from Egbert Eich <eich@suse.com> --- @Priyanka: I've looked into this a bit - it seems like devel:kubic:containers/kubic-pause-image needs the 3.9 tag in: https://build.opensuse.org/package/view_file/devel:kubic:containers/kubic-pa... I'm not sure what happened to the rest, devel:kubic/etcd-for-k8s1.27 should probably be updated it looks like the package openSUSE:Factory/kubernetes needs to be updated to the new version of etcd: # etcdversion - version of etcd %define etcdversion 3.5.7 # etcdversionminus1 - version of etcd for versionminus1 %define etcdversionminus1 3.5.6 and possibly others. I'm not sure why containers for the newer version of kubernetes1.24 are not being published, though. -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1213786 https://bugzilla.suse.com/show_bug.cgi?id=1213786#c8 --- Comment #8 from Egbert Eich <eich@suse.com> --- It looks like we only publish containers for the latest version of kubernetes sub-packages - ie, once kubernetes1.N becomes available, we stop publishing containers for kubernetes1.N-1 sub-packages while upstream seems to support multiple versions of kubernetes. This explains why registry.opensuse.org/kubic/kube-apiserver:v1.24.16 registry.opensuse.org/kubic/kube-controller-manager:v1.24.16 registry.opensuse.org/kubic/kube-scheduler:v1.24.16 registry.opensuse.org/kubic/kube-proxy:v1.24.16 and for this matter any kubernetes subpackages of kubernetes beyond 1.24.3. To mitigate this, we most likely need to have separate versioned container packages like: openSUSE:Containers:Tumbleweed/kubic-kube-apiserver-image1.N with: <package name="kubernetes1.N-apiserver"/> openSUSE:Containers:Tumbleweed/kube-controller-manager1.N with: <package name="kubernetes1.N-controller-manager"/> openSUSE:Containers:Tumbleweed/kubic/kube-scheduler1.N with: <package name="kubernetes1.N-scheduler"/> openSUSE:Containers:Tumbleweed/kube-proxy1.N with: <package name="kubernetes1.N-proxy"/> Also the currently 'unversioned' dependency package would be required 'per version'. An easier way would be to not announce the latest 'upstream' versions through https://kubic.opensuse.org/release/stable-1.<N>.txt but the latest for which a container is available. However, if the later releases were security-related, this may not be the way to go. Unfortunately. right now, the efforts to maintain separate code stream of kubernetes are largely in vain as the registry does not require the needed containers. It would still allow users to build their own containers - but who would seriously consider that? -- You are receiving this mail because: You are on the CC list for the bug.
https://bugzilla.suse.com/show_bug.cgi?id=1213786 https://bugzilla.suse.com/show_bug.cgi?id=1213786#c10 --- Comment #10 from Egbert Eich <eich@suse.com> --- Hello Priyanka, let me also address the different topics individually (for brevity reasons I've deleted some lines) - (In reply to Priyanka Saggu from comment #9)
Hello eich@suse.com, apologies for the delay. I got caught up in resolving another issue upstream.
Addressing below each point separately, but just want to mention that I am also relatively new to Kubernetes packaging with O/IBS. So, I have been going back and forth to gather context & still be missing information fully.
That's fine. I've been with this company for quite a while, now, and have handled a lot of difficult packaging issues and still don't know everything - this is true in particular around building containers.
we currently don't have any new tags available for k8s versions beyond N-1 (v1.26), because the above source kiwi files are no longer tracking them.
Exactly, and here lies the problem: we still update older code streams (ie k8s minor versions) - must likely to fix security vulnerabilities - but we do not push updated containers for these. So, despite the effort that went into maintaining all these different code streams - they are not consumable (or if they were, users would potentially still get vulnerable containers). This disadvantages the openSUSE Leap users in particular since they are stuck with kubernetes1.24.
---
Regarding `kubic-pause-image`, yes, it seems that adding another tag (3.9) in kiwi file[3] is required, and versions of kuberetes-pause[4] package need to be bumped (but I'm still digging further on pause image).
[3] https://build.opensuse.org/package/view_file/devel:kubic:containers/kubic- pause-image/kubic-pause-image.kiwi?expand=1 [4] https://build.opensuse.org/package/show/devel:kubic/kubernetes-pause
In my understanding, only the former needs an additional 'Tag' while the latter may remain the same: pause is really a simple application - and it seems we have our own, which has seen fewer updates than upstream. I have not done any research on 'pause' - would it make sense to sync (the sources) with upstream?
---
As for `devel:kubic/etcd-for-k8s1.27`, it appears to be up-to-date based on the dependencies[5] from upstream project. The meta (unversioned) kubernetes packages have the etcd values[5][6] set properly.
[5] etcdversion (wrt v1.27.4) - https://github.com/kubernetes/kubernetes/blob/v1.27.4/build/dependencies. yaml#L63 [6] etcdversionminus1 (wrt v1.26.7) -https://github.com/kubernetes/kubernetes/blob/v1.26.7/build/dependencies. yaml#L56
Indeed. There may have been a different issue involved that is unrelated to the package and the available containers that I was not able to get the correct version - or I was just confused.
---
I totally agree that the current situation of maintaining multiple versions of Kubernetes packaging in OBS needs improvement on multiple levels, to make it barely usable.
For starters, one suggestion I received is to consider moving the k8s container images to registry.suse.com and utilizing `BCI-dockerfile-generator`[7] for that purpose. I am currently exploring this option.
Using BCI-dockerfile-generator might be an option, but I'm skeptical about moving to registry.suse.com: 1. It may be a good idea to keep enterprise and openSUSE activities separate and let the former be derived from the latter. This gives openSUSE more independence. 2. registry.opensuse.com comes with a lot more strings attached. For once, you need to build in IBS. 3. At least currently, SLE users do not get any of the kubernetes packages (except kubernetesXX-client) - not even through PackageHub. Thus, none of them will be able to consume these containers. 4. There's no browsable interface for registry.suse.de - at least I haven't found one, yet. 5. I don't see how this would resolve the problems at hand - we would still have to fix them, wouldn't we? Using registry.suse.com should not be a prerequisite to use the dockerfile generator, so you may still use it while staying on registry.opensuse.org. -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com