http://bugzilla.opensuse.org/show_bug.cgi?id=1198064 http://bugzilla.opensuse.org/show_bug.cgi?id=1198064#c1 Enno Gotthold changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |egotthold@suse.com --- Comment #1 from Enno Gotthold --- So I am using homerack-10-server:~ # uname -r 5.17.1-1-default with homerack-10-server:~ # rpm -qa k3s k3s-1.22.3+k3s1-1.5.x86_64 and my coredns is working quite fine. Indeed this is a little out of date but I will work on that through packaging a new version for k3s soon. My k3s expertise is quite low probably but for me it seems to work. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1198064 http://bugzilla.opensuse.org/show_bug.cgi?id=1198064#c3 --- Comment #3 from Johannes Kastl --- Hi Enno, I managed to get one of my clusters working again by removing Cilium and replacing it with flannel. After some tweaks (had to copy /usr/libexec/cni/* to /opt/cni/bin/ manually, then flannel was happy) it seems to be running again. So I guess it is related to the more advanced things that cilium and weave are doing in comparison to flannel, that they both break using the newer kernel version. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1198064 http://bugzilla.opensuse.org/show_bug.cgi?id=1198064#c5 --- Comment #5 from Johannes Kastl --- Hi Enno, any news here? Did you check if your cluster was still working fine? All of my clusters with the newer kernel are still broken, the ones with flannel and the ones with cilium. From what I found it looks like the whole networking to the outside is broken: - DNS queries are not working - ping against an IP not working - wget against an IP address is not working Updating cilium to the latest version did not improve the situation. And, of course, this could be something else than the kernel, that was rolled back when I rolled back to the earlier snapshot containing the older kernel... Kind Regards, Johannes -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1198064 http://bugzilla.opensuse.org/show_bug.cgi?id=1198064#c7 --- Comment #7 from Johannes Kastl --- Hi Enno, I tried to reproduce this on some new tumbleweed and MicroOS machines last week, but all of the newly installed k3s clusters (using flannel) worked fine. Either this is something in my setups, or the bug is not triggered on new installations or the bug is fixed in the meantime. But my clusters (on machines that are being update and not frozen to the old working state) are still broken. Did you have more luck? Kind Regards, Johannes -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1198064 http://bugzilla.opensuse.org/show_bug.cgi?id=1198064#c9 Johannes Kastl changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution|--- |FIXED --- Comment #9 from Johannes Kastl --- Just to close this topic properly: Seems like this was caused by cilium being broken due to a missing sysctl parameter. The [latest 1.11.7 documentation](https://docs.cilium.io/en/v1.11/operations/system_requirements/) states that systemd >=245 causes issues: ``` Systemd 245 and above (systemctl --version) overrides rp_filter setting of Cilium network interfaces. This introduces connectivity issues (see [GitHub issue 10645](https://github.com/cilium/cilium/issues/10645) for details). To avoid that, configure rp_filter in systemd using the following commands: echo 'net.ipv4.conf.lxc*.rp_filter = 0' > /etc/sysctl.d/99-override_cilium_rp_filter.conf systemctl restart systemd-sysctl ``` I did not notice that warning in the docs at the time the problem started, or it was not present back then, I do not know. However I tried the workaround and my clusters started working properly again. Thanks for your help! -- You are receiving this mail because: You are the assignee for the bug.