[opensuse-kubic] CNI plugins binary path
Hi, Recently I'm trying to deploy Cilium on Kubic and to create k8s manifest which is using Kubic container images. Everything is going well except one thing - I'm not sure what to do with cilium-cni binary (a CNI plugin which needs to be visible by kubelet). I assumed that since we use (almost) vanilla Kubernetes with kubeadm, kubelet is going to look for plugins in /opt/cni/bin and that I can just mount that directory, then copy the binary there from the initContainer (or as a part of preStart step). But it seems that kubelet is lookng only at /usr/lib/cni, which is read only and there are only CNI plugins installed from packages. I see two potential solutions: 1. Changing the cni-bin-dir parameter for Kubelet to `/usr/lib/cni,/opt/cni/bin` to support both CNI plugins installed on host and those installed dynamically. 2. Adding cilium-cni package to patterns-containers. What do you think? BTW, I have the manifest here: https://github.com/kubic-project/k8s-manifests/blob/cilium/cilium.yaml It should work if you do `transactional-update pkg install cilium-cni` and restart nodes before. Cheers, Michal -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Hi, On Thu, Apr 18, Michal Rostecki wrote:
1. Changing the cni-bin-dir parameter for Kubelet to `/usr/lib/cni,/opt/cni/bin` to support both CNI plugins installed on host and those installed dynamically.
As /opt/cni/bin is the upstream default and mentioned everywhere in the documentation, we should add that if cni-bin-dir allows a comma seperated list of directories. Everything else will only be confusing and limited the usage of third party solutions.
2. Adding cilium-cni package to patterns-containers.
devel:kubic/cilium contains for cilimum-cni: Requires: cilium That would mean, we pull in a huge amount of code. Is this requires really correct? I cannot imagine if I look at the cilium-cni package. If yes, adding cilium-cni is a no-go due to the size. If no, remove that Requires and add additional cilium-cni to the container patterns. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
On Thu, Apr 18, 2019 at 02:37:53PM +0200, Thorsten Kukuk wrote:
1. Changing the cni-bin-dir parameter for Kubelet to `/usr/lib/cni,/opt/cni/bin` to support both CNI plugins installed on host and those installed dynamically.
As /opt/cni/bin is the upstream default and mentioned everywhere in the documentation, we should add that if cni-bin-dir allows a comma seperated list of directories. Everything else will only be confusing and limited the usage of third party solutions.
Cool, so we are on the same page. I will prepare the change soon. Thanks, Michal -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
I realized that with other runtimes than Docker, --cni-bin-dir option in kubelet is irrelevant and the plugins_dir option in crio.conf needs to be used instead. Please take a look: https://build.opensuse.org/request/show/695814 After this change, Cilium works perfectly. Thanks, Michal -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Hi, I see that the update is already been accepted by openSUSE:Factory so please bear with me that I'm quite late. I'm not sure If this kind of adaptions to the package are the right way to solve the configuration problems, so I would propose an additional change for the next upcoming major version of CRI-O: As far as I understood was the problem that CRI-O does not start if there is a CNI plugin directory configured which does not exist. Other projects like Cilium would run as DeamonSets on Kubernetes where CRI-O surely has to be started before. To fix this, I changed the upstream behavior for the CNI plugin directory validation that not existing directories will be created automatically by CRI-O [1]. This makes the following lines in the spec obsolete for the next major (v1.15.0) release: ``` %post %service_add_post %{name_source1} # This is the additional directory where cri-o is going to look up for CNI # plugins installed by DaemonSets running on Kubernetes (i.e. Cilium). mkdir -p /opt/cni/bin ``` If you're fine with that I will keep track of that proposal. Nevertheless, we would still keep the "/opt/cni/bin" directory in the CRI-O configuration: ``` plugin_dir = ["/usr/lib/cni", "/opt/cni/bin"] ``` If the CRI-O configuration (in the future) is not flexible enough for our needs, we could consider using the `crio config` subcommand with different arguments for different distributions. All the best, Sascha [1] https://github.com/cri-o/cri-o/pull/2284 On 4/18/19 2:37 PM, Thorsten Kukuk wrote:
Hi,
On Thu, Apr 18, Michal Rostecki wrote:
1. Changing the cni-bin-dir parameter for Kubelet to `/usr/lib/cni,/opt/cni/bin` to support both CNI plugins installed on host and those installed dynamically.
As /opt/cni/bin is the upstream default and mentioned everywhere in the documentation, we should add that if cni-bin-dir allows a comma seperated list of directories. Everything else will only be confusing and limited the usage of third party solutions.
2. Adding cilium-cni package to patterns-containers.
devel:kubic/cilium contains for cilimum-cni: Requires: cilium
That would mean, we pull in a huge amount of code. Is this requires really correct? I cannot imagine if I look at the cilium-cni package. If yes, adding cilium-cni is a no-go due to the size. If no, remove that Requires and add additional cilium-cni to the container patterns.
Thorsten
-- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
Hi, On Thu, Apr 25, Sascha Grunert wrote:
I'm not sure If this kind of adaptions to the package are the right way to solve the configuration problems, so I would propose an additional change for the next upcoming major version of CRI-O: As far as I understood was the problem that CRI-O does not start if there is a CNI plugin directory configured which does not exist.
Let's say it this way: configuring standard directories, which do not exist, is always a bad idea. This could even become a security problem.
Other projects like Cilium would run as DeamonSets on Kubernetes where CRI-O surely has to be started before. To fix this, I changed the upstream behavior for the CNI plugin directory validation that not existing directories will be created automatically by CRI-O [1]. This makes the following lines in the spec obsolete for the next major (v1.15.0) release:
``` %post %service_add_post %{name_source1} # This is the additional directory where cri-o is going to look up for CNI # plugins installed by DaemonSets running on Kubernetes (i.e. Cilium). mkdir -p /opt/cni/bin
I would prefer to fix it "correct": Add /opt/cni/bin as directory to the filelist of the "right" RPM, whichever this is. Or use systemd-tmpfiles to create it. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
On Thu, Apr 25, 2019 at 10:10:26PM +0200, Thorsten Kukuk wrote:
Other projects like Cilium would run as DeamonSets on Kubernetes where CRI-O surely has to be started before. To fix this, I changed the upstream behavior for the CNI plugin directory validation that not existing directories will be created automatically by CRI-O [1]. This makes the following lines in the spec obsolete for the next major (v1.15.0) release:
``` %post %service_add_post %{name_source1} # This is the additional directory where cri-o is going to look up for CNI # plugins installed by DaemonSets running on Kubernetes (i.e. Cilium). mkdir -p /opt/cni/bin
I would prefer to fix it "correct": Add /opt/cni/bin as directory to the filelist of the "right" RPM, whichever this is. Or use systemd-tmpfiles to create it.
/opt is not a part of the / ro BTRFS parition. Wouldn't be that a problem for transactional updates and RPM packages? Using systemd-tmpfiles is probably better. But Sascha already fixed the problem with his upstream patch, so I guess we can just remove the %post step with the next release of CRI-O and we don't need to think about anything else. Thanks Sascha! Cheers, Michal -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
On Fri, Apr 26, Michal Rostecki wrote:
On Thu, Apr 25, 2019 at 10:10:26PM +0200, Thorsten Kukuk wrote:
I would prefer to fix it "correct": Add /opt/cni/bin as directory to the filelist of the "right" RPM, whichever this is. Or use systemd-tmpfiles to create it.
/opt is not a part of the / ro BTRFS parition. Wouldn't be that a problem for transactional updates and RPM packages?
Directories are no problem. All informations about a directory are part of the RPM database, so we can create them at the next boot. In opposite to %pre/%post sections and files: here we don't have all informations in the RPM database.
Using systemd-tmpfiles is probably better.
It has pro and cons. At least it is more portable. Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany GF: Felix Imendoerffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nuernberg) -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
On Thu, Apr 18, 2019 at 02:25:27PM +0200, Michal Rostecki wrote:
I see two potential solutions:
1. Changing the cni-bin-dir parameter for Kubelet to `/usr/lib/cni,/opt/cni/bin` to support both CNI plugins installed on host and those installed dynamically. 2. Adding cilium-cni package to patterns-containers.
I forgot to mention - I'm more in favor of the 1st solution. If we decide to support more CNI plugins, especially the ones which might be heavier and pull dependencies - like CNI plugins supporting DPDK, VPP, SDNs (OpenDaylight, Tungsten Fabric etc.), having all of them on the node will be a pain. -- To unsubscribe, e-mail: opensuse-kubic+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kubic+owner@opensuse.org
participants (3)
-
Michal Rostecki -
Sascha Grunert -
Thorsten Kukuk