Cilium installation

Question

I'm trying to install Cilium in Windows 10 with these configuration:

• minikube version: v1.26.0
• Kubectl version:

Client Version: version.Info { Major: "1", Minor: "24", 
 GitVersion: "v1.24.1", 
 GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", 
 GitTreeState: "clean", 
 BuildDate: "2022年05月24日T12:26:19Z", 
 GoVersion: "go1.18.2", 
 Compiler: "gc", 
 Platform: "windows/amd64" }
Kustomize Version: v4.5.4

• Cilium version:

cilium-cli: v0.11.11 compiled with go1.18.3 on windows/amd64
cilium image (default): v1.11.6

When I do

minikube start --network-plugin=cni --cni=false

And then download the latest Cilium version:

Cilium -> download cilium-windows-amd64.tar.gz
Release v0.11.11 Latest

And then install Cilium:

cilium install

this error appears

🔮 Auto-detected Kubernetes kind: minikube
✨ Running "minikube" validation checks
✅ Detected minikube version "1.26.0"
i️ Using Cilium version 1.11.6
🔮 Auto-detected cluster name: minikube
🔮 Auto-detected datapath mode: tunnel
i️ helm template --namespace kube-system cilium cilium/cilium --version 1.11.6 --set cluster.id=0,cluster.name=minikube,encryption.nodeEncryption=false,kubeProxyReplacement=disabled,operator.replicas=1,serviceAccounts.cilium.name=cilium,serviceAccounts.operator.name=cilium-operator,tunnel=vxlan
i️ Storing helm values file in kube-system/cilium-cli-helm-values Secret
🔑 Created CA in secret cilium-ca
🔑 Generating certificates for Hubble...
🚀 Creating Service accounts...
🚀 Creating Cluster roles...
🚀 Creating ConfigMap for Cilium version 1.11.6...
🚀 Creating Agent DaemonSet...
level=warning msg="spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use \"kubernetes.io/os\" instead" subsys=klog
🚀 Creating Operator Deployment...
⌛ Waiting for Cilium to be installed and ready...
 / ̄ ̄\
 / ̄ ̄\__/ ̄ ̄\ Cilium: 2 errors
 \__/ ̄ ̄\__/ Operator: OK
 / ̄ ̄\__/ ̄ ̄\ Hubble: disabled
 \__/ ̄ ̄\__/ ClusterMesh: disabled
 \__/
DaemonSet cilium Desired: 1, Unavailable: 1/1
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 1
Cluster Pods: 0/0 managed by Cilium
Image versions cilium quay.io/cilium/cilium:v1.11.6@sha256:f7f93c26739b6641a3fa3d76b1e1605b15989f25d06625260099e01c8243f54c: 1
Errors: cilium cilium 1 pods of DaemonSet cilium are not ready
 cilium cilium-d2jp4 unable to retrieve cilium status: unable to upgrade connection: container not found ("cilium-agent")
↩️ Rolling back installation...
Error: Unable to install Cilium: timeout while waiting for status to become successful: context deadline exceeded

If I try to do

minikube start --network-plugin=cni --cni=cilium

The same problem occurs. Cilium status:

←[33m / ̄ ̄\
←[36m / ̄ ̄←[33m\__/←[32m ̄ ̄\←[0m Cilium: ←[31m2 errors←[0m
←[36m \__←[31m/ ̄ ̄\←[32m__/←[0m Operator: ←[32mOK←[0m
←[32m / ̄ ̄←[31m\__/←[35m ̄ ̄\←[0m Hubble: ←[36mdisabled←[0m
←[32m \__←[34m/ ̄ ̄\←[35m__/←[0m ClusterMesh: ←[36mdisabled←[0m
←[34m \__/
←[0m
Deployment cilium-operator Desired: 1, Ready: ←[32m1/1←[0m, Available: ←[32m1/1←[0m
DaemonSet cilium Desired: 1, Unavailable: ←[31m1/1←[0m
Containers: cilium Running: ←[32m1←[0m
 cilium-operator Running: ←[32m1←[0m
Cluster Pods: 0/1 managed by Cilium
Image versions cilium quay.io/cilium/cilium:v1.9.9@sha256:a85d5cff13f8231c2e267d9fc3c6e43d24be4a75dac9f641c11ec46e7f17624d: 1
 cilium-operator quay.io/cilium/operator-generic:v1.9.9@sha256:3726a965cd960295ca3c5e7f2b543c02096c0912c6652eb8bbb9ce54bcaa99d8: 1
Errors: cilium cilium-5pjc5 unable to retrieve cilium status: unable to upgrade connection: container not found ("cilium-agent")
 cilium cilium 1 pods of DaemonSet cilium are not ready

These are the logs extracted from Lens:

• cilium-5pjc5 container logs:

level=info msg="Endpoints restored" failed=0 restored=0 subsys=daemon
level=info msg="Addressing information:" subsys=daemon
level=info msg=" Cluster-Name: default" subsys=daemon
level=info msg=" Cluster-ID: 0" subsys=daemon
level=info msg=" Local node-name: minikube" subsys=daemon
level=info msg=" Node-IPv6: <nil>" subsys=daemon
level=info msg=" External-Node IPv4: 192.168.49.2" subsys=daemon
level=info msg=" Internal-Node IPv4: 10.244.0.199" subsys=daemon
level=info msg=" IPv4 allocation prefix: 10.244.0.0/24" subsys=daemon
level=info msg=" Loopback IPv4: 169.254.42.1" subsys=daemon
level=info msg=" Local IPv4 addresses:" subsys=daemon
level=info msg=" - 192.168.49.2" subsys=daemon
level=info msg=" - 10.244.0.199" subsys=daemon
level=info msg="Creating or updating CiliumNode resource" node=minikube subsys=nodediscovery
level=info msg="Adding local node to cluster" node="{minikube default [{InternalIP 192.168.49.2} {CiliumInternalIP 10.244.0.199}] 10.244.0.0/24 <nil> 10.244.0.122 <nil> 0 local 0 map[beta.kubernetes.io/arch:amd64 beta.kubernetes.io/os:linux kubernetes.io/arch:amd64 kubernetes.io/hostname:minikube kubernetes.io/os:linux minikube.k8s.io/commit:f4b412861bb746be73053c9f6d2895f12cf78565 minikube.k8s.io/name:minikube minikube.k8s.io/primary:true minikube.k8s.io/updated_at:2022_07_10T12_53_48_0700 minikube.k8s.io/version:v1.26.0 node-role.kubernetes.io/control-plane: node.kubernetes.io/exclude-from-external-load-balancers:] 6}" subsys=nodediscovery
level=info msg="Annotating k8s node" subsys=daemon v4CiliumHostIP.IPv4=10.244.0.199 v4Prefix=10.244.0.0/24 v4healthIP.IPv4=10.244.0.122 v6CiliumHostIP.IPv6="<nil>" v6Prefix="<nil>" v6healthIP.IPv6="<nil>"
level=info msg="Initializing identity allocator" subsys=identity-cache
level=info msg="Cluster-ID is not specified, skipping ClusterMesh initialization" subsys=daemon
level=info msg="Setting up BPF datapath" bpfClockSource=jiffies bpfInsnSet=v3 subsys=datapath-loader
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
level=warning msg="Failed to sysctl -w" error="could not open the sysctl file /proc/sys/net/core/bpf_jit_enable: open /proc/sys/net/core/bpf_jit_enable: no such file or directory" subsys=datapath-loader sysParamName=net.core.bpf_jit_enable sysParamValue=1
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=net.ipv4.conf.all.rp_filter sysParamValue=0
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.unprivileged_bpf_disabled sysParamValue=1
level=info msg="Setting sysctl" subsys=datapath-loader sysParamName=kernel.timer_migration sysParamValue=0
level=info msg="All pre-existing resources related to policy have been received; continuing" subsys=k8s-watcher
level=error msg="Command execution failed" cmd="[iptables -w 5 -t raw -A CILIUM_PRE_raw -m mark --mark 0x00000200/0x00000f00 -m comment --comment cilium: NOTRACK for proxy traffic -j NOTRACK]" error="exit status 2" subsys=iptables
level=warning msg="iptables v1.8.4 (legacy): Couldn't load target `NOTRACK':No such file or directory" subsys=iptables
level=warning subsys=iptables
level=warning msg="Try `iptables -h' or 'iptables --help' for more information." subsys=iptables
level=error msg="Error while initializing daemon" error="cannot add static proxy rules: exit status 2" subsys=daemon
level=fatal msg="Error while creating daemon" error="cannot add static proxy rules: exit status 2" subsys=daemon

• coredns-6d4b75cb6d-wvlmp container logs:

Failed to load logs: container "coredns" in pod "coredns-6d4b75cb6d-wvlmp" is waiting to start: ContainerCreating
Reason: BadRequest (400)

What can be happen

Answer 1

You need to change the minikube start command to enable the cni flag. Like this:

minikube start --network-plugin=cni --cni=cilium

This will enable the cilium networking plugin in minikube allowing you to install the cilium components.

• Daniel: The same problem when I do a Cilium status. I have added these steps in my question with your suggestion also
• zer0: Ah, seems like you need to mount the bpf filesystem as well. You can do it this way: minikube ssh -- sudo mount bpffs -t bpf /sys/fs/bpf

Answer 2

Finally, I could launch it doing this:

minikube start --network-plugin=cni --cni=false --driver=hyperv