ArgoCD

This repository holds the basic k3s cluster installation

Getting started

Node installation

• Use OpenSUSE Micro
• Set up a loadbalancer
• Set up 3 control planes
• Set up 3 worker nodes

Source: https://blog.stonegarden.dev/articles/2024/02/bootstrapping-k3s-with-cilium/#bootstrapping-k3s

Gateway API: https://blog.stonegarden.dev/articles/2023/12/cilium-gateway-api/#infrastructure-provider

Loadbalancer installation

Install HAProxy and SELinux tools

transactional-update pkg install haproxy policycoreutils-python-utils
reboot

Configure HAproxy

In /etc/haproxy/haproxy.cfg

global
 log /dev/log local0 info
 maxconn 32768
 chroot /var/lib/haproxy
 user haproxy
 group haproxy
 daemon
 stats socket /run/haproxy/stats.sock user haproxy group haproxy mode 0640 level operator
 tune.bufsize 32768
 tune.ssl.default-dh-param 2048
 ssl-default-bind-ciphers ALL:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK:!RC4:!ADH:!LOW@STRENGTH
 log 127.0.0.1:514 local0 info
defaults
 log global
 mode http
 option log-health-checks
 option log-separate-errors
 option dontlog-normal
 option dontlognull
 option httplog
 option socket-stats
 retries 3
 option redispatch
 maxconn 10000
 timeout connect 5s
 timeout client 50s
 timeout server 450s
listen stats
 bind 0.0.0.0:80
 bind :::80 v6only
 stats enable
 stats uri /
 stats refresh 5s
#
# Kube API
#
frontend k3s-frontend-api
 bind *:6443
 mode tcp
 option tcplog
 default_backend k3s-backend-api
backend k3s-backend-api
 mode tcp
 balance roundrobin
 server k3s-cp1 <control-plane-ip>:6443 check
 server k3s-cp2 <control-plane-ip>:6443 check
 server k3s-cp3 <control-plane-ip>:6443 check
#
# Regular http/https traffic
#
frontend k3s-frontend-ingress
 bind *:443
 mode tcp
 option tcplog
 default_backend k3s-backend-ingress
backend k3s-backend-ingress
 mode tcp
 balance roundrobin
 server k3s-cilium <shared-ip>:443 check

Start and enable HAProxy

systemctl enable --now haproxy

If HAProxy crashes, you probably need to allow it to open a port on 6443 and connect to port 6443. You may need to run and apply this twice:

cat /var/log/audit/audit.log | grep AVC | grep denied | audit2allow -M haproxy
semodule -i haproxy.pp

First control-plane installation

Because we are using Cilium, install a very minimal version of k3s:

curl -sfL https://get.k3s.io | sh -s - \
 --flannel-backend=none \
 --disable-kube-proxy \
 --disable servicelb \
 --disable-network-policy \
 --disable traefik \
 --cluster-init

Wait for it all to complete, then:

cat > /etc/transactional-update.conf << "EOF"
REBOOT_METHOD=kured
EOF
reboot

After this has been done, also create a config file and add the loadbalancer IP as SAN:

/etc/rancher/k3s/config.yaml:

tls-san:- "192.168.122.207"- "k3s.intern.kn0x.org"flannel-backend:"none"disable-kube-proxy:truedisable-network-policy:truecluster-init:truedisable:- servicelb- traefik

You can now get the KUBECONFIG from /etc/rancher/k3s/k3s.yaml and integrate it into your own ~/.kube/config

Install Cilium

Ensure Helm is installed, I do this from my local machine but you could do it from the control plane as well.

helm repo add cilium https://helm.cilium.io/
helm repo update

Check if the LoadBalancer IP is correct in applications/cilium/values.yaml. If not, edit it.

Then install Cilium using the value files we have present:

kubectl create ns cilium
helm install cilium cilium/cilium -f applications/cilium/values.yaml -n cilium

Gateway API CRDs

Then apply the Gateway API CRDS

for crd in applications/gateway-api/templates/*.yaml; do
 kubectl apply -f $file -n cilium
done

Cilium command-line

On the controlplane, k3s-cp1, install Cilium and check the status:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz

Then check the installation of Cilium:

cilium -n cilium status --wait

Note: be sure to specify the namespace using -n cilium.

Your output should look something like this:

 / ̄ ̄\
 / ̄ ̄\__/ ̄ ̄\ Cilium: OK
 \__/ ̄ ̄\__/ Operator: OK
 / ̄ ̄\__/ ̄ ̄\ Envoy DaemonSet: OK
 \__/ ̄ ̄\__/ Hubble Relay: disabled
 \__/ ClusterMesh: disabled
DaemonSet cilium Desired: 6, Ready: 6/6, Available: 6/6
DaemonSet cilium-envoy Desired: 6, Ready: 6/6, Available: 6/6
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 6
 cilium-envoy Running: 6
 cilium-operator Running: 1
 clustermesh-apiserver
 hubble-relay
Cluster Pods: 3/3 managed by Cilium
Helm chart version: 1.17.3
Image versions cilium quay.io/cilium/cilium:v1.17.3@sha256:1782794aeac951af139315c10eff34050aa7579c12827ee9ec376bb719b82873: 6
 cilium-envoy quay.io/cilium/cilium-envoy:v1.32.5-1744305768-f9ddca7dcd91f7ca25a505560e655c47d3dec2cf@sha256:a01cadf7974409b5c5c92ace3d6afa298408468ca24cab1cb413c04f89d3d1f9: 6
 cilium-operator quay.io/cilium/operator-generic:v1.17.3@sha256:8bd38d0e97a955b2d725929d60df09d712fb62b60b930551a29abac2dd92e597: 1

Adding additional control-plane nodes:

Get the join token on k3s-cp1 from /var/lib/rancher/k3s/server/token.

Then, on the other control-plane nodes:

K3S_TOKEN=<TOKEN>
API_SERVER_IP=<LOAD-BALANCER-IP>
API_SERVER_PORT=6443
curl -sfL https://get.k3s.io | sh -s - server \
 --token ${K3S_TOKEN} \
 --server "https://${API_SERVER_IP}:${API_SERVER_PORT}" \
 --flannel-backend=none \
 --disable-kube-proxy \
 --disable servicelb \
 --disable-network-policy \
 --disable traefik

Wait for it all to complete, then:

cat > /etc/transactional-update.conf << "EOF"
REBOOT_METHOD=kured
EOF
reboot

Adding additional agents

Get the join token on k3s-cp1 from /var/lib/rancher/k3s/server/token.

Then, on the worker nodes:

K3S_TOKEN=<TOKEN>
API_SERVER_IP=<LOAD-BALANCER-IP>
API_SERVER_PORT=6443
curl -sfL https://get.k3s.io | sh -s - agent \
 --token "${K3S_TOKEN}" \
 --server "https://${API_SERVER_IP}:${API_SERVER_PORT}"

Wait for it all to complete, then:

cat > /etc/transactional-update.conf << "EOF"
REBOOT_METHOD=kured
EOF
reboot

Getting the cluster up and running

Now we'll get the cluster up and running by installing Bitnami Sealed Secrets and ArgoCD

Bitnami Sealed Secrets

Add the helm repo:

helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets
helm repo update

Install sealed secrets:

kubectl create ns sealed-secrets
helm install sealed-secrets --include-crds -n "sealed-secrets" -f "applications/sealed-secrets/values.yaml" sealed-secrets/sealed-secrets

Install the kubeseal CLI tool and alias

VERSION="$(curl -sL -H 'Accept: application/vnd.github+json' \
 'https://api.github.com/repos/bitnami-labs/sealed-secrets/releases/latest' | \
 jq -r '.tag_name')"

cert-manager

helm install cert-manager cert-manager/cert-manager --namespace cert-manager --create-namespace -f applications/cert-manager/values.yaml

System Upgrade Controller

Follow official k3s docs for system upgrade controller