Setup a Kubernetes HomeLab on MacBook/iMac (M1)

Overview

What's my Rig for this tutorial and Recommendation

Tools used to Deploy the K8s Cluster

Additional topics in this tutorial: Install Cilium, Hubble, Istio Service Mesh, Security Benchmark Tools

Is there single push Automated Script?

Installation Steps for a Multi-node Kubernetes Cluster

A] Download and setup Kubespray

📝 We will use the iMac/Macbook as the Ansible control node.

💡 Set the desired environment parameters.

cat > ~/.k8slab_env <<EOF
# Set desired Project Home
export PROJECT_HOME=$HOME/Projects/k8s
# No Change
export MULTIPASS_KEY=$HOME/.ssh/multipass.key
# Set Ubuntu version. 
export UBUNTU_REL=22.10
# Set desired number of nodes for multi-node K8s cluster. Set 2 if 8GB RAM.
# Best Practice: Odd number to prevent Split Brain 
NODE_COUNT=2
# Desired multipass node name prefix
export NODE_PREFIX=k8snode
# Compute for the Virtual nodes
CPU=2
MEM=3G
# Disk can be expanded if required but can't be shrinked
DISK=10G
# Set stable K8 version
export K8S_VERSION=v1.27.2
# Set Desired cluster name. Avoid using "." in cluster name for eg homelab.local. Cilium installation will fail.
export CLUSTER_NAME=homelab
EOF
source ~/.k8slab_env

📝 Install Multipass and copy the SSH private key to home directory of current Mac user.

brew install --cask multipass

sudo cp /var/root/Library/Application\ Support/multipassd/ssh-keys/id_rsa $MULTIPASS_KEY && sudo chown $USER $MULTIPASS_KEY

📝 Create the virtual nodes

count=1
until [ $count -gt $NODE_COUNT ]
do
 multipass launch $UBUNTU_REL -n "$NODE_PREFIX$count" -c $CPU -m $MEM -d $DISK
 ((count=count+1))
done 

📝 List the virtual nodes

multipass list

📝 Download Kubespray and install the pre-requisite packages with python-pip

💡 Python may already exist on MacOS

brew list python

📝 ...but if not then install it

brew install python3
brew postinstall python3

📝 Create Project home

mkdir -p $PROJECT_HOME && cd $PROJECT_HOME

📝 Download Kubespray

git clone https://github.com/kubernetes-sigs/kubespray.git && cd kubespray 

📝 Install dependency packages

pip3 install -r requirements.txt 

📝 Refresh the login shell environment

bash

📝 Set the Ansible Private key and Playbook log file

cd $PROJECT_HOME/kubespray
sed -i '' '/^\[defaults\]/a \
private_key_file = '"$MULTIPASS_KEY"' \
' ansible.cfg
sed -i '' '/^\[defaults\]/a \
log_path = '"$PROJECT_HOME"'/kubespray/playbook.log \
' ansible.cfg

📝 Generate the Ansible Host Inventory

cd $PROJECT_HOME/kubespray && \
cp -r inventory/sample inventory/k8cluster && \
declare -a IPS=( $(multipass list --format csv |tail -n +2 |cut -d "," -f3 | xargs) ) && \
CONFIG_FILE=inventory/k8cluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]}

📝 Update the Ubuntu nodes with Ansible play

cat > initialsetup.yml <<EOF
- hosts: all
 become: true
 tasks: 
 - name: Update and upgrade apt packages
 apt:
 upgrade: yes
 update_cache: yes
 - name: Install pip, net-tools 
 apt:
 name: 
 - pip
 - net-tools
 state: present
 update_cache: true
EOF

ansible-playbook -i inventory/k8cluster/hosts.yml ./initialsetup.yml -e ansible_user=ubuntu

B] Cluster Hardening Profile (OPTIONAL for StudyLab)

📝 Create the hardening yaml

C] Deploy the Kubernetes Cluster

📝 Backup original files

cd $PROJECT_HOME/kubespray/inventory/k8cluster/group_vars/k8s_cluster && cp k8s-cluster.yml k8s-cluster.yml.BAK ; cp addons.yml addons.yml.BAK

📝 Set K8s version

sed -i '' "s~^kube_version:.*~kube_version: $K8S_VERSION~g" k8s-cluster.yml

📝 Set cluster name

sed -i '' "s~^cluster_name:.*~cluster_name: $CLUSTER_NAME~g" k8s-cluster.yml

📝 Set network plugin to cni for now. We will install Cilium separately to be more interactive with the plugin.

sed -i '' 's~^kube_network_plugin:.*~kube_network_plugin: cni~g' k8s-cluster.yml

📝 Enable encryption of secret data at rest in etcd

sed -i '' 's~^kube_encrypt_secret_data:.*~kube_encrypt_secret_data: true~g' k8s-cluster.yml

📝 Install Kubernetes dashboard

sed -i '' "s~^# dashboard_enabled:.*~dashboard_enabled: true~g" addons.yml

📝 Install Helm client

sed -i '' "s~^helm_enabled:.*~helm_enabled: true~g" addons.yml

📝 Install metrics server

sed -i '' "s~^metrics_server_enabled:.*~metrics_server_enabled: true~g" addons.yml

📝 Enable NTP sync

cd $PROJECT_HOME/kubespray/inventory/k8cluster/group_vars/all && cp all.yml all.yml.BAK && cp etcd.yml etcd.yml.BAK
sed -i '' "s~^ntp_enabled:.*~ntp_enabled: true~g" all.yml 

📝 Deploy etcd with kubeadm

cd $PROJECT_HOME/kubespray/inventory/k8cluster/group_vars/all && cp etcd.yml etcd.yml.BAK
sed -i '' "s~^etcd_deployment_type:.*~etcd_deployment_type: kubeadm~g" etcd.yml 

📝 Run the deployment to create StudyLab

cd $PROJECT_HOME/kubespray && ansible-playbook -i ./inventory/k8cluster/hosts.yml ./cluster.yml -e ansible_user=ubuntu -b --become-user=root 

📝 Run the deployment with Hardening Profile

⚠️ Skip if previous option executed

cd $PROJECT_HOME/kubespray && ansible-playbook -i ./inventory/k8cluster/hosts.yml ./cluster.yml -e "@hardening.yml" -e ansible_user=ubuntu -b --become-user=root 

📝 Expected Output

PLAY RECAP 
localhost : ok=3 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 
node1 : ok=709 changed=144 unreachable=0 failed=0 skipped=1204 rescued=0 ignored=5 
node2 : ok=607 changed=115 unreachable=0 failed=0 skipped=1055 rescued=0 ignored=2 
kubernetes/control-plane : Joining control plane node to the cluster.
16.50s
download : download_container | Download image if required
11.64s
kubernetes/control-plane : kubeadm | Initialize first master
8.56s
etcd : reload etcd
8.17s
download : download_file | Download item
7.27s
kubernetes/preinstall : Ensure NTP package
6.57s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources
6.26s
kubernetes/preinstall : Install packages requirements
5.88s
download : download_container | Download image if required
5.84s
download : download_container | Download image if required
5.78s
etcd : Configure | Check if etcd cluster is healthy
5.38s
container-engine/crictl : download_file | Download item
5.36s
download : download_container | Download image if required
5.27s
download : download_container | Download image if required
5.24s
etcd : Configure | Ensure etcd is running
5.22s
download : download_container | Download image if required
4.75s
download : download_file | Download item
4.41s
kubernetes-apps/metrics_server : Metrics Server | Apply manifests
4.31s
container-engine/containerd : download_file | Download item
4.18s
download : download_container | Download image if required
3.72s

C] Setup CLI, Access the Cluster & Label the Nodes

📝 Connect to the first node

multipass shell "$NODE_PREFIX"1

📝 Create kubectl shortcut aliases

cat >> ~/.bashrc <<EOF
alias k="kubectl"
alias ksn="kubectl config set-context --current --namespace"
EOF
bash

📝 Copy the cluster admin config

mkdir .kube && sudo cp /etc/kubernetes/admin.conf .kube/config && sudo chown ubuntu .kube/config

📝 List the nodes and Label worker nodes as per your preference.

for i in $(k get nodes -o custom-columns=NAME:.metadata.name --no-headers)
do
 k label node $i node-role.kubernetes.io/worker=worker 
done
k get nodes

D] Remove kube-proxy and Install Cilium CNI Plugin

📝 Delete the daemonset and configmap

ksn kube-system && k delete ds kube-proxy && k delete cm kube-proxy

📝 Dump and Restore the iptables rules to remove the Kube proxy rules

💡 Run on all nodes from host ansible control node

cd $PROJECT_HOME/kubespray && \
ansible all -m shell -a "sudo iptables-save | grep -v KUBE | sudo iptables-restore" -i ./inventory/k8cluster/hosts.yml -e ansible_user=ubuntu 

📝 Connect back to the K8s node1

multipass shell "$NODE_PREFIX"1

📝 Download Cilium CLI

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/master/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

📝 Install cilium with Kube-proxy replacement set to strict

cilium install -n kube-system --version=v1.13.3 \
 --helm-set k8sServiceHost=localhost \
 --helm-set k8sServicePort=6443 \
 --helm-set kubeProxyReplacement=strict 

📝 Install hubble

cilium hubble enable --ui \
 --helm-set hubble.ui.enabled=true \
 --helm-set hubble.ui.service.type=NodePort \
 --helm-set hubble.relay.service.type=NodePort \
 --helm-set hubble.peerService.clusterDomain=homelab

📝 Verify Cilium pods are running

k -n kube-system get pods -l app.kubernetes.io/part-of=cilium -w

📝 Check cilium status

cilium status

📝 Output should look like below

 / ̄ ̄\
 / ̄ ̄\__/ ̄ ̄\ Cilium: OK
 \__/ ̄ ̄\__/ Operator: OK
 / ̄ ̄\__/ ̄ ̄\ Envoy DaemonSet: disabled (using embedded mode)
 \__/ ̄ ̄\__/ Hubble Relay: OK
 \__/ ClusterMesh: disabled

📝 Verify kube proxy replacement is set to strict and list the service map table

CILIUM_POD=$(k -n kube-system get pods -l k8s-app=cilium -o custom-columns=NAME:.metadata.name --no-headers | head -1)
k exec -it $CILIUM_POD -- cilium status | grep KubeProxyReplacement
k exec -it $CILIUM_POD -- cilium service list

📝 Verify Nodes are in Ready state

k get nodes

E] Access the cluster with Kubernetes Dashboard

📝 Create service account

ksn kube-system && k create sa dashboard 

📝 Assign cluster-admin role to the service account

k create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard

📝 Create the secret object of type token for the service account

k apply -f - <<EOF
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
 name: dashboard
 annotations:
 kubernetes.io/service-account.name: dashboard
EOF

📝 Expose the dashboard service as NodePort to access from outside the cluster network.

k patch svc kubernetes-dashboard --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
export DASHBOARD_PORT=$(k get svc kubernetes-dashboard -o jsonpath='{.spec.ports[].nodePort}')
export DASHBOARD_HOST=$(k get po -l k8s-app=kubernetes-dashboard -o jsonpath='{.items[0].status.hostIP}')

📝 Open the URL in your browser

echo https://$DASHBOARD_HOST:$DASHBOARD_PORT

📝 Login with the generated Token

k get secret dashboard -o jsonpath="{.data.token}" | base64 --decode

F] Install Hubble - Network, Service & Security Observability for Kubernetes

📝 Download Hubble

export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

📝 Verify Hubble is working through CLI

cilium hubble port-forward & 
sleep 5
hubble status

📝 Expected Output

Healthcheck (via localhost:4245): Ok
Current/Max Flows: 4,065/8,190 (49.63%)
Flows/s: 40.13
Connected Nodes: 2/2

📝 Verify Hubble is working through UI dashboard

export HUBBLE_PORT=$(k get svc hubble-ui -o jsonpath='{.spec.ports[].nodePort}')
export HUBBLE_HOST=$(k get po -l k8s-app=hubble-ui -o jsonpath='{.items[0].status.hostIP}')
echo "http://$HUBBLE_HOST:$HUBBLE_PORT/?namespace=kube-system"

Installation Steps for ISTIO Service Mesh

A] Install Istio

⚠️ Follow below steps for a Hardened cluster with POD Security Admission Controller and baseline policy.

curl -L https://istio.io/downloadIstio | sh - && \
cd istio-* && \
export PATH=$PWD/bin:$PATH && \
istioctl install --set components.cni.enabled=true -y

📝 Verify Istio components are running

ksn istio-system && k get pods 

⚠️ Follow below Helm steps for non-Hardened regular Cluster setup.

📝 Install the Istio Helm repo

helm repo add istio https://istio-release.storage.googleapis.com/charts && \
helm repo update 

📝 Istio components

helm install istio-base istio/base -n istio-system --create-namespace

📝 This will take few seconds to minutes

helm install istiod istio/istiod -n istio-system --wait

📝 Install the gateway

helm install istio-ingressgateway istio/gateway -n istio-system

📝 Verify Istio components are running

ksn istio-system && k get pods 

B] Install Addons: Kiali, Prometheus & Grafana

📝 Install the add-ons

k apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/kiali.yaml
k apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/prometheus.yaml
k apply -f https://raw.githubusercontent.com/istio/istio/master/samples/addons/grafana.yaml

📝 Verify the deployment is ready

k get deploy -n istio-system -o wide

📝 Change the service type to NodePort to make it accessible from the Host Laptop/Desktop

k patch svc istio-ingressgateway --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
k patch svc kiali --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
k patch svc prometheus --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'
k patch svc grafana --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"}]'

C] Install the sample Booking Micro-service App to test the Mesh

📝 Create the namespace and label for the sidecar injection

k create ns servicemesh && \
k label namespace servicemesh istio-injection=enabled && \
ksn servicemesh

⚠️ Deploy the Pod Security Admission compatibe sample Booking App for Hardened Cluster

k apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo-psa.yaml

⚠️ Deploy the regular sample Booking App for non-Hardened Cluster

k apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml

📝 Verify Booking App pod is running

k get pods

📝 Install the Booking App gateway (Required to expose the Booking app externally irrespective of a Hardened cluster)

k apply -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/networking/bookinfo-gateway.yaml

📝 Trigger synthetic load on the URL to generate the traffic graphs in Kiali and monitoring data in Grafana

for i in $(seq 1 100); do curl -s -o /dev/null "http://$BOOK_APP_URL/productpage";done

📝 Generate the External URLs

export KIALI_PORT=$(k -n istio-system get svc kiali -o jsonpath='{.spec.ports[?(@.name=="http")].nodePort}')
export PROM_PORT=$(k -n istio-system get svc prometheus -o jsonpath='{.spec.ports[].nodePort}')
export GRAFANA_PORT=$(k -n istio-system get svc grafana -o jsonpath='{.spec.ports[].nodePort}')
export BOOK_APP_PORT=$(k -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
export INGRESS_HOST=$(k get po -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}')
export BOOK_APP_URL=$INGRESS_HOST:$BOOK_APP_PORT
export KIALI_URL=$INGRESS_HOST:$KIALI_PORT
export PROMETHEUS_URL=$INGRESS_HOST:$PROM_PORT
export GRAFANA_URL=$INGRESS_HOST:$GRAFANA_PORT

echo "Booking App: http://$BOOK_APP_URL/productpage"

echo "Kiali Dashboard: http://$KIALI_URL"

echo "Prometheus Dashboard: http://$PROMETHEUS_URL"

echo "Grafana Dashboard: http://$GRAFANA_URL"

D] Uninstall the Booking App if needed.

ksn servicemesh && \
k delete -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo.yaml && \
k delete -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/platform/kube/bookinfo-psa.yaml && \
k delete -f https://raw.githubusercontent.com/istio/istio/master/samples/bookinfo/networking/bookinfo-gateway.yaml

Run CIS Kubernetes Benchmarks to secure the cluster

📝 Download latest kube-bench tool

cd ~ && \
curl -s \
 -H "Accept: application/vnd.github+json" \
 -H "X-GitHub-Api-Version: 2022年11月28日" \
 https://api.github.com/repos/aquasecurity/kube-bench/releases/latest |grep "browser_download_url.*arm64.deb" |
 cut -d : -f 2,3 |
 tr -d \" |
 wget -i - 

📝 Install kube-bench tool

cd ~ && \
KBENCH_NAME=`curl -s \
 -H "Accept: application/vnd.github+json" \
 -H "X-GitHub-Api-Version: 2022年11月28日" \
 https://api.github.com/repos/aquasecurity/kube-bench/releases/latest |grep -o "name.*arm64.deb" |
 cut -d : -f 2,3 |
 tr -d " \""` && \
sudo apt -y install ./$KBENCH_NAME

📝 Run the CIS benchmark with Kube-Bench and remediate any CRITICAL,HIGH findings

sudo kube-bench | tee kube-bench-results.txt

📝 Download & Install kubescape tool

curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash && \
echo "export PATH=\$PATH:/home/ubuntu/.kubescape/bin" >> .bashrc && \
bash

📝 Run the CIS benchmark with KubeScape and remediate any CRITICAL,HIGH findings

kubescape scan framework CIS --enable-host-scan -e kubescape -v | tee results.txt

Uninstall / Reset the Setup

📝 Uninstall Istio

istioctl uninstall --purge -y
k delete ns istio-system
k delete ns servicemesh
# If Installed with Helm Chart
helm delete -n istio-system $(helm ls -n istio-system --short | xargs) 

📝 Reset the cluster

cd $PROJECT_HOME/kubespray && ansible-playbook -i ./inventory/k8cluster/hosts.yml ./reset.yml -e ansible_user=ubuntu -b --become-user=root 

📝 Delete and Purge the Virtual Nodes

source ~/.k8slab_env
multipass delete $(multipass list --format csv |tail -n +2 |grep $NODE_PREFIX | cut -d "," -f1 | xargs)
multipass purge 

Feedback

Hopefully this Guide was resourceful, informative and easier to follow. In upcoming tutorials, we will look at ArgoCD, CrossPlane and Auto cluster node scaling with builtin controller as well as Karpenter tool.

If you have any questions, Ask here Discussions or message me on LinkedIn