Kubernetes Cluster with Proxmox VMs

With relatively little effort, K3s lets you set up a highly available Kubernetes cluster on Proxmox Virtual Environment (or similar). K3s is a lightweight Kubernetes distribution from Rancher that is especially well suited for edge computing, IoT, and homelab environments. By using three master nodes in an HA setup with embedded etcd, an HAProxy load balancer, and several worker nodes, you get a production-like cluster architecture that provides fault tolerance.

It’s advisable to set up HA for the master nodes right from the start using --cluster-init; adding this feature later is only possible via a backup/restore.

VM requirements (minimum):

HAProxy: 1 vCPU, 1 GB RAM, 10 GB disk
Master nodes: 2 vCPUs, 4 GB RAM, 20 GB disk
Worker nodes: 2 vCPUs, 2 GB RAM, 20 GB disk
OS: Debian 10 (or comparable)

Network:

Static IPs for all nodes
Open ports between the nodes

Step 1: Create VMs in Proxmox

Create VMs (HAProxy, 3 masters, 3 workers)
Cloud-Init makes it easier to assign static IPs
Install Debian
Updates:

sudo apt update && sudo apt upgrade -y

Step 2: Install HAProxy We use HAProxy to distribute traffic across the master nodes. Thanks to the cluster node setup, they become equivalent, since the ==embedded etcd== is synchronized automatically. This is the simplest supported HA architecture; alternatively, an external database (PostgreSQL/MySQL) could be used.

sudo apt install -y haproxy

MASTER1_IP="192.168.201.1"
MASTER2_IP="192.168.201.2"
MASTER3_IP="192.168.201.3"

cat <<EOF >> /etc/haproxy/haproxy.cfg

frontend k3s-api
    bind *:6443
    mode tcp
    default_backend k3s-masters
backend k3s-masters
    mode tcp
    balance roundrobin
    server master1 ${MASTER1_IP}:6443 check
    server master2 ${MASTER2_IP}:6443 check
    server master3 ${MASTER3_IP}:6443 check

EOF

Check the configuration for syntax errors; the ==WARNING option httplog== can be ignored:

haproxy -c -f /etc/haproxy/haproxy.cfg

Apply the configuration:

sudo systemctl reload haproxy

Step 3: Install master node On the first master node, install K3s as master:

HAPROXY_IP="192.168.201.0"

curl -sfL https://get.k3s.io | sh -s - server \
  --cluster-init \
  --tls-san=${HAPROXY_IP}

Retrieve the node token for additional nodes:

sudo cat /var/lib/rancher/k3s/server/node-token

Also install Master2 and Master3 as master nodes:

MASTER1_IP="192.168.201.1"
NODE_TOKEN="K10cf9d2625abcde0a83a53bc4f02192f3fcb4cc8d730ef4e248e023251bbd08139::server:7ba73ceb7480ee96763c43554819976b"

curl -sfL https://get.k3s.io | sh -s - server \
  --server https://${MASTER1_IP}:6443 \
  --token ${NODE_TOKEN}

Step 4: Add worker nodes Install as follows on each worker node:

HAPROXY_IP="192.168.201.0"
NODE_TOKEN="K10cf9d2625abcde0a83a53bc4f02192f3fcb4cc8d730ef4e248e023251bbd08139::server:7ba73ceb7480ee96763c43554819976b"

curl -sfL https://get.k3s.io | K3S_URL=https://${HAPROXY_IP}:6443 K3S_TOKEN=${NODE_TOKEN} sh -

Step 5: Verify the cluster On the master:

sudo kubectl get nodes

The cluster is now ready for use.

kubectl is required for management; the setup on macOS is shown below:

MASTER1_IP="192.168.201.1"
brew install kubectl

mkdir ~/.kube

scp root@${MASTER1_IP}:/etc/rancher/k3s/k3s.yaml ~/.kube/config

# Set permissions
sudo chown $USER:$USER ~/.kube/config
chmod 600 ~/.kube/config

# Adjust the IP, enter the HAProxy IP
code ~/.kube/config

kubectl get pods --all-namespaces

Load Balancer

Deploy MetalLB

METALLB_VERSION=$(curl -s https://api.github.com/repos/metallb/metallb/releases/latest | grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/"${METALLB_VERSION}"/config/manifests/metallb-native.yaml

Wait until the pods are running

kubectl wait --namespace metallb-system --for=condition=ready pod -l app=metallb

Configure the IP pool

IP_POOL="192.168.202.1-192.168.202.254"

cat <<EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
  - ${IP_POOL}
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
  - default-pool
EOF

Deploy a test application:

# 1. Create the Nginx deployment
kubectl create deployment nginx-test --image=nginx --replicas=2

# 2. Expose it as a service
kubectl expose deployment nginx-test --port=80 --type=LoadBalancer

# 3. Check status
kubectl get deployments
kubectl get pods -o wide
kubectl get svc nginx-test

Test it via browser or curl:

EXTERNAL_IP="192.168.202.2"

curl http://"${EXTERNAL_IP}"

Also, by design, the application is reachable on all IPs of the masters and agents, but only via NODEPORT there.

Remove the test application:

kubectl delete service nginx-test
kubectl delete deployment nginx-test

K8s K3s PVE