Configure a Highly Available Kubernetes Control Plane with kubeadm

A highly available Kubernetes control plane keeps the API reachable when one control-plane host fails. With kubeadm, that means a stable controlPlaneEndpoint behind a TCP load balancer, three stacked-etcd control-plane nodes, and joins where kubeadm init --upload-certs stores shared control-plane trust material in the temporary kubeadm-certs Secret. Each kubeadm join --control-plane downloads that material and starts local API server, scheduler, controller-manager, and etcd static Pods on the joining host.

What this HA cluster provides

The finished layout routes every client and node through one API address while three control-plane hosts each run a full control-plane stack:

API endpoint
            k8s-api.lab.local:6443
                         │
                 Virtual IP / LB
                 ┌───────┴───────┐
                 │               │
               lb-1            lb-2
              (MASTER)        (BACKUP)
                 │               │
                 └───────┬───────┘
                         │
       ┌─────────────────┼─────────────────┐
       │                 │                 │
      cp-1              cp-2              cp-3
   API server        API server        API server
   scheduler         scheduler         scheduler
   controller mgr    controller mgr    controller mgr
   etcd member       etcd member       etcd member

Five-VM lab layout:

Both load-balancer hosts must avoid the control-plane role — kube-apiserver already listens on TCP 6443 on every control-plane node. An external or cloud TCP load balancer can replace lb-1 and lb-2 when you already have one.

What high availability means here

High availability in this guide refers to the control plane, not every application Pod:

• Multiple API servers can receive requests through the shared endpoint.
• Scheduler and controller-manager replicas use leader election; only one replica is active at a time.
• Stacked etcd keeps quorum while two of three members remain healthy.
• Losing one control-plane node should not make the API unavailable when the load balancer and remaining members are healthy.
• Existing application Pods can continue running during a control-plane interruption because kubelet and container runtime on workers keep local containers alive.
• Application availability still depends on workload replicas, worker health, Services, storage, and application design.

Control-plane HA does not automatically make every application highly available.

Choose stacked or external etcd

kubeadm supports two production-style layouts. Both should use an odd number of etcd members for optimal quorum.

This article fully implements stacked etcd. The external-etcd section later shows the architectural and kubeadm configuration changes without repeating the full install walkthrough.

Plan the cluster before installation

Sketch hostnames, stable IPs, and the shared API address before you install packages. Every node must resolve the API DNS name to the load-balancer VIP, not to a single control-plane IP.

Prerequisites before you touch the load balancer or run kubeadm init:

• Unique hostnames on every node — set them with hostnamectl and map each IP in /etc/hosts:

192.168.56.108 cp-1 k8s-cp
192.168.56.109 lb-2
192.168.56.110 lb-1
192.168.56.112 cp-2
192.168.56.113 cp-3
192.168.56.200 k8s-api.lab.local

• Static or stable node addresses on the cluster network
• Time synchronization with timedatectl or chrony — certificate and join operations fail when clocks drift
• Open API server TCP 6443 from clients, nodes, and load balancers; etcd TCP 2379-2380 between control-plane members; and kubelet TCP 10250 where required between nodes and the control plane. The default scheduler and controller-manager ports 10259 and 10257 are self-access ports and do not need to be opened generally between hosts. Add any ports required by the selected CNI.
• Matching kubeadm, kubelet, and kubectl minor versions on every host
• Working containerd with CRI enabled
• Kernel modules overlay and br_netfilter, plus net.ipv4.ip_forward
• Outbound access to registry.k8s.io
• Full L3 connectivity between all node IPs

Node preparation — swap off, containerd, kernel settings, and Kubernetes packages — follows install Kubernetes with kubeadm. Run the linked Kubernetes node-preparation steps on all three control-plane hosts and on any optional worker nodes. The load-balancer hosts require only normal Linux networking, time synchronization, HAProxy, Keepalived, firewall configuration, and hostname resolution. Do not install kubelet, kubeadm, containerd, or a CNI on lb-1 and lb-2.

Do not convert an unsuitable existing cluster

The shared controlPlaneEndpoint must be configured during initial cluster creation. kubeadm does not support converting a single-control-plane cluster created without controlPlaneEndpoint into an HA cluster through a simple supported workflow. I reset my prior single-control-plane lab and built a fresh HA cluster for this walkthrough.

Configure the highly available API endpoint

If you already have an external or cloud TCP load balancer, point it at every control-plane API server on port 6443 and skip the HAProxy and Keepalived steps. The important part is that k8s-api.lab.local (or your chosen name) always reaches a healthy API server.

For this lab I used HAProxy and Keepalived on lb-1 (192.168.56.110, MASTER) and lb-2 (192.168.56.109, BACKUP). HAProxy runs in TCP mode and does not terminate Kubernetes API TLS — encrypted traffic passes through to the API servers.

Install packages on both load-balancer hosts

Run this on lb-1 and lb-2 before you create the HAProxy or Keepalived configuration:

sudo dnf install -y haproxy keepalived nmap-ncat psmisc

nmap-ncat supplies nc for endpoint tests. psmisc provides killall, which the Keepalived health script uses.

Allow HAProxy to bind the VIP on the BACKUP host

Both HAProxy configurations bind 192.168.56.200:6443. The BACKUP host does not normally own that VIP. HAProxy therefore cannot bind it unless non-local binding is enabled. Configure this on both load-balancer hosts before you start HAProxy:

cat <<'EOF' | sudo tee /etc/sysctl.d/99-haproxy-vip.conf
net.ipv4.ip_nonlocal_bind = 1
EOF

sudo sysctl --system

Enable net.ipv4.ip_nonlocal_bind=1 on both load-balancer hosts so HAProxy can remain running on the BACKUP while it does not own the VIP. Start and enable both HAProxy and Keepalived on both hosts. An alternative is bind :6443, but that also exposes HAProxy through each load balancer's normal node address.

Configure HAProxy backends

Install HAProxy on each load-balancer host. The front end listens on the VIP and port 6443; the back end lists every control-plane node:

Virtual endpoint :6443
├── cp-1:6443  (192.168.56.108)
├── cp-2:6443  (192.168.56.112)
└── cp-3:6443  (192.168.56.113)

On each load-balancer host, create /etc/haproxy/haproxy.cfg with the same contents:

global
    log /dev/log local0
    chroot /var/lib/haproxy
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode tcp
    option tcplog
    timeout connect 5s
    timeout client  50s
    timeout server  50s

frontend k8s-api
    bind 192.168.56.200:6443
    default_backend k8s-api-back

backend k8s-api-back
    balance roundrobin
    option tcp-check
    server cp-1 192.168.56.108:6443 check
    server cp-2 192.168.56.112:6443 check
    server cp-3 192.168.56.113:6443 check

HAProxy settings for this lab:

• TCP mode on port 6443 — do not terminate API TLS in the load balancer
• One backend entry per control-plane API server
• TCP health checks so failed backends stop receiving traffic

Validate the syntax before you enable the service:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg

Sample output:

Configuration file is valid

Open load-balancer firewall ports

When firewalld is active on the load-balancer hosts, open SSH, API forwarding, and VRRP on both lb-1 and lb-2:

sudo firewall-cmd --permanent --add-service=ssh

sudo firewall-cmd --permanent --add-port=6443/tcp

sudo firewall-cmd --permanent --add-rich-rule='rule protocol value="vrrp" accept'

sudo firewall-cmd --reload

VRRP traffic must pass between the Keepalived hosts. If VRRP is blocked, both nodes can incorrectly behave as MASTER or the VIP may never fail over.

For lab testing only, you can disable firewalld on both load-balancer hosts instead of opening individual ports:

sudo systemctl disable --now firewalld

Re-enable and configure firewalld before you move this pattern into production.

Configure the virtual IP

Install Keepalived on both load-balancer hosts. On lb-1 (MASTER), create /etc/keepalived/keepalived.conf:

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface enp0s8
    virtual_router_id 51
    priority 101
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass k8slab01
    }
    virtual_ipaddress {
        192.168.56.200/24
    }
    track_script {
        chk_haproxy
    }
}

On lb-2 (BACKUP), use the same file with only these changes:

state BACKUP
priority 100

Complete /etc/keepalived/keepalived.conf on lb-2:

vrrp_script chk_haproxy {
    script "killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s8
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass k8slab01
    }
    virtual_ipaddress {
        192.168.56.200/24
    }
    track_script {
        chk_haproxy
    }
}

Keepalived requires two hosts participating in the same VRRP instance for the VIP to move between them. Replace enp0s8 on both hosts with the interface that carries your cluster network.

Do not place either load balancer on a control-plane node. When the VIP moves to a host that already runs kube-apiserver on port 6443, HAProxy cannot bind the same port.

Enable and start both services on each load-balancer host with systemctl:

sudo systemctl enable --now haproxy keepalived

Verify HAProxy runs on both nodes while the VIP exists on only one:

sudo systemctl is-active haproxy keepalived

sudo ss -lntp | grep ':6443'

ip address show enp0s8 | grep 192.168.56.200

On the current MASTER (lb-1), sample output looks like this:

active
active
LISTEN 0 4096 192.168.56.200:6443 0.0.0.0:* users:(("haproxy",pid=999069,fd=9))
    inet 192.168.56.200/24 scope global secondary enp0s8

On the BACKUP host (lb-2), haproxy and keepalived should both be active, HAProxy should listen on 192.168.56.200:6443, and the VIP line should be absent until failover.

Test the endpoint before kubeadm init

From any node that resolves k8s-api.lab.local, test TCP connectivity to the shared endpoint:

nc -zv -w 2 k8s-api.lab.local 6443

Before kubeadm init, backends are not running yet. Interpreting the result:

• Connection refused on the backend path usually means the load balancer reached a target, but no API server is listening yet. That is normal before kubeadm init on the first control-plane node.
• Timeout means the VIP, firewall, or backend routing is wrong and must be fixed before you continue.

After the first API server starts, check readiness through the shared endpoint with curl. HTTP 200 from /readyz means at least one healthy API server is behind the load balancer:

curl -sk -o /dev/null -w "%{http_code}\n" https://k8s-api.lab.local:6443/readyz

Sample output:

200

Initialize the first control-plane node

On cp-1, create a kubeadm configuration file instead of a long command line. Set controlPlaneEndpoint to the load-balancer DNS name and port, match networking to your CNI, and list names clients use to reach the API in certSANs.

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.3
controlPlaneEndpoint: "k8s-api.lab.local:6443"
networking:
  podSubnet: 192.168.0.0/16
  serviceSubnet: 10.96.0.0/12
apiServer:
  certSANs:
  - k8s-api.lab.local
  - 192.168.56.200
etcd:
  local:
    dataDir: /var/lib/etcd
---
apiVersion: kubeadm.k8s.io/v1beta4
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.56.108
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock

The three settings serve different purposes:

Initialize and upload control-plane certificates so additional control-plane nodes can copy PKI material:

sudo kubeadm init --config kubeadm-config.yaml --upload-certs

On success, kubeadm prints join material you must save before the shell scrolls away:

• Worker join command (no --control-plane)
• Control-plane join command with --control-plane and --certificate-key
• The certificate key itself (expires in two hours by default)

Sample tail from my run:

Your Kubernetes control-plane has initialized successfully!

kubeadm join k8s-api.lab.local:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <certificate-key>

The join commands already use k8s-api.lab.local:6443 because controlPlaneEndpoint was set during init.

Understand --upload-certs and the certificate key

--upload-certs encrypts shared control-plane certificates and stores them temporarily in the kubeadm-certs Secret in kube-system. The certificate key decrypts those files on additional control-plane nodes during kubeadm join --control-plane.

• The Secret and key expire after approximately two hours.
• The certificate key grants access to sensitive cluster material and must not be published in tickets, chat, or version control.

To regenerate upload material on a healthy control-plane node:

sudo kubeadm init phase upload-certs --upload-certs

To generate a key without uploading:

kubeadm certs certificate-key

Manual certificate copying is an advanced alternative outside this walkthrough.

Configure kubectl and install the CNI

Configure cluster admin access on the first control-plane node:

mkdir -p "$HOME/.kube"

sudo cp -i /etc/kubernetes/admin.conf \
  "$HOME/.kube/config"

sudo chown "$(id -u):$(id -g)" \
  "$HOME/.kube/config"

chmod 600 "$HOME/.kube/config"

Install the same CNI used in the single-control-plane lab. The Calico operator manifests and custom-resources.yaml with 192.168.0.0/16 from install Kubernetes with kubeadm apply here without changes. Add nodeAddressAutodetectionV4 with kubernetes: NodeInternalIP when nodes have multiple interfaces.

Wait until the first control-plane node is Ready and core kube-system Pods are running before joining another control-plane member. A half-ready etcd ring makes the next join harder to debug.

List nodes:

kubectl get nodes

Sample output after CNI on cp-1:

NAME   STATUS   ROLES           AGE   VERSION
cp-1   Ready    control-plane   21s   v1.36.3

Join additional control-plane nodes

Prepare cp-2 and cp-3 with the same packages and host settings as the first node. Join one control-plane host at a time so the first failure is easy to spot.

On multi-interface VMs, set an explicit API advertise address on each join. My cp-2 and cp-3 VMs had only one vCPU; kubeadm normally requires two. I passed --ignore-preflight-errors=NumCPU for those joins. Production hosts should meet the CPU requirement instead of ignoring the check.

Join cp-2 with the control-plane join command from kubeadm init output:

sudo kubeadm join k8s-api.lab.local:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --control-plane --certificate-key <certificate-key> --apiserver-advertise-address 192.168.56.112 --cri-socket unix:///run/containerd/containerd.sock --ignore-preflight-errors=NumCPU

Wait until the new node reports Ready:

kubectl wait --for=condition=Ready node/cp-2 --timeout=180s

Sample output:

node/cp-2 condition met

Inspect the control-plane static Pods on the new host:

kubectl get pods -n kube-system -l tier=control-plane --field-selector spec.nodeName=cp-2 -o wide

Sample output:

NAME                           READY   STATUS    RESTARTS   AGE   IP               NODE
etcd-cp-2                      1/1     Running   0          3m    192.168.56.112   cp-2
kube-apiserver-cp-2            1/1     Running   0          3m    192.168.56.112   cp-2
kube-controller-manager-cp-2   1/1     Running   0          3m    192.168.56.112   cp-2
kube-scheduler-cp-2            1/1     Running   0          3m    192.168.56.112   cp-2

Repeat for cp-3 with --apiserver-advertise-address 192.168.56.113 only after cp-2 is healthy.

After the third control-plane join, list etcd members from a control-plane node:

ETCD_POD=$(kubectl get pod -n kube-system -l component=etcd --field-selector spec.nodeName=cp-1 -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n kube-system "$ETCD_POD" -- etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key member list --write-out=table

Sample output:

+------------------+---------+------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  | NAME |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+------+-----------------------------+-----------------------------+------------+
| 73f02e3a4230ca03 | started | cp-1 | https://192.168.56.108:2380 | https://192.168.56.108:2379 |      false |
| 83cbc0154f684033 | started | cp-2 | https://192.168.56.112:2380 | https://192.168.56.112:2379 |      false |
| f695f9f8c6ed37eb | started | cp-3 | https://192.168.56.113:2380 | https://192.168.56.113:2379 |      false |
+------------------+---------+------+-----------------------------+-----------------------------+------------+

Three started members confirm each control-plane join added one stacked-etcd member.

Join worker nodes (optional)

The five-VM lab focuses on control-plane HA and does not require a worker node. When you want to schedule application Pods, join one or more additional worker VMs with the worker join command from kubeadm init — the same API endpoint, token, and discovery hash, but without --control-plane or --certificate-key:

sudo kubeadm join k8s-api.lab.local:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash> --cri-socket unix:///run/containerd/containerd.sock

If the bootstrap token expired, create a new worker join line on any control-plane node:

sudo kubeadm token create --print-join-command

A correctly initialized HA cluster stores k8s-api.lab.local:6443 as controlPlaneEndpoint, so the printed command should already use that endpoint. If it prints a single node IP, the output came from a different cluster or the cluster configuration is wrong.

Confirm nodes and scheduling:

kubectl get nodes -o wide

Sample output:

NAME   STATUS   ROLES           AGE   VERSION   INTERNAL-IP
cp-1   Ready    control-plane   9m    v1.36.3   192.168.56.108
cp-2   Ready    control-plane   7m    v1.36.3   192.168.56.112
cp-3   Ready    control-plane   4m    v1.36.3   192.168.56.113

Verify the HA control plane

Three control-plane rows in kubectl get nodes are necessary but not sufficient. You also need a working load balancer, matching certificates, leader election for controllers, and a successful failover test.

Verify all API servers

List API server static Pods:

kubectl get pods -n kube-system -l component=kube-apiserver -o wide

Sample output:

NAME                  READY   STATUS    RESTARTS   AGE   IP               NODE
kube-apiserver-cp-1   1/1     Running   0          10m   192.168.56.108   cp-1
kube-apiserver-cp-2   1/1     Running   0          7m    192.168.56.112   cp-2
kube-apiserver-cp-3   1/1     Running   0          4m    192.168.56.113   cp-3

Check API readiness through the shared endpoint:

kubectl --server=https://k8s-api.lab.local:6443 get --raw=/readyz

Sample output:

ok

Verify etcd membership and health

Check endpoint health across all stacked members:

kubectl exec -n kube-system "$ETCD_POD" -- etcdctl --endpoints=https://192.168.56.108:2379,https://192.168.56.112:2379,https://192.168.56.113:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key endpoint health --write-out=table

Sample output:

+-----------------------------+--------+-------------+-------+
|          ENDPOINT           | HEALTH |    TOOK     | ERROR |
+-----------------------------+--------+-------------+-------+
| https://192.168.56.108:2379 |   true | 65.341268ms |       |
| https://192.168.56.112:2379 |   true | 105.668741ms|       |
| https://192.168.56.113:2379 |   true | 64.293652ms |       |
+-----------------------------+--------+-------------+-------+

All three endpoints report true. Plan etcd backup and restore separately before you rely on this cluster in production.

Verify controller-manager and scheduler leader election

Inspect Lease objects:

kubectl get lease kube-controller-manager kube-scheduler -n kube-system -o custom-columns=NAME:.metadata.name,HOLDER:.spec.holderIdentity,TRANSITIONS:.spec.leaseTransitions

Sample output:

NAME                      HOLDER                                         TRANSITIONS
kube-controller-manager   cp-2_401efe5b-4f26-47fe-be05-7c7508114d67    1
kube-scheduler            cp-3_65441d33-0814-4e22-ad78-ca639d5695cb    2

All replicas run, but only one scheduler and one controller-manager replica hold the active lease at a time. A standby becomes leader after the current holder fails.

Rebalance CoreDNS

After additional nodes join, CoreDNS Pods may still run only on the first control-plane node. Restart the Deployment so replicas can spread where scheduling permits:

kubectl -n kube-system rollout restart deployment coredns

Before the restart, both CoreDNS Pods were on cp-1. After the restart, new replicas scheduled onto cp-2 and cp-3 as those nodes became available.

Test control-plane failure and failover

Run one controlled failure at a time. Avoid stopping two of three stacked control-plane nodes because etcd quorum would be lost.

Test API-server backend failure

Move the kube-apiserver static Pod manifest off one control-plane node to simulate a dead API server process. The local etcd member on that host keeps running — this test proves API availability through the load balancer, not etcd quorum loss.

On cp-1:

sudo mv \
  /etc/kubernetes/manifests/kube-apiserver.yaml \
  /root/kube-apiserver.yaml.bak

From the workstation or another control-plane node, wait until the API server Pod is actually gone. Kubelet checks the static-Pod directory periodically, so the original API server may still be running for up to about 20 seconds:

kubectl wait \
  --namespace kube-system \
  --for=delete \
  pod/kube-apiserver-cp-1 \
  --timeout=120s

Only then repeat requests through the shared endpoint. If an http_proxy is set, add --noproxy '*':

for i in {1..5}; do
  curl -sk --noproxy '*' -o /dev/null -w '%{http_code}\n' \
    https://k8s-api.lab.local:6443/readyz
  sleep 1
done

Sample output (five consecutive attempts during my test):

200
200
200
200
200

Prove that etcd still accepts writes through the surviving API servers while all three etcd members remain healthy:

kubectl create configmap ha-write-check --from-literal=result=ok --dry-run=client -o yaml | kubectl apply -f -

kubectl get configmap ha-write-check -o jsonpath='{.data.result}{"\n"}'

Sample output:

configmap/ha-write-check created
ok

Restore the manifest when you finish:

sudo mv \
  /root/kube-apiserver.yaml.bak \
  /etc/kubernetes/manifests/kube-apiserver.yaml

Verify recovery:

kubectl wait \
  --namespace kube-system \
  --for=condition=Ready \
  pod/kube-apiserver-cp-1 \
  --timeout=120s

Test complete control-plane node failure

Power off or stop one control-plane VM to test etcd quorum failover. Unlike removing only kube-apiserver.yaml, this stops the local etcd member on that host as well. Then confirm:

• kubectl through k8s-api.lab.local still answers
• etcd retains quorum (two of three members)
• A write through the shared endpoint still succeeds
• scheduler and controller-manager leaders fail over or remain on surviving nodes

Restore the node and confirm its API server and etcd member return to Running.

Test load-balancer failover

Confirm the VIP starts on the MASTER (lb-1):

ip address show enp0s8 | grep 192.168.56.200

Sample output on lb-1 before failover:

inet 192.168.56.200/24 scope global secondary enp0s8

Stop VRRP ownership on the MASTER:

sudo systemctl stop keepalived

On lb-2, confirm the VIP moved:

ip address show enp0s8 | grep 192.168.56.200

Sample output on the standby after failover:

inet 192.168.56.200/24 scope global secondary enp0s8

From your workstation, repeat readiness checks through the shared endpoint. If an http_proxy is set, add --noproxy '*' so local lab traffic does not leave the host network:

for i in {1..10}; do
  curl -sk --noproxy '*' -o /dev/null -w '%{http_code}\n' \
    https://k8s-api.lab.local:6443/readyz
  sleep 1
done

Sample output during my failover test:

200
200
200
200
200
200
200
200
200
200

You should see 200 on every line while the VIP is on the surviving load balancer and at least one API server backend remains healthy. A short gap of connection errors can appear while Keepalived moves the VIP.

Restore the original MASTER when you finish:

sudo systemctl start keepalived

The VIP should return to lb-1 when its Keepalived priority is higher again.

What remains available during failure?

External etcd topology

When etcd runs outside the control-plane nodes, replace the etcd.local block in ClusterConfiguration with client endpoints and TLS files:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.36.3
controlPlaneEndpoint: "k8s-api.lab.local:6443"
etcd:
  external:
    endpoints:
    - https://etcd-1.example.com:2379
    - https://etcd-2.example.com:2379
    - https://etcd-3.example.com:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key

External etcd requires:

• Three or more dedicated etcd hosts with TLS and peer connectivity on ports 2379 and 2380
• A healthy etcd cluster before kubeadm init
• Before kubeadm init, securely copy the external-etcd CA certificate and apiserver-etcd-client certificate and key to the first control-plane node at the paths referenced by etcd.external. With --upload-certs, additional control-plane nodes receive shared certificate material during kubeadm join --control-plane. If you do not use --upload-certs, distribute the required files manually before each join.
• Independent etcd backup, upgrade, and lifecycle management

The first-control-plane init, CNI install, control-plane joins, worker joins, and verification steps remain the same after external etcd is prepared.

Load-balanced control plane
├── cp-1
├── cp-2
└── cp-3

External etcd cluster
├── etcd-1
├── etcd-2
└── etcd-3

Regenerate expired join material

Bootstrap tokens expire (default 24 hours). Certificate keys from --upload-certs expire faster (default two hours).

Upload a new certificate bundle for control-plane joins:

sudo kubeadm init phase upload-certs --upload-certs

Copy the new key, then run:

sudo kubeadm token create --print-join-command --certificate-key <new-certificate-key>

Sample output:

kubeadm join k8s-api.lab.local:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane \
  --certificate-key <new-certificate-key>

For worker joins only, omit --certificate-key:

sudo kubeadm token create --print-join-command

Common HA kubeadm problems

Final validation checklist

Confirm before you treat the cluster as HA:

• Shared API DNS or VIP resolves from every node
• HAProxy listens on the VIP and health-checks every control-plane backend
• Three control-plane nodes are Ready
• Three kube-apiserver Pods are Running
• Three etcd members are healthy
• Scheduler and controller-manager leader election works
• If worker nodes are added, they join through the shared API endpoint and reach Ready
• CNI works across nodes
• CoreDNS resolves Service names after rebalance
• A write succeeds through the shared endpoint during one API-server failure
• Load-balancer VIP failover works between lb-1 and lb-2
• etcd backup procedure has been tested separately

References

• Creating Highly Available Clusters with kubeadm — upstream stacked and external etcd procedures
• Options for Highly Available Topology — stacked versus external-etcd behavior
• kubeadm configuration (v1beta4) — controlPlaneEndpoint, certSANs, and etcd.external
• kubeadm token — generating worker and control-plane join commands
• PKI certificates and requirements — etcd health-check client certificates
• Operating etcd clusters for Kubernetes — member and endpoint health verification
• Leases — controller-manager and scheduler leader election
• etcd FAQ — Why an odd number of members?

Summary

Highly available kubeadm clusters center on one stable API address: controlPlaneEndpoint behind a TCP load balancer on port 6443, with certSANs that cover the VIP and DNS name clients use. Stacked etcd on three control-plane nodes is the default teaching path — each join after the first uses --control-plane and a fresh --certificate-key from --upload-certs.

I built the lab on five VMs: three stacked-etcd control-plane nodes (cp-1 192.168.56.108, cp-2 192.168.56.112, cp-3 192.168.56.113) and two dedicated load-balancer hosts (lb-1 192.168.56.110, lb-2 192.168.56.109). Load-balancer hosts must not share port 6443 with kube-apiserver, so HAProxy and Keepalived run on lb-1 and lb-2 rather than on control-plane nodes. After CNI was healthy on the first control plane, I joined cp-2 and cp-3 sequentially, confirmed three etcd members and three API servers, rebalanced CoreDNS, proved API availability through k8s-api.lab.local while one API server manifest was removed, verified VIP movement to lb-2 during a Keepalived failover test with ten consecutive /readyz responses returning 200, and confirmed writes still succeeded through the surviving API servers while all three etcd members remained healthy.

The usual production pitfalls are certificate SAN gaps, expired certificate keys, forgetting net.ipv4.ip_nonlocal_bind=1 on the BACKUP load balancer, blocked VRRP traffic, and placing a load balancer on a control-plane node. Join control-plane nodes one at a time, keep token and certificate-key material current, and run deliberate API-server and load-balancer failover tests before you call the cluster highly available. For etcd backups, upgrades, and certificate renewal, continue with the dedicated guides linked above.