Upgrade a Kubernetes Cluster with kubeadm

Upgrading a kubeadm cluster feels intimidating the first time. In practice it is a careful sequence you repeat node by node: move up one minor version, touch the control plane before the workers, and pin the package repositories so nothing on a node quietly jumps ahead of you. Get that order right and the upgrade is boring, which is exactly what you want on a production cluster.

I ran the RHEL-family single-control-plane workflow end to end on a v1.35.7 to v1.36.3 lab and captured the outputs below. If you use Debian or Ubuntu, or you run an HA control plane, the package commands differ slightly but the sequence is the same.

Use the quick reference table when you want the full sequence on one screen. Otherwise work through Steps 1–11 in order. On a single-control-plane cluster, skip Step 8.

Guide order:

1. Define the upgrade path and run preflight checks (cgroup v2, swap)
2. Record cluster state and create a demo workload
3. Prepare backups (etcd snapshot and /etc/kubernetes)
4. Confirm CNI compatibility
5. Switch the package repository to the destination minor
6. Upgrade kubeadm on the first control plane and run kubeadm upgrade plan
7. Run kubeadm upgrade apply on the first control plane
8. Upgrade additional control-plane nodes (HA clusters only)
9. Upgrade control-plane kubelet and kubectl
10. Upgrade each worker node, one at a time
11. Verify add-ons and workloads

Quick reference: kubeadm upgrade order

The table below maps each phase to the node and the commands involved. Steps 1–11 expand every row with the exact commands and the output I saw on the lab.

These rules apply in every phase:

• Upgrade one minor version at a time (here 1.35 → 1.36). Skipping minors is unsupported.
• Prefer the latest patch of the destination minor (here v1.36.3).
• Upgrade control-plane nodes before workers.
• Keep kubelet no newer than any API server it can contact while the cluster is mid-upgrade.
• In HA clusters, finish one control-plane node and confirm Ready before starting the next.

Step 1: Define the upgrade path and run preflight checks

Before you install a single package, decide exactly where you are going and in what order you will get there. Guessing halfway through is how you end up with a kubelet that is newer than the API server it talks to, so write the plan down first.

Read the destination release notes and check whether any of your workloads still call APIs scheduled for removal. See Kubernetes API deprecations for the common ones.

Confirm every Linux node uses cgroup v2 before upgrading its kubelet. Kubernetes 1.36 enables FailCgroupV1 by default:

stat -fc %T /sys/fs/cgroup

Expected output:

cgroup2fs

A tmpfs result indicates a cgroup v1 layout that must be addressed before continuing.

Kubeadm 1.36 also expects swap disabled for the standard workflow:

swapon --show

Expected result: no output. Any listed device or file means swap is active. When the cluster intentionally uses Kubernetes swap with failSwapOn: false, verify that node's existing swap and kubelet configuration separately before upgrading.

Step 2: Record the current cluster state

Take a snapshot of what healthy looks like right now, before you change anything. When something looks off later, you compare against this baseline instead of trying to remember how the cluster behaved. Start on the control plane:

kubectl get nodes -o wide

Sample output from the lab before the upgrade:

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                              CONTAINER-RUNTIME
k8s-cp     Ready    control-plane   4m    v1.35.7   192.168.56.108   <none>        Rocky Linux 10.2 (Red Quartz)   6.12.0-211.16.1.el10_2.0.1.x86_64 (amd64)   containerd://2.2.5
worker01   Ready    <none>          2m    v1.35.7   192.168.56.109   <none>        Rocky Linux 10.2 (Red Quartz)   6.12.0-211.34.1.el10_2.x86_64 (amd64)       containerd://2.2.5

Both nodes reported v1.35.7 and sat at Ready. That was the starting line I expected. Next, confirm the kubectl client and the API server agree on that version:

kubectl version

Client Version: v1.35.7
Kustomize Version: v5.8.1
Server Version: v1.35.7

Client and server both sat on the old minor, so there was no hidden skew to untangle before I began. Check which kubeadm binary is actually installed on the control plane:

kubeadm version

Trimmed output from my lab:

kubeadm version: &version.Info{Major:"1", Minor:"35", GitVersion:"v1.35.7", ... Platform:"linux/amd64"}

kubeadm still reported v1.35.7, which is what you want before switching the package repository. List every Pod in the cluster too, so you have a record of what was running before you touch any nodes:

kubectl get pods -A

You should see the kube-system control-plane Pods, CoreDNS, kube-proxy, and your CNI namespace all sitting at Ready or Running. If something is already unhealthy here, fix it before you upgrade. An upgrade never repairs a broken Pod. It only adds new variables to an already shaky cluster.

Ask the API server whether it considers itself healthy:

kubectl get --raw='/readyz?verbose'

A healthy API server ends that output with readyz check passed. While you are here, note everything outside core Kubernetes that has to survive the jump. No upgrade guide can check these for you:

• CNI version (this lab: Calico via the Tigera operator)
• CSI drivers and storage classes in use
• Ingress or Gateway controllers
• Helm releases and other add-ons
• Current package repository URL under /etc/yum.repos.d/ or /etc/apt/sources.list.d/

I also created a two-replica demo Deployment pinned to the worker so drain and uncordon behavior is visible later:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: upgrade-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: upgrade-demo
  template:
    metadata:
      labels:
        app: upgrade-demo
    spec:
      nodeSelector:
        kubernetes.io/hostname: worker01
      containers:
      - name: app
        image: busybox:1.36
        command: ["sh", "-c", "sleep 86400"]

Save that manifest as upgrade-demo.yaml, then apply and verify it:

kubectl apply -f upgrade-demo.yaml
kubectl rollout status deployment/upgrade-demo --timeout=120s
kubectl get pods -l app=upgrade-demo -o wide

Sample output:

deployment.apps/upgrade-demo created
deployment "upgrade-demo" successfully rolled out

The nodeSelector ensures replacement Pods remain Pending while worker01 is cordoned and return after it is uncordoned.

Step 3: Prepare backups

kubeadm upgrade rewrites the static Pod manifests and can rotate certificates. Those are files you want a copy of if the upgrade goes sideways. Before you touch any version, create a timestamped backup directory, take an etcd snapshot, copy /etc/kubernetes, and export the kubeadm ConfigMap. The complete etcd procedure lives in back up and restore etcd. On my lab I used this sequence:

BACKUP_DIR="/var/backups/kubernetes-pre-upgrade-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  snapshot save "$BACKUP_DIR/etcd.db"

etcdutl snapshot status "$BACKUP_DIR/etcd.db" -w table

cp -a /etc/kubernetes "$BACKUP_DIR/"

KUBECONFIG=/etc/kubernetes/admin.conf \
  kubectl get configmap kubeadm-config -n kube-system -o yaml \
  > "$BACKUP_DIR/kubeadm-config.yaml"

+----------+----------+------------+------------+---------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | VERSION |
+----------+----------+------------+------------+---------+
| 86254e4b |     1458 |        685 |     8.2 MB |   3.6.0 |
+----------+----------+------------+------------+---------+

A non-zero revision, key count, and expected size mean etcdutl can read the snapshot metadata. That confirms the file is structurally sound. Only a tested restore procedure proves the backup can actually recover your cluster.

Keep a copy of $BACKUP_DIR outside the node. A local backup on the same disk does not protect against node or filesystem loss.

That directory holds admin.conf, the manifests folder, the pki certificates, and the rest of the kubeadm files (about 164K of config on my lab).

A few boundaries worth keeping in mind:

• An etcd snapshot backs up cluster state, not application data. Anything in a PersistentVolume needs its own backup.
• These paths and certificates assume kubeadm-managed stacked etcd on the local control-plane node and host-installed etcdctl/etcdutl.
• For external etcd, use its actual endpoints and client certificates.
• In an HA control plane, copy /etc/kubernetes separately from every control-plane node because manifests and certificates are node-specific.

Step 4: Confirm CNI compatibility

Kubeadm does not upgrade the CNI provider. Before you change Kubernetes packages, confirm the installed CNI release supports both the current and destination Kubernetes minors, and download any operator or manifest upgrade instructions you might need.

On this lab, Calico runs through the Tigera operator. Record both the operator and data-plane images:

kubectl get deployment tigera-operator -n tigera-operator \
  -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

kubectl get daemonset calico-node -n calico-system \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="calico-node")].image}{"\n"}'

Sample output from my lab:

quay.io/tigera/operator:v1.42.3
quay.io/calico/node:v3.32.1

Extract the Calico version from the calico-node image. Open the Calico documentation version matching that installed release and confirm support for both the source and destination Kubernetes minors. For this lab I checked Calico 3.32 system requirements, which lists Kubernetes 1.34, 1.35, and 1.36 as tested. Recording only the Tigera operator version is insufficient because the operator and calico-node images can differ.

After a provider upgrade, verify it before proceeding:

kubectl rollout status daemonset/calico-node \
  -n calico-system --timeout=180s

kubectl get tigerastatus

Step 5: Configure the destination package repository

This detail trips up a lot of people: the community repositories on pkgs.k8s.io are split per minor version. A node still pointing at .../v1.35/rpm/ simply cannot see the 1.36.x packages, no matter how many times you run an update. On every node you plan to upgrade, repoint the repository at the destination minor. Keep the package excludes or holds in place as well, so a stray dnf update cannot drag a newer kubelet onto the node ahead of schedule.

On Rocky Linux and other RHEL-family nodes, replace the repo file so it targets v1.36:

cat > /etc/yum.repos.d/kubernetes.repo <<'EOF'
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.36/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

With the repo pointed at the new minor, list the kubeadm builds it now offers so you can pick the exact patch:

dnf list --showduplicates kubeadm --disableexcludes=kubernetes

On Fedora and other DNF5 systems, use --setopt=disable_excludes=kubernetes instead:

dnf list --showduplicates kubeadm \
  --setopt=disable_excludes=kubernetes

kubeadm.x86_64                   1.36.3-150500.1.1                   kubernetes

On Debian or Ubuntu, pkgs.k8s.io uses a separate repository URL for every Kubernetes minor release. Write and verify the destination repository explicitly:

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' \
  | sudo tee /etc/apt/sources.list.d/kubernetes.list >/dev/null

grep 'core:/stable:/v1.36/deb/' \
  /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
apt-cache madison kubeadm

Leave the holds or excludes on kubelet and kubectl in place until you deliberately upgrade that node in Steps 9 or 10. Step 5 only repoints the repository and confirms the destination packages are visible.

Step 6: Upgrade kubeadm on the first control plane

On the first control-plane node, install only kubeadm at the destination version for now. Do not install kubelet in the same command. You want kubelet left on the old minor until kubeadm upgrade apply has done its work.

dnf install -y kubeadm-1.36.3-150500.1.1 --disableexcludes=kubernetes

On DNF5:

dnf install -y kubeadm-1.36.3-150500.1.1 \
  --setopt=disable_excludes=kubernetes

On Debian or Ubuntu, install only kubeadm on the first control-plane node:

sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm='1.36.3-*'
sudo apt-mark hold kubeadm

Confirm the binary picked up the new version:

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"36", GitVersion:"v1.36.3", GitCommit:"0f29094e5b73085e3802ecc1298ecae13866bfe6", GitTreeState:"clean", BuildDate:"2026-07-22T18:09:52Z", GoVersion:"go1.26.5", Compiler:"gc", Platform:"linux/amd64"}

So kubeadm is now v1.36.3 while the cluster and the kubelets are all still v1.35.7. That gap looks wrong at a glance, but it is exactly the state you want at this step. kubeadm is the tool doing the upgrade, so it moves first. Before you apply anything, ask kubeadm for its plan and read the output:

kubeadm upgrade plan

Trimmed output from my lab:

[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/versions] Cluster version: v1.35.7
[upgrade/versions] kubeadm version: v1.36.3
...
You can now apply the upgrade by executing the following command:

    kubeadm upgrade apply v1.36.3

The plan confirms the cluster can move from v1.35.7 to v1.36.3. It shows the target versions and configuration states for kubeadm-managed components. On a stacked-etcd cluster, this includes the local etcd static Pod. Kubeadm does not upgrade an external etcd cluster; plan and perform that upgrade according to the external etcd deployment's own procedure. The plan also lists API server, controller-manager, scheduler, kube-proxy, and CoreDNS targets, and reminds you that kubelets are your job to upgrade by hand afterward. Read it carefully before you run upgrade apply.

Kubeadm manages the local etcd static Pod when stacked etcd is used, while external etcd follows a separate lifecycle.

If the plan refuses the upgrade, read the reported preflight or version-skew error. Common causes include:

• An unsupported minor jump
• Unhealthy nodes or control-plane components
• Component-configuration migration problems

A wrong package repository normally prevents you from installing the destination kubeadm binary earlier in the workflow.

Step 7: Apply the control-plane upgrade

This is the one command that actually rewrites the control plane. Run it on the first control-plane node only. Never run it on a second control-plane node, and never run it on a worker.

kubeadm upgrade apply v1.36.3 -y

Trimmed output from my lab:

[upgrade/versions] Cluster version: v1.35.7
[upgrade/versions] kubeadm version: v1.36.3
[upgrade/control-plane] Upgrading your static Pod-hosted control plane to version "v1.36.3" (timeout: 5m0s)...
[upgrade/staticpods] Component "etcd" upgraded successfully!
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.36.3".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.

In that one command, kubeadm:

• Rewrote the static Pod manifests
• Renewed kubeadm-managed control-plane certificates by default, unless certificate renewal was disabled or the cluster uses externally managed certificates
• Updated the kubelet configuration on this node
• Tucked away a copy of the old manifests under /etc/kubernetes/tmp/ (for example kubeadm-backup-manifests-2026-07-27-18-13-56)

On a stacked-etcd control-plane node, kubeadm can create both kubeadm-backup-manifests-* and kubeadm-backup-etcd-* directories under /etc/kubernetes/tmp. Kubeadm's documented phases include certificate renewal and timestamped backups of static manifests and local etcd data.

On this single-control-plane lab, kubeadm upgrade apply also upgraded CoreDNS and kube-proxy immediately. In an HA cluster, kubeadm normally waits until the final control-plane instance has completed kubeadm upgrade node before upgrading these add-ons.

Kubeadm does not upgrade Calico or another CNI implementation. Apply the provider-specific upgrade at the point recommended by its documentation, then confirm all CNI node agents are Ready before continuing with additional nodes.

Do not be surprised by the next part. Right after this, kubectl get nodes still showed v1.35.7 for both nodes. That is not a failure:

• The control-plane containers were already running v1.36.3
• The version column reflects the kubelet, and the kubelet package on this node had not moved yet

HA clusters continue with Step 8 below. Single-control-plane labs go straight to Step 9.

Step 8: Upgrade additional control-plane nodes (HA only)

My lab runs a single control plane, so I skipped this step. If you run a highly available control plane, work through two explicit passes before you upgrade any control-plane kubelet packages.

HA pass 1 — Upgrade control-plane components

1. Run kubeadm upgrade apply on the first control-plane node.
2. On each additional control-plane node, install the destination kubeadm.
3. Run kubeadm upgrade node.
4. Verify that node's static control-plane Pods before moving to the next node.
5. Keep all kubelets on the old minor during this pass when they use the shared controlPlaneEndpoint.

On an additional control-plane node named k8s-cp2, install destination kubeadm the same way as in Step 6. On Debian or Ubuntu:

sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm='1.36.3-*'
sudo apt-mark hold kubeadm

Then run the node upgrade on k8s-cp2:

kubeadm version
kubeadm upgrade node

Perform verification from the admin shell:

NODE=k8s-cp2

kubectl wait -n kube-system --for=condition=Ready \
  "pod/kube-apiserver-${NODE}" \
  "pod/kube-controller-manager-${NODE}" \
  "pod/kube-scheduler-${NODE}" \
  --timeout=180s

For stacked etcd:

kubectl wait -n kube-system --for=condition=Ready \
  "pod/etcd-${NODE}" --timeout=180s

Confirm the images actually running on that node:

kubectl get pods -n kube-system \
  --field-selector "spec.nodeName=${NODE}" \
  -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,IMAGE:.status.containerStatuses[*].image,IMAGE-ID:.status.containerStatuses[*].imageID'

kubeadm upgrade node upgrades the local static Pod manifests and kubelet configuration on an additional control-plane node.

Do not rely only on /readyz through a load balancer because that request may reach a different healthy API-server instance.

HA pass 2 — Upgrade control-plane kubelets

After all control-plane components are on the destination minor:

1. Drain one control-plane node.
2. Upgrade kubelet and kubectl.
3. Restart kubelet.
4. Verify the new kubelet version and Node readiness.
5. Uncordon it.
6. Repeat for the next control-plane node.

Use Step 9: Upgrade kubelet and kubectl on the control plane for the drain, package install, readiness checks, and uncordon workflow on each control-plane member.

The commands that trip people up most often:

• kubeadm upgrade apply runs exactly once, on the very first control-plane node
• Every other control-plane member uses kubeadm upgrade node to refresh local static Pods before any kubelet package upgrade
• Workers follow the same kubeadm upgrade node pattern
• Mixing apply and node up is one of the most common upgrade mistakes

HA topology choices are covered in highly available control plane with kubeadm.

Step 9: Upgrade kubelet and kubectl on the control plane

On a single-control-plane cluster, continue here immediately after the first kubeadm upgrade apply. HA clusters complete pass 1 in Step 8 first, then use this section for pass 2. For each control-plane member, set the node name once and reuse it in every drain, wait, and uncordon command. On my lab that is k8s-cp; for an additional HA member use k8s-cp2, k8s-cp3, and so on.

NODE=k8s-cp

Before you restart the kubelet for a minor bump, drain the node so anything reschedulable moves off it first. DaemonSet Pods stay put by design, which is why you pass --ignore-daemonsets. The full set of drain flags and how PodDisruptionBudgets affect eviction are covered in cordon, drain and uncordon.

kubectl drain "$NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data

On my single control-plane lab, drain stalled here. Calico's calico-apiserver PodDisruptionBudget refused to let its Pod be evicted until another replica could run somewhere else, and on a one-node control plane there was nowhere else. Your options:

• In a real cluster with spare capacity, wait it out
• In a tight lab, temporarily loosen the PDB
• Accept the disruption and pass --disable-eviction only when you understand the impact

If the drain keeps retrying because a PDB blocks eviction, press Ctrl+C. After confirming that disruption is acceptable, rerun the drain with --disable-eviction:

kubectl drain "$NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --disable-eviction

Once drain returns, the node is cordoned and emptied of movable Pods.

With the node drained, install the matching kubelet and kubectl packages:

dnf install -y kubelet-1.36.3-150500.1.1 kubectl-1.36.3-150500.1.1 --disableexcludes=kubernetes

On DNF5:

dnf install -y \
  kubelet-1.36.3-150500.1.1 \
  kubectl-1.36.3-150500.1.1 \
  --setopt=disable_excludes=kubernetes

On Debian or Ubuntu:

sudo apt-mark unhold kubelet kubectl
sudo apt-get update
sudo apt-get install -y \
  kubelet='1.36.3-*' \
  kubectl='1.36.3-*'
sudo apt-mark hold kubelet kubectl

Run the package and systemctl commands on the control-plane node. Run the readiness checks and kubectl uncordon from an admin shell. When using the same control-plane root shell, first set KUBECONFIG=/etc/kubernetes/admin.conf.

On the control-plane node:

systemctl daemon-reload
systemctl restart kubelet

if ! systemctl is-active --quiet kubelet; then
  systemctl status kubelet --no-pager
  exit 1
fi

From an admin shell:

# Needed only when using the control-plane root shell
export KUBECONFIG=/etc/kubernetes/admin.conf

if ! timeout 180s bash -c '
  until kubectl get --raw=/readyz 2>/dev/null | grep -qx ok; do
    sleep 2
  done
'; then
  echo "API server did not become ready within 180 seconds." >&2
  exit 1
fi

kubectl wait \
  --for=jsonpath='{.status.nodeInfo.kubeletVersion}'=v1.36.3 \
  "node/${NODE}" --timeout=180s

kubectl wait --for=condition=Ready \
  "node/${NODE}" --timeout=180s

kubectl uncordon "$NODE"

Sample output:

node/k8s-cp condition met
node/k8s-cp condition met
node/k8s-cp uncordoned

kubectl get nodes -o wide

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                              CONTAINER-RUNTIME
k8s-cp     Ready    control-plane   10m   v1.36.3   192.168.56.108   <none>        Rocky Linux 10.2 (Red Quartz)   6.12.0-211.16.1.el10_2.0.1.x86_64 (amd64)   containerd://2.2.5
worker01   Ready    <none>          8m    v1.35.7   192.168.56.109   <none>        Rocky Linux 10.2 (Red Quartz)   6.12.0-211.34.1.el10_2.x86_64 (amd64)       containerd://2.2.5

Now the control plane reports v1.36.3 while the worker is still on v1.35.7. If that mismatch made you nervous a few steps ago, this is where it should start feeling normal. The control plane leads, and the worker catches up in Step 10.

Step 10: Upgrade worker nodes

Start on the workers only after the control plane is healthy and fully on the destination version. Take them one at a time. Rushing this is how you turn a routine upgrade into an outage. Set the worker name once and reuse it in every drain, wait, and uncordon command:

NODE=worker01

On $NODE, point the repository at v1.36 exactly the way you did on the control plane, then install kubeadm:

dnf install -y kubeadm-1.36.3-150500.1.1 --disableexcludes=kubernetes

On DNF5:

dnf install -y kubeadm-1.36.3-150500.1.1 \
  --setopt=disable_excludes=kubernetes

On Debian or Ubuntu:

sudo apt-mark unhold kubeadm
sudo apt-get update
sudo apt-get install -y kubeadm='1.36.3-*'
sudo apt-mark hold kubeadm

kubeadm version

Trimmed output from my lab:

kubeadm version: &version.Info{Major:"1", Minor:"36", GitVersion:"v1.36.3", ... Platform:"linux/amd64"}

On the worker itself, refresh the local kubelet configuration before you drain:

kubeadm upgrade node

[upgrade] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/preflight] Skipping prepull. Not a control plane node.
[upgrade/control-plane] Skipping phase. Not a control plane node.
[upgrade/kubelet-config] The kubelet configuration for this node was successfully upgraded!

This is much quieter than the control plane. A worker has no static control-plane Pods to rewrite, so kubeadm upgrade node skips those phases and just updates the local kubelet config. The official Linux-node workflow updates that configuration first, then drains immediately before changing the kubelet binary.

Run drain from wherever you have an admin kubeconfig. The control plane is fine. You do not run drain from the worker itself. Move the application Pods off the worker next:

kubectl drain "$NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data

On my lab, drain stalled when Calico's calico-apiserver PodDisruptionBudget blocked eviction of both replicas. There was no other schedulable node to absorb them, so the command retried indefinitely. In production, wait for spare capacity or adjust the PDB. If the drain keeps retrying because a PDB blocks eviction, press Ctrl+C. After confirming that disruption is acceptable, rerun the drain with --disable-eviction:

kubectl drain "$NODE" \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --disable-eviction

Trimmed output when eviction succeeds normally:

node/worker01 cordoned
evicting pod ...
pod/upgrade-demo-68459554b8-ksm27 evicted
pod/upgrade-demo-68459554b8-xq4lm evicted
node/worker01 drained

You can watch the demo Pods being evicted:

• In this one-worker lab, replacement Pods may remain Pending until the worker is uncordoned unless the control-plane node is eligible to run them
• In a cluster with another schedulable worker, the replacements can move there during the drain
• In this lab, a Calico API server PDB blocked the drain. In other clusters, any restrictive PDB, unmanaged Pod, or local-storage constraint can be responsible

With the node drained, install the kubelet and kubectl packages and restart the service on $NODE:

dnf install -y kubelet-1.36.3-150500.1.1 kubectl-1.36.3-150500.1.1 --disableexcludes=kubernetes

On DNF5:

dnf install -y \
  kubelet-1.36.3-150500.1.1 \
  kubectl-1.36.3-150500.1.1 \
  --setopt=disable_excludes=kubernetes

On Debian or Ubuntu:

sudo apt-mark unhold kubelet kubectl
sudo apt-get update
sudo apt-get install -y kubelet='1.36.3-*' kubectl='1.36.3-*'
sudo apt-mark hold kubelet kubectl

On the worker:

systemctl daemon-reload
systemctl restart kubelet

if ! systemctl is-active --quiet kubelet; then
  systemctl status kubelet --no-pager
  exit 1
fi

Return to the control plane or another admin workstation after restarting kubelet on the worker. The following kubectl commands do not run on the worker unless that worker also has a configured admin kubeconfig.

Once the kubelet service is active, wait for the destination kubelet version to register before you uncordon:

kubectl wait \
  --for=jsonpath='{.status.nodeInfo.kubeletVersion}'=v1.36.3 \
  "node/${NODE}" --timeout=180s

kubectl wait --for=condition=Ready \
  "node/${NODE}" --timeout=180s

kubectl uncordon "$NODE"

kubectl get nodes -o wide

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                              CONTAINER-RUNTIME
k8s-cp     Ready    control-plane   17m   v1.36.3   192.168.56.108   <none>        Rocky Linux 10.2 (Red Quartz)   6.12.0-211.16.1.el10_2.0.1.x86_64 (amd64)   containerd://2.2.5
worker01   Ready    <none>          15m   v1.36.3   192.168.56.109   <none>        Rocky Linux 10.2 (Red Quartz)   6.12.0-211.34.1.el10_2.x86_64 (amd64)       containerd://2.2.5

Both nodes now report v1.36.3, so this two-node cluster is fully upgraded. If you have more workers, repeat this exact sequence on each one. Change only NODE=worker02, NODE=worker03, and so on, still strictly one at a time.

Step 11: Verify add-ons and workloads

Both nodes reporting the right version is a good sign, but it is not the whole story. Spend a few minutes confirming the cluster is genuinely healthy.

When cluster administration is performed from a separate workstation, upgrade that workstation's kubectl independently. Installing kubectl on the control-plane and worker nodes does not update an external admin machine. The sample below assumes the active client was also upgraded to v1.36.3. A kubectl client can be one minor newer or older than the API server, but matching the destination release removes avoidable client skew.

Start with whether the client and API server now agree on the destination version:

kubectl version

Client Version: v1.36.3
Kustomize Version: v5.8.1
Server Version: v1.36.3

Client and server both read v1.36.3, so the upgrade landed and there is no leftover skew between kubectl and the API server. Next, confirm the API server is serving normally again:

kubectl get --raw=/readyz

ok

A plain ok is all you need here. Anything else means the API server is not fully back. Hold off on further changes until it is.

Version numbers can lie if the underlying containers did not actually change, so look at the images the upgraded static Pods are really running:

kubectl get pods -n kube-system \
  -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,IMAGE:.status.containerStatuses[*].image,IMAGE-ID:.status.containerStatuses[*].imageID'

NAME                             READY   IMAGE                                             IMAGE-ID
coredns-589f44dc88-nhqxq         true    registry.k8s.io/coredns/coredns:v1.14.2           registry.k8s.io/coredns/coredns@sha256:e7e6440cfd1e919280958f5b5a6ab2b184d385bba774c12ad2a9e1e4183f90d9
coredns-589f44dc88-prt2g         true    registry.k8s.io/coredns/coredns:v1.14.2           registry.k8s.io/coredns/coredns@sha256:e7e6440cfd1e919280958f5b5a6ab2b184d385bba774c12ad2a9e1e4183f90d9
etcd-k8s-cp                      true    registry.k8s.io/etcd:3.6.8-0                      registry.k8s.io/etcd@sha256:397189418d1a00e500c0605ad18d1baf3b541a1004d768448c367e48071622e5
kube-apiserver-k8s-cp            true    registry.k8s.io/kube-apiserver:v1.36.3            registry.k8s.io/kube-apiserver@sha256:b4bc06c81fd76f81174e6c19ddacf477acdf1583e7a5846ebbd513493aef6e43
kube-controller-manager-k8s-cp   true    registry.k8s.io/kube-controller-manager:v1.36.3   registry.k8s.io/kube-controller-manager@sha256:ed56454bf514916079a227f5765b64524fde52106dfcc52978b28634765b78b8
kube-proxy-flm2h                 true    registry.k8s.io/kube-proxy:v1.36.3                registry.k8s.io/kube-proxy@sha256:919d710a0e8bf2bd67d347e512e88205cb82a53a48ab6ff380d014406300ad1b
kube-proxy-xnmv9                 true    registry.k8s.io/kube-proxy:v1.36.3                registry.k8s.io/kube-proxy@sha256:919d710a0e8bf2bd67d347e512e88205cb82a53a48ab6ff380d014406300ad1b
kube-scheduler-k8s-cp            true    registry.k8s.io/kube-scheduler:v1.36.3            registry.k8s.io/kube-scheduler@sha256:128fc07d278d64c4f2cce416ed0a9f37b23a30cdde6f97873d18c9c78e259df4

The API server, controller-manager, scheduler, and kube-proxy images all read v1.36.3, and CoreDNS moved to its bundled version as part of the add-on upgrade. Confirm the add-ons and demo workload rolled out successfully:

kubectl rollout status deployment/coredns \
  -n kube-system --timeout=180s

kubectl rollout status daemonset/kube-proxy \
  -n kube-system --timeout=180s

kubectl rollout status daemonset/calico-node \
  -n calico-system --timeout=180s

kubectl rollout status deployment/upgrade-demo \
  --timeout=180s

kubectl get pods -A

Test DNS and Pod networking from the rescheduled demo workload:

kubectl get pods -l app=upgrade-demo -o wide

kubectl exec deployment/upgrade-demo -- \
  nslookup kubernetes.default.svc.cluster.local.

Abridged sample output:

Server:    10.96.0.10
Address:   10.96.0.10:53

Name:      kubernetes.default.svc.cluster.local
Address:   10.96.0.1

Successful resolution from upgrade-demo confirms that the rescheduled workload has Pod networking and can reach cluster DNS. Addresses vary by cluster.

For clusters using CSI:

kubectl get csidrivers
kubectl get volumeattachments

That covers the kubeadm-managed pieces. The rest of the cluster is on you to check, because these live outside kubeadm's reach:

• CSI volume mounts still work for stateful apps
• Metrics API if you run Metrics Server
• Service networking and DNS resolution for a test Pod
• No unexpected deprecated API warnings for critical workloads

On my lab the demo Deployment returned to two Ready Pods on worker01 after uncordon.

Rollback boundaries

Before trouble hits, it helps to know honestly what kubeadm can and cannot walk back for you:

• On control-plane nodes, kubeadm upgrade apply and control-plane kubeadm upgrade node create timestamped backups under /etc/kubernetes/tmp, including static-Pod manifests and local etcd data where stacked etcd is used.
• On workers, kubeadm upgrade node updates the local kubelet configuration; it does not create control-plane static-Pod or etcd backups.
• If a kubeadm upgrade command fails, rerun the same target version because the upgrade workflow is designed to be idempotent.
• If automatic rollback fails on a control-plane node, restore only the documented manifest or local-etcd backup for the failed phase, or follow the tested etcd restoration procedure.
• Do not blindly overwrite the entire /etc/kubernetes directory on a running, partially upgraded cluster.
• A completed minor-version downgrade remains outside the normal supported recovery path.

Troubleshooting

Most upgrade problems come from a handful of recurring causes. Nearly all of them are recoverable if you stop and read the error instead of retrying blindly. These are the ones I hit most often and how I get past them:

What's Next

• Kubernetes RBAC with Examples
• Kubernetes Authentication, Authorization and Admission Control
• Kubernetes ServiceAccounts with Examples

References

• Upgrading kubeadm clusters
• Upgrading Linux nodes — worker-node command order
• Upgrade a cluster — Kubernetes 1.36 cgroup v2 requirement and post-upgrade tasks
• Operating etcd clusters for Kubernetes — snapshot creation and verification
• kubeadm upgrade command — phases, certificate renewal, and recovery behavior
• Version skew policy
• Changing the Kubernetes package repository
• Calico system requirements — Kubernetes versions tested with current Calico releases
• Upgrade Calico on Kubernetes — provider-specific upgrade procedure
• pkgs.k8s.io package repositories

Summary

This guide walked a real kubeadm minor upgrade from v1.35.7 to v1.36.3 in eleven steps: recorded node and add-on state, created the upgrade-demo workload, confirmed cgroup v2 and swap, checked the installed CNI release against its compatibility documentation, took timestamped etcd and /etc/kubernetes backups, switched pkgs.k8s.io to the destination minor, upgraded kubeadm, ran upgrade plan and upgrade apply on the first control plane, completed the HA component pass when applicable, then upgraded control-plane kubelets and workers with drain, verification, and uncordon.

The same rules apply in production:

• One minor hop at a time
• Upgrade every API server reachable through the shared controlPlaneEndpoint before upgrading those kubelets
• Workers after the control plane
• Package excludes or holds so repositories cannot race you
• In HA clusters, keep kubelets on the old minor while they can still reach an older API server through the shared controlPlaneEndpoint

Drain plus PodDisruptionBudgets is where small clusters stall. Plan capacity before maintenance windows, or know when --disable-eviction is acceptable.

If you need the backup restore path next, use the etcd snapshot guide. For HA control planes:

• Run kubeadm upgrade node on each extra member after the first apply
• Upgrade every API server reachable through the shared controlPlaneEndpoint before upgrading those kubelets
• Never upgrade every control-plane node in parallel

After the cluster is steady, clear unused demo workloads and review API deprecations before the next minor upgrade.