Monitor Kubernetes Pods and Nodes with kubectl top

kubectl top gives you a live read on CPU and memory without opening a dashboard. I use it when a node feels hot or a Pod might be starving its neighbors. This walkthrough installs Metrics Server on a kubeadm lab, verifies the Metrics API, then runs kubectl top on nodes, Pods, and individual containers while a small stress workload runs.

How kubectl top Collects Resource Metrics

kubectl top does not read cgroups on the node directly. The data path is:

Kubelet → Metrics Server → Metrics API → kubectl top

• The kubelet exposes resource usage for the node and each Pod.
• Metrics Server scrapes those kubelet endpoints and aggregates short-lived samples.
• The Metrics API (metrics.k8s.io) exposes node and Pod metrics to clients.
• kubectl top queries that API and prints human-readable columns.

CPU is reported as average core usage over the metric's sampling window. Memory is the working set measured when the sample was collected. Metrics Server aggregates these kubelet-provided values; it does not calculate them itself. Memory values such as 64Mi use Kubernetes binary units—Mi means mebibytes. The working set can include some file-backed cache and is not simply process RSS.

A few practical limits:

• Metrics reflect recent usage, not a long history.
• A Pod that just started may not have a sample until the pipeline publishes its first reading.
• kubectl top is ideal for spot checks during troubleshooting, not dashboards, alerting, or capacity planning over weeks.

Install and Verify Metrics Server

Check for an existing installation

Before you install anything, confirm whether Metrics Server already runs in kube-system.

Check the Deployment:

kubectl get deployment metrics-server -n kube-system

On a cluster without Metrics Server you see NotFound. After install you should see READY 1/1 and AVAILABLE 1.

List Metrics Server Pods:

kubectl get pods -n kube-system -l k8s-app=metrics-server

Sample output on a working cluster:

NAME                              READY   STATUS    RESTARTS   AGE
metrics-server-6c9576559c-kvzxm   1/1     Running   0          2m1s

Confirm the aggregated APIService:

kubectl get apiservice v1beta1.metrics.k8s.io

Sample output when the API is ready:

NAME                     SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io   kube-system/metrics-server   True        72s

AVAILABLE=True confirms that the API aggregation layer can reach the Metrics Server API. It does not prove that Metrics Server has successfully collected node and Pod samples from every kubelet. Confirm the full path with a successful kubectl top node request.

If Metrics Server and the APIService are already working, skip the installation section and continue with node metrics.

Install Metrics Server 0.8.1

Metrics Server 0.8.x supports Kubernetes 1.31+, so the pinned v0.8.1 version is suitable for this lab. Kubernetes 1.31+ alone is not sufficient. Before you install, confirm the cluster meets these requirements:

• API aggregation enabled
• Kubelet webhook authentication and authorization
• Network access from Metrics Server to each kubelet
• Network access from the control plane to Metrics Server
• Trusted kubelet serving certificates, or the lab-only insecure flag described below

Apply the pinned upstream manifest:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.1/components.yaml

Sample output (trimmed):

serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

The command creates RBAC objects, a Service, the Deployment, and the v1beta1.metrics.k8s.io APIService. The v0.8.1 manifest sets a 15-second metric resolution and --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname.

Check whether the lab has the certificate error:

kubectl logs -n kube-system -l k8s-app=metrics-server --tail=30

Fix kubelet certificate errors in the lab

On kubeadm labs, kubelet serving certificates often lack the node IP in SANs. Metrics Server logs then show kubelet x509 verification failures. Metrics Server requires trusted kubelet serving certificates—or --kubelet-insecure-tls for testing. The upstream manifest defines a /readyz probe that expects usable metrics, so the Deployment may remain unready until TLS is fixed or the testing-only flag is applied.

Only when the logs show kubelet x509 verification failures, apply:

kubectl patch deployment metrics-server -n kube-system --type=json -p='[
    {
      "op":"add",
      "path":"/spec/template/spec/containers/0/args/-",
      "value":"--kubelet-insecure-tls"
    }
  ]'

Sample output:

deployment.apps/metrics-server patched

Do not append a second --kubelet-preferred-address-types flag; the manifest already sets it.

Do not treat --kubelet-insecure-tls as a production default. It disables kubelet certificate verification and is only a lab workaround. Fix kubelet serving certificates or trust on real clusters instead.

Verify the Metrics API and first sample

Then run:

kubectl rollout status deployment/metrics-server -n kube-system --timeout=120s

Sample output:

deployment "metrics-server" successfully rolled out

kubectl wait --for=condition=Available apiservice/v1beta1.metrics.k8s.io --timeout=120s

Sample output:

apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io condition met

APIService Available=True confirms that the aggregated API can reach Metrics Server, but the first kubelet metrics collection may still be pending. Metrics Server normally collects on a short interval. Wait for the first node sample with bounded retries:

for attempt in {1..36}; do
  kubectl top node >/dev/null 2>&1 && break
  sleep 5
done

kubectl top node

Sample output:

NAME       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
k8s-cp     691m         23%      3120Mi          47%         
worker01   40m          4%       1155Mi          62%

The final command either prints metrics or exposes the error after a maximum wait of three minutes. A successful kubectl top node response confirms the full kubelet → Metrics Server → Metrics API path; CPU does not have to be non-zero. Metrics Server is intended for recent resource signals and kubectl top spot checks, not precision monitoring.

Monitor Node Resource Usage

Read node CPU and memory

List every node:

kubectl top node

By default, the node percentage columns use allocatable CPU and memory as their denominator. The absolute CPU(cores) and MEMORY(bytes) columns remain measured usage. --show-capacity changes the percentage denominator from allocatable resources to total capacity.

Restrict output to one node:

kubectl top node k8s-cp

Sample output:

NAME     CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
k8s-cp   650m         21%      3122Mi          47%

Sort nodes

Sort nodes by CPU to find the busiest worker:

kubectl top node --sort-by=cpu

Sample output:

NAME       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
k8s-cp     650m         21%      3122Mi          47%         
worker01   93m          9%       1168Mi          62%

Sort by memory instead:

kubectl top node --sort-by=memory

Allocatable versus capacity percentages

--show-capacity bases percentages on total node capacity rather than allocatable resources (which reserve space for system daemons):

kubectl top node --show-capacity

Sample output:

NAME       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
k8s-cp     650m         21%      3122Mi          46%         
worker01   93m          9%       1168Mi          59%

Memory percentages can drop slightly compared to the default view because total capacity is larger than allocatable memory.

Monitor Pod and Container Usage

Generate test CPU usage

Create a namespace and a Deployment with nginx plus a stress container so metrics move visibly.

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
  name: top-lab
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress
  namespace: top-lab
  labels:
    app: stress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stress
  template:
    metadata:
      labels:
        app: stress
        tier: demo
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.0
          resources:
            requests:
              cpu: 100m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 128Mi
        - name: stress
          image: busybox:1.36
          resources:
            requests:
              cpu: 100m
              memory: 32Mi
            limits:
              cpu: 500m
              memory: 64Mi
          command:
            - sh
            - -c
            - while :; do :; done
YAML

Sample output:

namespace/top-lab created
deployment.apps/stress created

Wait for the Deployment and the first metrics sample:

kubectl rollout status deployment/stress -n top-lab --timeout=120s

Sample output:

deployment "stress" successfully rolled out

for attempt in {1..36}; do
  kubectl top pod -n top-lab -l app=stress >/dev/null 2>&1 && break
  sleep 5
done

kubectl top pod -n top-lab -l app=stress

The final command either prints metrics or exposes the error after a maximum wait of three minutes.

Store the Pod name for later examples:

POD=$(kubectl get pod -n top-lab -l app=stress -o jsonpath='{.items[0].metadata.name}')

The CPU loop runs until the Deployment or namespace is deleted.

Monitor Pods by name, namespace, and label

List Pods in the default namespace:

kubectl top pod

CPU is commonly shown in millicores (500m = half a core). These values are current usage, not configured requests or limits.

Query the lab Pod by name:

kubectl top pod "$POD" -n top-lab

Sample output:

NAME                      CPU(cores)   MEMORY(bytes)   
stress-5756544dd7-z7v8h   500m         2Mi

List every Pod in the lab namespace:

kubectl top pod -n top-lab

List Pods across all namespaces:

kubectl top pod -A | head -8

Sample output:

NAMESPACE            NAME                                       CPU(cores)   MEMORY(bytes)   
calico-system        calico-apiserver-f786987bb-bt84s           4m           58Mi            
calico-system        calico-apiserver-f786987bb-x56xr           3m           48Mi            
calico-system        calico-kube-controllers-86bbc9c477-hfxkb   9m           60Mi            
calico-system        calico-node-hc7ph                          32m          136Mi           
calico-system        calico-node-sc7p2                          45m          108Mi           
calico-system        calico-typha-6545b7d97f-htqpt              5m           52Mi            
calico-system        csi-node-driver-54j2p                      1m           18Mi

Truncate with head when the cluster lists hundreds of system Pods.

Filter Pods using labels:

kubectl top pod -n top-lab -l tier=demo

Sort Pods by CPU or memory

Sort by CPU to surface the hottest Pod first:

kubectl top pod -n top-lab --sort-by=cpu

Switch to memory when you suspect a leak rather than CPU burn:

kubectl top pod -n top-lab --sort-by=memory

With only one replica in top-lab, both commands return the same row; the flags matter when many Pods share a namespace.

View per-container metrics

Pod-level totals hide which container consumed the CPU. Use --containers to split multi-container Pods.

For one Pod:

kubectl top pod "$POD" -n top-lab --containers

Sample output:

POD                       NAME     CPU(cores)   MEMORY(bytes)   
stress-5756544dd7-z7v8h   nginx    0m           2Mi             
stress-5756544dd7-z7v8h   stress   500m         0Mi

List every container in a namespace:

kubectl top pod -n top-lab --containers

Per-container rows help when:

• A Pod runs several containers and only one spikes
• A sidecar consumes more than the main app
• You need to justify separate limits per container

For sidecar layout patterns, see Kubernetes sidecar example.

Compare usage with requests and limits

kubectl top answers how much a workload uses right now. Requests and limits in the Pod spec answer how much the scheduler reserved and how much the kubelet may enforce.

High usage is not automatically wrong. Compare live metrics with configured resources before you raise limits.

Read current Pod usage:

kubectl top pod -n top-lab

Inspect requests and limits on the same Pod:

kubectl describe pod "$POD" -n top-lab | grep -A4 'Limits:\|Requests:'

Sample output:

Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        100m
      memory:     64Mi
--
    Limits:
      cpu:     500m
      memory:  64Mi
    Requests:
      cpu:        100m
      memory:     32Mi

Pull the same fields from YAML:

kubectl get pod "$POD" -n top-lab -o jsonpath='{.spec.containers[*].name}{"\n"}{.spec.containers[*].resources}{"\n"}'

Sample output:

nginx stress
{"limits":{"cpu":"500m","memory":"128Mi"},"requests":{"cpu":"100m","memory":"64Mi"}} {"limits":{"cpu":"500m","memory":"64Mi"},"requests":{"cpu":"100m","memory":"32Mi"}}

In the lab, the stress container should approach its 500m CPU limit, although sampled values will vary, while nginx stays near idle. For requests, limits, and QoS classes, see Kubernetes resources and requests.

Troubleshoot kubectl top

When Metrics Server misbehaves, inspect its logs:

kubectl logs -n kube-system -l k8s-app=metrics-server --tail=20

Confirm Metrics Server has ready EndpointSlice backends:

kubectl get endpointslices \
  -n kube-system \
  -l kubernetes.io/service-name=metrics-server \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{"\tready="}{.conditions.ready}{"\n"}{end}'

Sample output:

192.168.5.33    ready=true

Empty output usually means the Metrics Server Pod is not ready yet.

What kubectl top Does Not Provide

kubectl top is a thin CLI over short-lived kubelet metrics. It does not give you:

• Historical metrics or trend graphs
• Dashboards or visualization
• Alerts or notifications
• Application-level metrics (HTTP latency, queue depth, etc.)
• Long-term capacity planning data
• Persistent monitoring storage

When you need retention and alerting, adopt a monitoring platform such as Prometheus with Grafana or your vendor's observability stack. That installation is outside this chapter.

What's Next

• kubectl logs, Events and describe with Examples
• Debug Kubernetes Pods with kubectl exec and kubectl debug
• Kubernetes Pod Troubleshooting Workflow

References

• Metrics Server — upstream project and install manifest
• Resource metrics pipeline — how Metrics Server fits the cluster
• kubectl top — command reference

Summary

You verified whether Metrics Server is installed, applied the upstream manifest when needed, and confirmed the v1beta1.metrics.k8s.io APIService before trusting any kubectl top output. On the lab cluster, a kubelet TLS workaround was required; production clusters should fix serving certificates instead of skipping verification.

With metrics flowing, kubectl top node exposed allocatable CPU and memory percentages, while kubectl top pod and --containers separated Pod totals from per-container usage. The top-lab Deployment showed how one container can approach its CPU limit while a sibling container remains mostly idle, which is why per-container metrics matter.

Remember that these numbers are snapshots, not requests, limits, or history. Pair kubectl top with kubectl describe or Pod YAML when you decide whether to change resource settings, and use kubectl logs, Events and describe when usage looks wrong but the cause is still unclear.