Fix Kubernetes OOMKilled and Exit Code 137

Your container died and kubectl get shows OOMKilled or a climbing restart count with exit code 137. Before you raise limits, confirm Kubernetes recorded an out-of-memory termination—not a manual SIGKILL or eviction for another reason. This walkthrough reproduces a controlled OOM in oom-lab, debugs a growing-memory Pod with cgroup and node checks, and applies the fix patterns that match production failures.

What OOMKilled and Exit Code 137 Mean

OOMKilled means the Linux kernel terminated the container process because memory use exceeded what the cgroup allows or because node-level memory pressure selected that process. Kubernetes copies the kernel outcome into the container status as reason OOMKilled. Anonymous heap, page cache, and cgroup accounting all feed that decision; Linux memory management explains why RSS alone rarely tells the full story on a node.

Exit code 137 commonly means the process received SIGKILL. Linux encodes signal termination as 128 + signal number; signal 9 is SIGKILL, so 128 + 9 = 137. OOM kills frequently produce both OOMKilled and 137, but exit code 137 alone does not prove memory exhaustion.

Confirm termination reason, Pod events, and memory configuration before you change limits or profile the application.

Prepare and Confirm the OOM Lab

Create the OOM Pod

Create a namespace and a Pod with a small memory limit and a stress workload. The container runs stress --vm to allocate anonymous memory past the cap; the stress command covers the same --vm-bytes pattern on a Linux host. restartPolicy: Never keeps one termination on the record for inspection:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Namespace
metadata:
  name: oom-lab
---
apiVersion: v1
kind: Pod
metadata:
  name: mem-oom
  namespace: oom-lab
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: amd64
  containers:
  - name: stress
    image: polinux/stress:1.0.4
    resources:
      limits:
        memory: 64Mi
      requests:
        memory: 32Mi
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "128M", "--vm-hang", "0"]
YAML

Wait for the kernel to record the OOM termination:

kubectl wait pod/mem-oom \
  -n oom-lab \
  --for=jsonpath='{.status.containerStatuses[0].state.terminated.reason}'=OOMKilled \
  --timeout=120s

kubectl get pod mem-oom -n oom-lab

kubectl wait supports waiting for an exact JSONPath status value.

Sample output:

NAME      READY   STATUS      RESTARTS   AGE
mem-oom   0/1     OOMKilled   0          15s

STATUS OOMKilled on a Never Pod means the main container terminated and will not restart. With restartPolicy: Always, STATUS may show CrashLoopBackOff while lastState still records OOMKilled.

Read Termination Reason and Logs

Read the container block and events:

kubectl describe pod mem-oom -n oom-lab

The stress container shows Reason: OOMKilled and Exit Code: 137 under State: Terminated. Limits list memory: 64Mi while the workload requested 128M of anonymous memory—well above the cgroup cap.

Pull reason and exit code programmatically when several containers share the Pod:

kubectl get pod mem-oom -n oom-lab -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state.terminated.reason}{"\t"}{.state.terminated.exitCode}{"\n"}{end}'

Sample output:

stress	OOMKilled	137

After a restart, read lastState.terminated instead of state.terminated for the previous instance:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

On a Pod with restartPolicy: Always that keeps OOM looping, that jsonpath commonly prints OOMKilled and 137 while STATUS shows CrashLoopBackOff.

Because this Pod uses restartPolicy: Never, its terminated instance remains the current container instance and ordinary kubectl logs reads its captured output:

kubectl logs mem-oom -n oom-lab -c stress

Sample output (truncated):

stress: info: [1] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd

Logs may end abruptly because SIGKILL does not allow graceful shutdown.

For a restarting workload, read the most recently terminated instance with:

kubectl logs POD_NAME -n NAMESPACE -c CONTAINER_NAME --previous

For describe, jsonpath, and event ordering detail, see kubectl logs, Events and describe.

Inspect Requests and Limits

Inspect what the scheduler reserved and what the cgroup enforces:

kubectl get pod mem-oom -n oom-lab -o jsonpath='{.spec.containers[0].resources}{"\n"}'

Sample output:

{"limits":{"memory":"64Mi"},"requests":{"memory":"32Mi"}}

• requests.memory participates in scheduling and QoS classification
• limits.memory is the maximum the container may use; exceeding it can trigger a cgroup OOM kill
• Increasing the request alone does not raise the limit

A typical Burstable block looks like:

resources:
  requests:
    memory: 128Mi
  limits:
    memory: 256Mi

Full request, limit, and QoS design lives in Kubernetes resource requests and limits.

Distinguish Container OOM from Node Pressure

Verify a container-limit OOM when:

• A memory limit is configured on the failing container
• Usage approached or exceeded that limit
• Only one container or one Pod pattern fails repeatedly
• Last termination reason is OOMKilled

Check Node MemoryPressure

Inspect memory pressure on the node that ran mem-oom:

NODE=$(kubectl get pod mem-oom -n oom-lab -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" -o jsonpath='MemoryPressure={.status.conditions[?(@.type=="MemoryPressure")].status}{"\n"}'

Sample output:

MemoryPressure=False

When MemoryPressure is True, check cluster events and whether multiple Pods on the same node fail together:

kubectl get events -A --field-selector involvedObject.kind=Node | tail -5

Kernel OOM and eviction detail on the assigned worker is out of scope here—focus on Kubernetes signals first, then node logs if several unrelated workloads fail on one host.

Check Current Memory Usage

kubectl top shows recent usage, not the exact historical peak that caused a fast spike:

kubectl top node

Sample output:

NAME       CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
k8s-cp     868m         28%      3131Mi          47%         
worker01   17m          1%       1077Mi          57%

For a running Pod while Metrics Server is installed:

kubectl top pod <pod-name> -n <namespace> --containers

Metrics Server must be running. A terminated Pod such as mem-oom will not appear in top. A spike between scrapes may not appear in top even though the kernel killed the container. Compare each container separately in multi-container Pods. Command flags and install steps are in Monitor Pods and nodes with kubectl top.

Understand QoS Behaviour

Kubernetes assigns a QoS class from resource configuration:

Check the class on the lab Pod:

kubectl get pod mem-oom -n oom-lab -o jsonpath='{.status.qosClass}{"\n"}'

Sample output:

Burstable

QoS influences eviction and OOM priority during node memory pressure—BestEffort Pods are generally evicted first. Guaranteed does not protect a container that exceeds its own configured memory limit.

Identify the Source of Memory Growth

Before you change limits, look for what consumed memory:

• Application logs before termination (kubectl logs --previous on restarting workloads)
• Per-container usage from kubectl top while the Pod is still running
• Recent deploys, config changes, or traffic spikes
• Larger request payloads or higher concurrency
• Cache growth or suspected leaks
• Large in-memory buffers or files
• Memory-backed emptyDir volumes counting toward container memory
• Sidecar containers in the same Pod

Files written to a memory-backed emptyDir count against the memory usage of the container that wrote them. In a multi-container Pod, identify which container writes to the volume and compare that container's usage with its limit.

Kubernetes tracks tmpfs-backed emptyDir data as container memory rather than ephemeral storage.

Debug OOM Before You Raise Limits

Raising limits.memory is a valid fix when measured peak usage exceeds the cap. It is not the right first move when memory climbs on every restart without plateauing. The mem-leak Pod below allocates eight mebibytes every three seconds until the cgroup kills it, then restarts. Use it to practice the checks you run on a suspect workload before you edit YAML.

Deploy the Growing-Memory Lab Pod

Create a Pod with a 128Mi limit and a Python loop that keeps anonymous memory:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: mem-leak
  namespace: oom-lab
spec:
  restartPolicy: Always
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: amd64
  containers:
  - name: leak
    image: python:3.12-slim-bookworm
    resources:
      limits:
        memory: 128Mi
      requests:
        memory: 64Mi
    command: ["python3", "-c"]
    args:
    - |
      import time
      chunks = []
      mb = 0
      while True:
          chunks.append(bytearray(8 * 1024 * 1024))
          mb += 8
          print(f"allocated {mb} MiB", flush=True)
          time.sleep(3)
YAML

Wait until the container is running so you can inspect it before the next OOM:

kubectl wait pod/mem-leak -n oom-lab --for=condition=Ready --timeout=60s

kubectl get pod mem-leak -n oom-lab

Sample output:

NAME       READY   STATUS    RESTARTS   AGE
mem-leak   1/1     Running   0          12s

RESTARTS 0 with Running means you still have a live container to exec into. After the kernel kills it, the count climbs and lastState records OOMKilled.

Watch Memory Climb from kubectl

Application logs are often the fastest signal that memory is growing on purpose or leaking:

kubectl logs mem-leak -n oom-lab -c leak --tail=6

Sample output:

allocated 16 MiB
allocated 24 MiB
allocated 32 MiB
allocated 40 MiB
allocated 48 MiB
allocated 56 MiB

The counter rises every few seconds with no idle drop. That pattern points to runaway allocation rather than a one-time startup spike.

While the Pod is still Running, compare live usage against the limit. Metrics Server must be installed:

kubectl top pod mem-leak -n oom-lab --containers

Sample output:

POD        NAME   CPU(cores)   MEMORY(bytes)   
mem-leak   leak   2m           93Mi

93Mi against a 128Mi limit means the cgroup is nearly full. A terminated Pod such as mem-oom does not appear in top; run these checks while the suspect workload is alive.

Read the Cgroup Budget Inside the Container

Kubernetes enforces memory through the container cgroup. For how cgroup v1 and v2 map to Kubernetes limits, why page cache can count toward the cap, and why free inside a container misleads, see Linux memory limits in containers. On cgroup v2 nodes, read the limit, current use, and peak from inside the container:

kubectl exec mem-leak -n oom-lab -c leak -- cat /sys/fs/cgroup/memory.max

Sample output:

134217728

134217728 bytes is 128Mi, matching limits.memory in the Pod spec.

kubectl exec mem-leak -n oom-lab -c leak -- cat /sys/fs/cgroup/memory.current

Sample output:

76808192

Current use is roughly 73Mi and still climbing toward the 128Mi cap.

kubectl exec mem-leak -n oom-lab -c leak -- cat /sys/fs/cgroup/memory.peak

Sample output:

82509824

memory.peak records the high-water mark for this container instance. When peak approaches memory.max, the next allocation can trigger a cgroup OOM kill.

On cgroup v1 nodes, read memory.limit_in_bytes and memory.usage_in_bytes under /sys/fs/cgroup/memory/ instead.

Find Which PID Owns the Memory

Do not assume PID 1 is your application. Shell wrappers (sh -c), init helpers such as tini, and entrypoint scripts often sit at PID 1 while the worker that allocates memory is a child process. When the image ships ps, the ps command can list RSS with aux or -o rss; on minimal images, rank every process by resident set size using only /proc:

kubectl exec mem-leak -n oom-lab -c leak -- sh -c '
for p in /proc/[0-9]*; do
  pid=${p#/proc/}
  rss=$(awk "/^VmRSS:/{print \$2}" "$p/status" 2>/dev/null) || continue
  cmd=$(tr "\0" " " < "$p/cmdline" 2>/dev/null)
  [ -z "$cmd" ] && cmd=$(awk "/^Name:/{print \$2}" "$p/status")
  printf "%s\t%s kB\t%s\n" "$pid" "$rss" "$cmd"
done | sort -t"$(printf "\t")" -k2 -nr | head -5
'

Sample output when Python is PID 1:

1	83340 kB	python3 -c import time
47	1624 kB	sh -c

Sample output when a shell wrapper starts the worker (common with command: ["sh", "-c", "..."]):

7	34240 kB	python3 -c 
1	1784 kB	sh -c python3 -c "

The first column is the PID to inspect. The highest VmRSS row is usually the process the kernel will score first inside the cgroup.

Set PID from that list before the checks below. The examples use 1 because mem-leak runs Python as PID 1; replace it when your sort shows a child PID on top:

PID=1

Read process identity, parent, and memory counters for that PID. The follow-up checks filter /proc files with grep command extended patterns:

kubectl exec mem-leak -n oom-lab -c leak -- grep -E '^(Name|Pid|PPid|VmRSS|VmSize|VmPeak|Threads):' "/proc/${PID}/status"

Sample output:

Name:	python3
Pid:	1
PPid:	0
VmPeak:	  112388 kB
VmSize:	  112388 kB
VmRSS:	  107928 kB
Threads:	1

PPid is the parent PID. When PPid is 1 and PID 1 is sh, the shell spawned the heavy child you found in the sort output.

Read the full command line (null bytes in /proc/PID/cmdline render as spaces):

kubectl exec mem-leak -n oom-lab -c leak -- sh -c 'tr "\0" " " < /proc/'"$PID"'/cmdline; echo'

Sample output (truncated):

python3 -c import time
chunks = []
mb = 0
while True:
    chunks.append(bytearray(8 * 1024 * 1024))

smaps_rollup breaks RSS into anonymous heap mappings versus file-backed pages. RSS, PSS, and USS on bare metal are covered in check memory usage per process in Linux:

kubectl exec mem-leak -n oom-lab -c leak -- grep -E '^(Rss|Pss|Anonymous):' "/proc/${PID}/smaps_rollup"

Sample output:

Rss:              116124 kB
Pss:              112975 kB
Anonymous:        110488 kB

Large Anonymous with climbing VmRSS usually means heap or anonymous allocations, not a memory-mapped file on disk.

Confirm which binary is running and whether ulimit caps apply inside the container:

kubectl exec mem-leak -n oom-lab -c leak -- readlink "/proc/${PID}/exe"

Sample output:

/usr/local/bin/python3.12

kubectl exec mem-leak -n oom-lab -c leak -- grep -E '^(Max resident set|Max address space|Max processes)' "/proc/${PID}/limits"

Sample output:

Max resident set          unlimited            unlimited            bytes     
Max address space         unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes

When Max address space is unlimited, the cgroup memory.max value is the hard ceiling, not ulimit. For setting prlimit or entrypoint ulimit before your app starts, see ulimit in Kubernetes Pods.

Optional container-wide view from /proc/meminfo (shows the node memory the container can see, not the cgroup cap):

kubectl exec mem-leak -n oom-lab -c leak -- head -4 /proc/meminfo

Sample output:

MemTotal:        2008296 kB
MemFree:          192008 kB
MemAvailable:     790504 kB
Buffers:           56908 kB

Prefer memory.current and memory.max from the cgroup files above for limit comparisons. Use /proc/${PID}/status and smaps_rollup to find which process is consuming the cgroup budget. When the Pod restarts faster than kubectl exec allows, attach a debug container with kubectl debug Pods.

Minimal images may not ship ps, top, or free. /proc, /sys/fs/cgroup, and kubectl exec cover the checks you need on stripped production images.

Record the Kill After Restart

Wait until Kubernetes records an OOM on the previous instance:

kubectl wait pod/mem-leak -n oom-lab --for=jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'=OOMKilled --timeout=120s

kubectl get pod mem-leak -n oom-lab

Sample output:

NAME       READY   STATUS    RESTARTS      AGE
mem-leak   1/1     Running   1 (12s ago)   58s

RESTARTS 1 with Running means a new container started after the kill. Read the previous instance, not the fresh one:

kubectl get pod mem-leak -n oom-lab -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

Sample output:

leak	OOMKilled	137

Logs from the killed instance stop at the last successful print:

kubectl logs mem-leak -n oom-lab -c leak --previous --tail=8

Sample output:

allocated 64 MiB
allocated 72 MiB
allocated 80 MiB
allocated 88 MiB
allocated 96 MiB
allocated 104 MiB
allocated 112 MiB
allocated 120 MiB

The process died just after 120 MiB while the limit was 128Mi. Overhead from the Python runtime and libc fills the remaining cgroup headroom, so the kill lands below the limit string in YAML.

Inspect the Worker Node with SSH

When you have node access, kernel logs distinguish a container cgroup kill from node-wide memory exhaustion. The worker checks below use ssh command from your workstation; resolve the node name first:

NODE=$(kubectl get pod mem-leak -n oom-lab -o jsonpath='{.spec.nodeName}')
echo "$NODE"

Sample output:

worker01

On that node, search dmesg for recent OOM lines. Resolve the node IP when the hostname does not resolve from your workstation:

NODE_IP=$(kubectl get node "$NODE" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')
echo "$NODE_IP"

Sample output:

192.168.56.109

ssh root@"$NODE_IP" "dmesg -T | grep -iE 'oom|killed process' | tail -5"

Sample output:

[Mon Jul 27 07:53:51 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-b1ce3267795e982509f8acafd660e6b665de15ee7454367144aaed25468d0820.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc808fa49_7e50_41b0_8940_5465c7903885.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc808fa49_7e50_41b0_8940_5465c7903885.slice/cri-containerd-b1ce3267795e982509f8acafd660e6b665de15ee7454367144aaed25468d0820.scope,task=python3,pid=579578,uid=0
[Mon Jul 27 07:53:51 2026] Memory cgroup out of memory: Killed process 579578 (python3) total-vm:145172kB, anon-rss:130060kB, file-rss:5292kB, shmem-rss:0kB, UID:0 pgtables:316kB oom_score_adj:968

CONSTRAINT_MEMCG and Memory cgroup out of memory confirm the container exceeded its cgroup budget. Node-wide kills omit CONSTRAINT_MEMCG and often list several unrelated processes in the OOM summary.

You can read the same cgroup counters from the node when exec is not available. Resolve the Pod UID and cgroup scope path:

POD_UID=$(kubectl get pod mem-leak -n oom-lab -o jsonpath='{.metadata.uid}')
POD_UID_ESC=${POD_UID//-/_}

ssh root@"$NODE_IP" "ls -d /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod${POD_UID_ESC}.slice/cri-containerd*.scope"

Sample output:

/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc808fa49_7e50_41b0_8940_5465c7903885.slice/cri-containerd-08b2da4c56f1b24cda2edbe91bbf0eb29d9175f3684ac8156a1e01b129e34a9e.scope

ssh root@"$NODE_IP" 'SCOPE=$(ls -d /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod'"${POD_UID_ESC}"'.slice/cri-containerd*.scope | head -1); echo "limit=$(cat "$SCOPE/memory.max")"; echo "current=$(cat "$SCOPE/memory.current")"; echo "peak=$(cat "$SCOPE/memory.peak")"'

Sample output (values change while the container is running):

limit=134217728
current=13508608
peak=13680640

Node checks need root SSH on the worker. Skip them when you only have namespace-scoped RBAC; the in-container cgroup files and kubectl describe usually suffice for container-limit OOM.

Decide: Raise the Limit, Fix the Workload, or Fix the Node

Use the signals together instead of editing limits by default:

Doubling the limit on a leak only buys time. Deploy a one-shot copy with a 256Mi cap to prove the point:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: mem-leak-wide
  namespace: oom-lab
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: amd64
  containers:
  - name: leak
    image: python:3.12-slim-bookworm
    resources:
      limits:
        memory: 256Mi
      requests:
        memory: 64Mi
    command: ["python3", "-c"]
    args:
    - |
      import time
      chunks = []
      mb = 0
      while True:
          chunks.append(bytearray(8 * 1024 * 1024))
          mb += 8
          print(f"allocated {mb} MiB", flush=True)
          time.sleep(1)
YAML

kubectl wait pod/mem-leak-wide -n oom-lab --for=jsonpath='{.status.containerStatuses[0].state.terminated.reason}'=OOMKilled --timeout=120s

kubectl get pod mem-leak-wide -n oom-lab

Sample output:

NAME            READY   STATUS      RESTARTS   AGE
mem-leak-wide   0/1     OOMKilled   0          34s

kubectl logs mem-leak-wide -n oom-lab -c leak --tail=5

Sample output:

allocated 216 MiB
allocated 224 MiB
allocated 232 MiB
allocated 240 MiB
allocated 248 MiB

The wider limit delayed the kill but did not stop it. Logs still climb until the cgroup kills the process. For a leak, cap caches, fix the code path, or set runtime heap limits below the cgroup cap. Raise Kubernetes limits only after usage plateaus at a level you accept.

Fix OOMKilled

Raise an Undersized Limit

When normal usage exceeds the configured limit under representative load:

1. Measure usage under realistic load with kubectl top or your metrics stack
2. Add headroom for short spikes above steady usage
3. Raise request and limit together when scheduling and cap should move together
4. Apply the updated Deployment or Pod template
5. Watch restart count and rollout status
6. Confirm new Pods stay Ready and termination reason no longer flips to OOMKilled

Apply a safer limit for the same stress pattern:

kubectl apply -f - <<'YAML'
apiVersion: v1
kind: Pod
metadata:
  name: mem-ok
  namespace: oom-lab
spec:
  restartPolicy: Never
  nodeSelector:
    kubernetes.io/os: linux
    kubernetes.io/arch: amd64
  containers:
  - name: stress
    image: polinux/stress:1.0.4
    resources:
      limits:
        memory: 256Mi
      requests:
        memory: 64Mi
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "64M", "--vm-hang", "30", "--timeout", "10s"]
YAML

kubectl wait pod/mem-ok -n oom-lab --for=jsonpath='{.status.phase}'=Succeeded --timeout=120s
kubectl get pod mem-ok -n oom-lab

A Succeeded phase means every container terminated successfully and will not restart.

Sample output:

NAME     READY   STATUS      RESTARTS   AGE
mem-ok   0/1     Completed   0          15s

Completed with exit code 0 confirms the workload fit inside the new limit. Do not remove all memory limits to silence OOMKilled—that trades one failure mode for node-wide risk.

Fix an Application Memory Leak

The mem-leak lab shows what a leak looks like in practice: logs climb on every restart, memory.current approaches memory.max, and doubling the limit only delays the next OOMKilled.

Leak indicators:

• Memory usage climbs on every run without plateauing
• The container survives briefly after each restart, then dies again
• Raising the limit only delays the next kill
• Usage does not return to a stable baseline after idle time

Profile the application, review recent code or dependency changes, cap caches and worker pools, and set runtime heap limits below the container limit. For Valgrind, heap trackers, and host-level leak workflows on Linux, see how to find memory leaks. Language-specific profilers stay outside this article. Treat leaks as an application fix, not only a limit bump.

Configure Runtime Memory

Some runtimes need heap or allocator caps below limits.memory to leave room for:

• Native allocations and thread stacks
• Shared libraries and runtime overhead
• Network and I/O buffers

Java heap and Node.js --max-old-space-size are common examples—set them under the cgroup limit so the runtime fails predictably before the kernel OOM killer intervenes.

Resolve Node Memory Pressure

When the container did not exceed its own limit but the node reports pressure:

• List heavy consumers with kubectl top pod --sort-by=memory across namespaces
• Verify workloads declare realistic memory requests so the scheduler spreads load
• Scale down or reschedule noisy neighbors
• Add node capacity when the fleet is chronically full
• Read kubelet eviction events when Pods disappear with Evicted instead of OOMKilled

Detailed node administration belongs in CKA node and capacity material—not expanded here.

Verify the Fix

Confirm recovery across several checks:

• Restart count stops increasing on Deployments with restartPolicy: Always
• Last termination reason no longer becomes OOMKilled
• Pod reaches Ready when probes are configured
• kubectl top stays comfortably below limits.memory during normal work
• Node MemoryPressure returns to False when the issue was fleet-wide
• Deployment kubectl rollout status completes when you changed a template

Monitor long enough to cover the traffic pattern that previously triggered the kill—a spike-only workload may look healthy at idle.

Troubleshoot Common OOM Patterns

What's Next

• Delete a Stuck or Terminating Kubernetes Pod
• Fix a Kubernetes Deployment Not Creating Pods
• Fix Kubernetes Node NotReady and kubelet Errors

References

• Manage memory for containers — requests, limits, and OOM behaviour
• Configure Quality of Service for Pods — Guaranteed, Burstable, and BestEffort
• Node-pressure eviction — kubelet behaviour when nodes run low on memory

Summary

OOMKilled tells you Kubernetes recorded a kernel out-of-memory kill—not merely that the exit code was 137. The mem-oom lab Pod hit a 64Mi limit while stress allocated 128M, producing reason OOMKilled and exit code 137 in one clear chain. Always read reason, limit, and events together before you change YAML.

The mem-leak lab adds a full debugging path: watch logs and kubectl top while memory climbs, read memory.current and memory.peak inside the container, confirm CONSTRAINT_MEMCG in node dmesg when you have SSH, and use lastState plus kubectl logs --previous after each restart. Doubling the limit on mem-leak-wide still ends in OOMKilled, which is why leak fixes beat repeated limit bumps.

Container-limit OOM usually isolates one workload; node MemoryPressure and multiple failing Pods on the same host point toward fleet or node capacity. kubectl top helps with steady usage but can miss spikes between Metrics Server scrapes—do not treat a quiet top line as proof the limit is generous enough.

Fixes follow the diagnosis: raise the memory limit with measured headroom when legitimate peak usage exceeds it, and adjust the request separately when sustained usage justifies reserving more memory. Profile the application when memory climbs every restart, and set runtime heaps below the cgroup cap when the process dies below the Kubernetes limit. When restarts mix OOM with other errors, continue with the CrashLoopBackOff troubleshooting workflow for the full restart path.