Kubernetes Liveness, Readiness and Startup Probes

Tested on Rocky Linux 10.2 (Red Quartz) workstation
Package kubectl 1.36.3
Applies to Any host with kubectl configured; Kubernetes 1.27+ cluster with Linux worker nodes
Cert prep CKA · CKAD · CKS
Lab environment Multi-node kubeadm cluster with containerd — install Kubernetes with kubeadm
Privilege Normal user (no sudo required on the workstation)
Scope Liveness, readiness, and startup probes; HTTP, TCP, exec, and gRPC probe types; timing fields; probe failure effects; controlled readiness and liveness demos; decision tables; and common misconfiguration mistakes. Does not cover full Pod troubleshooting runbooks, Service YAML, application health-endpoint development, load balancer health checks, readiness gates, lifecycle hooks, or PodDisruptionBudgets.

Probes are how the kubelet checks container health. This walkthrough configures HTTP probes on a web Deployment in probe-lab, then shows shorter TCP, exec, and gRPC examples so you can match the probe type to what your application actually exposes.


How Kubernetes Probes Work

A probe is a diagnostic performed periodically by the kubelet against a container. An exec probe runs a command inside the container, while HTTP, TCP, and gRPC probes make network requests to the Pod. HTTP and gRPC probes target the Pod IP, TCP checks originate from the node, and exec commands run inside the container.

Probe Main purpose Result of repeated failure
Liveness Detect an unhealthy running container Container is restarted
Readiness Decide whether the Pod can receive traffic Pod becomes NotReady
Startup Protect slow-starting containers Container is restarted

Kubernetes supports HTTP GET, TCP socket, exec, and gRPC probes. Built-in gRPC probes have been stable since Kubernetes v1.27. Native gRPC probes are a Kubernetes feature and do not need a special container-runtime capability.

Liveness vs Readiness vs Startup

Use the probes for different failure modes:

  • Readiness — temporary inability to serve requests. Traffic should pause, but a restart may not help.
  • Liveness — the process is alive but stuck. Restart only when that restart can recover the application.
  • Startup — initialization takes longer than normal liveness timing allows.

When a startup probe is defined, liveness and readiness checks do not run until startup succeeds.

A Pod can show phase Running while the Ready condition is false. kubectl get pods may display 0/1 in the READY column even though STATUS is Running. Phase and readiness are separate signals — see Kubernetes Pods and Pod lifecycle for conditions and STATUS meanings.


Configure Kubernetes Probe Types

HTTP Probe

This Deployment uses nginx with a ConfigMap that returns HTTP 200 on /startup, /ready, and /health. Each probe targets a different path on the named port http.

Save probe-nginx.conf:

nginx
server {
  listen 80;
  root /usr/share/nginx/html;
  default_type text/plain;

  location = /startup {
    try_files /startup-ok =503;
  }

  location = /ready {
    try_files /ready-ok =503;
  }

  location = /health {
    try_files /health-ok =500;
  }
}

Create the ConfigMap and Deployment:

bash
kubectl create namespace probe-lab
bash
kubectl create configmap probe-nginx-conf -n probe-lab --from-file=default.conf=probe-nginx.conf

Save web-probes.yaml:

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: probe-lab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.0
          command: ["/bin/sh", "-c"]
          args:
            - |
              printf 'startup ok\n' > /usr/share/nginx/html/startup-ok
              printf 'ready ok\n' > /usr/share/nginx/html/ready-ok
              printf 'health ok\n' > /usr/share/nginx/html/health-ok
              exec nginx -g 'daemon off;'
          ports:
            - name: http
              containerPort: 80
          volumeMounts:
            - name: nginx-conf
              mountPath: /etc/nginx/conf.d/default.conf
              subPath: default.conf
          startupProbe:
            httpGet:
              path: /startup
              port: http
              scheme: HTTP
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /ready
              port: http
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: http
            periodSeconds: 10
            failureThreshold: 3
      volumes:
        - name: nginx-conf
          configMap:
            name: probe-nginx-conf

The startup command recreates the marker files whenever the container restarts.

Important httpGet fields:

  • path — URL path the kubelet requests
  • port — number or named port from the container spec
  • schemeHTTP or HTTPS
  • httpHeaders — optional extra headers (not shown here)

Apply the manifest:

bash
kubectl apply -f web-probes.yaml

Sample output:

output
deployment.apps/web created

Wait until the Deployment reports available:

bash
kubectl wait --for=condition=Available deployment/web -n probe-lab --timeout=120s
bash
kubectl get pods -n probe-lab -l app=web

Sample output:

output
NAME                   READY   STATUS    RESTARTS   AGE
web-6f48c76c7c-qbrhv   1/1     Running   0          3s

Inspect probe configuration on the Pod:

bash
kubectl describe pod -n probe-lab -l app=web

The Liveness, Readiness, and Startup sections list probe type, path, port, and timing. Events at the bottom show probe failures when they occur.

Watch READY change during later tests:

bash
kubectl get pods -n probe-lab -l app=web --watch

Stop the watch with Ctrl+C when you finish observing changes.


TCP Socket Probe

A TCP probe succeeds when the kubelet opens a TCP connection to the port. It does not read HTTP bodies or validate application logic.

yaml
livenessProbe:
  tcpSocket:
    port: 80
  periodSeconds: 5

Use TCP when the process listens on a port but does not expose a dedicated HTTP health path. An open socket does not prove the application returns correct responses.


exec Probe

An exec probe runs a command inside the container namespace. Exit code 0 means success. Any non-zero exit code means failure.

yaml
livenessProbe:
  exec:
    command:
      - cat
      - /tmp/healthy
  periodSeconds: 5

The command and any binaries it calls must exist in the container image. A typical pattern creates /tmp/healthy during container start, then probes with cat.

Shell quoting, command versus args, and entrypoint interaction are covered in commands, args and environment variables.


gRPC Probe

gRPC probes use the gRPC Health Checking Protocol. The application must implement the health service on the target port.

yaml
livenessProbe:
  grpc:
    port: 50051
    service: health
  periodSeconds: 5

The port field must be numeric. Named ports are not supported for gRPC probes. If the service name is wrong or the health RPC is not implemented, the probe fails.

Lab example with agnhost. Save the manifest as grpc-probe.yaml:

yaml
apiVersion: v1
kind: Pod
metadata:
  name: grpc-probe
  namespace: probe-lab
spec:
  containers:
    - name: agnhost
      image: registry.k8s.io/e2e-test-images/agnhost:2.53
      args:
        - grpc-health-checking
        - --port=50051
      ports:
        - containerPort: 50051
      readinessProbe:
        grpc:
          port: 50051
        periodSeconds: 2
        failureThreshold: 3
      livenessProbe:
        grpc:
          port: 50051
        periodSeconds: 5
bash
kubectl apply -f grpc-probe.yaml

Sample output:

output
pod/grpc-probe created
bash
kubectl wait --for=condition=Ready pod/grpc-probe -n probe-lab --timeout=60s

Sample output:

output
pod/grpc-probe condition met
bash
kubectl describe pod grpc-probe -n probe-lab

The Readiness and Liveness sections should show grpc <pod>:50051. Without a readiness probe, the container can become Ready before its first gRPC liveness check succeeds; readiness controls the Pod Ready condition, while liveness controls container restarts. The wait command above verifies that the gRPC health service actually returns SERVING. The agnhost grpc-health-checking command supports --port, so the --port=50051 argument is valid. This article does not cover implementing the gRPC health service in application code.


Tune Probe Timing and Thresholds

Field Purpose
initialDelaySeconds Delay before the first probe
periodSeconds Interval between checks
timeoutSeconds Time allowed for one check
failureThreshold Consecutive failures before action
successThreshold Consecutive successes required after failure

Approximate startup allowance is initialDelaySeconds + (failureThreshold × periodSeconds). Probe execution and timeout timing can make the observed wall-clock interval vary slightly.

For the HTTP Deployment manifest above:

0 + (30 × 2 seconds) = approximately 60 seconds

Kubernetes documents startup sizing using the initial delay plus the failure-threshold and period window. Liveness and startup probes require successThreshold: 1. Readiness can use a higher successThreshold when you want several consecutive successes before marking the Pod ready.

There is no single timing preset that fits every application. Tune from observed startup time and failure behavior rather than copying defaults blindly.


How Probe Failures Affect the Pod

Readiness probe failure

When readiness fails repeatedly:

  • The container keeps running.
  • The Pod Ready condition becomes False.
  • Endpoints for matching Services and EndpointSlices drop the Pod until readiness succeeds again.

Traffic stops, but the process is not restarted.

Liveness probe failure

When liveness fails past failureThreshold:

  • The kubelet stops and restarts the container.
  • RESTARTS increases on kubectl get pods.
  • Repeated failures can lead to CrashLoopBackOff backoff between restarts.

Startup probe failure

When startup fails past its threshold:

  • The container is restarted before liveness and readiness run.
  • Fix slow startup by widening startup timing, not by disabling liveness without analysis.

Test Readiness and Liveness Failures

This controlled demo uses the web Deployment from the HTTP section. Readiness and liveness are tested on the same running container by removing marker files the nginx config serves.

Confirm the Pod is ready:

bash
kubectl get pods -n probe-lab -l app=web

Sample output:

output
NAME                   READY   STATUS    RESTARTS   AGE
web-6f48c76c7c-qbrhv   1/1     Running   0          3s

Readiness failure test

bash
POD=$(kubectl get pod -n probe-lab -l app=web -o jsonpath='{.items[0].metadata.name}')
bash
kubectl exec -n probe-lab "$POD" -- rm -f /usr/share/nginx/html/ready-ok
bash
kubectl wait --for=condition=Ready=false pod/"$POD" -n probe-lab --timeout=30s
bash
kubectl get pod "$POD" -n probe-lab

Expected state:

output
NAME                   READY   STATUS    RESTARTS   AGE
web-6f48c76c7c-qbrhv   0/1     Running   0          2m

Restore readiness:

bash
kubectl exec -n probe-lab "$POD" -- sh -c "printf 'ready ok\n' > /usr/share/nginx/html/ready-ok"
bash
kubectl wait --for=condition=Ready pod/"$POD" -n probe-lab --timeout=30s

A readiness failure marks the container and Pod not ready without restarting it.

Liveness failure test

bash
BEFORE=$(kubectl get pod "$POD" -n probe-lab -o jsonpath='{.status.containerStatuses[0].restartCount}')
bash
kubectl exec -n probe-lab "$POD" -- rm -f /usr/share/nginx/html/health-ok

Wait for the restart instead of immediately assuming RESTARTS=1:

bash
DEADLINE=$((SECONDS + 120))

while true; do
CURRENT=$(kubectl get pod "$POD" -n probe-lab -o jsonpath='{.status.containerStatuses[0].restartCount}')

  if [ "$CURRENT" -gt "$BEFORE" ]; then
    break
  fi

  if [ "$SECONDS" -ge "$DEADLINE" ]; then
    echo "Timed out waiting for the liveness restart" >&2
    exit 1
  fi

  sleep 2
done
bash
kubectl get pod "$POD" -n probe-lab

The Pod name remains unchanged while the container restart count increases. The container startup command then recreates health-ok.

Inspect probe events:

bash
kubectl describe pod "$POD" -n probe-lab

Sample Events lines:

output
Warning  Unhealthy  ...  Liveness probe failed: HTTP probe failed with statuscode: 500

The kubelet restarted the container because the liveness probe could not read a healthy response from /health.


Choose the Correct Probe

Requirement Probe
Do not send traffic until startup is complete Readiness (after startup succeeds)
Temporarily stop traffic during overload or maintenance Readiness
Restart an application stuck in an unrecoverable state Liveness
Allow a slow application extra startup time Startup
Check a web endpoint HTTP httpGet
Check whether a port accepts connections TCP tcpSocket
Run an application-specific command exec
Check a gRPC health service gRPC

Common Probe Configuration Mistakes

Symptom Likely cause Fix
Same strict dependency check on liveness and readiness Restart loops during external outages Use readiness for dependencies; keep liveness focused on the local process
Container restarts during slow startup Liveness runs before the app listens Add a startup probe with a longer failureThreshold × periodSeconds window
Probe always fails on a healthy app Wrong port, path, or scheme Match probe settings to the listening port and health URL
exec probe always fails Command or file missing in the image Verify binaries and paths inside the container filesystem
Pod Running but not Ready Readiness probe failing kubectl describe pod — check readiness Events and probe fields
Rising RESTARTS or CrashLoopBackOff Liveness too aggressive or app exits Inspect probe timing, logs, and previous container logs

For describe, logs, conditions, and restart-count workflows, use Kubernetes Pods and Pod lifecycle. When restarts coincide with memory limits, see OOMKilled and exit code 137. Kubernetes Services explains how ready Pods become Service endpoints.


What's Next


References


Summary

You configured liveness, readiness, and startup probes on one Deployment, using separate HTTP paths so each check can target a different concern. Readiness failures removed the Pod from the ready set without killing the container. Liveness failures increased the restart count when the kubelet could not reach the health endpoint.

The main configuration mistake is using liveness for problems a restart cannot fix. Pair readiness with Service traffic decisions, startup with slow initialization, and liveness only when restarting the container is a valid recovery step. Validate probe YAML with kubectl apply --dry-run=server before production changes — see kubectl apply, edit, patch and replace for server-side dry-run mechanics.

When probes look correct but traffic still misbehaves, read Pod conditions and Service endpoints together.


Frequently Asked Questions

1. What is the difference between liveness and readiness probes?

Liveness decides whether the kubelet should restart a container that is stuck in an unrecoverable state. Readiness decides whether the Pod should receive Service traffic. A failing readiness probe marks the Pod NotReady without restarting the container.

2. When should I use a startup probe?

Use a startup probe when application initialization can take longer than your normal liveness timing allows. Until the startup probe succeeds, liveness and readiness checks are disabled for that container.

3. Can a Pod be Running but not Ready?

Yes. Pod phase Running means the Pod is scheduled and at least one container is active. The Ready condition stays false when readiness probes fail or containers are not ready.

4. What HTTP status codes count as probe success?

The kubelet treats HTTP response codes from 200 through 399 as success for httpGet probes. Connection errors and codes outside that range count as failure.

5. Should liveness and readiness use the same check?

Usually not. Readiness can reflect temporary dependencies such as load or maintenance. Liveness should only restart the container when that restart can recover the process itself, not when an external dependency is down.

6. Does a TCP probe verify application health?

No. A TCP probe only confirms that something accepts connections on the port. It does not validate HTTP responses, command output, or gRPC health status.
Deepak Prasad

R&D Engineer

Founder of GoLinuxCloud with more than 15 years of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels across development, DevOps, networking, and security, delivering robust and efficient solutions for diverse projects.

  • Go (programming language)
  • Python (programming language)
  • DevOps
  • Computer Security
  • Cloud Computing
  • Kubernetes
  • Linux
  • Ansible (software)