Kubernetes StatefulSet with Examples

Tested on Rocky Linux 10.2 (Red Quartz) workstation
Package kubectl 1.36.3
Applies to Any host with kubectl configured; any Kubernetes cluster
Cert prep CKA · CKAD
Lab environment Multi-node kubeadm cluster with containerd — install Kubernetes with kubeadm
Privilege Normal user (no sudo required on the workstation)
Scope StatefulSet and headless Service YAML, stable identity, volumeClaimTemplates, ordered scale and rollout, PVC retention policy, DNS verification, and troubleshooting. Does not cover storage provisioner install, database clustering, backup, probes, or full Services tutorial.
Related guides Kubernetes volumes
Choose a Kubernetes workload resource

This walkthrough uses one StatefulSet named database with three replicas in the sts-lab namespace. You need a cluster StorageClass that supports dynamic volume provisioning. The lab uses local-path when that StorageClass is present.

What Is a Kubernetes StatefulSet?

A StatefulSet manages Pods that require stable identity. Unlike a Deployment, each replica is distinguishable by name and ordinal rather than treated as fully interchangeable.

  • Each Pod receives a predictable name such as database-0, database-1, database-2
  • Pods can retain persistent storage across recreation when volumeClaimTemplates is configured
  • With the default OrderedReady policy, StatefulSet Pods are created and scaled in ordinal order, and rolling updates normally proceed from the highest ordinal downward
  • Deleting the StatefulSet object directly does not guarantee ordered, graceful Pod termination; scale it to zero first when ordered shutdown is required

A StatefulSet normally pairs with a headless Service so each Pod has a stable DNS hostname. The StatefulSet manages the Pods. The governing headless Service provides their network identity, while volumeClaimTemplates creates ordinal-associated PVCs when configured.

When to use a StatefulSet

Reach for a StatefulSet when the application needs stable ordinal identity, predictable per-Pod DNS, ordered lifecycle, per-replica storage, or a combination of these properties.

  • databases and distributed data stores
  • message brokers with per-node data directories
  • cluster members that must be addressed by predictable names
  • applications that require one persistent volume per replica

Using persistent storage alone does not always require a StatefulSet. A single Pod with a PVC, or a Deployment with shared storage when the backend supports concurrent access, may be enough. A StatefulSet does not require volumeClaimTemplates; per-replica claims are optional. Direct StatefulSet deletion does not guarantee ordered Pod termination.

Use a Deployment when replicas are interchangeable. Use a StatefulSet when replicas need stable identity, per-ordinal storage, or ordered lifecycle. See Deployment versus StatefulSet for the complete decision guide.

Create a StatefulSet Lab

Check the StorageClass

Before you save the StatefulSet manifest, confirm that a StorageClass exists for dynamic provisioning:

bash
kubectl get storageclass

Sample output:

output
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  2d

Use the exact name shown by your cluster. If no StorageClass is available, the PVCs will remain Pending until a provisioner or suitable pre-created volumes are available. StatefulSet storage must be dynamically provisioned through the requested StorageClass or provided by an administrator.

Create the headless Service

StatefulSets commonly pair with a headless Service because:

  • it does not assign a single cluster virtual IP for load balancing across Pods
  • it creates DNS records for individual Pod endpoints
  • it gives each Pod a predictable network identity through serviceName

Save this manifest as database-headless.yaml:

yaml
apiVersion: v1
kind: Service
metadata:
  name: database
  namespace: sts-lab
spec:
  clusterIP: None
  selector:
    app: database
  ports:
    - port: 80
      name: web

Create the namespace, then apply the Service:

bash
kubectl create namespace sts-lab

Sample output:

output
namespace/sts-lab created
bash
kubectl apply -f database-headless.yaml

Sample output:

output
service/database created

For ClusterIP, NodePort, and LoadBalancer behaviour, see Kubernetes Services. This article only needs the headless pattern.

Create the StatefulSet

Save the StatefulSet manifest as database-sts.yaml. It references the headless Service through serviceName, declares three replicas, and provisions one PVC per Pod through volumeClaimTemplates:

yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
  namespace: sts-lab
spec:
  serviceName: database
  replicas: 3
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
        - name: nginx
          image: nginx:1.27-alpine
          volumeMounts:
            - name: data
              mountPath: /var/lib/data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: local-path
        resources:
          requests:
            storage: 1Gi

Replace storageClassName with a StorageClass available in your cluster. Omit the field only when a default StorageClass exists.

Apply the StatefulSet:

bash
kubectl apply -f database-sts.yaml

Sample output:

output
statefulset.apps/database created

Verify Pods and PVCs

Watch ordinal creation:

bash
kubectl get pods -n sts-lab --watch

Observe database-0, then database-1, and finally database-2. Press Ctrl+C after all three reach Running.

Wait before you trust the StatefulSet status columns:

bash
kubectl wait --for=jsonpath='{.status.readyReplicas}'=3 statefulset/database -n sts-lab --timeout=180s

Sample output:

output
statefulset.apps/database condition met

Under OrderedReady, the controller creates Pods in increasing ordinal order and waits for each preceding Pod to become Ready.

bash
kubectl get statefulsets -n sts-lab

Sample output:

output
NAME       READY   AGE
database   3/3     21s

List the claims the StatefulSet created:

bash
kubectl get pvc -n sts-lab

Sample output:

output
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-database-0   Bound    pvc-1a67f90f-f0b7-40f9-be08-d24e0bde673d   1Gi        RWO            local-path     21s
data-database-1   Bound    pvc-a53958cf-6e37-4806-bb66-7aa5b7efd0e4   1Gi        RWO            local-path     14s
data-database-2   Bound    pvc-c3eb88cd-afe4-4536-bab3-3c1047951742   1Gi        RWO            local-path     7s

Each claim name follows <claimTemplate>-<statefulset>-<ordinal>.

Understand StatefulSet Identity and YAML

Field Rule
spec.serviceName Governing Service name used for Pod network identity
spec.replicas Desired ordinal count; defaults to one if omitted
spec.selector Required, must match template labels, and is immutable
spec.template.spec.restartPolicy Must be Always or omitted
spec.volumeClaimTemplates Creates ordinal-associated claims and cannot be updated on an existing StatefulSet
spec.podManagementPolicy OrderedReady by default; Parallel changes scaling behavior

Stable Pod names and ordinals

Three properties define StatefulSet identity:

  • Stable Pod name — database-0 keeps the same name after recreation
  • Stable DNS hostname — database-0.database within the namespace when the governing headless Service is configured
  • Stable persistent storage — ordinal 0 reattaches to data-database-0 when volumeClaimTemplates is configured

A recreated Pod keeps the same StatefulSet ordinal but receives a new UID and may receive a new IP address. Clients should target DNS names and PVCs, not Pod IPs.

Governing Service and DNS

serviceName must match the headless Service that provides stable network identity for Pods. Kubernetes uses it when forming per-Pod DNS names. If serviceName points to a missing Service or a Service whose selector does not match Pod labels, Pod DNS records will not resolve as expected.

Within the cluster, the fully qualified name follows:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

On clusters using the default cluster.local domain, the complete name is <pod>.<service>.<namespace>.svc.cluster.local. Clusters can use a different DNS domain.

For this lab:

  • database-0.database.sts-lab.svc.cluster.local
  • database-1.database.sts-lab.svc.cluster.local

Short names resolve inside the same namespace:

  • database-0.database
  • database-1.database

Headless Services return the endpoint addresses of individual selected Pods rather than providing a load-balanced virtual IP.

By default, Service DNS normally publishes ready endpoints. Applications that require peer discovery before readiness may need publishNotReadyAddresses: true on the headless Service. That field is intended primarily for StatefulSet peer discovery through headless Services.

Per-replica volume claims

Kubernetes creates one PersistentVolumeClaim per Pod ordinal from volumeClaimTemplates. For the manifest above, claims are named:

  • data-database-0
  • data-database-1
  • data-database-2

Each Pod binds to its own claim for the life of that ordinal. volumeMounts attach each Pod's claim at the same container path. Every replica sees /var/lib/data, but the backing volume is unique per ordinal.

Selector and template rules

spec.selector.matchLabels must match spec.template.metadata.labels. Additional template labels are allowed. A mismatch is rejected at apply time.

Copy the working manifest and change only the template label to test a mismatch:

bash
cp database-sts.yaml bad-database-sts.yaml

Edit bad-database-sts.yaml: set metadata.name to bad-db and change the Pod template labels:

yaml
template:
  metadata:
    labels:
      app: db

A server dry-run surfaces the mismatch before you apply:

bash
kubectl apply --dry-run=server -f bad-database-sts.yaml

Sample output:

output
The StatefulSet "bad-db" is invalid: spec.template.metadata.labels: Invalid value: {"app":"db"}: `selector` does not match template `labels`

StatefulSet selectors are required and immutable in apps/v1.

Verify DNS and Persistent Storage

Resolve individual Pods

Create a temporary client Pod:

bash
kubectl run dns-test --image=busybox:1.36 --restart=Never --namespace=sts-lab --command -- sleep 3600

Sample output:

output
pod/dns-test created
bash
kubectl wait --for=condition=Ready pod/dns-test -n sts-lab --timeout=60s

Resolve an individual StatefulSet Pod:

bash
kubectl exec dns-test -n sts-lab -- nslookup database-0.database

Sample output (trimmed):

output
Name:      database-0.database.sts-lab.svc.cluster.local
Address:   10.244.1.25

IP addresses vary between clusters. For CoreDNS troubleshooting, see Kubernetes DNS troubleshooting.

Recreate a Pod and verify its data

Write test data into the mounted volume:

bash
kubectl exec database-0 -n sts-lab -- sh -c "echo pod0-data > /var/lib/data/test.txt && cat /var/lib/data/test.txt"

Sample output:

output
pod0-data

Optionally record the UID before deletion:

bash
kubectl get pod database-0 -n sts-lab -o custom-columns='NAME:.metadata.name,UID:.metadata.uid'

Sample output:

output
NAME         UID
database-0   add76b1c-3038-4d61-8352-e9b739afbe8c

Delete the Pod:

bash
kubectl delete pod database-0 -n sts-lab

Sample output:

output
pod "database-0" deleted

Wait for the replacement Pod to be created and Ready. In Kubernetes 1.36, kubectl wait can wait for resource creation and a status condition in the same command:

bash
kubectl wait --for=create --for=condition=Ready pod/database-0 -n sts-lab --timeout=180s
bash
kubectl get pod database-0 -n sts-lab -o custom-columns='NAME:.metadata.name,UID:.metadata.uid'

Sample output:

output
NAME         UID
database-0   4500923e-7fce-47b5-86ca-e7c3236ad68f

The name stayed database-0, but the UID changed.

Read the file again:

bash
kubectl exec database-0 -n sts-lab -- cat /var/lib/data/test.txt

Sample output:

output
pod0-data

The Pod object is new, but it reattached to data-database-0, so the file persisted across recreation.

Scale StatefulSet Pods

Scale up to four replicas:

bash
kubectl scale statefulset database -n sts-lab --replicas=4

Sample output:

output
statefulset.apps/database scaled

Wait before you read four Ready replicas:

bash
kubectl wait --for=jsonpath='{.status.readyReplicas}'=4 statefulset/database -n sts-lab --timeout=180s
bash
kubectl get pods -n sts-lab -l app=database

Sample output:

output
NAME         READY   STATUS    RESTARTS   AGE
database-0   1/1     Running   0          10s
database-1   1/1     Running   0          28s
database-2   1/1     Running   0          21s
database-3   1/1     Running   0          7s

Scale down to two:

bash
kubectl scale statefulset database -n sts-lab --replicas=2

Sample output:

output
statefulset.apps/database scaled
bash
kubectl wait --for=jsonpath='{.status.readyReplicas}'=2 statefulset/database -n sts-lab --timeout=180s
bash
kubectl get pods -n sts-lab -l app=database

Sample output:

output
NAME         READY   STATUS    RESTARTS   AGE
database-0   1/1     Running   0          14s
database-1   1/1     Running   0          32s

Scaling down does not delete data-database-2 or data-database-3 under the default PVC-retention policy. Scaling back up can reuse those claims for the same ordinals.

bash
kubectl get pvc -n sts-lab

You can also change spec.replicas in database-sts.yaml and re-apply for a declarative scale.

kubectl scale changes the live StatefulSet but does not update database-sts.yaml. The file still declares replicas: 3; applying it later restores three replicas. StatefulSet scaling uses ordinal order under OrderedReady, and the application should be healthy before scale-down.

OrderedReady behaviour

The default podManagementPolicy is OrderedReady:

  • Pods are created in ascending ordinal order (0, then 1, then 2)
  • the previous Pod must become Ready before the next Pod starts
  • scale-down and rolling updates normally proceed from the highest ordinal downward

Parallel Pod management

podManagementPolicy: Parallel relaxes ordering for scale-up and scale-down. It does not by itself change the selected updateStrategy or turn a rolling update into an unordered update. The Parallel policy changes scaling behavior while preserving stable identity.

PVC behavior during scale-down

Under the default retention policy, PVCs for removed ordinals remain after scale-down. Ordinals that scale back up rebind to the same claim names when those claims still exist.

Update a StatefulSet

RollingUpdate and OnDelete

Change the container image on the running StatefulSet:

bash
kubectl set image statefulset/database nginx=nginx:1.28-alpine -n sts-lab

Sample output:

output
statefulset.apps/database image updated

Wait for the rollout to finish:

bash
kubectl rollout status statefulset/database -n sts-lab

Sample output:

output
partitioned roll out complete: 2 new pods have been updated...

Only after rollout completes, verify the image on running Pods:

bash
kubectl get pods -n sts-lab -l app=database -o custom-columns="NAME:.metadata.name,IMAGE:.spec.containers[0].image"

Sample output:

output
NAME         IMAGE
database-0   nginx:1.28-alpine
database-1   nginx:1.28-alpine

kubectl set image updates the live StatefulSet but not database-sts.yaml. Reapplying the unchanged file later can restore the old image and the file's declared replica count. In a declarative workflow, update the manifest instead.

RollingUpdate is the default strategy. OnDelete changes the Pod template in the StatefulSet object but waits for you to delete each Pod manually before the new spec takes effect.

Partitioned rolling updates

spec.updateStrategy.rollingUpdate.partition limits automated updates to Pods with ordinal greater than or equal to the partition. Ordinals below the partition keep the old template until you lower the partition.

Pods with ordinals below the partition remain on the previous revision. Even if one of those lower-ordinal Pods is deleted, the controller recreates it from the previous revision until the partition is lowered. That is an important behavior of partitioned StatefulSet rollouts. General rollout concepts also appear in Deployments and rolling updates.

Delete a StatefulSet and Manage PVCs

Default retention

Deleting the StatefulSet removes its Pods:

bash
kubectl delete statefulset database -n sts-lab

Sample output:

output
statefulset.apps "database" deleted

Wait for its Pods to disappear:

bash
kubectl wait --for=delete pod -l app=database -n sts-lab --timeout=180s

PVCs created from volumeClaimTemplates remain by default:

bash
kubectl get pvc -n sts-lab

Sample output:

output
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-database-0   Bound    pvc-1a67f90f-f0b7-40f9-be08-d24e0bde673d   1Gi        RWO            local-path     49s
data-database-1   Bound    pvc-a53958cf-6e37-4806-bb66-7aa5b7efd0e4   1Gi        RWO            local-path     42s
data-database-2   Bound    pvc-c3eb88cd-afe4-4536-bab3-3c1047951742   1Gi        RWO            local-path     35s
data-database-3   Bound    pvc-8c54fe49-3329-434e-9f8c-1f00c61976cd   1Gi        RWO            local-path     21s

PersistentVolumeClaim retention policy

You can configure retention explicitly:

yaml
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
Field Retain Delete
whenDeleted Keep claims when the StatefulSet is deleted Delete claims when the StatefulSet is deleted
whenScaled Keep claims for removed ordinals Delete claims belonging to scaled-down ordinals

The default is Retain for both.

Delete lab claims when you intend to discard application data. In this dedicated lab namespace:

bash
kubectl delete pvc --all -n sts-lab

Depending on your StorageClass reclaim policy, deleting a claim may permanently remove the underlying volume. Confirm your storage policy before deleting production data. See PersistentVolume and PVC for reclaim policy detail.

Ordered shutdown considerations

Direct StatefulSet deletion starts Pod termination but does not guarantee reverse-ordinal, graceful shutdown. Scale to zero before deletion when application shutdown order matters.

Troubleshoot Common StatefulSet Problems

Start with describe output and Pod status:

bash
kubectl describe statefulset <name> -n <namespace>
kubectl describe pod <pod> -n <namespace>
Symptom Likely cause First check
Only one Pod appears at first Default OrderedReady policy Wait for previous ordinal Ready
Next ordinal never starts Previous Pod not Ready kubectl describe pod database-N
PVCs remain after StatefulSet delete Default Retain policy or explicit retention policy kubectl get pvc and persistentVolumeClaimRetentionPolicy
DNS name fails Headless Service, serviceName, or readiness timing Service selector, serviceName, and Pod Ready state
Pod Pending Unbound PVC, missing StorageClass, or scheduling block kubectl describe pod Events and kubectl get pvc
Template change ignored on running Pods OnDelete strategy spec.updateStrategy.type
Selector rejected on apply Selector and template labels differ Server-side dry run

For general Pending and ContainerCreating diagnosis, see Kubernetes Pods and Pod Lifecycle.

What's Next

References

Summary

A Kubernetes StatefulSet runs Pods that need stable names, predictable DNS hostnames through a governing headless Service, and optional per-ordinal persistent storage through volumeClaimTemplates. You set serviceName, declare replicas, and verify ordered creation under OrderedReady before trusting READY counts. The persistence test proves that ordinal 0 reattaches to data-database-0 after Pod recreation.

Scale-up adds higher ordinals in order; scale-down removes the highest ordinals first while default PVC retention keeps claims for removed ordinals. Image updates roll through replicas in reverse ordinal order with RollingUpdate. Use a Deployment when replicas are interchangeable. Reach for a StatefulSet when clients or cluster software must address distinct ordinals with optional per-replica disks. When Pods stay Pending, inspect StorageClass availability and PVC binding before chasing DNS or rollout issues.

Frequently Asked Questions

1. What is a Kubernetes StatefulSet used for?

A StatefulSet runs Pods that need stable network identity and stable per-replica storage. Each Pod gets a predictable name and ordinal such as database-0. When volumeClaimTemplates is configured, Kubernetes creates one PersistentVolumeClaim per Pod ordinal.

2. Why does a StatefulSet need a headless Service?

The serviceName field points to a headless Service with clusterIP: None. That Service creates DNS records for individual Pods so clients can reach database-0.database rather than a single load-balanced virtual IP.

3. Are PVCs deleted when I delete a StatefulSet?

PVCs created from volumeClaimTemplates are retained on StatefulSet deletion and scale-down by default. The optional persistentVolumeClaimRetentionPolicy can instead configure automatic deletion for either event. The defaults are Retain for both whenDeleted and whenScaled, but both can be set to Delete.

4. How does StatefulSet Pod creation order work?

With the default OrderedReady podManagementPolicy, Pods are created in ascending ordinal order. Each Pod must become Ready before the next starts. Scale-down and rolling updates normally proceed from the highest ordinal downward.

5. Should I use a StatefulSet or a Deployment?

Use a Deployment for interchangeable stateless replicas. Use a StatefulSet when replicas need stable ordinal names, predictable per-Pod DNS, ordered lifecycle, per-replica storage, or a combination of these properties.
Deepak Prasad

R&D Engineer

Founder of GoLinuxCloud with more than 15 years of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels across development, DevOps, networking, and security, delivering robust and efficient solutions for diverse projects.

Expertise
  • Go (programming language)
  • Python (programming language)
  • DevOps
  • Computer Security
  • Cloud Computing
  • Kubernetes
  • Linux
  • Ansible (software)

Related Articles

interview-questions

OpenShift Interview Questions and Answers

OpenShift interview questions for 2026: OpenShift vs Kubernetes, oc vs kubectl, projects, routes, SCC, …

27 min readinterview-questions

DevSecOps Interview Questions and Answers

DevSecOps interview questions for 2026: shift-left security, CI/CD gates, container and Kubernetes …

22 min readinterview-questions

Platform Engineer Interview Questions and Answers

Platform engineer interview questions for 2026: internal developer platforms, golden paths, self-service, …

21 min read