Kubernetes Scheduling, nodeSelector and Affinity

Tested on Rocky Linux 10.2 (Red Quartz) workstation
Package kubectl 1.36.3
Applies to Any host with kubectl configured; any Kubernetes cluster
Cert prep CKA · CKAD
Lab environment Multi-node kubeadm cluster with containerd — install Kubernetes with kubeadm. This guide uses two schedulable nodes (k8s-cp and worker01); the control-plane NoSchedule taint is removed for the lab so affinity across nodes is visible, then restored.
Privilege Normal user for kubectl when kubeconfig is available
Scope Scheduler filter and score overview, node labels, nodeSelector, required and preferred node affinity, IgnoredDuringExecution, pod affinity and anti-affinity with topologyKey, namespace selection for peer Pods, and scheduling failure diagnosis. Does not cover taints in depth, scheduler profiles, custom schedulers, topology spread constraints in depth, priority and preemption, the descheduler, or node autoscaling.
Related guides Kubernetes labels and selectors
Kubernetes Pods and Pod Lifecycle
Choose a Kubernetes workload resource
Cordon, drain and uncordon

When you need a Pod on a specific kind of node, or you want it near another workload, you use nodeSelector and affinity rules. In this guide you start with the simplest form of label matching, move on to required and preferred node affinity, and finish with pod affinity and anti-affinity. Each step uses real Pods on the lab cluster so you can see both Running and Pending outcomes.

IMPORTANT
This article covers nodeSelector, node affinity, and inter-Pod affinity or anti-affinity. It does not cover taints and tolerations in depth, topology spread constraints, custom schedulers, or preemption. For dedicated-node taints, use taints and tolerations.

How the scheduler selects a node

When a Pod has no node name yet, the default scheduler works through three stages:

text
Unscheduled Pod
Filter unsuitable nodes
Score suitable nodes
Bind Pod to selected node

Filtering removes nodes that fail a hard rule. Common filters include:

  • Resource requests the node cannot satisfy
  • Taints without a matching toleration
  • Volume topology that blocks attachment
  • nodeSelector labels that do not match
  • Required node affinity expressions
  • Required pod affinity or anti-affinity rules

Scoring ranks the nodes that passed filtering. Preferred affinity adds weight to matching nodes, but it does not remove non-matching nodes from the list.

Binding writes the winning node name into the Pod spec.

Affinity is only one part of that pipeline. A node can still be rejected when CPU or memory is insufficient, a taint has no toleration, or a volume cannot attach there, even when every label rule matches.


Label nodes for the lab

Before you schedule anything, label the nodes you want the scheduler to choose from. Use labels you control for application placement. Do not rely on built-in control-plane role labels as your app scheduling API.

This lab needs two schedulable nodes so anti-affinity can spread Pods across two hostnames. Remove the control-plane NoSchedule taint for the exercise, and restore it when you finish:

bash
kubectl taint nodes k8s-cp node-role.kubernetes.io/control-plane:NoSchedule-

You should see node/k8s-cp untainted when the taint is gone. Apply the lab labels next:

bash
kubectl label nodes worker01 storage=ssd environment=production topology.kubernetes.io/zone=zone-a --overwrite
bash
kubectl label nodes k8s-cp storage=hdd environment=staging topology.kubernetes.io/zone=zone-b --overwrite

Confirm the custom labels:

bash
kubectl get nodes --show-labels

You should see these custom labels:

  • worker01storage=ssd, environment=production, topology.kubernetes.io/zone=zone-a
  • k8s-cpstorage=hdd, environment=staging, topology.kubernetes.io/zone=zone-b

Create a namespace for the demos:

bash
kubectl create namespace affinity-lab \
  --dry-run=client -o yaml |
kubectl apply -f -

Sample output on the first run:

output
namespace/affinity-lab created

On another run:

output
namespace/affinity-lab unchanged

Schedule with nodeSelector

nodeSelector is the simplest placement rule. It is a map of required node labels, and every key/value pair must match before the scheduler can place the Pod.

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: select-ssd
  namespace: affinity-lab
spec:
  nodeSelector:
    storage: ssd
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF

Wait until the Pod is Ready before you inspect placement:

bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/select-ssd --timeout=120s

Sample output:

output
pod/select-ssd condition met

Check which node the scheduler chose:

bash
kubectl -n affinity-lab get pod select-ssd -o wide

Sample output:

output
NAME         READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
select-ssd   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>

The Pod landed on worker01 because that is the only node with storage=ssd. Next, request a label that no node carries and watch what happens:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: select-missing
  namespace: affinity-lab
spec:
  nodeSelector:
    storage: nvme
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF

Wait for the scheduler to record a FailedScheduling event before you run describe. The event may not appear immediately after you create the Pod:

bash
until kubectl get events -n affinity-lab \
  --field-selector involvedObject.name=select-missing,reason=FailedScheduling \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 2
done
bash
kubectl -n affinity-lab describe pod select-missing

Sample output:

output
Warning  FailedScheduling  ...  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector.

Neither node has storage=nvme, so the scheduler has nowhere to place the Pod. It stays Pending until you fix the selector or add the label.

nodeSelector is simple, but it is always a hard requirement. It cannot express:

  • NotIn — exclude nodes that carry a specific label value
  • OR across alternative terms — accept one of several label combinations
  • Soft preferences — prefer a node without blocking placement elsewhere

Node affinity adds all three options.


Use required node affinity

Required node affinity behaves like a stricter nodeSelector. It uses requiredDuringSchedulingIgnoredDuringExecution, and the scheduler must satisfy the rule before it places the Pod.

A few rules govern how expressions combine:

  • Inside one nodeSelectorTerms entry, every matchExpressions entry is ANDed
  • Separate terms are ORed
  • Supported operators include In, NotIn, Exists, DoesNotExist, Gt, and Lt

Start with a Pod that requires both storage=ssd and environment=production:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: required-aff
  namespace: affinity-lab
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: storage
            operator: In
            values: ["ssd"]
          - key: environment
            operator: In
            values: ["production"]
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/required-aff --timeout=120s
bash
kubectl -n affinity-lab get pod required-aff -o wide

Sample output:

output
NAME           READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
required-aff   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>

Only worker01 satisfies both expressions on this cluster. You can also exclude nodes by label value. The next Pod uses NotIn to reject HDD storage:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: notin-hdd
  namespace: affinity-lab
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: storage
            operator: NotIn
            values: ["hdd"]
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/notin-hdd --timeout=120s
bash
kubectl -n affinity-lab get pod notin-hdd -o wide

Sample output:

output
NAME        READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
notin-hdd   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>

k8s-cp is filtered out because it carries storage=hdd. The Pod still schedules on worker01.

If you set nodeSelector and node affinity on the same Pod, both must pass. The scheduler applies every hard rule together.


Use preferred node affinity

Preferred rules live under preferredDuringSchedulingIgnoredDuringExecution. Each preference has a weight from 1 to 100. Matching nodes gain score, but the scheduler can still place the Pod on a node that does not match.

That is an important distinction. Preferred placement is a hint, not a guarantee:

  • Preferred affinity weight is added to scores from other scheduler plugins
  • Kubernetes applies built-in soft topology-spread scoring to controller-owned Pods unless you change scheduler defaults
  • Four replicas may all land on worker01, or some may spread across both nodes

Deploy four replicas that prefer SSD storage and compare where they end up:

bash
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefer-ssd
  namespace: affinity-lab
spec:
  replicas: 4
  selector:
    matchLabels:
      app: prefer-ssd
  template:
    metadata:
      labels:
        app: prefer-ssd
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage
                operator: In
                values: ["ssd"]
      containers:
      - name: app
        image: registry.k8s.io/pause:3.10
EOF

Wait for the Deployment rollout to finish:

bash
kubectl rollout status deployment/prefer-ssd \
  -n affinity-lab --timeout=120s

Sample output:

output
deployment "prefer-ssd" successfully rolled out

These values can differ between runs:

  • ReplicaSet hashes
  • Pod suffixes
  • Pod IPs
  • Ages and output order
  • The node selected for preferred rules
bash
kubectl -n affinity-lab get pods -l app=prefer-ssd -o wide

Sample output:

output
NAME                                      READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
prefer-ssd-<replicaset-hash>-<suffix-1>   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>
prefer-ssd-<replicaset-hash>-<suffix-2>   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>
prefer-ssd-<replicaset-hash>-<suffix-3>   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>
prefer-ssd-<replicaset-hash>-<suffix-4>   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>

Your output may look different. The sample above shows one possible placement:

  • storage=ssd adds affinity score to worker01
  • That score does not guarantee every replica uses that node
  • The final decision combines every enabled score plugin

Next, prefer a label that does not exist. The Pod should still schedule somewhere:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: prefer-missing
  namespace: affinity-lab
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: storage
            operator: In
            values: ["nvme"]
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/prefer-missing --timeout=120s
bash
kubectl -n affinity-lab get pod prefer-missing -o wide

Sample output:

output
NAME             READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
prefer-missing   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>

No node had storage=nvme, so the preference added no useful score. The scheduler still bound the Pod:

  • The prefer-missing Pod can land on either node in this lab
  • Do not treat a specific node name as the expected result

Understand IgnoredDuringExecution

The IgnoredDuringExecution suffix trips people up. It means:

  • Affinity is checked at schedule time, not continuously on a running Pod
  • Changing node labels under a running Pod does not evict it
  • A new Pod created later must satisfy the rules again at schedule time

Create a Pod that requires storage=ssd through required node affinity:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: keep-running
  namespace: affinity-lab
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: storage
            operator: In
            values:
            - ssd
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/keep-running --timeout=120s

Confirm the Pod is Running on worker01, then remove the label from that node:

bash
kubectl label nodes worker01 storage-
bash
kubectl -n affinity-lab get pod keep-running -o wide

Sample output:

output
NAME           READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
keep-running   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>

The Pod keeps running even though the node no longer matches the rule. That is the behavior you should expect:

  • Removing storage=ssd does not evict an existing Pod
  • The required node-affinity rule no longer matches the node, but the running Pod stays put

Delete and recreate the same Pod while the label is still missing:

bash
kubectl -n affinity-lab delete pod keep-running

Re-apply the same manifest:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: keep-running
  namespace: affinity-lab
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: storage
            operator: In
            values:
            - ssd
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl -n affinity-lab get pod keep-running -o wide

Sample output:

output
NAME           READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
keep-running   0/1     Pending   0          <age>  <none>   <none>   <none>           <none>

Wait for the FailedScheduling event, then read the scheduler message:

bash
until kubectl get events -n affinity-lab \
  --field-selector involvedObject.name=keep-running,reason=FailedScheduling \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 2
done
bash
kubectl -n affinity-lab describe pod keep-running

Sample output:

output
Warning  FailedScheduling  ...  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector.

The recreation cannot schedule because storage=ssd is still missing from worker01. That is the difference IgnoredDuringExecution makes: the old Pod kept running, but a new Pod must pass the rule again.

Restore the label on worker01 before you continue:

bash
kubectl label nodes worker01 storage=ssd --overwrite

Schedule near other Pods with pod affinity

Pod affinity selects nodes based on where other Pods already run. You point the scheduler at peer Pod labels and a topologyKey that defines what "near" means.

Common topologyKey values:

  • kubernetes.io/hostname — place on the same node as the peer
  • topology.kubernetes.io/zone — place in the same zone as the peer

The scheduler evaluates pod affinity against the current location of matching Pods. Create and wait for the backend before you apply the frontend.

Apply the backend:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: backend
  namespace: affinity-lab
  labels:
    app: backend
spec:
  nodeSelector:
    storage: ssd
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/backend --timeout=120s

Apply the frontend that requires the same hostname:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: frontend-near
  namespace: affinity-lab
  labels:
    app: frontend
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values: ["backend"]
        topologyKey: kubernetes.io/hostname
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/frontend-near --timeout=120s
bash
kubectl -n affinity-lab get pods -l 'app in (backend,frontend)' -o wide

Sample output:

output
NAME            READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
backend         1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>
frontend-near   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>

Both Pods share worker01 because the frontend required the same hostname as the backend. Next, require affinity to a peer label that does not exist:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: frontend-missing
  namespace: affinity-lab
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values: ["missing-backend"]
        topologyKey: kubernetes.io/hostname
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF

Wait for the scheduler event, then inspect the Pod:

bash
until kubectl get events -n affinity-lab \
  --field-selector involvedObject.name=frontend-missing,reason=FailedScheduling \
  --no-headers 2>/dev/null | grep -q .; do
  sleep 2
done
bash
kubectl -n affinity-lab describe pod frontend-missing

Sample output:

output
Warning  FailedScheduling  ...  0/2 nodes are available: 2 node(s) didn't match pod affinity rules.

No node can satisfy the rule when the peer label does not exist. Required pod affinity leaves the Pod Pending until a matching peer appears in the right topology.


Spread workloads with pod anti-affinity

Pod anti-affinity does the opposite of pod affinity. It tells the scheduler to keep replicas apart within a topology domain.

Rule Result
Required Scheduling fails when separation cannot be met
Preferred Scheduler tries to separate but can still colocate

Deploy three replicas with hard anti-affinity on a two-node cluster:

bash
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hard-spread
  namespace: affinity-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hard-spread
  template:
    metadata:
      labels:
        app: hard-spread
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values: ["hard-spread"]
            topologyKey: kubernetes.io/hostname
      containers:
      - name: app
        image: registry.k8s.io/pause:3.10
EOF

Wait until two replicas are available. The third stays Pending because only two hostnames exist in this lab:

bash
kubectl wait -n affinity-lab \
  --for=jsonpath='{.status.availableReplicas}'=2 \
  deployment/hard-spread --timeout=120s

Pick the wait command that matches what you are checking:

  • kubectl wait — checks a Pod condition or a JSONPath value on a resource
  • kubectl rollout status — watches the current Deployment rollout to completion
  • hard-spread — use the JSONPath wait here because the rollout never reaches three available replicas
bash
kubectl -n affinity-lab get pods -l app=hard-spread -o wide

Sample output:

output
NAME                              READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
hard-spread-<replicaset-hash>-<pod-suffix>   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>
hard-spread-<replicaset-hash>-<pod-suffix>   1/1     Running   0          <age>  <pod-ip>   k8s-cp     <none>           <none>
hard-spread-<replicaset-hash>-<pod-suffix>   0/1     Pending   0          <age>  <none>     <none>     <none>           <none>

Two replicas occupy the two hostnames. The third stays Pending because no third hostname is free:

bash
kubectl get events -n affinity-lab \
  --field-selector reason=FailedScheduling

Sample output:

output
Warning  FailedScheduling  ...  0/2 nodes are available: 2 node(s) didn't match pod anti-affinity rules.

The third replica has no hostname left where it can satisfy the hard anti-affinity rule. That is the expected behavior on a two-node cluster with three required-spread replicas.

Switch to preferred anti-affinity with the same replica count. All three replicas should schedule, and colocation is allowed:

bash
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: soft-spread
  namespace: affinity-lab
spec:
  replicas: 3
  selector:
    matchLabels:
      app: soft-spread
  template:
    metadata:
      labels:
        app: soft-spread
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values: ["soft-spread"]
              topologyKey: kubernetes.io/hostname
      containers:
      - name: app
        image: registry.k8s.io/pause:3.10
EOF
bash
kubectl rollout status deployment/soft-spread \
  -n affinity-lab --timeout=120s

Sample output:

output
deployment "soft-spread" successfully rolled out
bash
kubectl -n affinity-lab get pods -l app=soft-spread -o wide

Sample output:

output
NAME                              READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
soft-spread-<replicaset-hash>-<pod-suffix>   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>
soft-spread-<replicaset-hash>-<pod-suffix>   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>
soft-spread-<replicaset-hash>-<pod-suffix>   1/1     Running   0          <age>  <pod-ip>   <node>     <none>           <none>

On a two-node cluster, preferred anti-affinity usually produces uneven placement:

  • One node will normally host two replicas
  • Either k8s-cp or worker01 may be that node
  • Equal final scheduler scores can be resolved without a predictable node choice
  • Preferred anti-affinity does not guarantee even distribution

Understand namespace selection

By default, pod affinity looks for peer Pods in the same namespace as the Pod being scheduled. You can widen or narrow that search:

  • Omit namespaces and namespaceSelector — search the Pod's own namespace (default)
  • Set namespaces: [other-ns] — search those namespaces explicitly
  • Use namespaceSelector — match namespaces by label
  • Combine filters — namespace selection works together with the pod labelSelector

Create a peer namespace and backend in affinity-peer:

bash
kubectl create namespace affinity-peer \
  --dry-run=client -o yaml |
kubectl apply -f -
bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: peer-backend
  namespace: affinity-peer
  labels:
    app: peer-backend
spec:
  nodeSelector:
    storage: ssd
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-peer \
  --for=condition=Ready pod/peer-backend --timeout=120s

Apply a consumer in affinity-lab that lists namespaces: [affinity-peer]:

bash
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cross-ns-aff
  namespace: affinity-lab
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values: ["peer-backend"]
        topologyKey: kubernetes.io/hostname
        namespaces:
        - affinity-peer
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
EOF
bash
kubectl wait -n affinity-lab \
  --for=condition=Ready pod/cross-ns-aff --timeout=120s
bash
kubectl get pod cross-ns-aff \
  -n affinity-lab -o wide
bash
kubectl get pod peer-backend \
  -n affinity-peer -o wide

Sample output:

output
NAME           READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
cross-ns-aff   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>
output
NAME           READY   STATUS    RESTARTS   AGE   IP         NODE       NOMINATED NODE   READINESS GATES
peer-backend   1/1     Running   0          <age>  <pod-ip>   worker01   <none>           <none>

Both Pods scheduled on worker01 because the consumer in affinity-lab could see the peer in affinity-peer. When you write cross-namespace affinity, keep the namespace list small and explicit. That makes it clear which namespaces are part of the placement contract.


Diagnose affinity scheduling failures

When a Pod stays Pending, start with these three checks:

  • kubectl describe pod — read the scheduler message on the Pod
  • kubectl get events — list recent FailedScheduling events in the namespace
  • kubectl get nodes --show-labels — confirm the node labels your rule expects
bash
kubectl -n affinity-lab describe pod select-missing
bash
kubectl get events -n affinity-lab --field-selector reason=FailedScheduling
bash
kubectl get nodes --show-labels
Symptom or event Likely cause Fix
didn't match Pod's node affinity/selector Missing or wrong node labels; conflicting nodeSelector and affinity Fix labels or relax the selector; verify with kubectl get nodes --show-labels
didn't match pod affinity rules Required peer Pod missing or wrong namespace Create the peer; set namespaces or namespaceSelector
didn't match pod anti-affinity rules More replicas than topology domains Add nodes or zones, lower replicas, or switch to preferred anti-affinity
Pending with nodeSelector No node has every listed label Fix labels or relax the selector
Pending with required node affinity Expressions too strict or OR terms incomplete Adjust operators and values
Preferred affinity ignored Soft rule lost to other scores or capacity Raise weight, add capacity, or accept nonpreferred placement
Running Pod after label removed Expected IgnoredDuringExecution behavior Relabel for new Pods; do not expect auto-eviction
Required anti-affinity Pending More replicas than topology domains Add nodes or zones, lower replicas, or switch to preferred
Affinity Pending across namespaces Peer searched in the wrong namespace Set namespaces or namespaceSelector
Matching node still refused Taint or insufficient CPU or memory Add a toleration or lower requests

Taints are a separate hard filter. When a node matches every affinity rule but still rejects the Pod, check taints and tolerations next.


What's Next


References


Summary

In this guide you labeled nodes for the lab, pinned Pods with nodeSelector, and expressed the same placement ideas with required and preferred node affinity. You saw the difference between hard and soft rules:

  • Required rules leave Pods Pending when nothing matches
  • Preferred rules add score without blocking placement elsewhere
  • IgnoredDuringExecution keeps a running Pod in place even when labels change underneath it

Pod affinity colocated a frontend with a backend on the same hostname. Hard anti-affinity capped replicas at one per node, so a third replica stayed Pending on a two-node cluster. Preferred anti-affinity scheduled every replica and still allowed colocation. Do not treat preferred anti-affinity as a hard spread guarantee.

When you finish, restore the control-plane NoSchedule taint and clean up the affinity-lab and affinity-peer namespaces. For dedicated nodes that should reject most workloads, combine these selectors with taints rather than overloading affinity alone.


Frequently Asked Questions

1. What is the difference between nodeSelector and node affinity?

nodeSelector is a simple required match on every listed node label. Node affinity adds operators, OR across terms, and preferred weighted rules. If both fields are set, the Pod must satisfy both.

2. Does preferred node affinity guarantee placement on a matching node?

No. Preferred rules add score to matching nodes. The scheduler can still place the Pod on a nonmatching node when that is the best available choice or when no preferred node fits.

3. What does IgnoredDuringExecution mean for affinity?

Affinity is evaluated when the Pod is scheduled. If node or peer labels change afterward, the running Pod is not automatically evicted. A new Pod created later must satisfy the rules again.

4. Why does required pod anti-affinity leave a replica Pending?

Hard anti-affinity with topologyKey kubernetes.io/hostname allows at most one matching Pod per node. When every node already hosts a replica, additional replicas stay Pending until capacity or the rule changes.

5. Which namespace does pod affinity search by default?

When namespaces and namespaceSelector are omitted, the rule looks for peer Pods in the same namespace as the Pod being scheduled. Use namespaces or namespaceSelector to search other namespaces.
Deepak Prasad

R&D Engineer

Founder of GoLinuxCloud with more than 15 years of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels across development, DevOps, networking, and security, delivering robust and efficient solutions for diverse projects.

  • Go (programming language)
  • Python (programming language)
  • DevOps
  • Computer Security
  • Cloud Computing
  • Kubernetes
  • Linux
  • Ansible (software)