Fix Podman `no space left on device`

A pull, build, or container write fails with no space left on device long before you run out of RAM. The message is accurate — something on disk refused the write — but the path in the error line tells you which filesystem actually filled. This guide walks that path: identify the exhausted resource, reclaim safely, and relocate graphroot when the host partition is too small.

The no space left on device error from Podman

Podman surfaces the kernel ENOSPC errno when any write in the pull, build, or runtime path fails. The exact wording varies by operation:

Error: creating runtime volume path directory: mkdir /path/to/storage/volumes: no space left on device
Error: unable to copy from source docker://registry.example/image:tag: copying system image from manifest list: writing blob: setting up to decompress blob: write /var/tmp/container_images_storage123456/1: no space left on device

The path after write, mkdir, or open is your first clue. Copy it before you run df on / by habit.

On this lab host I reproduced block exhaustion on a disposable 8 MiB loopback filesystem — not on the production root disk — by pointing CONTAINERS_STORAGE_CONF at a tiny graphroot:

CONTAINERS_STORAGE_CONF=/tmp/podman-enospc-blocks/storage.conf podman pull quay.io/podman/hello:latest

Sample output when the loop mount was already full:

Error: creating runtime volume path directory: mkdir /tmp/podman-enospc-blocks/mnt/storage/volumes: no space left on device

That failure happened while Podman created the volume directory under graphroot, before any image layer landed. Your error may name /var/lib/containers/storage, /var/tmp, or $HOME/.local/share/containers/storage instead — always trust the path Podman prints.

ENOSPC does not always mean graphroot is full

Treat no space left on device as a branch, not a single root cause. Podman can exhaust space on several independent paths:

• graphroot — persistent container/image storage; named volumes normally live beneath it by default, but volume_path can be configured separately
• runRoot — runtime state (often tmpfs)
• /var/tmp or TMPDIR — blob staging during pulls and builds

Inode exhaustion on any of those mounts returns the same ENOSPC string.

no space left on device
        ↓
disk blocks exhausted on the path in the error?
        ↓
inode pool exhausted on that filesystem?
        ↓
temporary directory full (/var/tmp or TMPDIR)?
        ↓
runRoot on a separate tmpfs full?
        ↓
XFS user, group, or project quota hit?

Only after you know which branch applies should you prune images or edit storage.conf.

Identify which Podman storage path failed

Do not assume every host uses /var/lib/containers/storage. Ask the running Podman instance:

podman info --format '{{.Store.GraphRoot}}'

Sample output on this rootful lab host:

/var/lib/containers/storage

runRoot holds ephemeral runtime state and may sit on a different filesystem — often tmpfs under /run:

podman info --format '{{.Store.RunRoot}}'

Sample output:

/run/containers/storage

For volume data and the active configuration file in one pass:

podman info --format 'graphRoot={{.Store.GraphRoot}} runRoot={{.Store.RunRoot}} volumePath={{.Store.VolumePath}} configFile={{.Store.ConfigFile}}'

Sample output:

graphRoot=/var/lib/containers/storage runRoot=/run/containers/storage volumePath=/var/lib/containers/storage/volumes configFile=/usr/share/containers/storage.conf

RHEL 10.2 ships the default in /usr/share/containers/storage.conf when no override exists under /etc/containers/. Config scope depends on how you invoke Podman:

• Rootful — /etc/containers/storage.conf when present
• Rootless — $HOME/.config/containers/storage.conf when present

After any config change, re-run podman info and confirm graphRoot moved before you prune or migrate data.

Measure Podman disk usage with system df

Before deleting anything, see what Podman accounts for locally. Start with the summary:

podman system df

Sample output on the lab host:

TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         7           0           353.8MB     353.8MB (100%)
Containers     0           0           0B          0B (0%)
Local Volumes  3           0           122.2MB     122.2MB (100%)

The verbose view lists each image, container layer, and volume:

podman system df -v

Sample output (trimmed):

Images space usage:

REPOSITORY                                   TAG         IMAGE ID      CREATED     SIZE        SHARED SIZE  UNIQUE SIZE  CONTAINERS
quay.io/podman/hello                         latest      5dd467fce50b  2 years     787kB       0B           787kB        0
registry.access.redhat.com/ubi9/ubi-minimal  latest      591c6dfb4400  5 days      108.8MB     0B           108.8MB      0

Local Volumes space usage:

VOLUME NAME                                                       LINKS       SIZE
5794e3ac2db59056ad54d085fffc79a921173522a544d754f5a6de0cc50ed339  0           40.72MB

RECLAIMABLE is an estimate. Shared image layers count toward several unused images at once, so pruning one tag may free less than the summary suggests. See Clean up with system prune and df for prune flags and the safe cleanup sequence.

Check block space on graphroot and runRoot

Map graphroot to the host filesystem:

GRAPHROOT=$(podman info --format '{{.Store.GraphRoot}}')
df -h "$GRAPHROOT"

Sample output when the root filesystem was nearly full during testing:

Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   25G   25G  125M 100% /

Avail at zero or a few megabytes on the mount that backs graphroot explains pull and layer-write failures even if other partitions have space.

runRoot can fill independently when it sits on tmpfs:

RUNROOT=$(podman info --format '{{.Store.RunRoot}}')
df -h "$RUNROOT"

Sample output:

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.5G  2.0M  1.5G   1% /run

Runtime state and persistent image storage are separate resources. A full / partition does not always mean /run is full — and the reverse is also true.

To see how much of graphroot Podman owns versus other daemons on the same mount, a top-level du is enough when permissions allow:

du -xsh "$GRAPHROOT"

Avoid deep recursive du across a production container store during peak hours; it walks every layer directory and can contend with running workloads. Use podman system df -v first, then targeted du on volumes you already identified.

Check inode exhaustion

A filesystem can report free blocks while inode usage is maxed out. Check the same mount as graphroot:

df -i "$GRAPHROOT"

Sample output on the lab root filesystem (not inode-starved):

Filesystem             Inodes  IUsed  IFree IUse% Mounted on
/dev/mapper/rhel-root 690648 435723 254925   64% /

When IFree reaches 0 and IUse% hits 100%, new files fail with the same no space left on device string:

Filesystem  Inodes  IUsed  IFree IUse%
/dev/...    ...     ...    0     100%

Image extraction and overlay mounts create many small files. Reclaim with podman image prune, podman container prune, and reviewed volume removal — not by deleting random directories under storage/overlay.

To find directories with huge file counts on a test host, find with -printf helps; on production, coordinate a maintenance window. The goal is to confirm inode pressure, then free objects through Podman commands.

Check /var/tmp and TMPDIR

Podman stages downloaded image content under a temporary directory. Upstream documentation identifies /var/tmp as the default location for that traffic. The error path often includes container_images_storage under the temp directory.

Check the default temp area:

df -h /var/tmp

On this host /var/tmp shares the root XFS mount:

Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   25G   25G  125M 100% /

graphroot can show a sliver of free space while a pull still fails if the temp write path fills first — or if you point TMPDIR at a small filesystem.

I reproduced that pattern on a 2 MiB tmpfs with TMPDIR set and a larger Red Hat base image:

TMPDIR=/tmp/podman-tmpdir-full podman pull registry.access.redhat.com/ubi9/ubi-minimal:latest

Sample output:

Trying to pull registry.access.redhat.com/ubi9/ubi-minimal:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob sha256:c282501e7b1aa336a39e8eb09b91d97b94921c378c6a6fbfe7b56d2db572c305
Copying config sha256:591c6dfb4400e422f10f911ff8d95aead6e1d80975bb11cfabd1ba918133f8c5
Error: unable to copy from source docker://registry.access.redhat.com/ubi9/ubi-minimal:latest: copying system image from manifest list: writing blob: setting up to decompress blob: write /tmp/podman-tmpdir-full/container_images_storage484539573/1: no space left on device

The failure path is under /tmp/podman-tmpdir-full, not under graphroot. Free that filesystem, enlarge it, or unset TMPDIR so Podman falls back to /var/tmp on a partition with room.

Rootless storage on a small /home partition

Rootless Podman stores persistent data under the invoking user's home unless storage.conf overrides it:

$HOME/.local/share/containers/storage

A full /home mount fails pulls and builds while /var still shows free space. Always read the actual path:

podman info --format '{{.Store.GraphRoot}}'

Rootless and rootful stores are independent. Cleaning images as root does not reclaim a rootless user's disk under their home directory. See Rootless Podman for subuid ranges and loginctl enable-linger — those concerns are separate from ENOSPC but matter when you manage both modes on one host.

Inspect volume consumption

Application data in named volumes can exceed image cache size. List volumes:

podman volume ls

Cross-check sizes in the verbose disk report:

podman system df -v

For one volume's host path:

podman volume inspect --format '{{.Mountpoint}}' VOLUME_NAME

Replace VOLUME_NAME with the name from podman volume ls. Large SIZE with LINKS at 0 means no container currently references the volume — but the data may still be valuable. Back up or export before podman volume rm. Do not delete a named volume only because system df lists it as reclaimable.

Check XFS quotas when the host uses them

When graphroot sits on XFS, confirm the mount and whether quotas are enabled:

findmnt -T "$GRAPHROOT"

Sample output on the lab host:

TARGET SOURCE                FSTYPE OPTIONS
/      /dev/mapper/rhel-root xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

noquota on this VM means project and user quotas are not limiting writes. On hosts that mount with prjquota, pquota, or uquota, a quota ceiling can reject writes before df -h shows the whole filesystem full.

Inspect quota usage when your mount options include quota support:

xfs_quota -x -c 'report -h' /mountpoint

User, group, and project quotas are filesystem administration — Podman does not create project IDs for you. The troubleshooting point is simple: the effective limit may be a quota, not the visible size of /var.

Reclaim Podman storage safely

Start from measurement, then prune targeted object types:

podman system df -v

Remove stopped containers nobody needs:

podman container prune

Remove dangling images first; add -a only when you accept deleting every unused tagged image:

podman image prune

When you need to drop every unused image, not only dangling layers, run the same command with -a:

podman image prune -a

Unused networks are usually small but safe to drop:

podman network prune

Volumes are destructive — review names and mountpoints before any volume prune. podman system prune without flags does not remove volumes by default.

Do not reach for podman system reset during a routine full-disk incident. Reset wipes local Podman metadata and images for that storage scope. Use prune steps and host-level free space first; reserve reset for deliberate rebuilds when corruption is confirmed out of band.

Move graphroot to a larger filesystem

When the partition backing graphroot is too small, relocate the store to a bigger local filesystem. High-level order:

1. Stop containers and Quadlet-managed units that use Podman.
2. Copy the existing store using a migration method that preserves ownership, permissions, hard links, and extended attributes — or start with an empty store and re-pull images.
3. Edit the correct storage.conf for rootful (/etc/containers/storage.conf) or rootless ($HOME/.config/containers/storage.conf).
4. On SELinux-enforcing hosts, apply file-context equivalence (next section).
5. Confirm with podman info and run a test container.

Minimal rootful override:

[storage]
graphroot = "/srv/podman/storage"

Do not edit storage.conf while containers are running and expect a live migration. A wrong path can make Podman open an empty store or leave metadata pointing at missing layers. Full precedence rules, imagestore, and copy semantics live in Podman storage location — this troubleshooting page only covers the ENOSPC motivation and the SELinux step people skip.

Apply SELinux file-context equivalence on the new path

On RHEL-family hosts with SELinux enforcing, a relocated graphroot needs the same labeling semantics as the default store. The shipped storage.conf documents the pattern:

sudo semanage fcontext -a -e /var/lib/containers/storage /srv/podman/storage

-e creates equivalence: new files under /srv/podman/storage receive the same context rules as under /var/lib/containers/storage. Apply labels to the migrated tree:

sudo restorecon -R -v /srv/podman/storage

Skipping this step often produces Permission denied inside containers even when ls -l permissions look correct. After relabel, confirm contexts on the store root:

ls -Zd /srv/podman/storage

Use the default source path from your active storage.conf comments if your distribution documents a different reference tree.

Verify storage after cleanup or migration

Confirm Podman picked up the new configuration:

podman info --format '{{.Store.GraphRoot}}'

Check free space on the mount that now backs the store:

df -h /srv/podman/storage

Run a short pull or start a throwaway container:

podman run --rm quay.io/podman/hello:latest

If ENOSPC persists, re-read the path in the new error — a successful graphroot move does not enlarge /var/tmp or a rootless home partition.

Troubleshooting

References

• Podman system df manual
• Podman system prune manual
• Podman info manual
• containers-storage.conf(5)
• Red Hat Podman storage documentation

Summary

no space left on device from Podman always names a path — read that path before you prune or resize the wrong partition. podman info gives you graphRoot, runRoot, and volumePath for the user and config scope you are running; podman system df -v shows which images and volumes consume space inside the store.

Each branch produces the same ENOSPC errno with a different fix:

• block exhaustion on graphroot or a volume mountpoint
• inode exhaustion on the same filesystem
• a full /var/tmp or TMPDIR during blob staging
• a saturated runRoot tmpfs
• XFS user, group, or project quota

Temp-directory failures are easy to miss because graphroot still shows free space while the blob write fails under /var/tmp or a small TMPDIR.

Reclaim with targeted prune commands before considering graphroot relocation. When you move the store on SELinux systems, add file-context equivalence with semanage fcontext -e and restorecon — UNIX permissions alone are not enough. For migration details and storage.conf precedence, continue with Podman storage location; for day-to-day cleanup commands, see Clean up with system prune and df.