Repair Podman Storage Corruption and Use `system reset` Safely

podman rm and podman image prune stop helping when metadata still references layers that no longer exist on disk — or when a shared image store path disappeared overnight. Before you reach for rm -rf under overlay or podman system reset, capture what Podman thinks it owns, run the supported consistency tools, and decide which branch you are on:

• on-disk corruption (missing layers)
• a missing additionalimagestores mount
• SELinux labeling after a graphroot move
• automation deleting images under you (prune, reset, or rmi --all)

Literal errors that point to storage inconsistency

Symptoms vary by driver and break point, but the message usually names a missing layer or overlay path:

Error: faccessat /path/to/storage/overlay/LAYERID: no such file or directory
Error: layer LAYERID: readlink /path/to/storage/overlay/LAYERID/diff: no such file or directory
Error: damage detected in local storage
Error: ... image not known ...

On this lab host I reproduced overlay breakage on a disposable loop-mounted store — never on the production /var/lib/containers/storage tree. A separate storage.conf pointed graphroot at the loop mount; after pulling quay.io/podman/hello:latest, I deleted one layer directory under overlay to simulate damage.

Re-run the container against that disposable store:

CONTAINERS_STORAGE_CONF=/tmp/podman-corrupt-lab/storage.conf podman run --rm quay.io/podman/hello:latest

Sample output:

Error: faccessat /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: no such file or directory

podman images could still list the tag even though the layer backing it was gone — metadata and on-disk layers had diverged. Use CONTAINERS_STORAGE_CONF, --root, or a dedicated test user when you need to practice system check and reset without touching production storage.

Capture storage configuration before changing anything

Record the active store layout while Podman still answers podman info:

podman info --format 'graphRoot={{.Store.GraphRoot}} runRoot={{.Store.RunRoot}} driver={{.Store.GraphDriverName}} volumePath={{.Store.VolumePath}} configFile={{.Store.ConfigFile}}'

Sample output on the lab host:

graphRoot=/var/lib/containers/storage runRoot=/run/containers/storage driver=overlay volumePath=/var/lib/containers/storage/volumes configFile=/usr/share/containers/storage.conf

For the full store block including graph options and image-store settings:

podman info --format '{{json .Store}}'

Read these fields from the store block:

• graphRoot
• runRoot
• graphDriverName
• graphOptions
• volumePath
• imageStore
• configFile

If you use imagestore or additionalimagestores, record those paths too — see Podman storage location.

Inventory what may be lost:

podman ps -a

List local images that a full reset would remove:

podman images

List named volumes that reset would delete:

podman volume ls

Copy container definitions you care about with podman inspect when they are not already declared in Quadlet, Compose, or version-controlled YAML.

Do not start with rm -rf overlay

Deleting overlay internals by hand is a common bad suggestion:

rm -rf /var/lib/containers/storage/overlay/*

That can remove layer data Podman metadata still references — or leave metadata pointing at directories you already deleted. You make podman system check and --repair less useful because the tool can no longer classify what was damaged versus deliberately removed.

Use Podman consistency commands and targeted podman rm / podman rmi first. Manual deletion inside graphroot is not a supported recovery path.

Run podman system check

Start with the non-destructive consistency scan:

podman system check

On a healthy primary store, Podman 5.8.2 exits silently with status 0 and prints nothing — absence of output means no damage was detected, not that the command failed.

For a faster pass that skips expensive checks:

podman system check --quick

--quick is useful on busy hosts when you want a fast signal, but it can miss issues the full scan would catch. Treat a clean --quick run as a first pass only, not proof the store is healthy.

When damage exists, Podman names the layer and image IDs. After the lab corruption test on disposable storage:

Damaged layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d:
2 errors occurred:
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: creating file-getter: readlink /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d/diff: no such file or directory
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: faccessat /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: no such file or directory

Damaged image 5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0:
2 errors occurred:
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: creating file-getter: readlink /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d/diff: no such file or directory
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: faccessat /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: no such file or directory

Error: damage detected in local storage

The layer ID in the error is the anchor for repair — not a random path to delete by hand.

Repair damaged images with --repair

When system check reports damage, let Podman remove inconsistent images:

podman system check --repair

On the lab disposable store, repair deleted the damaged image reference:

Damaged layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d:
2 errors occurred:
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: creating file-getter: readlink /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d/diff: no such file or directory
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: faccessat /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: no such file or directory

Damaged image 5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0:
2 errors occurred:
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: creating file-getter: readlink /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d/diff: no such file or directory
	* layer 2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: faccessat /tmp/podman-corrupt-lab/mnt/storage/overlay/2114fc8b70586b9325dde6fd26066d9951414dcdfb3995f41d51d1995cf3bd9d: no such file or directory

Deleted damaged image: 5dd467fce50b56951185da365b5feee75409968cbab5767b9b59e325fb2ecbc0

By default, damaged images still referenced by a container are left alone. After repair, re-pull or rebuild images you removed and recreate containers from your saved definitions.

Run a full check and a disposable workload to confirm the store is usable:

podman system check

podman pull quay.io/podman/hello:latest
podman run --rm quay.io/podman/hello:latest

Run a full podman system check again after repair. Then pull and start a disposable image to confirm that image storage, layer creation, and container startup all work. If the check still reports damage, do not assume the repair succeeded just because the damaged image disappeared.

When --force removes dependent containers

--repair together with --force can remove containers that depend on damaged images:

podman system check --repair --force

This is more destructive than repair alone but still narrower than podman system reset. Use it only when you accept losing containers tied to broken image layers — for example on a rebuild host with no unique volume data.

Be especially cautious if another container engine shares this storage: Podman may detect dependent containers created by that engine but may not know how to stop them cleanly.

Remove abandoned layers with --max

Interrupted pulls and builds can leave unreferenced layers on disk. system check accepts a maximum age for unreferenced layers:

podman system check --repair --max 1h

--max applies only to layers that are not referenced by any image or container. Layers older than the duration you specify are treated as remnants of cancelled pulls and considered damaged. Shorter durations make younger unreferenced layers eligible to be treated as abandoned. Use conservative values on active build hosts. Do not copy an aggressive --max value from documentation onto a machine running CI builds without understanding what it may delete.

image not known and concurrent cleanup

One upstream-documented failure class is a race, not silent disk corruption. An image is created or pulled, then another process runs aggressive cleanup — podman system prune --all, podman system reset, or podman rmi --all — before the first operation reopens the image. The follow-up command can fail with:

Error: ... image not known ...

Before declaring on-disk corruption, check cron jobs, CI pre-steps, and configuration-management tasks that prune or reset Podman on a schedule. Stagger cleanup automation away from deploy windows, or pin pulls to a registry mirror that repopulates predictably.

Additional image store and missing mount failures

When storage.conf lists read-only additional image stores:

additionalimagestores = [
  "/mnt/shared/container-images"
]

and that path is not mounted, Podman may report lower-layer or stat failures that resemble corruption. Diagnose configuration and mounts before reset:

podman info --format '{{json .Store}}'

Confirm whether the shared path is actually mounted:

findmnt /mnt/shared/container-images

Search configuration files for additional store entries:

grep -R additionalimagestores /etc/containers ~/.config/containers 2>/dev/null

Restore or remount the shared store, or remove the stale entry from storage.conf after confirming images are available elsewhere. Do not delete metadata assuming the layer is corrupt when the NFS or shared disk simply is not present.

SELinux labeling is not storage corruption

A relocated graphroot without file-context equivalence often produces Permission denied inside containers even when UNIX permissions look correct. That is labeling, not missing layers.

Check contexts on the store root:

ls -Zd "$(podman info --format '{{.Store.GraphRoot}}')"

Review AVC denials in journalctl or ausearch when SELinux is enforcing. Fix equivalence with semanage fcontext -e and restorecon as documented in Podman storage location and Fix no space left on device — do not run podman system reset for a labeling mistake.

Check underlying filesystem health

Storage errors can originate below Podman:

• I/O errors and read-only remounts after disk faults
• missing bind mounts after reboot
• network filesystem disconnects
• XFS or ext4 corruption reported in kernel logs

Scan recent kernel messages:

journalctl -k --since "1 hour ago" --no-pager | tail -30

Confirm mounts are present:

findmnt -T "$(podman info --format '{{.Store.GraphRoot}}')"

podman system reset does not repair a failing disk or restore a missing NFS export. Fix the filesystem or mount first, then reassess Podman storage.

Back up persistent data before destructive recovery

Images can usually be re-pulled or rebuilt. Named volumes often hold the irreplaceable state.

List volumes before any repair or reset:

podman volume ls

Export critical volume data with podman run mount copies or application-native backup tools — see Podman volumes. Capture podman inspect output for containers whose environment variables, labels, and mount lists are not stored elsewhere.

podman system reset is the last resort

When system check --repair cannot restore a coherent store and you accept total loss of local Podman objects, reset removes everything under the configured storage scope.

On a disposable lab store, the interactive prompt on Podman 5.8.2 reads:

podman system reset

Sample output:

WARNING! This will remove:
        - all containers
        - all pods
        - all images
        - all networks
        - all build cache
        - all machines
        - all volumes
        - the graphRoot directory: "/tmp/podman-corrupt-lab/mnt/storage"
        - the runRoot directory: "/tmp/podman-corrupt-lab/runroot"
Are you sure you want to continue? [y/N]

Answering n aborts with no changes. The paths listed for graphRoot and runRoot are the directories Podman will delete — read them every time.

podman system reset --force only skips the prompt

--force skips the confirmation dialog but still wipes the same store paths:

podman system reset --force

--force skips the confirmation dialog. It does not:

• preserve volumes or images
• repair more intelligently than reset already does
• back up container definitions
• scope the deletion differently

Use --force only in automation after you have already verified the graphRoot and runRoot paths and exported anything you need.

Verify graphRoot and runRoot before reset

Reset deletes the configured storage directories. Confirm they point where you expect:

podman info --format '{{.Store.GraphRoot}} {{.Store.RunRoot}}'

If storage.conf was mistyped to a broad or shared path — for example a parent directory that holds non-Podman data — reset can remove more than you intend. Never run reset without reading the warning lines that name graphRoot and runRoot.

For planned changes to storage-related settings such as driver, static_dir, tmp_dir, or volume_path, Podman documents that podman system reset must be run before changing the configuration. Reset reads the current configuration to locate storage; editing those paths first can leave the old store behind. That is a planned migration step, not emergency corruption recovery — plan it using Podman storage location.

Restart the Podman socket after reset when needed

podman system reset does not restart podman.service or podman.socket. On hosts that serve the Podman API through systemd, restart the socket if clients hang after reset:

sudo systemctl restart podman.socket

Rootless API setups use the user socket unit instead. Only restart services that are actually part of your deployment.

Recovery ladder

Follow this order rather than jumping to reset or manual deletion:

1. podman version / podman info — capture graphRoot, runRoot, driver, image stores
2. Confirm mounts, filesystem health, and SELinux labeling
3. Stop concurrent prune / reset / rmi --all automation
4. podman system check
5. podman system check --quick only when a fast pass is enough
6. podman system check --repair
7. podman system check --repair --force when you accept losing dependent containers
8. Run podman system check again and test a pull/run
9. Fix missing additional image store mounts or config
10. Back up volumes and inspect output for critical containers
11. podman system reset only if the store remains unusable
12. After reset, verify podman info and test a pull/run

Do not insert raw rm -rf on overlay between these steps.

Troubleshooting

References

• Podman system check manual
• Podman system reset manual
• Podman info manual
• containers-storage.conf(5)
• Red Hat — Managing container storage

Summary

Storage corruption in Podman means local metadata and on-disk layers disagree — not that every error requires podman system reset. Capture podman info, inventory containers/images/volumes, and run podman system check before touching overlay directories by hand. Full check is the authoritative scan; --quick is only a fast first pass.

podman system check --repair removes damaged images Podman can classify, leaving containers that still depend on broken layers until you add --force. Missing additionalimagestores mounts, SELinux labeling after a graphroot move, and concurrent prune automation can mimic corruption — rule those out before reset.

podman system reset on Podman 5.8.2 deletes containers, pods, images, networks, build cache, machines, volumes, and the configured graphRoot and runRoot directories after you confirm the warning. --force only skips the prompt. Back up volumes first, read the paths in the warning, and use Podman storage location when you are changing storage layout rather than recovering from damage.