Troubleshoot High I/O Wait in Linux

top shows 56.7 wa in the CPU line while %usr stays in single digits — CPUs are idle, but they are idle because they are waiting on disk, not because the host has spare capacity. That wa column is I/O wait, and it is one of the first signals that a storage bottleneck is slowing applications.

The workflow below confirms iowait is really elevated, names the busy device with iostat, ties writes to a PID with pidstat, checks for uninterruptible D state blockers, and rules out swap thrash before you resize disks or blame application code.

What I/O wait means

I/O wait (%iowait in mpstat, wa in top and vmstat) is the fraction of CPU time where a processor had nothing to run because tasks were waiting on block I/O to finish. The CPU is not executing user or kernel code during that slice — it is parked until the storage stack returns.

I/O wait is not the same as disk utilization. A disk can be busy at %util 100 while %iowait is moderate if enough work stays in CPU cache or if I/O completes quickly. Conversely, high %iowait with rising w_await in iostat means applications are stalling on slow or saturated storage.

I/O wait also differs from load average. Load includes tasks in uninterruptible sleep (D state) that may not show as high %iowait on every sample. Use both metrics together — see high load average troubleshooting when load and iowait diverge.

Confirm I/O wait is high

Start with mpstat from the sysstat package. It prints %iowait per interval so you catch brief spikes a single top snapshot might miss:

mpstat 1 3

On an idle lab host, %iowait stays near zero:

Average:     all    0.17    0.00    0.33    0.00    1.50    0.33    0.00    0.00    0.00   97.66

The fifth numeric column after %sys is %iowait; the last column is %idle. Values above roughly twenty percent sustained while users report lag deserve a storage investigation on most servers.

top in batch mode shows the same signal in the summary line — look for wa:

top -b -n 1 | head -5

During a heavy sync-write test on this VM, wa climbed past fifty percent:

%Cpu(s):  3.3 us,  3.3 sy,  0.0 ni, 30.0 id, 56.7 wa,  6.7 hi,  0.0 si,  0.0 st

Low us with high wa is the classic iowait pattern — the box is not CPU-starved; it is disk-starved. Re-run mpstat or top during the incident window, not after users say performance recovered.

Identify the busy storage device

Once %iowait is elevated, find which block device is saturated. iostat -xz prints per-disk stats and hides idle devices:

iostat -xz 1 3

During stress --hdd on the lab host, dm-0 and sda showed full utilization:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          10.53    0.00   84.21    0.00    0.00    5.26

Device            r/s     rkB/s     w/s     wkB/s  w_await  aqu-sz  %util
dm-0            53.00    212.00 2244.00 294956.00     1.97    4.49 100.00
sda             53.00    212.00 2290.00 292396.00     1.77    4.12   90.70

%util near one hundred on sda / dm-0 pinpoints local disk pressure. On systems with NFS, iSCSI, or multipath, map the busy /dev node back to mounts with lsblk and findmnt before chasing the wrong LUN.

The avg-cpu header in the same iostat output also lists %iowait — useful when you want disk and CPU breakdown on one screen.

Check latency with iostat

Utilization alone does not tell you whether latency is acceptable. Read w_await and r_await (milliseconds per I/O) and aqu-sz (average queue depth):

iostat -xz 1 2

Under sustained writes in the lab, queue depth and await rose with utilization:

Device            r/s     w/s  w_await  aqu-sz  %util
dm-0           793.07 405315.35     3.74    2.97   98.61
sda            989.11 406752.97     4.80    4.82   86.24

w_await of a few milliseconds on SSD-backed VMs is normal; double-digit or triple-digit await on production arrays under load explains application timeouts. Compare against your storage baseline — a sudden jump matters more than an absolute number.

If %iowait is high but every local disk shows low %util, look at network storage mounts, hypervisor datastore contention, or RAID rebuild activity before assuming the tool is wrong.

Find processes generating I/O

pidstat -d attributes read and write throughput to processes. Run it while %iowait is elevated:

pidstat -d 1 3

During stress --hdd, the stress workers dominated disk writes:

Average:      UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s iodelay  Command
Average:        0     23198      0.00 712589.47 522778.95       0  stress
Average:        0     23199      0.00 637978.95 522778.95       0  stress

kB_wr/s and kB_rd/s are kilobytes per second per process. High kB_ccwr/s means cache-flushing writes (fsync-heavy workloads). Note the PID, then confirm the command line:

ps -p 23198 -o pid,user,stat,etime,cmd

PID USER     STAT     ELAPSED CMD
  23198 root     D          00:08 stress --hdd 2 --hdd-bytes 512M --timeout 15

Parallel dd jobs with oflag=direct produced a similar pattern — several dd PIDs with hundreds of megabytes per second in pidstat. Short cron bursts may appear only in pidstat loops, so sample for at least thirty seconds during the slowdown.

Check processes in D state

Processes in D state (uninterruptible sleep) wait on kernel I/O and cannot be killed until the driver returns. They inflate load average and often accompany storage stalls:

ps -eo state,pid,cmd | awk '$1 ~ /D/ {print}'

During the stress --hdd run on the lab host:

D   23198 stress --hdd 2 --hdd-bytes 512M --timeout 15
D   23199 stress --hdd 2 --hdd-bytes 512M --timeout 15

Occasional kworker lines in D during heavy flush are common. Many application PIDs stuck in D for minutes, especially with NFS mounts, point at hung storage or network paths — not a simple CPU tuning problem.

vmstat column b counts blocked processes waiting for I/O:

vmstat 1 3

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  3      0 1281924   5052 4163932    0    0     4 266208 3273 2200  0 46 31 22  0
 0  2      0 1242324   5052 4206980    0    0     0 260596 1260  897  0 16 39 45  0

Rising b with high wa and large bo (blocks written out) matches a write-heavy bottleneck. bi/bo show system-wide block I/O rates in kilobytes per interval.

Check swap activity

Swap traffic is disk I/O. When memory is exhausted, paging drives %iowait on the swap device even if application disks look idle:

free -h

Mem:           7.5Gi       2.6Gi       1.9Gi       5.3Mi       3.3Gi       4.9Gi
Swap:          3.0Gi          0B       3.0Gi

On this lab VM swap is unused (0B used). Production hosts with swap in use need vmstat swap columns:

vmstat 1 3

Watch si (swap in) and so (swap out). Non-zero sustained values mean the kernel is paging — relieve memory pressure before tuning disk schedulers. See check memory usage per process when free shows little available memory and %iowait spikes without an obvious write workload.

List active swap devices with:

swapon --show

NAME      TYPE      SIZE USED PRIO
/dev/dm-1 partition   3G   0B   -2

If swap I/O correlates with iowait on dm-1, add RAM or reduce process footprint before buying faster application disks.

Identify the I/O bottleneck

Match the dominant signal to the fix — adding CPU rarely helps when %iowait and disk %util lead the chart.

After you address the root cause, confirm %iowait dropped:

mpstat 1 2

Average:     all    0.17    0.00    0.33    0.00    1.50    0.33    0.00    0.00    0.00   97.66

Return to idle-level %iowait within a few minutes of stopping a heavy writer is the recovery pattern you want. For ongoing disk monitoring beyond incident response, continue with monitor disk I/O performance and the iostat command reference.

Troubleshooting

References

• mpstat(1) — Linux manual page
• iostat(1) — Linux manual page
• pidstat(1) — Linux manual page
• vmstat(8) — Linux manual page
• ps(1) — Linux manual page

Summary

High I/O wait on Linux means CPUs are idle while tasks wait on block storage — not that you need more processor cores. Confirm the spike with mpstat or top during the incident, then use iostat -xz to see which device hits high %util and whether w_await latency explains the lag.

pidstat -d ties megabytes per second to a PID so you can distinguish a runaway backup from kernel flush noise. Processes in D state and a rising vmstat b column point at blocking I/O; swap-in and swap-out traffic means memory pressure is masquerading as a disk problem.

Use the bottleneck table to pick storage tuning, mount fixes, or RAM — not all three at once. When load is high but the story is unclear, cross-check high load average and high CPU usage guides so you do not chase the wrong resource.