Fix "watchdog: BUG: soft lockup - CPU stuck" on Linux

watchdog: BUG: soft lockup - CPU#N stuck for ... means the kernel detected a CPU whose watchdog thread did not run within the expected interval. The message does not identify a single cause. Overloaded or paused VMs, host CPU starvation, storage stalls, sustained guest workload, and real kernel or driver bugs can all produce soft lockups.

This guide shows how to distinguish those cases from the kernel log and stack trace. The worked example comes from a RHEL 10.2 VirtualBox VM where several vCPUs reported lockups at the same time.

Quick answer: what a soft lockup means

A soft lockup is a watchdog warning, not a panic by default. A soft lockup means the watchdog thread on a CPU did not get a chance to run for roughly 2 * kernel.watchdog_thresh seconds. watchdog_thresh defaults to 10 upstream, giving a soft-lockup threshold of roughly 20 seconds unless your distro or admin changed it.

This commonly happens when a CPU spends too long executing kernel code without scheduling other tasks, but a virtual CPU that stops receiving host CPU time can produce the same warning. The process name in brackets is whoever was current on that CPU when the detector fired. It is a clue, not a verdict.

Use the pattern in the log, not the bracketed name alone, to choose the next check:

Do not restart a daemon just because its name appeared in brackets. Read the stack trace first.

Common causes of Linux soft lockups

The same warning can come from several different problems. Start with the row that matches where the system is running.

Reports from real systems show why there is no single fix for this message. Virtual machines may be affected by host scheduling, snapshots, storage latency, or hypervisor bugs, while bare-metal systems may point to a specific kernel driver or hardware problem. Use the stack trace and the timing pattern to decide which branch applies before changing VM or kernel settings.

What the watchdog is measuring

The kernel normally runs the soft-lockup watchdog on each online CPU. A helper thread updates a timestamp whenever it is scheduled. If that timestamp goes stale for 2 * kernel.watchdog_thresh seconds because the watchdog thread did not run, the soft lockup detector fires.

The stuck for 37s value in the log is how long the detector thinks that CPU went without a watchdog heartbeat, not a separate sysctl.

The word BUG in the line is the kernel's printk tag, not a panic by itself. Whether detection can trigger panic() depends on kernel.softlockup_panic and kernel.panic; see the next section.

Can a soft lockup cause a kernel panic?

Yes, if panic-on-soft-lockup is enabled. With kernel.softlockup_panic=0, the detector prints the warning and stack trace without deliberately calling panic(). Check the current policy with the sysctl command:

sudo sysctl kernel.softlockup_panic kernel.panic

kernel.softlockup_panic = 0
kernel.panic = 0

kernel.softlockup_panic=0 means a soft lockup does not trigger panic() merely because the watchdog detected it. If it is set to 1, the soft-lockup detector calls panic() when it detects the lockup.

kernel.panic controls what happens after a kernel panic. A positive value tells the kernel to reboot after that many seconds, 0 leaves the system in the panicked state, and a negative value causes an immediate reboot.

For example, a system configured with kernel.softlockup_panic = 1 and kernel.panic = 30 can panic when a soft lockup is detected and then reboot about 30 seconds later.

Do not change these values just to suppress the warning. On production systems they may be deliberately configured so kdump or another recovery mechanism can capture a crash for analysis.

softlockup_panic=0 means the soft-lockup detector itself does not deliberately panic the kernel. The underlying condition that caused the soft lockup could still lead to another fatal kernel error, hard lockup, hardware failure, OOM-related problems, or an unrecoverable hang.

How to read "CPU stuck for 20s" messages

A typical console burst looks like this:

Message from syslogd@vm1 at Sep 15 08:48:34 ...
 kernel:watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [NetworkManager:1122]

Read it in pieces:

• watchdog is the lockup detector, not a hardware WDT daemon such as watchdog.service
• CPU#0 is the CPU that missed its scheduling heartbeat
• stuck for 37s is elapsed time in the detector's view
• [NetworkManager:1122] is the task that was current on that CPU: comm and PID

Message from syslogd@hostname means the kernel logged the line at emergency severity and syslog wrote it to every logged-in terminal. That is why it interrupts an SSH session. It is the same event you will find with journalctl -k.

Check the kernel log and stack trace

The wall message is one sentence. The journal holds timestamps across CPUs, Hardware name, RIP, and RCU lines that narrow the cause.

Search the kernel log for the detector line. Full-system journal access needs sudo or root:

sudo journalctl -k -g 'watchdog: BUG: soft lockup' --since '2026-09-15 08:40' --until '2026-09-15 09:30' --no-pager

Sep 15 08:48:34 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [NetworkManager:1122]
Sep 15 08:48:34 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 37s! [systemd:1]
Sep 15 08:48:34 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 36s! [dockerd:227800]
Sep 15 09:09:44 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [swapper/0:0]
Sep 15 09:09:44 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [containerd-shim:273706]
Sep 15 09:21:54 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 56s! [containerd-shim:269313]
Sep 15 09:27:55 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 37s! [containerd-shim:266994]
Sep 15 09:27:55 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 37s! [containerd-shim:267343]
Sep 15 09:27:55 vm1.lab.example kernel: watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [swapper/0:0]

Three vCPUs complained at 08:48:34, and later bursts again shared a single timestamp. That pattern points to a guest-wide scheduling problem rather than three independent application crashes. At this point, the names NetworkManager, systemd, and dockerd are less important than the fact that three different vCPUs reported the same problem at the same time.

Pull the stack trace for the same second. Match any value in the RIP: line instead of hardcoding RIP: 0010:, because that number is itself diagnostic and filtering on it hides one of the useful cases:

sudo journalctl -k -o cat --since '2026-09-15 08:48:34' --until '2026-09-15 08:48:35' --no-pager | grep -E 'watchdog: BUG: soft lockup|Hardware name|CPU: [0-9]+ UID|RIP: [0-9a-f]{4}:'

CPU: 2 UID: 0 PID: 0 Comm: swapper/2 Kdump: loaded Not tainted 6.12.0-211.47.1.el10_2.x86_64 #1 PREEMPT(voluntary)
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
RIP: 0010:default_idle+0xf/0x20
watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [NetworkManager:1122]
CPU: 0 UID: 0 PID: 1122 Comm: NetworkManager Kdump: loaded Not tainted 6.12.0-211.47.1.el10_2.x86_64 #1 PREEMPT(voluntary)
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
RIP: 0010:e1000_xmit_frame+0x3d6/0x9d0 [e1000]
watchdog: BUG: soft lockup - CPU#1 stuck for 37s! [systemd:1]
CPU: 1 UID: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G             L     ------  ---  6.12.0-211.47.1.el10_2.x86_64 #1 PREEMPT(voluntary)
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
RIP: 0010:handle_softirqs+0x8d/0x2a0
watchdog: BUG: soft lockup - CPU#2 stuck for 36s! [dockerd:227800]

RIP is the instruction pointer: it records where the CPU was executing when the watchdog captured the stack. Read the four hex digits before the colon as well as the symbol after it. That value is the code segment selector, and it tells you whether the CPU was running kernel code at all.

Every trace in the burst above shows 0010:, so those CPUs really were in the kernel. That is the case most soft-lockup write-ups describe, and it is the one where a repeated function name points somewhere useful.

A userspace RIP means something different, and the same lab VM produced one on a later burst. The guest was restarted after that hang, so the evidence sits in the previous boot and needs -b -1:

sudo journalctl -k -b -1 -o cat --no-pager | grep -E 'watchdog: BUG: soft lockup|CPU: [0-9]+ UID|RIP: [0-9a-f]{4}:'

watchdog: BUG: soft lockup - CPU#1 stuck for 37s! [containerd-shim:267788]
CPU: 1 UID: 0 PID: 267788 Comm: containerd-shim Kdump: loaded Tainted: G             L     ------  ---  6.12.0-211.47.1.el10_2.x86_64 #1 PREEMPT(voluntary)
RIP: 0033:0x46c03a
watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [containerd-shim:267701]
CPU: 0 UID: 0 PID: 267701 Comm: containerd-shim Kdump: loaded Tainted: G             L     ------  ---  6.12.0-211.47.1.el10_2.x86_64 #1 PREEMPT(voluntary)
RIP: 0033:0x40c0a4

Both CPUs reported 0033:, so neither was executing kernel code when the detector fired. No guest code was holding a CPU in a kernel loop. The watchdog thread simply never got scheduled, which is what you expect when a vCPU stops receiving host CPU time. On a virtual machine that reading points at the host or hypervisor, and it carries far more weight than the containerd-shim name in brackets.

On older journalctl builds without -g, pipe sudo journalctl -k --since … through grep. Note that -k implies the current boot, so add -b -1 when the lockup happened before a restart. The journalctl command covers boot ranges and kernel filters if you need a wider window.

Don't blame the process name without checking the stack trace

The name in brackets identifies the task that was current when the watchdog reported the lockup. By itself, it does not prove that process caused the stall.

If several CPUs show unrelated names at the same timestamp, look for a guest-wide problem first: hypervisor scheduling, host overload, or storage delay. If repeated events stop in the same kernel function, however, that function or driver deserves investigation. Genuine soft-lockup bugs do occur in storage, network, GPU, and other kernel drivers, which is why the stack trace matters more than the userspace task name.

swapper/N is the per-CPU idle task. If the captured RIP is default_idle, the CPU was in its idle path when the watchdog captured the stack. That tells you where execution was observed, not necessarily where the CPU spent the entire reported lockup interval. When that appears in the same timestamp burst as soft lockups on other vCPUs, it supports investigating a VM-wide scheduling or timing interruption rather than blaming one userspace daemon.

containerd-shim and dockerd showed up on the lab VM because Docker Engine was active. They were on the CPU when the watchdog sampled, not proof of a containerd kernel bug. For how Engine and containerd split that work, see Docker vs containerd.

Restarting the named service may temporarily change what appears in the next warning, but it does not address guest CPU starvation if the real problem is outside that service. The journal from earlier bursts remains either way.

Multiple CPUs at once: check the VM or hypervisor first

When several vCPUs lock at the same second, start with the hypervisor and host before chasing individual daemons. The lab VirtualBox case below is one worked example of that branch.

Same-timestamp bursts and mixed RIPs

Hardware name: innotek GmbH VirtualBox tells you this guest runs under VirtualBox. systemd-detect-virt may report oracle for VirtualBox even when lscpu shows Hypervisor vendor: KVM, because VirtualBox can expose KVM interfaces on some hosts. Those outputs are not contradictory by themselves.

Seeing unrelated paths such as default_idle and e1000_xmit_frame on different vCPUs in the same burst makes a guest-wide scheduling pause much more likely than two unrelated bugs occurring at the same instant. CPU 0 was executing e1000_xmit_frame in the context of NetworkManager when the stack was captured. That tells us where execution was observed; it does not prove that either NetworkManager or the e1000 driver caused the guest-wide stall.

If the lockups appear specifically after host suspend or resume or a long VM pause, record the guest's clocksource as part of the evidence, but do not switch clocksources merely because a soft lockup occurred.

RCU starvation alongside the lockups

The same second on the lab VM also logged an RCU stall:

sudo journalctl -k -g 'rcu_preempt' --since '2026-09-15 08:48:33' --until '2026-09-15 08:48:35' --no-pager

Sep 15 08:48:34 vm1.lab.example kernel: rcu: INFO: rcu_preempt self-detected stall on CPU
Sep 15 08:48:34 vm1.lab.example kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 17462841 jiffies! g10591629 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402
Sep 15 08:48:34 vm1.lab.example kernel: rcu: rcu_preempt kthread starved for 17462844 jiffies! g10591629 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=2
Sep 15 08:48:34 vm1.lab.example kernel: rcu:         Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Sep 15 08:48:34 vm1.lab.example kernel: task:rcu_preempt     state:I stack:0     pid:16    tgid:16    ppid:2      task_flags:0x208040 flags:0x00004000

The RCU messages are stronger evidence than the bracketed process name. The kernel explicitly reports that the rcu_preempt thread was starved of CPU time. Combined with several vCPUs reporting soft lockups at the same timestamp, this strongly points to a guest-wide scheduling or timing interruption rather than a NetworkManager-only problem.

In this VirtualBox case, host sleep, VM pause, a long snapshot operation, or severe host starvation are therefore stronger candidates than a 37-second userspace hang. Sustained guest workload pressure can also starve RCU, so correlate the warning with CPU, memory, and I/O metrics when possible.

Later lockups on this VM set the L taint flag, which means a soft lockup already happened in this boot. It does not name a new vendor module by itself.

Check the driver named by the stack

When the stack repeatedly samples network transmit code, note which driver backs the interface:

ethtool -i enp0s3

driver: e1000
version: 6.12.0-211.47.1.el10_2.x86_64
firmware-version:

If separate lockups repeatedly stop in a function such as e1000_xmit_frame, first check for kernel and hypervisor updates that address that driver path. If the problem persists, testing a supported paravirtualized network adapter is a reasonable next step rather than repeatedly restarting NetworkManager.

One CPU or same RIP repeatedly: check kernel or driver

When only one CPU locks, or the same RIP appears across multiple events while sibling CPUs keep scheduling normally, treat it as a possible kernel or driver defect.

The important pattern is repetition: if separate lockups repeatedly stop in the same kernel function while other CPUs continue working normally, investigate that function, module, and kernel version.

A kernel or driver bug becomes more likely when:

• One CPU locks while the others keep scheduling and logging normally
• RIP stays in the same function across events
• There is no matching RCU stall with millions of jiffies and no default_idle on sibling CPUs at the same timestamp
• Host pause, snapshots, laptop sleep, and sustained guest overload are already ruled out

Capture journalctl -k around the event, note uname -r and taint flags, and check kernel errata for your distro. The kernel version article lists the usual files besides uname. If the box is down hard, you need kdump or a vmcore, which is a different workflow from these console broadcasts.

Check CPU, memory, swap, and I/O

Metrics collected after the guest recovers can look healthy even when the journal still shows a long RCU stall. When you can sample while the problem is happening, sustained r above available CPUs, continuous swap activity, high I/O wait, or persistent steal time change the diagnosis. In that case, reduce the workload or allocate enough CPU and memory before assuming a kernel bug.

The vmstat command helps while the incident is active. In vmstat, watch r for runnable tasks, si/so for swap activity, wa for I/O wait, and st for CPU time taken by the hypervisor.

vmstat 1 5

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
 4  0 905264 800760      0 2990688   17   32   153   498  948    5  2 11 87  0  0  0
 8  0 905264 783484      0 2994128   12    0  3132    88 3880 5791 30 44 23  2  0  0
 6  0 905008 745312      0 3018908   28    0 24232   276 4909 7062 27 61 10  1  0  0
 3  0 905008 734808      0 3020756    0    0  1580   146 2438 4361 25 38 33  3  0  0
 8  0 905008 716104      0 3020780    0    0    20    35 2045 3214 26 29 46  0  0  0

st was 0 here because the sample ran after the lab stalls had ended. Pair vmstat with mpstat per-CPU lines and iostat when disk latency is suspect. High wa during the incident means the guest is spending significant time waiting for I/O. Use iostat and hypervisor or storage metrics to determine whether the delay comes from the guest workload, virtual disk, or backing storage.

Check memory and swap:

free -h

               total        used        free      shared  buff/cache   available
Mem:           7.5Gi       3.7Gi       1.8Gi        12Mi       2.2Gi       3.8Gi
Swap:          3.0Gi       888Mi       2.1Gi

Almost 900 MiB is stored in swap, but that alone does not show that memory pressure is occurring now. Previously swapped pages can remain there long after pressure subsides. The more useful signal is active swapping: if si and so remain non-zero during the lockup, investigate memory pressure and reduce the workload or add RAM. Swap policy is covered in the Linux swap and swappiness guide.

Fix VM or host resource problems

The bullets below apply when several CPUs lock together or RCU shows long starvation, as in the VirtualBox lab example.

On the hypervisor host:

• If you intentionally pause a VM for longer than the watchdog threshold, expect the guest to report a soft lockup when it resumes. Avoid long pauses when you need clean watchdog diagnostics
• Prevent unintended host sleep or hibernate while important VMs are running
• Check whether the lockup timestamp matches a snapshot, backup, migration, or other hypervisor operation that may briefly pause or delay the VM
• Avoid overcommitting the host CPU. If the VM has more vCPUs than it needs, test a smaller allocation and make sure the host still has capacity for other workloads
• If the lockup coincides with high I/O wait, check virtual-disk and host-storage latency. Severe storage delays can contribute to long stalls

When the stack repeatedly implicates one driver, update the kernel or test a supported paravirtual device for that subsystem. When lockups track sustained load inside the guest, reduce the workload or allocate enough CPU and memory before chasing a hypervisor-only explanation.

Temporarily increase the soft-lockup threshold

Increasing watchdog_thresh makes the watchdog wait longer before reporting a soft lockup. It does not give the VM more CPU time or correct the underlying stall.

The live knob is kernel.watchdog_thresh. Older articles may mention kernel.softlockup_thresh. That sysctl is not present on the RHEL 10 kernel tested here; kernel.watchdog_thresh is the threshold used by this system.

Note the current value before changing it. Read the watchdog tunables with the sysctl command:

sudo sysctl kernel.watchdog_thresh kernel.watchdog kernel.softlockup_panic kernel.nmi_watchdog kernel.soft_watchdog kernel.hung_task_timeout_secs

kernel.watchdog_thresh = 10
kernel.watchdog = 1
kernel.softlockup_panic = 0
kernel.nmi_watchdog = 0
kernel.soft_watchdog = 1
kernel.hung_task_timeout_secs = 120

Soft lockup delay is twice this value, so 10 means roughly 20 seconds. Raising watchdog_thresh to 30 makes the soft window about 60 seconds. Use that only while you fix host sleep, overcommit, storage, or workload pressure, or on a lab VM you expect to pause.

Apply it until reboot:

sudo sysctl -w kernel.watchdog_thresh=30

kernel.watchdog_thresh = 30

Restore the value that was configured before your test. On this lab it was 10:

sudo sysctl -w kernel.watchdog_thresh=10

kernel.watchdog_thresh = 10

If you have a legitimate reason to keep a different threshold permanently, use a sysctl drop-in after you understand why this system regularly exceeds the default threshold. The sysctl reload article covers --system versus -p when several drop-ins are in play.

Setting watchdog_thresh to 0 disables lockup detection; do not do that on a machine you still want to debug. Changing kernel.softlockup_panic or kernel.panic is a separate policy decision; see Can a soft lockup cause a kernel panic? above.

Summary

A watchdog: BUG: soft lockup warning means the CPU's watchdog thread did not run within the expected interval. Do not assume the process shown in brackets caused it. First compare timestamps and stack traces: several CPUs reporting soft lockups at the same time suggest guest-wide starvation or host or hypervisor delay, while repeated lockups in the same kernel function make a driver or kernel problem more likely.

Check CPU pressure, swap, I/O wait, steal time, VM operations, and RCU messages before changing kernel settings. kernel.watchdog_thresh can delay the warning on an intentionally slow or paused VM, but it does not fix the underlying stall.

References

• Softlockup detector and hardlockup detector (Linux kernel documentation)
• Documentation for /proc/sys/kernel/ (watchdog_thresh, softlockup_panic, hung_task_timeout_secs)