Linux Troubleshooting Interview Questions and Answers

What interviewers are testing: whether you classify the bottleneck before guessing at application bugs. They want a spoken triage path, not a random tool list.

Open your answer with: scope (one host vs whole fleet), timing (when it started), and that you will split CPU, memory, disk, and network before touching config.

Walk through these checks in that order and say what each result means as you go:

1. uptime + nproc — use load relative to CPU count as a pressure clue, then determine whether the queue contains runnable CPU work or blocked D-state tasks
2. top or mpstat 1 3 — split %usr, %sys, %iowait, %idle
3. free -h — available memory and swap use
4. df -h and df -i — block and inode pressure
5. iostat -xz 1 3 (if installed) — disk %util and await
6. ss -s or ss -tunap — connection pile-up
7. Recent change — deploy, backup, cron, traffic spike

Mention before you finish: ask about deploys, backups, or cron in the same window—interviewers reward correlating symptoms with change.

A strong answer is:

"I compare load to CPU count, then read top for iowait versus user CPU, check memory and disk headroom, and only then drill into the hottest process or disk. I also ask what changed in the last hour."

What interviewers are testing: whether you know load average is not CPU percent. Junior candidates blame the wrong subsystem; seniors name D state and I/O.

Open with: "Load is high but CPUs look idle, so I check for tasks blocked in uninterruptible sleep before I add cores."

On Linux, load average counts runnable (R) and uninterruptible (D) tasks—not only CPU burners. See proc_loadavg(5).

Check in this order:

1. uptime and nproc — is load sustained above CPU count?
2. mpstat -P ALL 1 3 — low %usr with high %iowait points at disk
3. ps aux | awk '$8 ~ /D/' — processes stuck in D state
4. vmstat 1 5 — rising b column (blocked) with high load
5. iostat -xz 1 — saturated disk %util or high await
6. NFS or multipath — D state on mount-related PIDs; see NFS server and client setup when the wait traces to an export

High %iowait supports an I/O diagnosis, but low %iowait does not rule it out; confirm D-state tasks, blocked queues, and device/NFS latency.

Say explicitly: if ps shows D-state PIDs or vmstat shows a rising blocked queue, the problem is I/O wait—not a CPU shortage.

A strong answer is:

"High load with relatively idle CPUs often points to tasks accumulating in uninterruptible sleep (D state), commonly because of storage or network-filesystem I/O. I confirm task states with ps and vmstat rather than inferring the cause from load alone."

What interviewers are testing: whether you understand why SIGKILL fails and that the fix is the I/O path, not more signals.

Open with: "D state means the task is stuck in an uninterruptible kernel wait—commonly storage or NFS—and signals will not clear it until that wait completes."

A process in D (uninterruptible sleep) is in uninterruptible sleep inside the kernel—commonly waiting on block storage, NFS, or another kernel resource—and ignores kill -9 until the I/O completes or the underlying device recovers.

1. ps -o pid,stat,wchan:30,comm -p <pid> — D in STAT, wchan hints at the wait (for example nfs_wait)
2. cat /proc/<pid>/stack — kernel stack for the blocked syscall
3. lsof -p <pid> — inspect files, sockets, and mount-backed objects the process currently has open
4. dmesg -T | tail — I/O errors, NFS server not responding
5. Do not reboot first unless the whole node is frozen—fix storage or unmount the stuck NFS export when safe

Recovery interview points:

• Lazy unmount (umount -l) only after stopping writers and accepting risk
• NFS hard mounts can block indefinitely; server or network repair is the real fix
• Killing the parent does not clear D state on the blocked child until I/O returns

Close with: reboot only if the node is fully frozen; otherwise fix storage, NFS, or unmount the stuck export after stopping writers.

A strong answer is:

D state means the task is stuck in an uninterruptible kernel wait, commonly I/O. I inspect wchan, the kernel stack, relevant files/mounts, and kernel logs to identify the blocked resource rather than repeatedly sending signals.

What interviewers are testing: Whether you can distinguish runnable CPU demand from uninterruptible waits and explain why load average and CPU utilization can diverge.

Load can rise when tasks pile up in D state even while %usr stays low. Treat load, %iowait, and CPU percent as related but distinct signals—see high load and high I/O wait for full walkthroughs.

A strong answer is:

"Load counts R and D tasks; CPU percent measures execution. They diverge when I/O blocks work in D state—I confirm task states and device latency instead of treating load as a CPU-only metric."

What interviewers are testing: Whether you separate one hot process/core from host-wide user/system CPU and know when to move from process metrics to profiling.

Open with: "I sort by CPU, split user versus system time, and check if one PID or one core dominates before I scale out."

Split user CPU from system CPU before blaming application code:

1. top — press 1 for per-CPU, then P to sort by CPU
2. mpstat -P ALL 1 3 — which logical CPU is hot; %sys vs %usr
3. pidstat -u 1 5 — thread-level CPU if one PID dominates
4. perf top or perf record -g (if allowed) — kernel vs userspace stack
5. Recent change — deploy, cron, virus scan, log parser

One core saturated, overall CPU looks low: press 1 in top or use mpstat -P ALL—a single-threaded process can pin one logical CPU at 100% while the rest stay idle. Sort by CPU per core, then check thread count with pidstat -t -p <pid>.

See high CPU usage for a full walkthrough.

Narrate the fork: one PID at 90% points to app or query tuning; high %sys everywhere points to syscalls, context switching, or driver work.

A strong answer is:

"I sort top by CPU, confirm whether it's one PID or many, and check %sys versus %usr. If one process owns the core, I inspect its command line and recent deploy; if %sys is high, I look for syscall storms or kernel-side work before scaling out."

What interviewers are testing: time correlation—steady outages are easy; intermittent slowness separates admins who guess from those who prove cause.

Open with: "I narrow the incident window first, then pull metrics and logs for only that slice before I change tunables."

Intermittent issues need time correlation and multiple signal types.

1. Define pattern — time of day, one user vs all, one API vs whole site
2. Metrics — CPU, memory, disk latency, error rate for the incident window
3. sar history (if sysstat enabled) — sar -u, sar -d, sar -n DEV
4. journalctl --since "2026-01-01 14:00" --until "2026-01-01 15:00" — errors in the window
5. Deploy and cron alignment — backup, batch job, cache expiry
6. External dependency — DNS TTL, remote API latency spikes
7. Capture while hot — perf record, short packet capture, application trace

Say out loud: intermittent slowness is often a batch job, backup, cache expiry, or external API—not a missing sysctl.

A strong answer is:

"I narrow the time window and pull sar or monitoring graphs for that slice, then match journal errors and deploy logs. Intermittent slowness is often a scheduled job or an external dependency—I prove correlation before changing kernel tunables."

What interviewers are testing: whether you distinguish real memory pressure from healthy page cache. Restarting services before reading available is a red flag.

Open with: "I read free -h and available, then find top RSS consumers—I do not treat cached memory as a leak."

Read free -h and /proc/meminfo before restarting services:

Next steps:

1. ps aux --sort=-%mem | head — top memory consumers
2. vmstat 1 5 — si/so swap activity
3. Cgroup limits — systemd-cgtop or cat /sys/fs/cgroup/.../memory.* on container hosts
4. OOM history — dmesg \| grep -i oom or journalctl -k \| grep -i oom

For cgroup-limited workloads, Linux container memory limits and cgroups walks through memory.events, MemoryMax, and host-wide pressure.

If on containers: say you would check cgroup memory.max and memory.events, not only host free.

A strong answer is:

"I use available not just free, identify the top RSS processes, watch swap in/out, and check whether a cgroup cap—not host RAM—is the limit."

What interviewers are testing: whether you trace who was killed, why, and whether the limit was host-wide or cgroup-scoped—not just restart the pod.

Open with: "I pull the OOM line from kernel logs, identify the victim process, then decide if the fix is more RAM, a higher limit, or a leak."

The kernel OOM killer picks victims when memory cannot be satisfied. Walk the evidence chain:

1. Kernel log — dmesg -T | grep -i 'killed process' or journalctl -k -b | grep -i oom
2. Which process — log line names PID, comm, and score
3. Host vs cgroup — for cgroup-limited workloads, inspect cgroup memory.events and the configured memory limit first; correlate with kernel/journal messages where available
4. Limits — systemctl show <unit> -p MemoryMax or K8s limits.memory
5. Leak vs legit — RSS trend over time, restart pattern

Mitigation interview points:

• Raise limit only if justified; fix leak or right-size workload
• Tune oom_score_adj only when you understand side effects
• Add alerting on MemAvailable from /proc/meminfo before OOM

Mention mitigation: alert on MemAvailable before OOM; raising limits without root-cause analysis is a weak senior answer.

A strong answer is:

"I pull the OOM line from dmesg or journal when present, check cgroup memory.events and limits for containerized workloads, then decide if we need more RAM, a higher limit, or a code fix for a leak—not just restart the pod."

What interviewers are testing: Whether you identify sustained paging through si/so rather than treating any nonzero swap usage as a problem.

Open with: "Sustained swap in and out in vmstat means the working set does not fit RAM—I find the memory hogs before I tune swappiness."

Swap thrashing occurs when the kernel pages memory in and out repeatedly because RAM cannot hold the working set.

1. free -h — swap used vs available
2. vmstat 1 10 — high si and so (swap in/out) every second
3. ps aux --sort=-%mem | head — top RSS consumers
4. Was cache mistaken for free? — compare available trend over time
5. Cgroup cap — container limit lower than host RAM

Mitigation talking points:

• Add RAM or reduce workload footprint
• Tune vm.swappiness only after measuring—not as first reflex
• Kill or restart the leak; temporary swap file on wrong disk worsens iowait

See Linux swap and swappiness before tuning vm.swappiness.

Say: adding swap on a slow disk can make iowait worse; right-size RAM or stop the leak first.

A strong answer is:

"High si/so in vmstat with crawling response times means we're paging constantly. I identify the memory hogs, relieve pressure, and treat swappiness tuning as secondary to right-sizing RAM or fixing the leak."

What interviewers are testing: the classic df versus du gap—open deleted files and mount overlays—not "mysterious disk gremlins."

Open with: "I confirm I'm measuring the same mount, then look for deleted files still held open and for a mount hiding older data."

Common causes when df and du disagree:

First commands — the lsof command finds deleted files still held open:

df -h /var
du -xh --max-depth=1 /var | sort -h | tail
lsof +L1 | grep /var

Say: space is not freed until the last file descriptor closes—lsof +L1 is the proof interviewers expect.

A strong answer is:

"I verify I'm on the same mount, then look for open-but-deleted files with lsof +L1—space is not reclaimed until the last file descriptor closes. I also check for a mount hiding an older data directory underneath."

What interviewers are testing: Unix unlink semantics—deleting a file does not free blocks while a process still holds it open.

Open with: "The log was unlinked from the directory but a daemon still has the inode open, so df stays full."

Find holders:

Unix does not free disk blocks until the inode link count hits zero and no process holds the file open.

Find holders:

lsof +L1 | grep deleted
# or
lsof | grep '<filename>'

Fix paths:

• Restart or HUP the process holding the descriptor (logrotate copytruncate vs create trade-off)
• Truncate via fd — truncate -s 0 /proc/<pid>/fd/<n> (last resort, know the process)

Mention logrotate: copytruncate versus create matters when daemons keep descriptors open across rotation.

A strong answer is:

"The file is unlinked from the directory but still open—lsof +L1 shows it. Space returns when the process closes the fd or exits. I fix logrotate strategy so we don't delete files still held open."

What interviewers are testing: whether you know inode exhaustion is separate from block space—df -h can show free GB while df -i is 100%.

Open with: "Create failed with no space left on device but df -h has room—I check inode use next."

Likely inode exhaustion—every file consumes an inode even when block space remains. Confirm with:

df -i /
df -h /

Find many small files — count files in a directory when you need to rank heavy subtrees under /var:

find /var -xdev -type f | cut -d/ -f1-3 | sort | uniq -c | sort -n | tail

Common culprits: mail spool, tmp sessions, container layers, millions of cache files.

Say: the fix is deleting or archiving millions of small files (mail spool, cache, sessions), not hunting one huge file.

A strong answer is:

"I run df -i. If IUse% is 100%, I hunt directories with huge file counts—often mail, cache, or session dirs—not large single files."

What interviewers are testing: whether you treat read-only remount as a protective kernel response to I/O or metadata errors—not something to force back to rw blindly.

Open with: "Writes fail because the filesystem entered a protective failure mode after an error—I read dmesg for the trigger before I remount writable."

Serious filesystem or block-device errors can force a filesystem into a protective failure mode. For example, ext4 may remount read-only, while XFS can shut the filesystem down and return I/O errors.

1. dmesg -T | tail -50 or journalctl -k -b | tail — EXT4-fs error, I/O error, Remounting filesystem read-only
2. findmnt -o TARGET,OPTIONS /path — confirm ro in mount options
3. smartctl -a /dev/sdX (if available) — disk health
4. cat /proc/mounts | grep <mount> — ro flag on device
5. Underlying storage — SAN path, cloud volume, full LVM thin pool

Recovery interview points:

• Do not force mount -o remount,rw on a corrupted filesystem without understanding the error
• Run the filesystem-specific repair tool offline when required—for example e2fsck/fsck for ext4 or xfs_repair for XFS—after fixing the underlying storage problem and taking appropriate backup/snapshot precautions
• Fix disk or multipath before remounting writable

Say: on production you snapshot or backup before offline repair; remount rw without understanding the error risks corruption.

A strong answer is:

"Read-only or failed writes usually mean the kernel hit a filesystem or block I/O error. I read dmesg for the remount reason, verify with findmnt, check disk health, and only remount read-write after I know whether offline repair with the correct filesystem tool is required."

What interviewers are testing: Whether you recognize per-user/project allocation limits after filesystem block and inode capacity have been ruled out.

Open with: "After ruling out full filesystem and full inodes, I check quotas for the writing user."

Check disk quotas when per-user or per-group limits apply:

1. Error — Disk quota exceeded or EDQUOT
2. quota -v <user> or repquota -a — user/group block and inode quotas
3. xfs_quota -x -c 'report -h' on XFS with project quotas
4. df -h and df -i — still rule out filesystem-wide full
5. Project quota on shared tree — app user limited under /data

A strong answer is:

"After df and df -i, I check quota -v for the writing user. Quotas can block writes while the volume still shows free space for other users."

What interviewers are testing: Whether you find what is growing, distinguish journal/app logs/open-deleted files, and fix retention/rotation rather than performing a one-time deletion.

Open with: "I rank what under /var grew, separate journal from app logs, then fix rotation so it does not refill next week."

Investigate:

1. df -h /var and du -xh --max-depth=1 /var | sort -h | tail — which subtree grew; check disk space covers df/du interpretation
2. journalctl --disk-usage — systemd journal size
3. find /var/log -type f -size +100M -ls — oversized files
4. logrotate -d /etc/logrotate.d/<app> — dry-run the rotation config
5. Application logging level — debug turned on in production?
6. Open deleted logs — lsof +L1 | grep /var/log if df full but du low

Sustainable fixes:

• Tune logrotate size/rotate/compress and copytruncate vs create when daemons hold file descriptors
• Cap journal with /etc/systemd/journald.conf (SystemMaxUse=) — enable persistent logging in journald when logs vanish after reboot
• Ship logs off-host instead of infinite local retention

Say: also check open-but-deleted logs with lsof +L1 if du and df disagree.

A strong answer is:

"I rank /var children with du, check journal disk usage, and find the largest log files. Then I fix rotation or logging verbosity—not just rm once—so the partition does not refill next week."

What interviewers are testing: whether you treat %iowait as a clue and correlate device latency and per-process I/O—not proof the disk is fine because %util is under 100%.

Open with: "High iowait means workloads may be waiting on storage—I map it to a device, then to a PID."

High %iowait is a clue that storage I/O may be delaying workloads. Correlate it with device latency, queueing, and per-process I/O rather than treating iowait alone as proof of a saturated disk.

1. iostat -xz 1 3 — %util, await, r/s, w/s per device
2. pidstat -d 1 5 — which PIDs issue reads/writes
3. iotop -o (if available) — live I/O by process
4. Filesystem type — NFS, EBS, local SSD behave differently
5. Recent job — backup, find, log rotation, database checkpoint

Disk slow but %util is not 100%: queue depth, latency spikes, or one slow volume behind LVM/RAID can still hurt workloads—read await and per-process I/O, not %util alone.

See monitor disk I/O performance for iostat, pidstat, and related tooling.

A strong answer is:

"I treat iowait as a pressure signal, then map it to a device with iostat and to a process with pidstat or iotop. High await with moderate %util still points at a storage bottleneck—I correlate workload timing before assuming the disk is fine."

What interviewers are testing: NFS as a client-side hang (D state, stale handle) and whether you check server, network, and mount options.

Open with: "Commands hang on an NFS path—I confirm mount options, retransmits, and D-state processes on that mount."

NFS problems often show up as D state processes and high load with low CPU—the same pattern as local disk stalls.

1. mount | grep nfs — server, export and mount options (hard vs soft, timeo, retrans, NFS version); see NFS mount options
2. nfsstat -c or nfsiostat — retransmits and latency to server
3. dmesg | grep -i nfs — server not responding, stale file handle
4. ps aux | awk '$8 ~ /D/' — PIDs blocked on the mount
5. Server side — export health, rpcinfo -p on NFS server; Firewall/RPC path — for NFSv4 verify TCP 2049; older NFS versions may also depend on rpcbind/mountd and additional RPC ports

Say: killing random PIDs does not fix a dead server; lazy umount is last resort after stopping writers.

A strong answer is:

If I see a stale file handle, I treat it as a client/server namespace or file-handle validity problem and remount or repair the affected export safely after checking what changed server-side.

What interviewers are testing: safe rebuild discipline—one disk at a time, backups verified, awareness that RAID5 rebuild is a second-failure window.

Open with: "Degraded means normal single-disk redundancy is already gone—I confirm which slot failed and that I have a replacement before rebuild."

Interviewers want safe order of operations and awareness of double-failure risk.

1. cat /proc/mdstat — software RAID state ([U_] degraded)
2. mdadm --detail /dev/mdX — failed slot, spare presence — the mdadm command covers replace and rebuild steps
3. Hardware RAID — vendor tool (storcli, megacli) for physical disk status
4. dmesg — I/O errors on specific /dev/sdX
5. Backup verify — a RAID5 array already operating degraded has exhausted its normal single-disk failure tolerance

Replace flow (MD RAID):

• Mark failed → remove → add replacement → watch rebuild in /proc/mdstat
• Do not reboot unnecessarily during rebuild

A strong answer is:

With degraded RAID5, normal single-disk redundancy is already gone. I identify the failed member, verify backups, replace only the correct disk, and closely monitor reconstruction because another member failure during the degraded/rebuild window is high risk.

What interviewers are testing: whether you know zombies are reaped by the parent, not killed directly, and that they do not consume CPU or RAM.

Open with: "Zombies are dead children waiting for the parent to call wait—I find the PPID and fix the supervisor."

A zombie is a process that exited but whose parent has not called wait()—it consumes a PID slot, not CPU or memory.

1. ps aux | awk '$8 ~ /Z/ {print}' | wc -l — count zombies
2. ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/' — map zombie to parent PID
3. ps -p <ppid> -o pid,comm,stat — identify the parent service
4. Fix — restart or fix the parent so it reaps children; killing zombies directly does nothing

See Linux process management for parent/child lifecycle and signal basics.

A strong answer is:

"Zombies mean the parent isn't reaping exits. I find the PPID, restart or patch that supervisor, and investigate why it forked so many children—often a service manager bug or a script in a tight fork loop."

What interviewers are testing: file descriptor limits versus leaks—ulimit, systemd LimitNOFILE, and actual FD count.

Open with: "EMFILE means the process hit its FD cap—I compare /proc/pid/limits to how many descriptors it actually holds."

Linux enforces per-process and system-wide file descriptor limits.

1. Application error — Too many open files, EMFILE, accept: Too many open files
2. ulimit -n — soft limit for current shell; check the service limit under systemd
3. cat /proc/<pid>/limits | grep 'open files' — effective limit for the process
4. ls /proc/<pid>/fd | wc -l — how many FDs the process actually holds
5. ss -s — socket summary; lsof -p <pid> | wc -l if permitted
6. systemd — LimitNOFILE= in unit file; systemctl show <unit> -p LimitNOFILE
7. If the actual error is process/thread exhaustion rather than EMFILE — check TasksMax=, RLIMIT_NPROC, cgroup pids.max, and kernel.pid_max

Root causes in interviews:

• Connection leak (not closing sockets)
• Log file handles never rotated
• Thread explosion exhausting TasksMax or kernel.pid_max
• fs.inotify.max_user_watches on file-watcher heavy apps (related but different error)

See Linux file descriptors for limit mechanics.

Say: raising limits without fixing a socket or log leak is a temporary band-aid.

A strong answer is:

"I read the error, check /proc/<pid>/limits and FD count, then compare to LimitNOFILE for the service. If we're at the cap, I fix the leak or raise the limit with justification—not blindly set unlimited."

What interviewers are testing: identifying the listening PID versus confusing TIME_WAIT with a live listener.

Open with: "EADDRINUSE means something already owns the port—I use ss -tlnp to name the process before I kill anything."

Work through:

1. Error — bind: Address already in use or EADDRINUSE
2. ss -tlnp | grep :<port> — TCP listeners with PID
3. ss -ulnp | grep :<port> — UDP (no connection state)
4. lsof -i :<port> — process name if ss lacks -p permission
5. TIME_WAIT — do not confuse many TIME_WAIT sockets with an existing listening socket; first identify whether another process actually owns the listening address/port
6. systemd socket activation — systemctl status <port>.socket

Say: if it is an old instance of the same unit, stop it cleanly with systemd—not kill -9 on a guess.

A strong answer is:

"ss -tlnp shows the owning PID. If it's an old instance of the same service, I stop the unit cleanly; if it's another app, I change the port or retire the conflicting service—not kill blindly without identifying the owner."

What interviewers are testing: ephemeral port exhaustion from connection churn—app-level keep-alive before sysctl hacks.

Open with: "Many TIME_WAIT sockets or a narrow local port range can block new outbound connections—I count them before I tune the kernel."

High connection churn can exhaust ephemeral ports or fill TIME_WAIT queues.

1. ss -s — TCP summary; many timewait
2. sysctl net.ipv4.ip_local_port_range — available local port span
3. ss -tan state time-wait | wc -l — count TIME_WAIT sockets
4. Connection reuse — HTTP keep-alive, connection pooling in the app
5. Tune carefully — prefer connection pooling/keep-alive and validate whether ephemeral-port exhaustion is actually occurring; only then consider widening ip_local_port_range or changing TCP reuse settings after checking behavior on the deployed kernel

A strong answer is:

"I check TIME_WAIT counts and the local port range. The fix is usually keep-alive and pooling in the application—not only sysctl tweaks. I widen ephemeral range only after confirming legitimate churn."

What interviewers are testing: the DNS versus routing split—direct IP access works, so the break is name resolution or NSS.

Open with: "I can reach the IP but not the hostname, so I test the resolver with dig or getent and read resolv.conf and NSS—not the default gateway first."

This pattern isolates name resolution from routing — the same split as ping IP works but hostname fails.

1. Confirm IP path — curl -v --connect-timeout 5 http://<server-ip>/ or nc -vz <server-ip> <port> (ICMP can be blocked while TCP works)
2. Test DNS — dig example.com or getent hosts example.com
3. /etc/resolv.conf — nameserver IPs, search domain, stub resolver (systemd-resolved)
4. /etc/nsswitch.conf — hosts: files dns order
5. Firewall to resolver — UDP/TCP 53 blocked?
6. Application cache — stale resolver in long-running process

dig works but the application cannot resolve: the app may use a different resolver library, hard-coded nameserver, stale cache, or NSS order (files before dns with a stale /etc/hosts entry). Compare getent hosts with the app's resolver and check whether the process needs a restart after resolv.conf changed.

See temporary failure in name resolution for resolver-focused fixes.

Say: if dig works but the app fails, mention stale resolver cache or that the long-running process needs a restart after resolv.conf changed.

A strong answer is:

If the same destination works by IP but not hostname, I focus on DNS/NSS first: getent, dig, resolver configuration, and resolver reachability. If dig works but the application fails, I compare the application's resolver path and caches.

What interviewers are testing: error classification—refused means TCP reached a host that rejected the port; timeout and unreachable are different stories.

Open with: "Connection refused means the packet reached something that rejected port 22—I verify sshd is listening on the right address before I blame the network."

Connection refused means the TCP SYN reached a host that actively rejected the port—different from timeout (filtered or silent drop) or network unreachable (no route).

Classify before you fix:

• Refused — host reached; no listener or explicit reject on that port
• Timeout — packets dropped, filtered, or path failure
• No route / network unreachable — routing or local network problem

Do not require ping—ICMP is often blocked while SSH works.

See test SSH connection for layered client-side checks when the error message is unclear.

Say: do not require ping—test TCP with nc -vz or ssh -vvv because ICMP is often blocked.

A strong answer is:

"Refused means TCP reached a host that rejected the port—I verify sshd is listening on the expected address with ss -tlnp, test with nc -vz, then check firewall and SELinux. I separate refused from timeout and unreachable instead of blaming SSH when routing failed."

What interviewers are testing: routing path from ip route get, then firewalls and return-path symmetry—not jumping to application config.

Open with: "I ask the kernel which interface and gateway it would use, test the next hop, then check ACLs if the route looks correct."

Work local to remote:

1. Local IP and link — ip -br a, interface UP
2. Route to destination — ip route get <dest-ip>
3. Default gateway — ip route | grep default
4. Same-subnet works? — test gateway and remote IP with nc or ping when ICMP is allowed
5. Firewall — host nftables/iptables, network ACLs
6. Reverse path — asymmetric routing breaks some flows
7. ARP on L2 — ip neigh for next-hop on same VLAN
8. Gateway correct but remote still fails — upstream router ACL, missing return route, or NAT/stateful firewall on a middle box
9. After a network change — new VLAN, VPN default route, or policy routing (ip rule) overriding the path you expect—compare ip route get before and after the change

See default route troubleshooting and destination host unreachable when ip route get looks correct but remote subnets still fail.

A strong answer is:

"I use ip route get to see which interface and gateway the kernel picks. If the gateway is right but the remote subnet still fails, I check ACLs on routers and return-path symmetry—not only the local ip route table."

What interviewers are testing: mapping errno / client message to layer—refused, timeout, and unreachable each imply different fixes.

Open with: "I classify the error before I change the app—I use ip route get and a verbose nc or curl to see refused versus silent drop versus no route."

Each error points at a different layer—interviewers want you to name that mapping.

Commands:

ip route get <dest-ip>
nc -vz <host> <port>
curl -v --connect-timeout 5 http://<host>/

See no route to host when the kernel reports routing failure, and connection refused for refused versus unreachable errors.

A strong answer is:

"Refused means I reached a host that rejected the port—service or local firewall. Timeout means silence—often a drop en route. Unreachable means the kernel has no path. I use ip route get and a verbose curl or nc to classify before touching the app config."

What interviewers are testing: separating latency, throughput, and loss—and ruling out DNS or TLS before blaming bandwidth.

Open with: "I baseline RTT and look for where delay or retransmits start—mtr and socket retrans counts, not just ping once."

Separate latency, throughput, and loss—fixing the wrong one wastes time.

1. ping -c 20 <gateway> — RTT min/avg/max, jitter
2. mtr -rwzbc 100 <dest> or traceroute — where latency or apparent loss begins and continues through later hops
3. iperf3 (if allowed) — TCP/UDP throughput between endpoints
4. ss -ti — retransmits on established sockets (retrans: field)
5. DNS — rule out slow resolver before blaming the path
6. NIC/driver — ethtool -S, errors on ip -s link

See slow network troubleshooting.

A strong answer is:

"I baseline RTT to the gateway and the remote host, then use mtr or socket retransmit counts to see if loss or bufferbloat starts at a hop. If ping to IP is fast but the app is slow, I check DNS and TLS next—not just bandwidth."

What interviewers are testing: confirming loss on real TCP traffic, not only ICMP (routers often rate-limit ping).

Open with: "I confirm loss with sustained probes, then check TCP retransmits on live connections—not only ping percentages."

Localize loss:

1. ping -c 50 <dest> — % packet loss at each interval
2. mtr -rwzbc 200 <dest> — loss per hop (watch for ICMP rate-limit false positives)
3. ss -ti dst <ip> — TCP retransmits on live connections
4. ethtool -S <iface> | grep -i drop — NIC-level drops
5. Duplex/speed mismatch — legacy but still appears on physical links
6. Firewall shaping — policers dropping bursts

See packet loss troubleshooting.

A strong answer is:

"I confirm loss with sustained ping or mtr, then check TCP retransmits on real traffic. If only ICMP looks lossy but TCP is clean, it may be ICMP rate limiting on a router—not true path loss."

What interviewers are testing: bind address (127.0.0.1 vs 0.0.0.0) versus firewall when local nc works but remote does not.

Open with: "The daemon is up locally but remote clients fail—I check ss for loopback-only bind, then host firewall and cloud security groups."

The process is up, but it is not accepting traffic on the interface clients use.

1. ss -tlnp | grep :<port> — 127.0.0.1:<port> vs 0.0.0.0:<port> vs :::
2. Application config — bind, listen, ListenAddress (nginx, sshd, database)
3. Host firewall — nft list ruleset, firewall-cmd --list-all; see open a port on Linux when the service listens locally but remote TCP fails
4. Cloud/security-group ACL — allow inbound on the port from client subnets
5. Test locally vs remotely — nc -vz 127.0.0.1 <port> on host; nc -vz <public-ip> <port> from client

Contrast with connection refused on all paths (nothing listening) and timeout (filtered en route).

A strong answer is:

"Local nc works but remote does not—I check whether the daemon binds loopback only, then host firewall and cloud ACLs. Fixing ListenAddress or opening the right zone is the answer, not restarting the service blindly."

What interviewers are testing: path debugging—localhost works, so the app is fine; the break is VIP, NAT, health check, or TLS at the load balancer.

Open with: "I compare curl to 127.0.0.1 with curl through the VIP and verify LB health checks and return-path firewall rules."

Compare paths:

1. Direct vs proxied path — curl -v http://127.0.0.1:<port>/ on server vs curl -v http://<vip-or-lb>/ from client
2. Backend health — LB marks pool members down; health check port/path wrong
3. NAT and return path — asymmetric routing after SNAT
4. Stateful firewall — allow established return traffic; idle timeout shorter than long requests
5. TLS termination — certificate/SNI mismatch at LB; HTTP vs HTTPS backend port
6. Source IP preservation — backend allowlist missing LB subnet when using SNAT
7. MTU / PMTUD — large responses black-holed through tunnel or VPN

For host firewall rules on RHEL-family systems, the firewalld cheat sheet lists permanent rich rule and zone patterns.

A strong answer is:

"I compare curl to localhost with curl through the VIP, then verify LB health checks and firewall rules for front-to-back and return traffic. Local success means the app is fine—the break is almost always path, NAT, or LB config."

What interviewers are testing: systemd literacy—systemctl --failed, journal lines, port conflicts, SELinux, and environment differences from an interactive shell.

Open with: "I check for other failed units, read systemctl status and journalctl -u for the exact exit reason, then verify ports and SELinux before I edit the unit file."

Start with systemctl command basics—status, start, and unit dependencies—then narrow to the failing unit:

1. systemctl --failed — any failed units on the host before you zoom in on one service
2. systemctl status <unit> — exit code, last lines of log
3. journalctl -u <unit> -b --no-pager -n 50 — full startup error
4. Config syntax — nginx -t, sshd -t, app-specific --check
5. Port in use — ss -tlnp | grep :<port>
6. Process running but not listening — ps shows the binary, but ss -tlnp has no socket on the expected port (wrong bind address, startup race, or crash after fork)
7. Permissions — file ownership, SELinux (ausearch -m avc -ts recent)
8. Dependencies — systemctl list-dependencies <unit>, After= / Requires=
9. Resource limits — TasksMax, MemoryMax in unit file
10. Restart loop — systemctl status shows repeated Start/Failed; read the first error in journal, not only the last restart
11. Works manually, fails under systemd — compare environment (Environment=, WorkingDirectory=, User=, PATH) with your shell; run systemd-run or sudo -u <service-user> ... to reproduce

Say: if the same command works manually but not under systemd, compare User=, WorkingDirectory=, and Environment= to your shell.

A strong answer is:

"I run systemctl --failed, then systemctl status and journalctl -u for the unit—the error is usually in the last twenty lines. I check listening sockets, SELinux, and whether the same command works as the service user with the unit's environment."

What interviewers are testing: cron environment—wrong user, bad PATH, minimal environment, or a systemd timer that replaced cron.

Open with: "I verify crond logged an entry for that minute, confirm the crontab user and schedule, then reproduce as that user with full paths."

Cron syntax and user context trip many interviews—see the crontab command for field order and % escaping before you debug the daemon:

1. Cron ran at all? — grep CRON /var/log/cron or journalctl -u crond --since today
2. User crontab — crontab -l for the intended user (root vs app user)
3. Schedule syntax — five-field vs six-field; Percent sign — in traditional crontab command fields, an unescaped % is treated specially; escape a literal % when required
4. Environment — cron has minimal PATH; use full paths to binaries
5. Permissions — /etc/cron.d files need correct owner and chmod (no world-writable)
6. Overlap — anacron on laptops; @reboot vs time-based
7. systemd timers — job may have migrated to systemctl list-timers

Works when run manually, not from cron/timer: cron uses a minimal environment—full paths, correct user, and MAILTO or logging to catch failures. For timers, compare systemctl cat <timer> and the service unit's User=, Environment=, and WorkingDirectory= with your manual shell.

A strong answer is:

"I confirm the cron daemon logged an entry, verify the crontab user and schedule, and reproduce as that user with the same PATH. If manual works but cron does not, I fix paths and environment—or move the job to a systemd timer with explicit unit settings."

What interviewers are testing: NTP and clock sync as root cause for Kerberos, TLS, and distributed cluster failures.

Open with: "Auth broke after a time jump—I check timedatectl and chrony offset before I regenerate certificates."

Clock skew can break Kerberos and other timestamp-sensitive authentication protocols. A badly incorrect clock can also make TLS certificates appear not yet valid or expired and can disrupt distributed systems that depend on time ordering or leases.

1. timedatectl — System clock synchronized, NTP service, timezone
2. chronyc tracking or ntpq -p — offset from source
3. Hypervisor / VM — clock drift after snapshot or suspend
4. Leap seconds and manual date -s — sudden jumps break Kerberos tickets
5. Containers — inherit host clock; check node, not only pod

Fix path:

• Enable chronyd or systemd-timesyncd — the timedatectl command shows sync state and active NTP services
• On VMs: sync after resume; consider chronyc makestep once if far off

A strong answer is:

"I run timedatectl and chronyc tracking to see offset and whether NTP is active. Auth failures after reboot often trace to unsynchronized clocks—I fix NTP on the host before regenerating certificates or keytabs."

What interviewers are testing: MAC beyond chmod—AVC denials, contexts, and booleans; not permanent setenforce 0 in production.

Open with: "Permission denied but mode looks fine on SELinux hosts—I search audit logs for AVC denials and compare file context to the process domain."

On SELinux-enforcing hosts, labels and booleans block access even when chmod is wide open — start with SELinux modes and contexts when AVC lines appear in the journal.

1. ausearch -m avc -ts recent or grep avc /var/log/audit/audit.log
2. sealert -a /var/log/audit/audit.log (if setroubleshoot installed)
3. ls -Z /path/to/file — SELinux context matches policy
4. ps -eZ | grep <service> — process domain
5. Temporary diagnosis — inspect or change context only when necessary; Persistent fix — define the expected labeling with semanage fcontext, then apply it with restorecon

Also check:

• Immutable flag — lsattr /path
• ACLs — getfacl /path
• AppArmor on Debian/Ubuntu — aa-status, journal denials

A strong answer is:

"After chmod checks out, I look for AVC denials in audit logs and compare file context to what the daemon domain allows. I fix policy with the right context or boolean—not permanent setenforce 0."

What interviewers are testing: path traversal—execute bit on every parent directory, ACL masks, groups, and NFS squash—not only the final file mode.

Open with: "I run namei -l on the full path to see which directory blocks traverse, then check groups, ACLs, and SELinux."

Work through identity, POSIX permissions, ACLs, and MAC in order:

1. Who are they? — id <user> — uid, groups, SELinux login context
2. Path traversal — execute bit on every parent directory
3. Ownership and mode — ls -la, namei -l /full/path
4. ACLs — getfacl /path — masks can deny despite chmod 777
5. SELinux / AppArmor — AVC or aa denials
6. NFS root squashing — root_squash maps remote root to nobody

See Linux file permissions and ACL examples.

A strong answer is:

"I run namei -l on the full path to find which directory blocks traverse, then getfacl and SELinux denials. Shared trees often fail on missing group membership or NFS squash—not the final file mode alone."

What interviewers are testing: hang versus lockup—D state on storage, OOB console, and evidence before power cycle.

Open with: "Before reboot I try out-of-band console and check if processes are stuck in D state on a mount—storage hangs mimic CPU freezes."

Distinguish full lockup from severe I/O or memory pressure before hitting reset.

1. Magic SysRq (if enabled) — last resort, one action per write:

echo s > /proc/sysrq-trigger   # sync
echo u > /proc/sysrq-trigger   # remount filesystems read-only where supported
echo b > /proc/sysrq-trigger   # immediate reboot

b reboots immediately without a normal shutdown sequence. 2. Out-of-band — IPMI/iLO serial console—can you get a login prompt? 3. Storage — NFS or SAN hang often freezes anything touching that mount 4. OOM or D state — from serial: top if responsive; all in D → I/O wait 5. Kernel soft lockup in dmesg after reboot — BUG: soft lockup

Before reboot in production:

• Capture vmcore or at least note time for post-mortem
• Alert stakeholders; check if HA failover is safer

A strong answer is:

"I try out-of-band console first. If processes are in D state on a mount, I suspect storage not CPU. I sync and remount read-only via SysRq if I must reboot, and I schedule kernel log review after recovery—not just power cycle without evidence."

What interviewers are testing: Whether you know evidence from the previous boot is more valuable than current-boot guesses and can distinguish panic, watchdog, OOM, power loss, and hypervisor reset.

Open with: "I confirm boot time with last reboot, then read the previous boot's kernel journal for panic, OOM, or watchdog lines."

Correlate boot time with the previous boot's journals before you assume hardware failure — Linux boot, reboot, and shutdown covers clean versus unclean restarts:

1. last reboot and who -b — when the boot happened
2. journalctl -b -1 -p err — errors from the previous boot before crash
3. journalctl -k -b -1 | tail — kernel messages before reboot (panic, OOM, watchdog)
4. dmesg -T on current boot — can reveal continuing hardware/storage symptoms, but previous-boot journal, persistent pstore/kdump data, IPMI/BMC SEL, or hypervisor logs are stronger evidence for the actual reboot cause
5. Out-of-band logs — IPMI SEL, hypervisor console (watchdog reset, host panic)
6. OOM — journalctl -k -b -1 | grep -i oom — kernel kill vs clean shutdown
7. Power — datacenter event, UPS, accidental reboot

A strong answer is:

"I confirm the boot time, then read the previous boot's kernel and service journals for panic, OOM, or watchdog lines. If logs are empty, I check IPMI or the hypervisor—unclean resets often leave little in userspace journal."