| Tested on | RHEL 10.2 (Coughlan) — vm1.lab.example |
|---|---|
| Package | bash-5.2.26-6.el10.x86_64coreutils-9.5-8.el10_2.x86_64systemd-257-23.el10_2.2.x86_64jq-1.7.1-11.el10_2.2 (optional JSON mode) |
| Applies to | Linux distributions with Bash installed |
| Privilege | Normal user for most checks; reading all mount points and systemctl --failed may need elevated rights on hardened hosts |
| Scope | One-time Linux health snapshot covering host identity, CPU utilization from /proc/stat, load averages in CPU context, available memory and swap, filesystem and inode use, failed systemd units, optional critical-service list, readable report, exit codes 0/1/2, optional JSON via jq, fixture-based threshold tests, and cron scheduling. Does not rank top CPU or memory processes or replace time-series monitoring. |
| Related guides | Shell scripting course hub |
A useful server health check needs more than raw uptime, free, and df output. This script turns CPU, load, memory, swap, disk, inode, and service state into OK, WARNING, CRITICAL, ERROR, or SKIP results with a predictable exit code.
Simply printing uptime, free, and df leaves the reader to decide what counts as unhealthy. This guide measures host-level CPU utilization and load as separate signals, uses available memory instead of naive used/total math, treats swap and inodes as their own checks, and only runs the systemd checks on hosts where systemd is available.
What "System Health" Means in This Script
This script reports a point-in-time view of host health. Each check becomes OK, WARNING, CRITICAL, ERROR, or SKIP, and those results determine the overall exit status.
| Check | What it measures | What it does not measure |
|---|---|---|
| Host | Hostname, kernel, uptime | Hardware serial, cloud instance ID |
| CPU | Host-wide utilization from /proc/stat |
Per-process CPU share |
| Load | 1/5/15-minute load vs CPU count | Same thing as CPU utilization |
| Memory | MemAvailable as percent of total |
Per-process RSS (see dedicated guide) |
| Swap | Swap used as percent of total | Whether swap I/O is thrashing |
| Disk | Block use and inode use per mount | Growth rate over time |
| systemd | Failed units, optional critical list | Container health, remote dependencies |
One snapshot cannot replace graphs or alerting on trends. Use this script for cron mail, pre-deploy gates, or a quick SSH sanity check. When you need who is consuming CPU or RAM, use the top memory and CPU processes walkthrough instead of duplicating ps logic here.
Define Thresholds and Exit Codes
Put thresholds in variables or a small config file so you can tune per environment without editing every comparison.
| Variable | Default | Meaning |
|---|---|---|
CPU_WARN_PCT / CPU_CRIT_PCT |
80 / 95 | Warn or fail when host CPU utilization is at or above this percent |
MEM_WARN_PCT / MEM_CRIT_PCT |
15 / 5 | Warn or fail when available memory is at or below this percent of total |
SWAP_WARN_PCT / SWAP_CRIT_PCT |
50 / 80 | Warn or fail when swap use is at or above this percent |
LOAD_WARN_MULT / LOAD_CRIT_MULT |
1.0 / 2.0 | Warn or fail when 1-minute load is at or above this multiple of CPU count |
DISK_WARN_PCT / DISK_CRIT_PCT |
85 / 95 | Filesystem block use at or above this whole-number percent |
INODE_WARN_PCT / INODE_CRIT_PCT |
85 / 95 | Inode use at or above this whole-number percent on the same mount |
Boundary comparisons are inclusive: exactly 80% CPU utilization triggers WARNING, exactly 85% disk use triggers WARNING, and exactly 15% available memory triggers WARNING.
Exit contract (aligned with Bash exit codes practice for monitoring scripts):
| Code | Meaning |
|---|---|
0 |
All checks OK or SKIP |
1 |
At least one WARNING or CRITICAL threshold breached |
2 |
Invalid config, missing required tool, or a required check could not run |
WARNING and CRITICAL both return 1 so cron, CI jobs, and simple monitoring wrappers can treat any unhealthy result as failure. Exit 2 is reserved for configuration or execution errors. The report text shows which level fired, so a wrapper can still read the detail even when the exit code is 1.
Example config file health.conf:
CPU_WARN_PCT=75
CPU_CRIT_PCT=90
MEM_WARN_PCT=20
MEM_CRIT_PCT=10
LOAD_WARN_MULT=1.5
LOAD_CRIT_MULT=3.0
CRITICAL_SERVICES="sshd,crond"Load the file after parsing options:
if [[ -n "${CONFIG_FILE}" && ! -f "${CONFIG_FILE}" ]]; then
echo "Config file not found: ${CONFIG_FILE}" >&2
exit 2
fi
[[ -f "${CONFIG_FILE}" ]] && source "${CONFIG_FILE}"Only load configuration files you trust. source executes the file as Bash code; it is not a passive key/value parser.
Validate thresholds before any collector runs:
validate_config() {
# reject non-numeric values, out-of-range percentages,
# integer-only disk/inode thresholds, and inverted warn/critical pairs
...
}
validate_configA bad value such as MEM_WARN_PCT=abc should exit 2:
./system-health.sh -c /tmp/bad-health.confInvalid threshold configuration: MEM_WARN_PCT=abcPoint -c at a missing file and the script should refuse to run:
./system-health.sh -c /nonexistent.confConfig file not found: /nonexistent.confThe exit status is 2, which tells a wrapper the thresholds were never applied.
Collect Host and Uptime Information
Include the hostname, kernel version, and uptime so you can immediately tell which server a report came from.
Read uptime seconds from /proc/uptime and format a short human string:
collect_host() {
local hostname kernel uptime_secs days hours
hostname="$(hostname -s 2>/dev/null || hostname)"
kernel="$(uname -r)"
uptime_secs="$(awk '{print int($1)}' /proc/uptime)"
days=$((uptime_secs / 86400))
hours=$(((uptime_secs % 86400) / 3600))
printf '%s | kernel %s | up %dd %dh\n' "${hostname}" "${kernel}" "${days}" "${hours}"
}On my lab host the collector produced:
vm1 | kernel 6.12.0-211.47.1.el10_2.x86_64 | up 0d 20hHost facts are informational only; they do not change the exit code unless you add your own policy later.
Check CPU Utilization and Load Average
CPU utilization and load average are related, but they are not the same measurement. Utilization measures how busy the processors were during a sample window. Load counts runnable and uninterruptible tasks averaged over 1, 5, and 15 minutes. A healthy host can show moderate load with low utilization (many threads waiting on I/O) or high utilization with modest load (a few CPU-bound jobs). Measure both.
Host CPU utilization from /proc/stat
Read the aggregate cpu line from /proc/stat, sample twice with a one-second gap, and compute utilization from the delta:
read_cpu_jiffies() {
local cpu user nice system idle iowait irq softirq steal _rest
read -r cpu user nice system idle iowait irq softirq steal _rest < /proc/stat
local idle_all=$((idle + iowait))
local non_idle=$((user + nice + system + irq + softirq + steal))
local total=$((idle_all + non_idle))
printf '%s %s\n' "${idle_all}" "${total}"
}After two samples:
cpu_usage_pct = 100 * (total_delta - idle_delta) / total_deltaCompare the result to CPU_WARN_PCT (default 80) and CPU_CRIT_PCT (default 95). On my lab host the one-second sample reported:
utilization 23.6%With an 80% warning threshold, this sample remains OK. This is host-wide utilization, not per-process ranking.
This delta calculation is suitable for a lightweight health snapshot, but /proc/stat CPU accounting — especially iowait — has kernel-documented limitations, so do not treat the percentage as precision performance telemetry.
Load averages in CPU context
Collect CPU count and the three load figures together:
collect_cpu_load() {
local cpus load1 load5 load15
cpus="$(nproc)"
read -r load1 load5 load15 _ < /proc/loadavg
printf 'cpus=%s load=%s/%s/%s (1/5/15m)\n' "${cpus}" "${load1}" "${load5}" "${load15}"
}Threshold logic multiplies CPU count by your warning and critical multipliers, then compares only the 1-minute figure:
warn_thresh="$(awk -v c="${cpus}" -v m="${LOAD_WARN_MULT}" 'BEGIN{printf "%.2f", c*m}')"
crit_thresh="$(awk -v c="${cpus}" -v m="${LOAD_CRIT_MULT}" 'BEGIN{printf "%.2f", c*m}')"On the same quiet three-CPU VM I saw:
cpus=3 load=0.34/0.21/0.20 (1/5/15m)With defaults (1.0× warn, 2.0× crit), thresholds are 3.00 and 6.00. Both CPU utilization and load stayed OK. Checking them separately helps distinguish high CPU utilization from a high load average.
Check Available Memory and Swap
Do not score memory with (total - used) / total from the first line of free. Linux can reclaim cache; MemAvailable in /proc/meminfo (and the "available" column in free -h) estimates what new workloads can still allocate.
mem_avail_kb="$(awk '/^MemAvailable:/ {print $2}' /proc/meminfo)"
mem_total_kb="$(awk '/^MemTotal:/ {print $2}' /proc/meminfo)"
avail_pct="$(awk -v a="${mem_avail_kb}" -v t="${mem_total_kb}" \
'BEGIN{if(t>0) printf "%.1f", (a/t)*100; else print 0}')"Swap is a separate check. Swap usage shows how much configured swap currently contains pages, but high usage alone does not prove the system is under memory pressure now. Treat it as a separate signal alongside MemAvailable; diagnosing active swapping requires I/O or paging-rate metrics.
swap_total_kb="$(awk '/^SwapTotal:/ {print $2}' /proc/meminfo)"
swap_used_kb="$(awk '/^SwapTotal:|^SwapFree:/ {
if ($1=="SwapTotal:") t=$2; if ($1=="SwapFree:") f=$2
} END {print t-f}' /proc/meminfo)"Lab output for the combined memory line:
available 65.7% (5164380 KiB of 7863248 KiB); swap 8.8% usedAvailable memory is well above its 15% warning floor, while swap usage is below its 50% warning threshold, so the combined memory check remains OK. When SwapTotal is zero, report swap not configured instead of swap 0% used. If MemAvailable falls to or below MEM_CRIT_PCT (default 5%), mark memory CRITICAL even when the legacy "used" bar in free still looks tolerable.
Check Disk Space and Inodes
Full disks and exhausted inode tables fail in different ways. A small-file mail spool can hit inode limits while df -h still shows free gigabytes. Run both checks per mount.
Skip pseudo filesystems you do not want in a server report, then collect the mount list before looping. If enumeration fails, record ERROR and exit 2 instead of reporting an empty OK:
if ! mounts="$(df -P -x tmpfs -x devtmpfs -x squashfs 2>/dev/null |
awk 'NR>1 { print $6 }')"; then
set_check disk ERROR "unable to enumerate filesystems"
return 0
fiFor each remaining mount, read block percent and inode percent. Handle df failures explicitly so set -e does not terminate the script before the report is printed:
if ! use_pct="$(df -P "${mount}" 2>/dev/null |
awk 'NR==2 { gsub(/%/,"",$5); print $5 }')"; then
set_check disk ERROR "unable to read block usage for ${mount}"
return 0
fi
if ! inode_pct="$(df -Pi "${mount}" 2>/dev/null |
awk 'NR==2 { gsub(/%/,"",$5); print $5 }')"; then
set_check disk ERROR "unable to read inode usage for ${mount}"
return 0
fiValidate that both values are numeric before comparing them to thresholds. GNU df returns non-zero on failure, so a mount that disappears between enumeration and inspection should produce [ERROR] disk: ... and exit 2, not a silent abort.
On my lab host the disk summary looked like:
/: 63% used, 3% inodes; /boot: 33% used, 1% inodesEither metric crossing DISK_WARN_PCT or INODE_WARN_PCT marks that mount at WARNING; critical thresholds default to 95%.
Check Failed systemd Units
Failed units are systemd-specific. Do not use systemctl is-system-running as a gate: a degraded system returns non-zero precisely because units failed, and you would skip the check you need most.
Check that systemctl exists and that the systemd runtime directory is present before running systemd-specific checks:
systemd_available() {
command -v systemctl >/dev/null 2>&1 && [[ -d /run/systemd/system ]]
}
collect_failed_units() {
local output f
if ! systemd_available; then
echo "SKIP|systemd not available"
return
fi
if ! output="$(systemctl --failed --no-legend --no-pager 2>&1)"; then
printf 'ERROR|unable to query failed systemd units: %s\n' "${output}"
return
fi
f="$(printf '%s\n' "${output}" | awk 'NF {print $1}' | paste -sd, -)"
[[ -z "${f}" ]] && echo "OK|" || echo "FAIL|${f}"
}The systemctl command cheat sheet covers unit states; here you only need the failed list. An empty result means OK. Any unit name in that list marks the check CRITICAL. If systemctl --failed itself fails, record an ERROR result and exit 2 instead of letting set -e terminate the script without a report.
Optional critical services (-s sshd,nginx) reuse the same systemd_available() guard before calling systemctl is-active. Treat the textual state is-active prints (inactive, failed, and similar) as a policy failure, and reserve ERROR for cases where systemd could not be queried at all. A stopped but enabled web server might not appear under --failed yet still violates your policy.
./system-health.sh -s sshd[OK] critical_services: all critical units activePass a unit that is not running and the check names it in the report with exit 1.
Aggregate Overall Status
Keep per-check results in parallel arrays or an associative map, then walk them once:
set_check() {
local name="$1" level="$2" detail="$3"
CHECK_STATUS["${name}"]="${level}"
CHECK_DETAIL["${name}"]="${detail}"
case "${level}" in
ERROR)
overall_status="ERROR"
overall_exit="${EXIT_CONFIG}"
;;
CRITICAL)
if [[ "${overall_status}" != "ERROR" ]]; then
overall_status="CRITICAL"
fi
if [[ "${overall_exit}" -ne "${EXIT_CONFIG}" ]]; then
overall_exit="${EXIT_PROBLEM}"
fi
;;
WARNING)
if [[ "${overall_status}" == "OK" ]]; then
overall_status="WARNING"
fi
if [[ "${overall_exit}" -eq "${EXIT_OK}" ]]; then
overall_exit="${EXIT_PROBLEM}"
fi
;;
esac
return 0
}set_check() records each result and updates the overall status. The script returns the final exit code only after all available checks have run.
Precedence is ERROR > CRITICAL > WARNING > OK/SKIP. An ERROR sets overall status to ERROR and exit 2. Later threshold breaches do not downgrade that. SKIP does not change the exit code (for example when systemd is absent).
Run collectors, then evaluators, in a fixed order so the text report is stable: host, CPU, load, memory, disk, failed units, critical services.
Produce a Readable Report
Human output is the default. Print overall status first, then one line per check with the level in brackets:
./system-health.sh=== System health snapshot ===
Overall: OK (exit 0)
[OK] host: vm1 | kernel 6.12.0-211.47.1.el10_2.x86_64 | up 0d 20h
[OK] cpu: utilization 23.6%
[OK] load: cpus=3 load=0.34/0.21/0.20 (1/5/15m)
[OK] memory: available 56.9% (4474036 KiB of 7863248 KiB); swap 8.8% used
[OK] disk: /: 63% used, 3% inodes; /boot: 33% used, 1% inodes
[OK] systemd_failed: no failed units
[SKIP] critical_services: not configuredWhen a threshold fires, the bracketed level changes to WARNING or CRITICAL and the detail line states which threshold was crossed. A scheduler or monitoring hook can grep for CRITICAL or react to the non-zero exit status.
Optional Machine-Readable JSON Output
For log aggregators or CI gates, add -j and build JSON with one jq -n call. Pass each status and detail string through --arg so jq handles escaping. That pattern matches the safe construction style in Parse JSON in Bash with jq.
./system-health.sh -j{
"overall": "OK",
"exit_code": 0,
"checks": {
"host": {
"status": "OK",
"detail": "vm1 | kernel 6.12.0-211.47.1.el10_2.x86_64 | up 0d 20h"
},
"cpu": {
"status": "OK",
"detail": "utilization 32.6%"
},
"load": {
"status": "OK",
"detail": "cpus=3 load=0.31/0.20/0.20 (1/5/15m)"
},
"memory": {
"status": "OK",
"detail": "available 56.9% (4473028 KiB of 7863248 KiB); swap 8.8% used"
},
"disk": {
"status": "OK",
"detail": "/: 63% used, 3% inodes; /boot: 33% used, 1% inodes"
},
"systemd_failed": {
"status": "OK",
"detail": "no failed units"
},
"critical_services": {
"status": "SKIP",
"detail": "not configured"
}
}
}If jq is not installed, print a clear error and exit 2. Parsers should read exit_code from JSON and still verify $? after the script exits.
Test Threshold Logic with Fixtures
Do not burn CPU or fill disks to test warnings. Split collectors (read real /proc) from evaluators (compare numbers to thresholds). Unit-test the evaluators by passing fake pipe-delimited fixture strings.
Create test-thresholds.sh that sources the evaluator functions and asserts expected levels:
./test-thresholds.shPASS cpu -> OK
PASS cpu -> WARNING
PASS cpu -> WARNING
PASS cpu -> CRITICAL
PASS load -> OK
PASS load -> OK
PASS load -> WARNING
PASS load -> CRITICAL
PASS memory -> OK
PASS memory -> WARNING
PASS memory -> WARNING
PASS memory -> CRITICAL
PASS memory -> CRITICAL
---
Passed: 13 Failed: 0Thirteen passing assertions confirm CPU, load, and memory math without touching live resources, including exact-boundary cases such as 80% CPU and 4.0 load on four CPUs. Add disk and systemd evaluators the same way when you extend the script.
Schedule the Report
Install the script on each server, then call it from cron or a systemd timer. Scheduled jobs need absolute paths, a minimal PATH, and log redirection; see Schedule Bash scripts for PATH, flock, and timer units.
Example user crontab line (daily at 07:00, append to a log in your home directory):
0 7 * * * /usr/local/bin/system-health.sh >>"$HOME/system-health.log" 2>&1A normal user cannot write to /var/log without a pre-created path and permissions. Point the redirect somewhere you own, or use a root crontab with a dedicated log file. External monitoring can react to the non-zero exit status without mailing from cron.
Complete Modular Script
The full script combines option parsing, config load, threshold validation, collectors, evaluators, text and JSON reporters, and a final exit "${overall_exit}". Save it as system-health.sh, mark executable, and place thresholds in a config file you trust on production hosts.
#!/usr/bin/env bash
set -euo pipefail
CPU_WARN_PCT=80
CPU_CRIT_PCT=95
MEM_WARN_PCT=15
MEM_CRIT_PCT=5
SWAP_WARN_PCT=50
SWAP_CRIT_PCT=80
LOAD_WARN_MULT=1.0
LOAD_CRIT_MULT=2.0
DISK_WARN_PCT=85
DISK_CRIT_PCT=95
INODE_WARN_PCT=85
INODE_CRIT_PCT=95
CRITICAL_SERVICES=""
OUTPUT_FORMAT="text"
CONFIG_FILE=""
EXIT_OK=0
EXIT_PROBLEM=1
EXIT_CONFIG=2
overall_status="OK"
overall_exit="${EXIT_OK}"
declare -A CHECK_STATUS CHECK_DETAIL
usage() { echo "Usage: $0 [-c FILE] [-j] [-s units] [-h]"; }
while getopts ":c:js:h" opt; do
case "${opt}" in
c) CONFIG_FILE="${OPTARG}" ;;
j) OUTPUT_FORMAT="json" ;;
s) CRITICAL_SERVICES="${OPTARG}" ;;
h) usage; exit 0 ;;
\?) echo "Invalid option: -${OPTARG}" >&2; exit "${EXIT_CONFIG}" ;;
:) echo "Option -${OPTARG} requires an argument." >&2; exit "${EXIT_CONFIG}" ;;
esac
done
if [[ -n "${CONFIG_FILE}" ]]; then
[[ -f "${CONFIG_FILE}" ]] || { echo "Config file not found: ${CONFIG_FILE}" >&2; exit "${EXIT_CONFIG}"; }
# shellcheck source=/dev/null
source "${CONFIG_FILE}"
fi
validate_config() {
local key val
for key in CPU_WARN_PCT CPU_CRIT_PCT MEM_WARN_PCT MEM_CRIT_PCT SWAP_WARN_PCT SWAP_CRIT_PCT; do
val="${!key}"
[[ "${val}" =~ ^[0-9]+([.][0-9]+)?$ ]] || { echo "Invalid threshold configuration: ${key}=${val}" >&2; exit "${EXIT_CONFIG}"; }
awk -v v="${val}" 'BEGIN{exit !(v>=0 && v<=100)}' || { echo "Invalid threshold configuration: ${key}=${val}" >&2; exit "${EXIT_CONFIG}"; }
done
for key in DISK_WARN_PCT DISK_CRIT_PCT INODE_WARN_PCT INODE_CRIT_PCT; do
val="${!key}"
[[ "${val}" =~ ^[0-9]+$ ]] || { echo "Invalid threshold configuration: ${key}=${val}" >&2; exit "${EXIT_CONFIG}"; }
awk -v v="${val}" 'BEGIN{exit !(v>=0 && v<=100)}' || { echo "Invalid threshold configuration: ${key}=${val}" >&2; exit "${EXIT_CONFIG}"; }
done
for key in LOAD_WARN_MULT LOAD_CRIT_MULT; do
val="${!key}"
[[ "${val}" =~ ^[0-9]+([.][0-9]+)?$ ]] || { echo "Invalid threshold configuration: ${key}=${val}" >&2; exit "${EXIT_CONFIG}"; }
awk -v v="${val}" 'BEGIN{exit !(v>0)}' || { echo "Invalid threshold configuration: ${key}=${val}" >&2; exit "${EXIT_CONFIG}"; }
done
awk -v w="${MEM_WARN_PCT}" -v c="${MEM_CRIT_PCT}" 'BEGIN{exit !(w>c)}' || { echo "Invalid threshold configuration: MEM_WARN_PCT must exceed MEM_CRIT_PCT" >&2; exit "${EXIT_CONFIG}"; }
awk -v w="${SWAP_WARN_PCT}" -v c="${SWAP_CRIT_PCT}" 'BEGIN{exit !(c>w)}' || { echo "Invalid threshold configuration: SWAP_CRIT_PCT must exceed SWAP_WARN_PCT" >&2; exit "${EXIT_CONFIG}"; }
awk -v w="${CPU_WARN_PCT}" -v c="${CPU_CRIT_PCT}" 'BEGIN{exit !(c>w)}' || { echo "Invalid threshold configuration: CPU_CRIT_PCT must exceed CPU_WARN_PCT" >&2; exit "${EXIT_CONFIG}"; }
awk -v w="${DISK_WARN_PCT}" -v c="${DISK_CRIT_PCT}" 'BEGIN{exit !(c>w)}' || { echo "Invalid threshold configuration: DISK_CRIT_PCT must exceed DISK_WARN_PCT" >&2; exit "${EXIT_CONFIG}"; }
awk -v w="${INODE_WARN_PCT}" -v c="${INODE_CRIT_PCT}" 'BEGIN{exit !(c>w)}' || { echo "Invalid threshold configuration: INODE_CRIT_PCT must exceed INODE_WARN_PCT" >&2; exit "${EXIT_CONFIG}"; }
awk -v w="${LOAD_WARN_MULT}" -v c="${LOAD_CRIT_MULT}" 'BEGIN{exit !(c>w)}' || { echo "Invalid threshold configuration: LOAD_CRIT_MULT must exceed LOAD_WARN_MULT" >&2; exit "${EXIT_CONFIG}"; }
}
validate_config
set_check() {
local name="$1" level="$2" detail="$3"
CHECK_STATUS["${name}"]="${level}"
CHECK_DETAIL["${name}"]="${detail}"
case "${level}" in
ERROR)
overall_status="ERROR"
overall_exit="${EXIT_CONFIG}"
;;
CRITICAL)
if [[ "${overall_status}" != "ERROR" ]]; then
overall_status="CRITICAL"
fi
if [[ "${overall_exit}" -ne "${EXIT_CONFIG}" ]]; then
overall_exit="${EXIT_PROBLEM}"
fi
;;
WARNING)
if [[ "${overall_status}" == "OK" ]]; then
overall_status="WARNING"
fi
if [[ "${overall_exit}" -eq "${EXIT_OK}" ]]; then
overall_exit="${EXIT_PROBLEM}"
fi
;;
esac
return 0
}
systemd_available() {
command -v systemctl >/dev/null 2>&1 && [[ -d /run/systemd/system ]]
}
read_cpu_jiffies() {
local cpu user nice system idle iowait irq softirq steal _rest
read -r cpu user nice system idle iowait irq softirq steal _rest < /proc/stat
local idle_all=$((idle + iowait))
local non_idle=$((user + nice + system + irq + softirq + steal))
local total=$((idle_all + non_idle))
printf '%s %s\n' "${idle_all}" "${total}"
}
collect_cpu_util() {
local idle1 total1 idle2 total2 idle_d total_d
read -r idle1 total1 < <(read_cpu_jiffies)
sleep 1
read -r idle2 total2 < <(read_cpu_jiffies)
idle_d=$((idle2 - idle1))
total_d=$((total2 - total1))
if [[ "${total_d}" -le 0 ]]; then
echo "CONFIG_ERROR|/proc/stat delta invalid"
return 0
fi
awk -v id="${idle_d}" -v td="${total_d}" 'BEGIN{printf "%.1f", (100*(td-id)/td)}'
}
collect_cpu_load() {
local cpus load1 load5 load15
cpus="$(nproc)"
read -r load1 load5 load15 _ < /proc/loadavg
printf '%s|%s|%s|%s' "${cpus}" "${load1}" "${load5}" "${load15}"
}
collect_host() {
local h k u d hr
h="$(hostname -s 2>/dev/null || hostname)"
k="$(uname -r)"
u="$(awk '{print int($1)}' /proc/uptime)"
d=$((u / 86400)); hr=$(((u % 86400) / 3600))
printf '%s|%s|%sd %sh' "${h}" "${k}" "${d}" "${hr}"
}
collect_memory() {
awk '/^MemAvailable:|^MemTotal:|^SwapTotal:|^SwapFree:/ {
if ($1=="MemAvailable:") a=$2; if ($1=="MemTotal:") t=$2
if ($1=="SwapTotal:") st=$2; if ($1=="SwapFree:") sf=$2
} END {printf "%s|%s|%s|%s", a,t,st-sf,st}' /proc/meminfo
}
eval_host() {
IFS='|' read -r h k up <<< "$1"
set_check host OK "${h} | kernel ${k} | up ${up}"
}
eval_cpu() {
local util="$1"
if [[ "${util}" == CONFIG_ERROR* ]]; then
set_check cpu ERROR "${util#*|}"
return 0
fi
local lvl=OK det="utilization ${util}%"
if awk -v u="${util}" -v c="${CPU_CRIT_PCT}" 'BEGIN{exit !(u>=c)}'; then
lvl=CRITICAL
det+=" — at or above ${CPU_CRIT_PCT}%"
elif awk -v u="${util}" -v w="${CPU_WARN_PCT}" 'BEGIN{exit !(u>=w)}'; then
lvl=WARNING
det+=" — at or above ${CPU_WARN_PCT}%"
fi
set_check cpu "${lvl}" "${det}"
}
eval_load() {
IFS='|' read -r c l1 l5 l15 <<< "$1"
local w cth lvl=OK det="cpus=${c} load=${l1}/${l5}/${l15} (1/5/15m)"
w="$(awk -v c="${c}" -v m="${LOAD_WARN_MULT}" 'BEGIN{printf "%.2f",c*m}')"
cth="$(awk -v c="${c}" -v m="${LOAD_CRIT_MULT}" 'BEGIN{printf "%.2f",c*m}')"
if awk -v l="${l1}" -v t="${cth}" 'BEGIN{exit !(l>=t)}'; then
lvl=CRITICAL
det+=" — 1m at or above ${cth}"
elif awk -v l="${l1}" -v t="${w}" 'BEGIN{exit !(l>=t)}'; then
lvl=WARNING
det+=" — 1m at or above ${w}"
fi
set_check load "${lvl}" "${det}"
}
eval_memory() {
IFS='|' read -r a t su st <<< "$1"
local ap sp=0 ml=OK sl=OK comb swap_detail
ap="$(awk -v a="${a}" -v t="${t}" 'BEGIN{printf "%.1f",(a/t)*100}')"
if [[ "${st}" -gt 0 ]]; then
sp="$(awk -v u="${su}" -v t="${st}" 'BEGIN{printf "%.1f",(u/t)*100}')"
swap_detail="swap ${sp}% used"
if awk -v p="${sp}" -v c="${SWAP_CRIT_PCT}" 'BEGIN{exit !(p>=c)}'; then
sl=CRITICAL
elif awk -v p="${sp}" -v w="${SWAP_WARN_PCT}" 'BEGIN{exit !(p>=w)}'; then
sl=WARNING
fi
else
sl=SKIP
swap_detail="swap not configured"
fi
if awk -v p="${ap}" -v c="${MEM_CRIT_PCT}" 'BEGIN{exit !(p<=c)}'; then
ml=CRITICAL
elif awk -v p="${ap}" -v w="${MEM_WARN_PCT}" 'BEGIN{exit !(p<=w)}'; then
ml=WARNING
fi
comb="${ml}"
if [[ "${sl}" == CRITICAL ]]; then
comb=CRITICAL
elif [[ "${sl}" == WARNING && "${comb}" == OK ]]; then
comb=WARNING
fi
set_check memory "${comb}" "available ${ap}% (${a} KiB of ${t} KiB); ${swap_detail}"
}
eval_disk_all() {
local worst=OK details="" m u i lvl note mounts
if ! mounts="$(df -P -x tmpfs -x devtmpfs -x squashfs 2>/dev/null |
awk 'NR>1 { print $6 }')"; then
set_check disk ERROR "unable to enumerate filesystems"
return 0
fi
if [[ -z "${mounts}" ]]; then
set_check disk ERROR "no filesystems found to check"
return 0
fi
while IFS= read -r m; do
if [[ -z "${m}" ]]; then
continue
fi
if ! u="$(df -P "${m}" 2>/dev/null | awk 'NR==2 { gsub(/%/,"",$5); print $5 }')"; then
set_check disk ERROR "unable to read block usage for ${m}"
return 0
fi
if ! i="$(df -Pi "${m}" 2>/dev/null | awk 'NR==2 { gsub(/%/,"",$5); print $5 }')"; then
set_check disk ERROR "unable to read inode usage for ${m}"
return 0
fi
if [[ -z "${u}" || ! "${u}" =~ ^[0-9]+$ ]]; then
set_check disk ERROR "invalid block usage value for ${m}"
return 0
fi
if [[ -z "${i}" || ! "${i}" =~ ^[0-9]+$ ]]; then
set_check disk ERROR "invalid inode usage value for ${m}"
return 0
fi
lvl=OK
note="${m}: ${u}% used, ${i}% inodes"
if [[ "${u}" -ge "${DISK_CRIT_PCT}" || "${i}" -ge "${INODE_CRIT_PCT}" ]]; then
lvl=CRITICAL
note+=" — critical"
elif [[ "${u}" -ge "${DISK_WARN_PCT}" || "${i}" -ge "${INODE_WARN_PCT}" ]]; then
lvl=WARNING
note+=" — warning"
fi
if [[ "${lvl}" == CRITICAL ]]; then
worst=CRITICAL
elif [[ "${lvl}" == WARNING && "${worst}" == OK ]]; then
worst=WARNING
fi
details+="${note}; "
done <<< "${mounts}"
set_check disk "${worst}" "${details%; }"
}
collect_failed_units() {
if ! systemd_available; then
echo "SKIP|systemd not available"
return 0
fi
local output f
if ! output="$(systemctl --failed --no-legend --no-pager 2>&1)"; then
output="${output//$'\n'/ }"
printf 'ERROR|unable to query failed systemd units: %s\n' "${output}"
return 0
fi
f="$(printf '%s\n' "${output}" | awk 'NF {print $1}' | paste -sd, -)"
if [[ -z "${f}" ]]; then
echo "OK|"
else
echo "FAIL|${f}"
fi
}
eval_failed_units() {
IFS='|' read -r st det <<< "$1"
case "${st}" in
SKIP) set_check systemd_failed SKIP "${det}" ;;
OK) set_check systemd_failed OK "no failed units" ;;
FAIL) set_check systemd_failed CRITICAL "failed: ${det}" ;;
ERROR) set_check systemd_failed ERROR "${det}" ;;
esac
}
collect_critical_services() {
if [[ -z "${CRITICAL_SERVICES}" ]]; then
echo "SKIP|"
return 0
fi
if ! systemd_available; then
echo "SKIP|systemd not available"
return 0
fi
local u output bad=""
IFS=',' read -ra units <<< "${CRITICAL_SERVICES}"
for u in "${units[@]}"; do
u="${u// /}"
if [[ -z "${u}" ]]; then
continue
fi
if output="$(systemctl is-active "${u}" 2>&1)"; then
continue
fi
output="${output//$'\n'/ }"
case "${output}" in
inactive|failed|activating|deactivating|maintenance|reloading|refreshing)
bad+="${u}:${output},"
;;
*)
printf 'ERROR|unable to query unit %s: %s\n' "${u}" "${output}"
return 0
;;
esac
done
bad="${bad%,}"
if [[ -z "${bad}" ]]; then
echo "OK|"
else
echo "FAIL|${bad}"
fi
}
eval_critical_services() {
IFS='|' read -r st det <<< "$1"
if [[ -z "${det}" ]]; then
det="not configured"
fi
case "${st}" in
SKIP) set_check critical_services SKIP "${det}" ;;
OK) set_check critical_services OK "all critical units active" ;;
FAIL) set_check critical_services CRITICAL "inactive: ${det}" ;;
ERROR) set_check critical_services ERROR "${det}" ;;
esac
}
print_text_report() {
echo "=== System health snapshot ==="
echo "Overall: ${overall_status} (exit ${overall_exit})"
echo
for n in host cpu load memory disk systemd_failed critical_services; do
printf '[%s] %s: %s\n' "${CHECK_STATUS[$n]:-SKIP}" "${n}" "${CHECK_DETAIL[$n]:-}"
done
}
print_json_report() {
command -v jq >/dev/null || { echo "jq is required for JSON output" >&2; exit "${EXIT_CONFIG}"; }
jq -n \
--arg overall "${overall_status}" \
--argjson exit_code "${overall_exit}" \
--arg hs "${CHECK_STATUS[host]}" --arg hd "${CHECK_DETAIL[host]}" \
--arg cs "${CHECK_STATUS[cpu]}" --arg cd "${CHECK_DETAIL[cpu]}" \
--arg ls "${CHECK_STATUS[load]}" --arg ld "${CHECK_DETAIL[load]}" \
--arg ms "${CHECK_STATUS[memory]}" --arg md "${CHECK_DETAIL[memory]}" \
--arg ds "${CHECK_STATUS[disk]}" --arg dd "${CHECK_DETAIL[disk]}" \
--arg ss "${CHECK_STATUS[systemd_failed]}" --arg sd "${CHECK_DETAIL[systemd_failed]}" \
--arg ks "${CHECK_STATUS[critical_services]}" --arg kd "${CHECK_DETAIL[critical_services]}" \
'{overall:$overall, exit_code:$exit_code, checks:{
host:{status:$hs,detail:$hd}, cpu:{status:$cs,detail:$cd},
load:{status:$ls,detail:$ld}, memory:{status:$ms,detail:$md},
disk:{status:$ds,detail:$dd}, systemd_failed:{status:$ss,detail:$sd},
critical_services:{status:$ks,detail:$kd}}}'
}
eval_host "$(collect_host)"
eval_cpu "$(collect_cpu_util)"
eval_load "$(collect_cpu_load)"
eval_memory "$(collect_memory)"
eval_disk_all
eval_failed_units "$(collect_failed_units)"
eval_critical_services "$(collect_critical_services)"
if [[ "${OUTPUT_FORMAT}" == json ]]; then
print_json_report
else
print_text_report
fi
exit "${overall_exit}"The complete script above was tested on the RHEL 10.2 host listed at the beginning of this article. Adjust thresholds in health.conf rather than editing the script body.
What This Script Cannot Replace
A snapshot script complements monitoring; it does not replace it.
| Gap | Use instead |
|---|---|
| CPU and RAM per process | Dedicated process-ranking script (see body above) |
| Historical trends, percentiles, SLOs | Prometheus, Datadog, Zabbix, or cloud vendor metrics |
| Log-based errors | Centralized logging and alert rules on log volume |
| Network, DNS, TLS expiry | Dedicated synthetic checks |
| Container/pod health | Orchestrator probes and CNI-aware tooling |
Treat exit 1 as "investigate now" on that host, not as a root-cause diagnosis. The report tells you which check failed, but not the root cause.
Summary
The script produces a point-in-time health snapshot from /proc, df, and systemd: host-wide CPU utilization, load averages, available memory, swap use, filesystem and inode usage, and failed units. Each line is scored against configurable thresholds.
CPU utilization and load average are separate checks with separate meanings. Memory decisions use MemAvailable, swap is scored separately, and inode exhaustion is checked independently of block use. The script exits 0 when healthy, 1 when a threshold fires, and 2 when configuration or required tools are wrong.
Fixture tests let you validate threshold math without stressing the machine. Schedule the same script from cron or a systemd timer and optionally emit JSON with jq for downstream automation. For historical trends and alerting, use this script alongside a proper monitoring system rather than trying to add those features to a one-time Bash health check.
References
- proc_stat(5) — CPU counter fields in
/proc/stat - proc_meminfo(5) —
MemAvailable, swap, and related memory fields - The /proc filesystem — kernel documentation for
/proc/loadavg,/proc/uptime, and other procfs files used here - systemctl(1) —
--failed,is-active, and unit inspection - df invocation (GNU Coreutils) — POSIX
-Poutput and inode reporting with-i - jq manual — safe JSON construction with
jq -n
Frequently Asked Questions
Is load average the same as CPU usage?
No. Load average counts runnable and uninterruptible tasks over time. CPU utilization measures how busy processors were during a sample window. This script checks both separately: utilization from /proc/stat deltas and load from /proc/loadavg compared with thresholds based on the CPU count.
Which memory value should I monitor?
Use MemAvailable from /proc/meminfo or the "available" column from free. It reflects memory the kernel could give new workloads without swapping. Simple used / total from free ignores reclaimable cache and often looks worse than reality.
How do I detect failed systemd units?
Run systemctl --failed when systemctl is available and /run/systemd/system exists. Do not gate the check on systemctl is-system-running, because a degraded system can return non-zero for that command while failed units still need reporting. Optionally verify named critical units with systemctl is-active even when they do not appear in the failed list.
Should a health check return non-zero?
Yes, when you want schedulers or monitoring hooks to act automatically. Return 0 for healthy, 1 for threshold breaches, and 2 for script or configuration errors. Keep warning and critical distinct in the report text even if both use exit 1.
Is this a replacement for node_exporter or monitoring?
No. This script is a single-point snapshot suitable for cron mail, quick SSH checks, or a simple gate. Time-series metrics, dashboards, and alert routing need a monitoring agent or service such as Prometheus node_exporter, Datadog, or your cloud provider's metrics platform.
