Linux Text Processing with grep, sed and awk

grep, sed, and awk are the core Linux text-processing utilities used in Bash pipelines. This guide uses one access log throughout so you can see where each command fits. cut, sort, and uniq appear as supporting tools when a pipeline needs simple field extraction or frequency counts.

The sample log is small enough to read in a terminal. These tools work well with large text files because commands such as grep, sed, cut, and many awk programs process input incrementally. Operations such as sort need to consider the full input and may use memory and temporary disk space for large files. awk programs that accumulate large associative arrays can also grow with the number of distinct keys.

For Linux text processing:

• grep 'error' file → find matching lines
• sed 's/old/new/g' file → replace text
• awk '{print $1}' file → extract or process fields

Use grep when you need matching lines, sed when you need to modify text, and awk when you need fields, conditions, calculations, or aggregation.

Which Text Processing Tool Should You Use?

Line-oriented Unix tools compose into Bash pipelines. The right command depends on whether you need to match lines, modify text, extract fields, or aggregate values:

Create the Sample Access Log

Create a lab directory and a combined log file you will reuse in every section:

mkdir -p /tmp/text-lab

Paste the sample access lines into the log file:

cat > /tmp/text-lab/access.log << 'EOF'
192.168.1.10 - - [11/Sep/2026:10:00:01 +0000] "GET /api/health HTTP/1.1" 200 128 "-" "curl/8.5"
10.0.0.5 - - [11/Sep/2026:10:00:02 +0000] "GET /index.html HTTP/1.1" 200 4521 "-" "Mozilla/5.0"
192.168.1.10 - - [11/Sep/2026:10:00:03 +0000] "POST /api/login HTTP/1.1" 401 89 "-" "curl/8.5"
10.0.0.8 - - [11/Sep/2026:10:00:04 +0000] "GET /static/app.js HTTP/1.1" 200 98340 "-" "Mozilla/5.0"
203.0.113.44 - - [11/Sep/2026:10:00:05 +0000] "GET /admin HTTP/1.1" 403 512 "-" "Mozilla/5.0"
10.0.0.5 - - [11/Sep/2026:10:00:06 +0000] "GET /api/health HTTP/1.1" 200 128 "-" "Mozilla/5.0"
192.168.1.10 - - [11/Sep/2026:10:00:07 +0000] "GET /missing HTTP/1.1" 404 162 "-" "curl/8.5"
203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "badbot/1.0"
10.0.0.8 - - [11/Sep/2026:10:00:09 +0000] "GET /index.html HTTP/1.1" 200 4521 "-" "Mozilla/5.0"
203.0.113.44 - - [11/Sep/2026:10:00:10 +0000] "GET /api/data HTTP/1.1" 502 0 "-" "badbot/1.0"
10.0.0.5 - - [11/Sep/2026:10:00:11 +0000] "GET /pricing HTTP/1.1" 200 2100 "-" "Mozilla/5.0"
192.168.1.10 - - [11/Sep/2026:10:00:12 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "curl/8.5"
10.0.0.8 - - [11/Sep/2026:10:00:13 +0000] "GET /static/app.js HTTP/1.1" 200 98340 "-" "Mozilla/5.0"
EOF

Point a variable at the file so later commands stay short:

LOG=/tmp/text-lab/access.log

With awk's default field separator, runs of whitespace separate fields. In this controlled log, the client IP is $1, request path $7, HTTP status $9, and byte count $10. These field numbers match the sample used here. For custom log formats or fields that can contain spaces, use the documented log format rather than assuming the same positions.

Filter Lines with grep

grep prints lines that match a pattern. Like sed and awk, it can read a file argument or standard input from the previous command in a pipeline. Prefer grep pattern file when the file is already on disk; use stdin when the data comes from another command. Filtering journalctl service output is a common example (journalctl -u nginx | grep 'error').

Start with a plain substring search on the sample log:

grep '/api/data' "$LOG"

203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "badbot/1.0"
203.0.113.44 - - [11/Sep/2026:10:00:10 +0000] "GET /api/data HTTP/1.1" 502 0 "-" "badbot/1.0"
192.168.1.10 - - [11/Sep/2026:10:00:12 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "curl/8.5"

A few everyday options cover most log-filtering jobs:

grep -i 'badbot' "$LOG"

203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "badbot/1.0"
203.0.113.44 - - [11/Sep/2026:10:00:10 +0000] "GET /api/data HTTP/1.1" 502 0 "-" "badbot/1.0"

-i ignores case. -v inverts the match and prints lines that do not contain the pattern:

grep -v '/api/' "$LOG"

10.0.0.5 - - [11/Sep/2026:10:00:02 +0000] "GET /index.html HTTP/1.1" 200 4521 "-" "Mozilla/5.0"
10.0.0.8 - - [11/Sep/2026:10:00:04 +0000] "GET /static/app.js HTTP/1.1" 200 98340 "-" "Mozilla/5.0"
203.0.113.44 - - [11/Sep/2026:10:00:05 +0000] "GET /admin HTTP/1.1" 403 512 "-" "Mozilla/5.0"
192.168.1.10 - - [11/Sep/2026:10:00:07 +0000] "GET /missing HTTP/1.1" 404 162 "-" "curl/8.5"
10.0.0.8 - - [11/Sep/2026:10:00:09 +0000] "GET /index.html HTTP/1.1" 200 4521 "-" "Mozilla/5.0"
10.0.0.5 - - [11/Sep/2026:10:00:11 +0000] "GET /pricing HTTP/1.1" 200 2100 "-" "Mozilla/5.0"
10.0.0.8 - - [11/Sep/2026:10:00:13 +0000] "GET /static/app.js HTTP/1.1" 200 98340 "-" "Mozilla/5.0"

-n prefixes each match with its line number, and -c prints only the count:

grep -n ' 500 ' "$LOG"

8:203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "badbot/1.0"
12:192.168.1.10 - - [11/Sep/2026:10:00:12 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "curl/8.5"

To print only the match count instead of the lines:

grep -c ' 500 ' "$LOG"

2

For an exact literal string, use -F so regex metacharacters are not interpreted:

grep -F '/api/data' "$LOG"

-F treats /api/data as a literal string rather than a regular expression. For server errors, anchor the status code after the quoted request so byte counts elsewhere on the line do not look like status codes:

grep -E 'HTTP/1\.1" 5[0-9]{2} ' "$LOG"

203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "badbot/1.0"
203.0.113.44 - - [11/Sep/2026:10:00:10 +0000] "GET /api/data HTTP/1.1" 502 0 "-" "badbot/1.0"
192.168.1.10 - - [11/Sep/2026:10:00:12 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "curl/8.5"

Basic regular expressions used by grep, sed, and awk

grep, sed, and awk all use regular expressions, although their default regex modes differ. Basic grep and sed use basic regular expressions (BRE) by default; grep -E, GNU sed -E, and awk use extended regular expressions (ERE). The examples use grep -E when extended syntax such as 5[0-9]{2} or error|warning makes the expression easier to read.

In HTTP/1\.1" 5[0-9]{2}, \. matches the literal dot in HTTP/1.1. The status-code pattern 5[0-9]{2} matches a literal 5 followed by exactly two digits, covering 500 through 599. See Bash regex for a fuller pattern reference.

The grep command cheat sheet covers more options. Use -F when you want a fixed substring, and extended regex (-E) when the match depends on field position or character classes.

Transform Text with sed

sed applies editing commands to each input line. Use s/pattern/replacement/ for substitution:

sed 's/badbot/unknown-bot/g' "$LOG" | grep unknown-bot

203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "unknown-bot/1.0"
203.0.113.44 - - [11/Sep/2026:10:00:10 +0000] "GET /api/data HTTP/1.1" 502 0 "-" "unknown-bot/1.0"

The s command means substitute. The pattern is badbot, the replacement is unknown-bot, and g replaces every occurrence on each line. Without g, only the first match on each line changes.

Delete lines that match a pattern:

sed '/badbot/d' "$LOG" | wc -l

11

The original log has 13 lines; two badbot lines were removed.

Print only selected lines with -n and the p command:

sed -n '/ 500 /p' "$LOG"

203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "badbot/1.0"
192.168.1.10 - - [11/Sep/2026:10:00:12 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "curl/8.5"

sed can also select by line number or range. The address 1,5 means lines 1 through 5:

sed -n '1,5p' "$LOG"

192.168.1.10 - - [11/Sep/2026:10:00:01 +0000] "GET /api/health HTTP/1.1" 200 128 "-" "curl/8.5"
10.0.0.5 - - [11/Sep/2026:10:00:02 +0000] "GET /index.html HTTP/1.1" 200 4521 "-" "Mozilla/5.0"
192.168.1.10 - - [11/Sep/2026:10:00:03 +0000] "POST /api/login HTTP/1.1" 401 89 "-" "curl/8.5"
10.0.0.8 - - [11/Sep/2026:10:00:04 +0000] "GET /static/app.js HTTP/1.1" 200 98340 "-" "Mozilla/5.0"
203.0.113.44 - - [11/Sep/2026:10:00:05 +0000] "GET /admin HTTP/1.1" 403 512 "-" "Mozilla/5.0"

GNU sed -i writes the edited result back to the input file. -i alone creates no backup; use -i.bak to keep the original as filename.bak:

cp "$LOG" /tmp/text-lab/access-copy.log

Apply the substitution directly to the copy:

sed -i 's/badbot/unknown-bot/' /tmp/text-lab/access-copy.log

BSD sed (macOS) requires an explicit backup extension: sed -i '' 's/.../' or sed -i.bak 's/.../'. This article uses GNU sed on Linux.

Process Fields with awk

awk treats each line as a record and splits it into fields. An awk program commonly has this form:

awk 'pattern { action }' file

An awk rule has the form pattern { action }. If you omit the pattern, the action runs for every input record. If you provide a pattern, awk runs the action only for records that match it. BEGIN runs before input is processed, while END runs after all input has been read.

The core field model:

Print the client IP from each line:

awk '{print $1}' "$LOG" | head -3

192.168.1.10
10.0.0.5
192.168.1.10

Print the request path and HTTP status together:

awk '{print $7, $9}' "$LOG" | head -3

/api/health 200
/index.html 200
/api/login 401

NR is useful when you want to label or limit output:

awk 'NR <= 3 {print NR, $1}' "$LOG"

1 192.168.1.10
2 10.0.0.5
3 192.168.1.10

When the delimiter is not whitespace, set it with -F:

awk -F ':' '{print $1}' /etc/passwd | head -3

root
bin
daemon

Filter lines by a field test:

awk '$9 >= 500 && $9 <= 599 {print $7, $9}' "$LOG"

/api/data 500
/api/data 502
/api/data 500

Because the status field contains numeric strings, awk performs a numeric comparison here.

Sum bytes in the last field:

awk '{sum += $10} END {print sum}' "$LOG"

208841

The END block runs once after the last line. awk can also keep counters in associative arrays, which is useful for counting occurrences of each IP, path, or status value.

See awk examples for more field patterns.

grep vs sed vs awk: Which Should You Use?

The following examples show the distinction between filtering with grep, transforming with sed, and field processing with awk.

Find lines with HTTP 500 responses using a fixed literal around the status code:

grep -F 'HTTP/1.1" 500 ' "$LOG"

203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "badbot/1.0"
192.168.1.10 - - [11/Sep/2026:10:00:12 +0000] "GET /api/data HTTP/1.1" 500 0 "-" "curl/8.5"

Including the end of the HTTP request before 500 makes the match specific to the status field instead of any other 500 appearing in the line.

Change 500 to ERROR in the displayed text:

sed 's/ 500 / ERROR /' "$LOG" | grep ERROR

203.0.113.44 - - [11/Sep/2026:10:00:08 +0000] "GET /api/data HTTP/1.1" ERROR 0 "-" "badbot/1.0"
192.168.1.10 - - [11/Sep/2026:10:00:12 +0000] "GET /api/data HTTP/1.1" ERROR 0 "-" "curl/8.5"

Print the path and status for every 5xx response using field numbers:

awk '$9 >= 500 && $9 <= 599 {print $7, $9}' "$LOG"

/api/data 500
/api/data 502
/api/data 500

In this example, grep selects complete lines, sed changes matching text, and awk selects fields based on their values.

awk can also filter lines, so you do not always need grep | awk. Use grep when line matching makes the pipeline clearer; use awk alone when you are already testing fields or calculating values.

Extract Simple Fields with cut

cut reads delimiter-separated columns. In this sample log, each line begins with the client IP followed by a space, so cut -d' ' -f1 extracts it:

cut -d' ' -f1 "$LOG" | head -3

192.168.1.10
10.0.0.5
192.168.1.10

Every : separates fields:

printf '%s\n' 'one:two:three' | cut -d: -f1-2

one:two

Unlike awk's default whitespace splitting, cut -d' ' treats every individual space as a delimiter, including consecutive spaces. It also does not understand quoted fields. Fixed field numbers can become unreliable when spacing varies or a field contains the delimiter. That is why cut cannot reliably pull the path from "GET /api/health HTTP/1.1" in one step on this log format.

cut -d, and awk -F, work for simple comma-delimited rows, but neither is a full CSV parser when fields can contain quoted commas or embedded newlines. The cut command page lists more field ranges. When the layout is more than “column n of a flat table,” switch to awk.

Sort and Count Repeated Values with sort | uniq -c

sort orders lines so identical values become neighbors. The sort command reference covers -n, -r, -k, and locale behavior in more detail. That adjacency matters because uniq only sees adjacent duplicates.

Sort status codes numerically when the field is numeric:

awk '{print $9}' "$LOG" | sort -n | uniq -c

      7 200
      1 401
      1 403
      1 404
      2 500
      1 502

Without -n, sort treats values as text ("10" before "2").

uniq only combines adjacent duplicates. Without sort, the same value may be counted more than once:

printf '%s\n' a b a | uniq -c

      1 a
      1 b
      1 a

After sorting, the count is correct:

printf '%s\n' a b a | sort | uniq -c

      2 a
      1 b

Rank paths by frequency:

awk '{print $7}' "$LOG" | sort | uniq -c | sort -nr | head -5

      3 /api/data
      2 /static/app.js
      2 /index.html
      2 /api/health
      1 /pricing

Count the most common client IPs:

awk '{print $1}' "$LOG" | sort | uniq -c | sort -nr | head -3

      4 192.168.1.10
      3 203.0.113.44
      3 10.0.0.8

If counts are tied, do not rely on their displayed order. Add a secondary sort key when tie order matters.

Combine grep, awk, sort, and uniq

This pipeline finds the client IPs responsible for 5xx responses and counts them:

grep -E 'HTTP/1\.1" 5[0-9]{2} ' "$LOG" | awk '{print $1}' | sort | uniq -c | sort -nr

      2 203.0.113.44
      1 192.168.1.10

1. grep keeps only 5xx lines.
2. awk '{print $1}' extracts the client IP.
3. sort places identical IPs next to each other.
4. uniq -c counts each consecutive group.
5. sort -nr ranks by count.

The same filter → extract → count pattern can be reused for paths, status codes, user agents, or other fields.

When One awk Command Is Clearer

The pipeline above is easy to read because each step is obvious. When filtering, field processing, and aggregation belong together, awk can perform the filtering and counting while reading the log once:

awk '$9 >= 500 && $9 <= 599 {c[$1]++} END {for (ip in c) print c[ip], ip}' "$LOG" | sort -nr

2 203.0.113.44
1 192.168.1.10

The associative array c stores one counter per client IP. Each time a 5xx line matches, c[$1]++ increments the element keyed by $1 (the client IP). The END block prints the totals after all input is read. The final sort -nr still orders the aggregated results.

Use one awk program when filtering and aggregation naturally belong together. Keep separate commands when the pipeline is easier to read that way.

Use grep Exit Status in Bash Scripts

grep -q suppresses output and sets the exit status from whether a match exists. Scripts use that for conditions:

if grep -q ' 500 ' "$LOG"; then echo "found 500"; else echo "no 500"; fi

found 500

Without the -q exception described below, GNU grep normally returns:

With GNU grep -q, output is suppressed and a successful match can cause grep to return 0 even if another input also produces an error. If your script must distinguish input errors from matches, do not rely on grep -q alone.

Check the exit status directly when you need the numeric code:

grep -q ' 599 ' "$LOG"

Read the exit status on the next line:

echo $?

1

Normally, an if grep -q ...; then ... else ... fi block does not distinguish “no match” from an error. If your script must distinguish those cases reliably, handle grep's status explicitly and remember the -q exception described above. For strict pipelines, remember that only the last command’s status is visible unless you enable pipefail — see PIPESTATUS and set -o pipefail in the Bash pipes guide.

Locale and Deterministic Sorting

sort order depends on locale rules. Letters with diacritics may sort differently from a plain ASCII expectation:

printf '%s\n' ä apple Ä banana | sort

The exact order depends on your current locale, so your result may differ.

Force bytewise C locale sorting when you need predictable byte order or slightly faster sorts on large ASCII logs:

printf '%s\n' ä apple Ä banana | LC_ALL=C sort

apple
banana
Ä
ä

LC_ALL=C trades locale-aware ordering for deterministic byte comparison. Use it when the downstream tool expects bytewise ordering; keep the normal locale when you want locale-aware collation for human-readable text.

Other Useful Text-Processing Commands

tr translates or deletes characters:

printf '%s\n' 'ERROR' | tr '[:upper:]' '[:lower:]'

error

For a single file, grep -c pattern file is simpler when you only need the number of matching lines. wc -l is useful more generally for counting lines produced by any pipeline:

grep -F '/api/data' "$LOG" | wc -l

3

What Not to Parse with Line-Oriented Tools

These tools work well with line-oriented text, especially logs and output with predictable patterns or fields. They are the wrong default for structured interchange formats.

For structured formats such as JSON, YAML, and XML, use a parser that understands the format rather than relying on line positions or regular expressions. Parse JSON in Bash with jq covers field extraction and filters for JSON; for other formats:

• YAML → yq
• XML → xmlstarlet or another format-aware parser

Do not scrape kubectl get, docker ps, or systemctl tables when JSON or API output exists (kubectl -o json, systemctl show, etc.).

grep on a JSON file might find a substring, but it cannot tell whether that string is a key, a value, or part of a larger token.

Common Text-Processing Mistakes

Summary

Use grep to filter lines, sed to transform text, and awk when field numbers, tests, or aggregation matter. Use cut for simple delimited fields, and sort | uniq -c to count repeated values after grouping identical lines. The examples run in Bash pipelines, but the utilities themselves are standard Linux text-processing commands.

These tools work well for logs and stable line-oriented text. For JSON, YAML, XML, or real CSV with quoted fields, use a format-aware parser instead.

References

• grep(1) — GNU grep manual
• sed(1) — GNU sed manual
• gawk manual — GNU Awk
• cut(1) — GNU coreutils
• sort(1) — GNU coreutils
• uniq(1) — GNU coreutils