Call REST APIs from Bash with curl and jq

Bash API scripts usually combine curl for HTTP requests with jq for JSON parsing. In a Bash API script, a request can fail before HTTP, return an HTTP error such as 404 or 500, or return a successful HTTP response containing invalid or unexpected JSON. The script should handle those cases separately.

A silent curl combined with set -e and a pipeline to jq can mark a run successful even when the HTTP request failed, so we treat download, status check, and JSON parsing as separate steps.

A Reliable Bash API Request Pattern

A reliable Bash API request usually needs:

• set -euo pipefail so unset variables and pipeline failures stop the script.
• curl -sS hides the progress meter while keeping curl error messages visible on stderr.
• Explicit timeouts so a hung socket does not block cron or CI forever.
• Bounded --retry only when repeating the request is safe for your application — curl retries on transient transport/HTTP conditions, not on idempotency alone.
• Capture the response body once, then validate JSON with jq in a second step.

GET JSON with curl and jq

Start with a read-only JSON endpoint. The curl command cheat sheet lists download and header options; here we focus on flags API scripts use every day.

Fetch a sample document without a progress bar:

curl -sS https://httpbin.org/json

{
  "slideshow": {
    "author": "Yours Truly", 
    "date": "date of publication", 
    "slides": [

-s hides the progress meter. Add -S with -s so curl still prints error messages for failures such as DNS, connection, or TLS errors. HTTP 4xx/5xx responses require --fail or --fail-with-body if you want them to make curl fail. Scripts that only use -s can fail quietly on DNS or TLS problems.

Pipe into jq only after you understand what -sS shows on failure:

curl -sS https://httpbin.org/json | jq -r '.slideshow.author'

Yours Truly

For anything beyond a one-liner, capture the body in a variable and parse it separately so a failed download cannot hide behind a successful jq run.

Make HTTP Errors Fail the Script

By default curl exits 0 when it successfully transfers a response, even if the HTTP status is 404 or 500. Bash automation usually needs a non-zero exit on application-level errors.

Ask curl to treat most HTTP 4xx/5xx responses as failure with -f / --fail. With --fail, HTTP responses of 400 or higher normally make curl return error 22 instead of treating the transfer as successful. Curl documents authentication-related exceptions around 401 and 407, so do not treat --fail as a complete HTTP policy engine.

curl -sS --fail https://httpbin.org/status/404
rc=$?
printf 'exit:%d\n' "$rc"

curl: (22) The requested URL returned error: 404
exit:22

Exit code 22 is curl's "HTTP returned error" status. Capture $? immediately after curl — $? only reflects the last command's exit status.

A successful JSON GET with --fail still returns exit 0 from curl:

curl -sS --fail https://httpbin.org/json >/dev/null
rc=$?
printf 'curl exit:%d\n' "$rc"

curl exit:0

Check curl's exit status directly — not through a pipeline, unless pipefail is enabled.

--fail vs --fail-with-body

Both options make curl exit non-zero on HTTP 4xx/5xx. The difference is what happens to the response body.

Use --fail when you only need pass/fail. Use --fail-with-body when an API returns JSON error details ({"error":"invalid token"}) that you want to log or parse.

On RHEL 10.2 the lab ships curl 8.12.1, which supports --fail-with-body. Gate newer options before relying on them in portable scripts:

curl --help all 2>&1 | grep -q -- '--fail-with-body' && echo 'fail-with-body: supported'

fail-with-body: supported

httpbin /status/404 returns no body, so --fail and --fail-with-body look similar. Real APIs often return JSON on 422 or 409; --fail-with-body keeps that text available while still failing the command:

if ! body=$(curl -sS --fail-with-body 'https://dummyjson.com/products/0'); then
  echo "request failed; body bytes: ${#body}" >&2
  [[ -n "$body" ]] && printf '%s\n' "$body" >&2
  exit 1
fi

When the server sends an error document, $body still holds it after the non-zero exit, which is the main reason to prefer --fail-with-body over --fail in automation.

Redirects

curl does not follow HTTP redirects automatically. If the API legitimately redirects requests, add -L / --location. Avoid adding it blindly to authenticated calls — credentials and request methods across redirects deserve deliberate handling.

Test a redirect without and with -L:

curl -sS -o /dev/null -w '%{http_code} redirect:%{num_redirects}\n' https://httpbin.org/redirect/1

302 redirect:0

Follow the redirect when that is what you intend:

curl -sS -L -o /dev/null -w '%{http_code} redirect:%{num_redirects}\n' https://httpbin.org/redirect/1

200 redirect:1

Set Connection and Request Timeouts

Two different clocks matter for API scripts:

• --connect-timeout SECONDS — limit the connection phase, including DNS lookup and TCP/TLS setup.
• --max-time SECONDS — limit how long each transfer attempt may take.

A slow API may connect quickly but respond slowly; set both limits in production scripts.

Verify a short connect budget against a working host:

curl -sS --connect-timeout 2 --max-time 5 -o /dev/null -w 'code:%{http_code} time:%{time_total}\n' https://httpbin.org/delay/1

code:200 time:2.423138

If a transfer attempt exceeds --max-time, curl exits with a timeout error instead of hanging your scheduler.

Retry Transient API Failures Safely

curl can retry certain transient failures when you add --retry:

curl -sS --retry 2 --retry-delay 1 --retry-max-time 10 -w '\nHTTP:%{http_code}\n' https://httpbin.org/status/429
rc=$?
printf 'exit:%d\n' "$rc"

HTTP:429
exit:0

With default retry rules, curl retries transient conditions such as timeouts and HTTP 408, 429, 500, 502, 503, 504, 522, and 524, and honors Retry-After when the server sends it. That logic is not limited to GET or to operations your application considers safe to repeat — restrict automatic retries to idempotent requests unless the API provides an idempotency mechanism.

Without --retry-delay, curl uses an increasing retry delay, starting at one second and doubling up to ten minutes. --retry-delay 1 replaces that default backoff with a one-second retry delay; curl can also honor a server-provided Retry-After header.

--retry-max-time limits how long curl continues starting retries. The timer begins before the first transfer and includes retry delays, but if the limit has not been reached when another attempt begins, that transfer can continue beyond the limit. Combine --retry-max-time with --max-time to bound each individual transfer.

A practical starting point on curl 8.12.x for a safe read:

curl -sS --fail-with-body \
  --connect-timeout 5 --max-time 20 \
  --retry 2 --retry-delay 1 --retry-max-time 30 \
  'https://dummyjson.com/products?limit=2&skip=0'

Adjust the retry count, delay, and timeout values to match how quickly your API normally responds and how long the script can afford to wait. Log the final HTTP status when a call still fails after retries.

--retry-all-errors expands retries to all curl errors. It does not by itself make every HTTP 4xx/5xx response an error; --fail or --fail-with-body controls that behavior. Curl warns against using --retry-all-errors as a default because retries can duplicate sent or received data.

Do not combine --retry-all-errors with create/update/delete calls unless the API is explicitly idempotent (for example a PUT with a stable idempotency key you control).

Authenticate API Requests

The API documentation determines the required scheme. Common patterns:

Curl warns that secrets on the command line can briefly be visible to other users on the host. For Basic credentials especially, avoid literal passwords in scripts when a safer mechanism exists.

Bearer tokens safely in Bash

Read tokens from the environment — see Bash environment variables for export and persistence patterns — not from literals in the script. The value below is a dummy token used only for demonstration:

export API_TOKEN='test-bearer-token-12345'

An environment variable keeps the token out of the script source, but expanding it in -H "Authorization: Bearer ${API_TOKEN}" still puts the expanded header in curl's argument list. That does not hide the token from process inspection on the host.

Confirm authentication without printing the secret:

curl -sS -H "Authorization: Bearer ${API_TOKEN}" https://httpbin.org/bearer | jq '{authenticated}'

{
  "authenticated": true
}

For stricter handling, curl supports reading headers from stdin with -H @-:

printf 'Authorization: Bearer %s\n' "$API_TOKEN" |
  curl -sS -H @- https://httpbin.org/bearer | jq '{authenticated}'

{
  "authenticated": true
}

Environment variables are convenient, not encrypted storage. Keep tokens out of Git commits, world-readable config files, and debug output that dumps full command lines.

If you enable xtrace for debugging, disable it before commands that expand credentials. Only re-enable tracing afterward if tracing was already enabled. Never enable xtrace around code that prints, transforms, or expands credentials.

For long-running or shared automation, prefer short-lived credentials or a secret-management mechanism appropriate to the environment.

Do not use -k / --insecure as ordinary TLS error handling. Fix trust store, corporate CA, or certificate renewal instead of disabling verification in automation.

Build JSON Request Bodies with jq

Never assemble JSON by quoting shell variables into a string. A quote or newline in user input breaks the document and opens injection bugs.

Build the body with jq -n and --arg / --argjson:

TITLE='Widget "Pro"'
jq -n --arg title "$TITLE" --argjson qty 2 '{title: $title, quantity: $qty}'

{
  "title": "Widget \"Pro\"",
  "quantity": 2
}

POST the generated document with explicit content type (--data-binary implies POST; no -X needed):

BODY=$(jq -n --arg title "$TITLE" --argjson qty 2 '{title: $title, quantity: $qty}')
curl -sS --fail-with-body https://httpbin.org/post \
  -H 'Content-Type: application/json' \
  --data-binary "$BODY" | jq -r '.json.title'

Widget "Pro"

--data-binary "$BODY" sends the generated JSON without modifying its contents. Keep "$BODY" quoted so Bash passes the complete JSON document as one argument.

GET, POST, PUT, PATCH and DELETE with curl

Most REST APIs use a small set of HTTP methods. The examples below use httpbin echo endpoints so you can inspect what curl sent.

Set a base URL and JSON body once:

URL='https://httpbin.org/anything/item/42'
BODY=$(jq -n --arg name 'Widget Pro' '{name: $name}')

GET:

curl -sS --fail-with-body "$URL"

POST JSON (--data-binary implies POST; -X POST is normally unnecessary):

curl -sS --fail-with-body \
  -H 'Content-Type: application/json' \
  --data-binary "$BODY" "$URL"

PUT:

curl -sS --fail-with-body \
  -X PUT \
  -H 'Content-Type: application/json' \
  --data-binary "$BODY" "$URL"

PATCH:

curl -sS --fail-with-body \
  -X PATCH \
  -H 'Content-Type: application/json' \
  --data-binary "$BODY" "$URL"

DELETE:

curl -sS --fail-with-body -X DELETE "$URL"

--data, --data-raw, --data-binary, and --json naturally imply POST, so -X POST is usually redundant. Use explicit -X when you intentionally select PUT, PATCH, or DELETE.

Encode Query Parameters Safely

Integer pagination parameters such as limit=2&skip=0 are safe to assemble manually when both values are under your control. Arbitrary user input is not — spaces, &, ?, and Unicode break hand-built query strings.

Let curl percent-encode query parameters:

query='red shoes'
curl -sS --get --data-urlencode "q=$query" 'https://httpbin.org/get' | jq -r '.args.q'

red shoes

Do not manually concatenate arbitrary user input into a query string. Let curl encode it with --data-urlencode (or --url-query on newer curl builds).

Parse and Validate JSON with jq

Once the HTTP layer succeeds, parse JSON with jq. The dedicated Parse JSON in Bash with jq guide covers filters, --arg, and loops; here we focus on API script checks.

Validate that the payload is an object and read a field from DummyJSON (no API key required for read demos at time of writing):

body=$(curl -sS --fail-with-body 'https://dummyjson.com/products?limit=2&skip=0')
jq -e '.products | type == "array"' <<<"$body" >/dev/null
jq -r '.products[0].title' <<<"$body"

Essence Mascara Lash Princess

jq -e makes jq exit non-zero when the filter result is false or null, which fits guard clauses before you mutate data.

Capture curl Status, HTTP Status and Body Separately

Some workflows need the curl exit code, HTTP status, and response body separately. Write the body to a file and capture the HTTP code from -w:

URL='https://httpbin.org/json'
if http_code=$(curl -sS -o /tmp/api-body.json -w '%{http_code}' "$URL"); then
  printf 'HTTP %s\n' "$http_code"
else
  rc=$?
  printf 'curl failed: %d\n' "$rc" >&2
fi

HTTP 200

Inspect the saved body when curl succeeds:

head -c 60 /tmp/api-body.json

{
  "slideshow": {
    "author": "Yours Truly", 

• curl exit code — transport/tool failure (DNS, TLS, timeout, or --fail HTTP error when enabled)
• http_code from -w — HTTP layer status returned by the server
• body file — application response payload

Combine with --fail-with-body when HTTP errors must fail the script while keeping the body for logging.

Handle HTTP 429 Rate Limits

HTTP 429 Too Many Requests means the client should reduce its request rate. Read the status without treating 429 as success in application logic:

code=$(curl -sS -o /tmp/ratelimit-body.txt -w '%{http_code}' https://httpbin.org/status/429)
printf 'http=%s\n' "$code"

http=429

In scripts, handle 429 using the API's documented retry/backoff policy: sleep according to Retry-After (or exponential backoff), log the event, and avoid tight loops. For write operations, queue work instead of retrying immediately.

Follow Paginated APIs

APIs may use page numbers, offset/limit, a next URL, HTTP Link headers, or opaque cursor tokens. Follow the mechanism documented by the API rather than inventing offsets yourself.

DummyJSON uses limit and skip and returns total in each response — a common offset/limit model. Inspect the first page:

curl -sS 'https://dummyjson.com/products?limit=2&skip=0' | jq '{total, skip, limit, count: (.products|length)}'

{
  "total": 194,
  "skip": 0,
  "limit": 2,
  "count": 2
}

A compact pagination loop stops when skip reaches total, a hard page cap is hit, or a page returns an empty array:

skip=0
limit=2
max_pages=2
total=999999
page=0

while (( skip < total && page < max_pages )); do
  body=$(curl -sS --fail-with-body "https://dummyjson.com/products?limit=${limit}&skip=${skip}")
  total=$(jq -er '.total' <<<"$body")
  count=$(jq -er '.products | length' <<<"$body")
  (( count == 0 )) && break
  jq -c '.products[] | {id, title}' <<<"$body"
  skip=$((skip + count))
  page=$((page + 1))
done

{"id":1,"title":"Essence Mascara Lash Princess"}
{"id":2,"title":"Eyeshadow Palette with Mirror"}
{"id":3,"title":"Powder Canister"}
{"id":4,"title":"Red Lipstick"}

Always cap max_pages. Advancing skip by the number of records returned (skip=$((skip + count))) handles a short final page more safely than assuming every page contains exactly limit items. When an API returns a next link or cursor token, follow that field instead of calculating offsets manually.

Keep curl Failures Separate from jq Failures

Transport success does not guarantee JSON. Separate the exit codes so alerts point at the right layer.

Download first:

body=$(curl -sS --fail-with-body https://httpbin.org/json)

Query a missing key with jq -e:

jq -e '.missing_key' <<<"$body"
jq_rc=$?
printf 'jq_exit:%d\n' "$jq_rc"

null
jq_exit:1

Here curl succeeded (200 JSON document) but the filter failed because .missing_key is null. Check curl's exit status immediately after the download; capture jq's exit status before running another command.

Enable pipefail if you still pipe — see Bash pipes — but capturing the body in a variable keeps failures easier to read in longer scripts.

Complete API Automation Script

The script below ties the patterns together: bounded retries on reads, JSON validation, payload construction with jq -n, paginated GETs against DummyJSON, and a final summary line. Replace the URL and field names with your API; add authentication headers in api_get when your service requires them.

Create the lab directory and save ~/api-lab/bin/sync-products.sh:

mkdir -p ~/api-lab/bin
cat > ~/api-lab/bin/sync-products.sh << 'SCRIPT'
#!/usr/bin/env bash
set -euo pipefail

API_BASE='https://dummyjson.com/products'
LOG_PREFIX='[sync-products]'
pages_done=0
records=0

api_get() {
  local url=$1
  local tmp http_code rc=0
  tmp=$(mktemp) || return 1
  # Body lands in $tmp; -w writes the HTTP status to stdout so Bash can log both on failure.
  if http_code=$(curl -sS --fail-with-body \
      --connect-timeout 5 --max-time 20 \
      --retry 2 --retry-delay 1 --retry-max-time 30 \
      -o "$tmp" -w '%{http_code}' "$url"); then
    if [[ "$http_code" == "200" ]]; then
      cat "$tmp"
    else
      echo "${LOG_PREFIX} unexpected HTTP ${http_code} for ${url}" >&2
      cat "$tmp" >&2
      rc=1
    fi
  else
    rc=$?
    echo "${LOG_PREFIX} GET failed url=${url} http=${http_code}" >&2
    cat "$tmp" >&2 || true
  fi
  rm -f "$tmp"
  return "$rc"
}

skip=0
limit=2
max_pages=2
total=999999
while (( skip < total && pages_done < max_pages )); do
  body=$(api_get "${API_BASE}?limit=${limit}&skip=${skip}")
  jq -e '.products | type == "array"' <<<"$body" >/dev/null
  total=$(jq -er '.total' <<<"$body")
  count=$(jq -er '.products | length' <<<"$body")
  (( count == 0 )) && break
  while IFS= read -r product; do
    id=$(jq -er '.id | select(type == "number")' <<<"$product")
    title=$(jq -er '.title | select(type == "string")' <<<"$product")
    payload=$(jq -n \
      --arg title "$title" \
      --argjson id "$id" \
      '{action: "catalog", product_id: $id, name: $title}')
    records=$((records + 1))
    echo "$payload"
  done < <(jq -c '.products[]' <<<"$body")
  pages_done=$((pages_done + 1))
  skip=$((skip + count))
done

echo "${LOG_PREFIX} pages=${pages_done} payloads=${records} status=ok"
SCRIPT

api_get removes its own temporary file before returning. Command substitution runs the function in a subshell, so a global temp-file array updated inside api_get would not survive back to the parent shell.

Make it executable:

chmod +x ~/api-lab/bin/sync-products.sh

Run the script:

~/api-lab/bin/sync-products.sh

{"action":"catalog","product_id":1,"name":"Essence Mascara Lash Princess"}
{"action":"catalog","product_id":2,"name":"Eyeshadow Palette with Mirror"}
{"action":"catalog","product_id":3,"name":"Powder Canister"}
{"action":"catalog","product_id":4,"name":"Red Lipstick"}
[sync-products] pages=2 payloads=4 status=ok

The skip += count increment matches DummyJSON's documented limit/skip contract. When your API returns a next URL or cursor token, follow that field instead.

Point api_get at your API base, add real authentication headers when required, and send the payload loop to your notifier or write endpoint.

When Bash Is No Longer the Right Client

Bash with curl and jq fits health checks, small sync jobs, and CI glue. Move to Python (requests, httpx) or Go when you need several of the following at once:

• Complex auth flows (OAuth refresh, mTLS, signed requests)
• Large response streaming or binary payloads
• Strict idempotency, deduplication, and structured error types across many endpoints
• Concurrent workers with shared rate-limit state
• Unit tests around HTTP client behavior

Move to a general-purpose HTTP client when authentication, concurrency, error handling, and response models make the Bash implementation difficult to test and maintain.

Summary

Reliable Bash API scripts treat curl and jq as separate failure domains. Use -sS to hide the progress meter while keeping curl transport errors visible on stderr; add --fail or --fail-with-body when HTTP status must fail automation (without treating --fail as a complete policy engine), and use bounded retries only when repeating the request is safe for your application.

Build JSON with jq -n, authenticate with the scheme your API documents, encode query parameters with --data-urlencode, paginate with hard stop conditions and empty-page checks, and handle 429 using the API's documented retry/backoff policy. The lab script walks two DummyJSON pages, validates arrays, builds payloads, and prints a one-line status summary you can extend with authenticated or write endpoints.

When authentication, pagination, and error models grow beyond a few functions, a typed HTTP client in Python or Go is usually easier to test and safer to change than a large Bash program.

References

• curl manual — retry, fail, and timeout options
• jq 1.7 manual
• RFC 9110 — HTTP Semantics (status codes, Retry-After)