Python subprocess: Run Shell Commands

The subprocess module is the standard-library way to execute shell commands and other programs from Python. For most scripts, start with subprocess.run(): it waits for completion, returns a CompletedProcess, and supports capturing output, checking exit status, timeouts, cwd, and env. Pair long-running wrappers with Python argparse when users pass flags at startup instead of hard-coded argv lists.

Tested on: Python 3.13.3; kernel 6.14.0-37-generic.

Quick answer: run a shell command in Python

Import subprocess, pass the program and arguments as a list, and call subprocess.run().

import subprocess

result = subprocess.run(["uname", "-r"], capture_output=True, text=True, check=True)
print(result.stdout.strip())

This runs the command, raises CalledProcessError on failure, and prints the kernel release string from stdout.

What is subprocess in Python?

subprocess is in the standard library. It starts external processes, connects optional stdin/stdout/stderr pipes, and exposes return codes. Typical uses: run a CLI tool, capture JSON from a helper command, wrap Git or build steps, or feed stdin to an interactive program (advanced).

Python subprocess quick reference

Why use subprocess instead of os.system?

os.system() runs a command in a subshell but does not give you structured access to stdout/stderr, timeouts, or safe argv splitting. Prefer subprocess.

Legacy alternatives: os.system and os.popen

os.system() runs a command in a subshell and returns an exit status. It does not give you structured stdout/stderr capture or timeouts.

import os

status = os.system("printf hello\\n")
print(status)

On POSIX, the return value encodes the exit status in a platform-specific way. Prefer subprocess.run() for new code.

os.popen() returns a file-like object for command output, but it is deprecated since Python 3.3. Use subprocess.run(..., capture_output=True, text=True) instead.

import subprocess

result = subprocess.run(["printf", "hello\n"], capture_output=True, text=True, check=True)
print(result.stdout, end="")

Third-party sh library (optional)

The sh package lets you call shell utilities with Python attribute syntax, for example from sh import ls. It requires pip install sh and is not part of the standard library. For most projects, stick with subprocess so you stay on stdlib APIs and explicit argv lists.

subprocess.run()

subprocess.run() executes a command, blocks until it finishes, and returns subprocess.CompletedProcess. It is the recommended entry point for “run this, maybe capture output, maybe fail on error.”

import subprocess

result = subprocess.run(["echo", "hello"], capture_output=True, text=True)
print(result.stdout.strip(), result.returncode)

This prints hello and 0.

Run a command with subprocess

Pass the program and each argument as separate list entries. That avoids quoting bugs and is the right default when shell=False (the default).

import subprocess

subprocess.run(["uname", "-r"])

Avoid shell=True unless you need shell syntax (pipes, globs, variables). With shell=True, never interpolate untrusted user input into the command string.

Capture command output

Use capture_output=True (or stdout=subprocess.PIPE). With text=True, result.stdout and result.stderr are strings; otherwise they are bytes unless you set encoding=.

import subprocess

r = subprocess.run(["printf", "line1\nline2"], capture_output=True, text=True)
print(r.stdout, end="")

Check command return code

Conventionally exit status 0 means success; non-zero indicates failure (meaning depends on the program). subprocess.run sets returncode on the result; with check=True, a non-zero code raises CalledProcessError.

import subprocess

r = subprocess.run(["true"])
print(r.returncode)

try:
    subprocess.run(["false"], check=True)
except subprocess.CalledProcessError as exc:
    print(exc.returncode)

You should see 0 then 1 (the exit code from false).

Handle errors with check=True

check=True turns a non-zero exit status into CalledProcessError, which is convenient when a failed command should stop the script—no manual if result.returncode != 0 after every call. If you also pass capture_output=True (or pipe stdout/stderr), the exception object includes whatever was captured so you can log or re-raise with context.

Successful commands return normally; failures jump into except:

import subprocess

subprocess.run(["true"], check=True)

This completes with no output.

import subprocess

try:
    subprocess.run(["false"], check=True)
except subprocess.CalledProcessError as exc:
    print("failed", exc.returncode, exc.cmd)

This prints failed 1 and the command tuple (exact cmd formatting may vary).

import subprocess

try:
    subprocess.run(
        ["sh", "-c", "echo stdout-msg; echo stderr-msg >&2; exit 9"],
        capture_output=True,
        text=True,
        check=True,
    )
except subprocess.CalledProcessError as exc:
    print(exc.returncode, repr(exc.stdout.strip()), repr(exc.stderr.strip()))

You should see return code 9 plus the captured stdout and stderr strings.

import subprocess

try:
    subprocess.run(
        ["grep", "pattern", "/nonexistent/file"],
        capture_output=True,
        text=True,
        check=True,
    )
except subprocess.CalledProcessError as exc:
    print(exc.returncode)

Here grep fails (typically exit status 2 for “file not found”); the exact message is on exc.stderr if you print it.

Run commands with timeout

timeout= stops waiting and raises TimeoutExpired if the child runs too long—useful for network tools or jobs that might hang.

import subprocess

try:
    subprocess.run(["sleep", "5"], timeout=0.3)
except subprocess.TimeoutExpired:
    print("timed out")

Pass command arguments safely

Build a list: ["program", "arg1", "arg2"]. Each list entry is one argv element, so spaces inside a path do not need shell quoting. Never interpolate untrusted user text into a shell=True string.

Path with spaces — one element carries the whole path:

import os
import subprocess
import tempfile

with tempfile.TemporaryDirectory() as tmp:
    path = os.path.join(tmp, "notes.txt")
    open(path, "w").write("data")
    subprocess.run(["wc", "-c", path], check=True)

This runs wc on a single filename that contains a space (no shell). To capture stdout into a file instead of the terminal, open the destination with write to file patterns (with open(..., "w") or stdout=fp on subprocess.run).

Dynamic argv — grow a list instead of pasting strings:

import subprocess

argv = ["printf", "%s\n", "alpha", "beta"]
extra = ["gamma"]  # imagine these come from your own logic, not raw shell
subprocess.run(argv + extra, check=True)

Filter then append — when users pick options, map them to a fixed allowlist instead of pasting their text into sh -c:

import subprocess

allowed = {"digits": r"^[0-9]+$", "words": r"^[a-z]+$"}
mode = "words"
pattern = allowed[mode]
subprocess.run(
    ["grep", "-E", pattern],
    input="foo\n123\nbar\n",
    capture_output=True,
    text=True,
    check=True,
)

Here grep still receives user-influenced data, but only after you validate or constrain mode; never pass arbitrary free-form text straight into shell=True.

If you must turn a trusted single-line command from config into argv (still no shell), shlex.split parses quoting the way a shell would—do not use that on untrusted input.

import shlex
import subprocess

line = 'printf "%s\\n" "one arg"'  # trusted single line from admin config only
subprocess.run(shlex.split(line), check=True)

shell=True vs shell=False

Default: list argv, no shell — the process receives exactly the strings you list; there is no glob or variable expansion.

import subprocess

subprocess.run(["uname", "-s"], check=True)

Shell pipeline (trusted string only) — here the whole command is a fixed literal in source, not built from user input:

import subprocess

r = subprocess.run(
    "printf 'a\nb\nc\n' | wc -l",
    shell=True,
    capture_output=True,
    text=True,
    check=True,
)
print(r.stdout.strip())

You should see a small positive integer line count.

shell=True plus user-controlled strings is a common injection vector. On Windows, the shell differs from POSIX; prefer list argv for portability.

When should you use shell=True?

Keep this subsection as a decision checklist—not a second tutorial. Use shell=True only when all of these are true:

• You need shell behavior (pipes, redirects, VAR=value cmd, globs, ~, or a built-in like cd in the same string).
• The command string is fixed in code or comes from a fully trusted source (not web form text, not file paths from users).
• You accept that quoting rules and the default shell differ across OS installs.

If you only need “run this program with these arguments,” you do not need shell=True—a list and shell=False is clearer and safer. If you need a pipeline, consider chaining Popen objects in Python instead of one big shell string.

subprocess.run vs Popen

run() is implemented in terms of Popen; it is the convenience wrapper. When you still want a child process but will call wait() or communicate() yourself, construct Popen directly:

import subprocess

proc = subprocess.Popen(
    ["printf", "line1\nline2\n"],
    stdout=subprocess.PIPE,
    text=True,
)
stdout, stderr = proc.communicate()
print(stdout, end="")
print("exit", proc.returncode)

You should see both lines on stdout, then exit 0.

When to use Popen

Use Popen instead of run() when you need more control than a single blocking call: long-running children, streaming stdout/stderr while the process is still alive, feeding stdin interactively, wiring processes together with stdin=PIPE / stdout=PIPE, or polling with poll() without blocking the whole interpreter. For one-shot commands, stay with run().

Capture live output from a running command

run() with capture_output=True buffers until the process exits. To react while bytes or lines arrive (build logs, progress text), use Popen with stdout=PIPE and read in a loop. On POSIX, iterating proc.stdout yields chunks separated by newlines when the child line-buffers; very chatty programs may need stdbuf or application flags for line buffering.

import subprocess

proc = subprocess.Popen(
    ["sh", "-c", "echo one; sleep 0.05; echo two"],
    stdout=subprocess.PIPE,
    text=True,
)
for line in proc.stdout:
    print("chunk:", line.rstrip())
proc.wait()

You should see chunk: one then chunk: two as lines arrive (small timing gaps are normal).

Send input to a subprocess

Pass input= as bytes or str (with text=True). Some CLIs read passwords or data from stdin.

import subprocess

r = subprocess.run(
    ["cat"],
    input="hello\n",
    capture_output=True,
    text=True,
    check=True,
)
print(r.stdout, end="")

This prints hello followed by a newline.

Use cwd to run a command in another directory

cwd= sets the child’s working directory without changing your Python process’s cwd—useful for Git, builds, or tools that expect a project root. Verify the target directory exists first with check if a file exists (Path(cwd).is_dir()) when a missing folder would make the child fail obscurely.

import subprocess
import tempfile
import os

with tempfile.TemporaryDirectory() as tmp:
    subprocess.run(["touch", "marker"], cwd=tmp)
    print("marker" in os.listdir(tmp))

This prints True when the file was created in the temp directory.

Pass environment variables to subprocess

Pass env= as a full mapping. Usually start with os.environ.copy() and update keys. Do not log secrets.

import os
import subprocess

env = os.environ.copy()
env["MY_FLAG"] = "1"
subprocess.run(["env"], env=env, capture_output=True, text=True)

Inspecting the captured output would show MY_FLAG=1 among other variables (trim for real apps).

Run a Python script with subprocess

Use the current interpreter when possible:

import sys
import subprocess

subprocess.run([sys.executable, "-c", "print('child')"], check=True)

For reusable code in the same project, normal imports are usually better than spawning a second interpreter.

Run multiple shell commands sequentially

Run commands one after another with separate run() calls and check return codes (or check=True). Shell pipelines (a | b) require a shell or explicit Popen plumbing; do not reach for shell=True until a plain list cannot express the pipeline.

Two steps in order — each call waits; check=True stops the script on the first failure:

import subprocess
import tempfile
import os

with tempfile.TemporaryDirectory() as tmp:
    path = os.path.join(tmp, "notes.txt")
    with open(path, "wb") as fp:
        subprocess.run(["printf", "line1\nline2\n"], stdout=fp, check=True)
    subprocess.run(["wc", "-l", path], check=True)

The first command writes the file; the second counts lines (you should see a small line count in the process stdout because wc prints to the parent’s terminal unless you capture it).

“Pipeline” without a shell — run the producer, pass its stdout into the consumer’s stdin (use text=True on both sides when you pass str, or keep everything as bytes without text=True):

import subprocess

packed = subprocess.run(
    ["printf", "a\nb\nc\n"],
    capture_output=True,
    text=True,
    check=True,
)
counted = subprocess.run(
    ["wc", "-l"],
    input=packed.stdout,
    capture_output=True,
    text=True,
    check=True,
)
print(counted.stdout.strip())

You should see a single integer (the line count). For streaming between long-running programs, use Popen with stdin=other.stdout instead of buffering everything in Python.

Start several processes, then wait — useful when each command should begin before the previous one finishes:

import subprocess

first = subprocess.Popen(["sleep", "0.1"])
second = subprocess.Popen(["sleep", "0.1"])
print(first.pid, second.pid)
first.wait()
second.wait()
print(first.returncode, second.returncode)

Both processes start, then wait() blocks until each exits. For parallel work with many jobs, consider a process pool or task queue instead of dozens of bare Popen calls.

Chained shell commands (trusted only) — && in one string requires shell=True:

import subprocess

subprocess.run("printf step1\\n && printf step2\\n", shell=True, check=True)

Prefer separate run() calls or Python-side logic when you do not need shell operators.

subprocess.call, check_call, and check_output

Older helpers still exist; prefer run() for new code.

Common subprocess errors

FileNotFoundError

Python could not find the executable: wrong name, not installed, missing from PATH, or you passed shell syntax (for example echo foo > bar) while shell=False (the first token must be a real executable path). Fix the program name, use an absolute path, or use a real shell only when you accept the security trade-offs. Missing files when opening paths (not spawning) are the same built-in—see FileNotFoundError in Python for errno 2, path hygiene, and try / except patterns.

import subprocess

try:
    subprocess.run(["/nonexistent/glc-demo-bin-zzzz"])
except FileNotFoundError as exc:
    print(exc.filename)

This prints the program path Python tried to execute. A wrong name on PATH fails the same way (exc.filename is the argv you passed).

import subprocess

try:
    subprocess.run(["echo hello > out.txt"], check=False)
except FileNotFoundError:
    print("no binary named like a shell line")

With shell=False, the whole string is treated as one program name, so you typically get FileNotFoundError instead of a shell running—use a list like ["sh", "-c", "..."] only when you really need a shell.

CalledProcessError

The process ran but exited with a non-zero status while check=True was set—often bad flags, missing inputs, permission denied inside the tool, or application-defined failure codes. Inspect returncode, stderr, and the tool’s docs. Designing your own failure modes is covered in Python custom exceptions.

TimeoutExpired

The child did not finish before timeout= elapsed—stuck work, slow network, or a prompt waiting on stdin are common causes. Increase the limit, fix the command, or supply non-interactive flags. The exception carries timeout, cmd, and often partial output / stdout / stderr depending on how the call was configured.

import subprocess

try:
    subprocess.run(["sleep", "60"], timeout=0.25)
except subprocess.TimeoutExpired as exc:
    print(round(exc.timeout, 2), exc.cmd[0])

You should see the timeout value (about 0.25) and the program name (sleep). On many POSIX systems the child receives SIGKILL after the timeout; do not rely on it having flushed partial work to disk. The same pattern appears under Run commands with timeout for day-to-day wrapping.

PermissionError

The path exists but is not executable (or the OS denied execution). Check file mode bits, mount options, and whether you pointed at a directory by mistake.

Common mistakes

These issues show up constantly in real scripts; you do not need a separate heading per row—a table keeps them easy to scan.

Best practices

Prefer subprocess.run with a list argv and shell=False. Combine capture_output=True, text=True, and check=True when you want string output and failures as exceptions. Add timeout for anything that might block. When debugging, log args, returncode, and captured streams (never log secrets). Use Python logging if you need structured logs. Reach for Popen only for streaming, pipelines, or interactive control.

Summary

Use the subprocess module to run shell commands and other programs from Python. For most tasks, subprocess.run() is the right default: pass the program and each argument as list elements (shell=False), turn on capture_output and text=True when you need readable strings, and use check=True when failure should raise CalledProcessError. Run multiple commands sequentially with separate run() calls or Popen plus wait(). Prefer subprocess over os.system() and deprecated os.popen(). Reserve shell=True for trusted commands that genuinely need shell syntax, and use Popen when you must stream I/O or orchestrate processes in more detail.

References

• subprocess — Subprocess management
• Security considerations