How to Read a File Line by Line in Python

In this article we will read a text file line by line in Python using the built-in open() function, starting with the pattern most scripts should use: iterate directly over the file object inside a with block.

Reading one line at a time is common when processing logs, configuration files, exports, and other large text files. To read a large file line by line in Python without loading the entire file into memory, iterate over the file object inside a with block instead of calling read() or readlines(). Fixed-width binary layouts (headers, integers, floats) belong in the Python struct module guide, not in a line loop.

Quick answer: read a file line by line in Python

Open the path with encoding="utf-8", then loop over the file object. Python reads ahead in a buffer, but it does not build a list of every line for you.

with open("example.txt", encoding="utf-8") as file:
    for line in file:
        print(line, end="")

Suppose example.txt contains:

Ubuntu
RHEL
Rocky Linux
Python

The loop prints those four lines exactly as stored. The with statement closes the file when the block ends. Each line still includes its newline character, so print(line, end="") avoids a blank line between rows. To write data instead, see write to file in Python.

read() vs readline() vs readlines() vs file iteration

All four methods sit on the same text-mode file object. They differ in how much they return at once and how much memory your script holds while you work.

For normal text processing, iterate directly over the file. Reserve readlines() and splitlines() for files that comfortably fit in memory. Neither is a streaming technique because both need the full text before the list exists.

Method 1: iterate directly over the file

This is the Pythonic default. The file object is an iterator: each pass through the loop asks the interpreter for the next line until end-of-file.

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "servers.txt"
    path.write_text(
        "web01.prod.example\nweb02.prod.example\ndb01.prod.example\n",
        encoding="utf-8",
    )
    with path.open(encoding="utf-8") as file:
        for line in file:
            print(line, end="")

That prints three hostnames, one per line. Nothing copies the full file into a list first, so memory stays roughly flat even when the path is large. You only pay for the current line string plus Python's read buffer.

A file that omits the final newline still works. If example.txt contains first\nsecond with no \n after second:

with open("example.txt", encoding="utf-8") as file:
    for line in file:
        print(repr(line))

The loop prints 'first\n' and then 'second'. Python still yields the final line even when the file does not end with a newline.

Remove the newline without removing other whitespace

Text-mode reads include the line ending. With Python's default universal-newline handling in text mode, \r\n and \r input line endings are translated to \n, so removing the terminator while preserving other whitespace is straightforward.

line = "    indented configuration\n"
print(repr(line.removesuffix("\n")))  # preferred: drops only the line ending
print(repr(line.strip()))             # removes leading and trailing whitespace
print(repr(line.rstrip()))            # removes trailing whitespace, not just \n

For indented config or code, strip() can erase meaningful leading spaces. Prefer line.removesuffix("\n") when you only want to drop the line terminator. line.rstrip("\n") works the same way once universal newlines have normalized the ending. The Python strip() guide walks through all three trimming methods.

If the goal is clean console output and the stored newline is fine, print(line, end="") is enough.

Read a file with line numbers

Wrap the file iterator with enumerate() and start at 1 when humans count lines from one.

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "app.log"
    path.write_text(
        "2026-09-11 10:01:10 INFO Service started\n"
        "2026-09-11 10:02:21 ERROR Database unavailable\n"
        "invalid line here\n"
        "2026-09-11 10:03:17 INFO Retrying\n",
        encoding="utf-8",
    )
    with path.open(encoding="utf-8") as file:
        for line_number, line in enumerate(file, start=1):
            if "invalid" in line:
                print(f"Invalid config on line {line_number}: {line.removesuffix('\n')}")

That reports Invalid config on line 3: invalid line here. Pairing enumerate() with streaming reads is how you cite exact line numbers in error reports without loading the file twice. See Python enumerate() for more counting patterns.

Method 2: read one line at a time with readline()

readline() returns the next line including its newline, or "" when the stream hits end-of-file.

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "example.txt"
    path.write_text("alpha\n\nbeta\n", encoding="utf-8")
    with path.open(encoding="utf-8") as file:
        while True:
            line = file.readline()
            if line == "":
                break
            print(repr(line))

Running that loop on alpha, a blank row, and beta prints repr() values in order: 'alpha\n', then '\n', then 'beta\n'. A blank line in the file is normally "\n", not "". Only end-of-file returns the empty string, so you can tell EOF apart from an empty row.

When readline() is better than a for loop

Use readline() when your program needs to decide what to do after each individual line:

• Read a header row, then iterate the body (common in CSV-like formats; for real CSV use the csv module).
• Consume exactly one logical line before deciding what to read next.
• Interleave file input with another stream where your code must handle one line at a time.

readline() is not faster than for line in file for everyday scans. Both use the same buffered reader underneath, which may read ahead internally even though your code consumes one logical line per call.

Method 3: use readlines() when you actually need a list

readlines() reads from the current position through EOF and returns a list of strings. To persist a list of lines you built in memory, the write-side mirror is writelines()—see Python writelines().

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "small.txt"
    path.write_text("one\n two\n three\n", encoding="utf-8")
    with path.open(encoding="utf-8") as file:
        lines = file.readlines()
    print(lines)

That prints ['one\n', ' two\n', ' three\n']. Each element keeps its trailing newline. This is fine for small files when you plan to index, sort, or pass the full list to another function. It is a poor default for multi-gigabyte logs because it builds and retains a list containing all remaining lines in memory.

Read a large file line by line without loading it into memory

The streaming pattern is the same loop you already saw:

with open("large.log", encoding="utf-8") as file:
    for line in file:
        process(line)

Python's buffered text I/O reads chunks from the OS and splits them into lines. The interpreter does not allocate a list of the whole file for a for loop.

There are two important limits to this memory advantage:

• Memory still depends on the current line and what your process() function stores. If you append every line to a growing list, you lose the streaming benefit.
• A file with one enormous line (no newline for megabytes) can still force a large single string into RAM. Line-by-line reading assumes reasonable line lengths.

Benchmark: file iterator vs readlines()

I generated a 26.2 MiB fixture (500,000 lines) and compared both approaches in separate subprocesses so each run starts with a clean process and RSS measurement.

Create the fixture once, then benchmark each mode in its own process with /usr/bin/time:

python3 -c "
LINE_COUNT = 500_000
LINE_TEXT = '2026-09-11 10:01:10 INFO sample log line for benchmark\n'
with open('large.log', 'w', encoding='utf-8') as f:
    for _ in range(LINE_COUNT):
        f.write(LINE_TEXT)
"

bench.py runs one mode per invocation:

import sys

FILE_PATH = "large.log"

def bench_iterate():
    count = 0
    with open(FILE_PATH, encoding="utf-8") as file:
        for line in file:
            count += 1
    return count

def bench_readlines():
    with open(FILE_PATH, encoding="utf-8") as file:
        lines = file.readlines()
    return len(lines)

if __name__ == "__main__":
    mode = sys.argv[1]
    if mode == "iterate":
        print(bench_iterate())
    else:
        print(bench_readlines())

Run each mode five times and read elapsed wall time plus maximum resident set size from time:

/usr/bin/time -f 'elapsed=%e maxrss_kb=%M' python3 bench.py iterate
/usr/bin/time -f 'elapsed=%e maxrss_kb=%M' python3 bench.py readlines

Median results on the lab host:

readlines() held about seven times more resident memory here because it builds the full list. Elapsed time was close on this small fixture, but for large logs the iterator is the safer default; wall-clock differences are usually smaller than the memory gap once files grow.

Read only lines that match a condition

Filter inside the loop so you never build a full list of non-matching rows.

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "app.log"
    path.write_text(
        "2026-09-11 10:01:10 INFO Service started\n"
        "2026-09-11 10:02:21 ERROR Database unavailable\n"
        "2026-09-11 10:03:17 INFO Retrying\n",
        encoding="utf-8",
    )
    with path.open(encoding="utf-8") as file:
        for line in file:
            if "ERROR" in line:
                print(line, end="")

That prints only the ERROR row. For stricter matching, compile a regular expression once outside the loop and call .search(line) per row.

Skip empty lines

Decide what "empty" means before you filter.

If whitespace-only lines should count as empty:

with open("sparse.txt", encoding="utf-8") as file:
    for line in file:
        if not line.strip():
            continue
        process(line)

If only newline-only records should be skipped, test the line directly:

with open("sparse.txt", encoding="utf-8") as file:
    for line in file:
        if line == "\n":
            continue
        process(line)

The first version drops lines that contain only spaces or tabs. The second keeps " \n" as real content. Pick the rule that matches your file format.

Stop reading when you find a match

Streaming lets you exit early without scanning the rest of a huge file.

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "fatal.log"
    path.write_text(
        "INFO ok\nWARN slow\nFATAL disk full\nINFO ignored\n",
        encoding="utf-8",
    )
    with path.open(encoding="utf-8") as file:
        for line_number, line in enumerate(file, 1):
            if "FATAL" in line:
                print(f"Found on line {line_number}")
                break

That stops at the first FATAL line and never reads INFO ignored. Use break when the task is "find first match" rather than "summarize entire file".

Handle file encodings correctly

Pass encoding="utf-8" on text files unless you know the source uses something else. Python 3 defaults to a locale-specific encoding when encoding is omitted, which breaks on machines with different settings.

with open("data.txt", encoding="utf-8") as file:
    for line in file:
        process(line)

If bytes are not valid UTF-8, open() raises UnicodeDecodeError. If invalid bytes should be preserved visibly rather than stopping the read, errors="replace" substitutes the Unicode replacement character:

with open("data.txt", encoding="utf-8", errors="replace") as file:
    for line in file:
        process(line)

UTF-8 files exported from Windows editors sometimes start with a byte-order mark. Use encoding="utf-8-sig" so the BOM is stripped from the first line:

with open("data.txt", encoding="utf-8-sig") as file:
    for line in file:
        process(line)

Avoid errors="ignore" as a default: it silently drops bytes and can merge lines without warning.

Handle file reading errors

Catch the exceptions you can recover from instead of a broad except Exception.

import sys

def process(line):
    pass

try:
    with open("data.txt", encoding="utf-8") as file:
        for line in file:
            process(line)
except FileNotFoundError:
    print("Path does not exist", file=sys.stderr)
except PermissionError:
    print("Read permission denied", file=sys.stderr)
except UnicodeDecodeError as exc:
    print(f"Encoding failure at byte {exc.start}: {exc.reason}", file=sys.stderr)

FileNotFoundError means the path is missing. PermissionError means the OS blocked the read. UnicodeDecodeError means the bytes do not match the declared encoding. See FileNotFoundError in Python and try / except for fuller patterns.

Read a file line by line with pathlib

Path.open() is the same open() call with a pathlib entry point.

from pathlib import Path

path = Path("app.log")
with path.open(encoding="utf-8") as file:
    for line in file:
        process(line)

Path.open() provides the same streaming behavior through a Path object. Do not use path.read_text().splitlines() on very large files because read_text() loads the entire contents into one string first. If the path may not exist, handle FileNotFoundError when opening it rather than relying on a separate existence check.

Read a gzip file line by line

Compressed logs use the same iterator pattern on a text wrapper around gzip.

import gzip

with gzip.open("app.log.gz", "rt", encoding="utf-8") as file:
    for line in file:
        process(line)

Mode "rt" decompresses to text. gzip.open() decompresses incrementally, so iterating over it does not require loading the complete uncompressed log into memory.

Practical example: process a large log file

This mini-script streams a log, counts ERROR rows, prints them with line numbers, and reports the total.

import tempfile
from pathlib import Path

LOG = """2026-09-11 10:01:10 INFO Service started
2026-09-11 10:02:21 ERROR Database unavailable
2026-09-11 10:03:17 INFO Retrying
2026-09-11 10:03:22 ERROR Connection timed out
"""

def scan_log(path):
    error_count = 0
    with path.open(encoding="utf-8") as file:
        for line_number, line in enumerate(file, start=1):
            if "ERROR" in line:
                error_count += 1
                print(f"{line_number}: {line.rstrip()}")
    return error_count

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp) / "app.log"
    path.write_text(LOG, encoding="utf-8")
    total = scan_log(path)
    print(f"Total ERROR records: {total}")

The scanner prints 2: 2026-09-11 10:02:21 ERROR Database unavailable, then 4: 2026-09-11 10:03:22 ERROR Connection timed out, and finishes with Total ERROR records: 2. It keeps only a counter in memory, not a list of every match. If you need to retain matching lines, memory then scales with the number and size of those matches.

Common mistakes when reading files line by line

Which method should you use?

Summary

Reading a text file line by line in Python usually means opening the path with encoding="utf-8" and looping for line in file inside a with block. That pattern streams through buffered I/O without building a list of every row, which is what you want for logs and exports that do not fit comfortably in RAM.

readline() still matters when you must read a header separately or stop after an exact number of lines. readlines() and read().splitlines() are fine when the file is small and you genuinely need every line in a list. On a 500,000-line lab fixture measured in separate subprocesses, iteration peaked near 8.7 MiB RSS while readlines() peaked near 58 MiB.

Watch newlines and encodings as closely as you watch the loop itself. Keep meaningful indentation with removesuffix("\n"), pick utf-8-sig when a BOM is present, and catch FileNotFoundError, PermissionError, and UnicodeDecodeError on paths you do not fully control. When the job is grep-style filtering or early exit on a match, stay in the same streaming loop and avoid collecting lines you will throw away.

References

• Python open() function
• Python tutorial: methods of file objects
• Python io — core tools for working with streams
• Python pathlib — reading and writing files
• Python gzip — read and write GNU zip files
• Python Unicode HOWTO