Golang Read File (Line by Line, Entire File, Large Files Explained)

Golang Read File (Quick Cheat Sheet)

Read Entire File (os.ReadFile)

Use os.ReadFile when you want to read the complete file content into memory in a simple and efficient way.

package main

import (
    "fmt"
    "os"
)

func main() {
    data, err := os.ReadFile("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(string(data))
}

This method is best suited for small to medium-sized files where loading the entire content into memory is acceptable.

Read File Line by Line (bufio.Scanner)

Use bufio.Scanner when you want to read a file line by line, especially for large files or logs.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }
}

This approach is memory-efficient and commonly used for processing large files incrementally.

How to Read File in Golang

Reading a file in Golang typically involves opening the file and then choosing the appropriate method depending on your use case, such as reading the full content or streaming data.

Open File using os.Open

Use os.Open to open a file for reading. It returns a file descriptor that can be used with different reading methods.

package main

import (
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer file.Close()

    fmt.Println("File opened successfully")
}

Opening a file is usually the first step before performing operations like reading line by line or reading in chunks.

Read Entire File using os.ReadFile (Recommended)

Use os.ReadFile when you want a quick and modern way to read the entire file content without manually opening and closing the file.

package main

import (
    "fmt"
    "os"
)

func main() {
    data, err := os.ReadFile("file.txt")
    if err != nil {
        fmt.Println("Error:", err)
        return
    }

    fmt.Println("File content:\n", string(data))
}

This method is recommended for most use cases where file size is manageable and simplicity is preferred.

Read File Line by Line in Golang

Reading a file line by line in Golang is the most efficient approach when working with large files such as logs or datasets. It avoids loading the entire file into memory and processes content incrementally.

Using bufio.Scanner

Use bufio.Scanner when you want a simple and clean way to read a file line by line.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        fmt.Println("Error reading file:", err)
    }
}

This method is commonly used for reading log files, configuration files, or any large text file line by line.

Using bufio.Reader

Use bufio.Reader when you need more control over how data is read, such as reading until a specific delimiter.

package main

import (
    "bufio"
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    reader := bufio.NewReader(file)

    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            break
        }
        fmt.Print(line)
    }
}

This approach is useful when working with custom formats or when you need more flexibility than bufio.Scanner.

Read Large Files Efficiently

Read File in Chunks

Reading a file in chunks allows you to process large files without loading the entire content into memory.

package main

import (
    "fmt"
    "os"
)

func main() {
    file, err := os.Open("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }
    defer file.Close()

    buffer := make([]byte, 1024)

    for {
        n, err := file.Read(buffer)
        if err != nil {
            break
        }
        fmt.Print(string(buffer[:n]))
    }
}

This method is ideal for processing very large files or streaming data.

Avoid High Memory Usage

Avoid reading entire files into memory when working with large datasets. Instead, use streaming approaches like bufio.Scanner, bufio.Reader, or chunk-based reading.

Choosing the right method depends on your use case:

• Use os.ReadFile for small files
• Use bufio.Scanner for line-by-line processing
• Use chunk reading for very large files

This ensures better performance and prevents memory-related issues in your Golang applications.

Read File into String / Bytes

In Golang, file content is typically read as a byte slice, which can then be converted into a string when needed. Choosing between string and byte representation depends on how you plan to process the data.

Convert File to String

Use os.ReadFile to read the file and convert the content into a string for easy manipulation and display.

package main

import (
    "fmt"
    "os"
)

func main() {
    data, err := os.ReadFile("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }

    content := string(data)
    fmt.Println(content)
}

This method is commonly used when working with text files such as configuration files, logs, or templates.

Read File into Byte Slice

Reading a file into a byte slice is useful when working with binary data or when you need precise control over file processing.

package main

import (
    "fmt"
    "os"
)

func main() {
    data, err := os.ReadFile("file.txt")
    if err != nil {
        fmt.Println(err)
        return
    }

    fmt.Println(data)
}

This approach is preferred when dealing with encoding, parsing, or streaming data operations.

Real-World Use Cases

Read Logs File

file, _ := os.Open("app.log")
defer file.Close()

scanner := bufio.NewScanner(file)
for scanner.Scan() {
    fmt.Println(scanner.Text())
}

This pattern is commonly used in applications to process logs line by line efficiently.

Read Config (JSON / CSV)

data, _ := os.ReadFile("config.json")
fmt.Println(string(data))

This method is useful for loading configuration files or structured data into your application.

Process Large Data Files

file, _ := os.Open("data.txt")
defer file.Close()

buffer := make([]byte, 1024)

for {
    n, err := file.Read(buffer)
    if err != nil {
        break
    }
    fmt.Print(string(buffer[:n]))
}

This approach is ideal for processing large files in chunks without consuming excessive memory.

Common Errors and Fixes

When reading files in Golang, you may encounter common issues related to file paths, permissions, or limitations of certain libraries. Understanding these errors helps in debugging and writing reliable code.

file not found error

This error occurs when the file path is incorrect or the file does not exist in the specified location.

file, err := os.Open("missing.txt")
if err != nil {
    fmt.Println("Error:", err)
}

To fix this issue, ensure that:

• The file exists in the given path
• The correct relative or absolute path is used
• The working directory is correctly set

Using absolute paths or verifying the file location can help avoid this error.

permission denied

This error occurs when the program does not have sufficient permissions to read the file.

data, err := os.ReadFile("/root/secure.txt")
if err != nil {
    fmt.Println("Error:", err)
}

To resolve this issue:

• Check file permissions using ls -l
• Ensure the current user has read access
• Avoid restricted directories unless running with proper privileges

Reading files from accessible directories like the current folder or /tmp can prevent permission issues.

bufio.Scanner token too long

This error occurs when bufio.Scanner tries to read a line that exceeds its default buffer size.

scanner := bufio.NewScanner(file)
for scanner.Scan() {
    fmt.Println(scanner.Text())
}

If a line is too long, the scanner will return an error.

To fix this, increase the buffer size:

scanner := bufio.NewScanner(file)
buffer := make([]byte, 1024)
scanner.Buffer(buffer, 1024*1024)

for scanner.Scan() {
    fmt.Println(scanner.Text())
}

This allows the scanner to handle larger lines without failing.

Summary

Golang provides multiple ways to read files, including reading the entire file, processing line by line, or reading in chunks. The right method depends on the file size and use case. Using os.ReadFile is ideal for small files, while bufio.Scanner, bufio.Reader, and chunk-based reading are better suited for large files and streaming data.

Understanding file handling patterns and common errors ensures that your applications remain efficient, reliable, and easy to maintain.

Official Documentation

• os.ReadFile Documentation
• bufio.Scanner Documentation
• os.Open Documentation