golang Reading Files in Go (Read File Line by Line)

[Linux Hint]

File operation is probably one of the most common operation when it comes to program, except for dealing with bugs.

In Go, you will encounter such instances where you need to read a file line by line especially when dealing with text files.

In this tutorial, we will walk you through all the methods and techniques on how we can read a file line by line.

We start with the basic file reading techniques and progress to the more advanced features such as bufio and scanner libraries. We will also cover an example to help cement our understanding.

Sample File

Before we dive into the step of reading a file line by line, let us set up a basic file that we will use for demonstration purposes.

In our case, we use a basic Terraform configuration as shown in the following output:

# Define the AWS provider configuration
provider "aws" {
  region = "us-east-1"
}
# Create a VPC
resource "aws_vpc" "example_vpc" {
  cidr_block = "10.0.0.0/16"
  enable_dns_support = true
  enable_dns_hostnames = true
}

# Create two subnets in different availability zones
resource "aws_subnet" "example_subnet1" {
  vpc_id                  = aws_vpc.example_vpc.id
  cidr_block              = "10.0.1.0/24"
  availability_zone       = "us-east-1a"
  map_public_ip_on_launch = true
}
resource "aws_subnet" "example_subnet2" {
  vpc_id                  = aws_vpc.example_vpc.id
  cidr_block              = "10.0.2.0/24"
  availability_zone       = "us-east-1b"
  map_public_ip_on_launch = true
}
# Create a security group for the EC2 instance
resource "aws_security_group" "example_sg" {
  name        = "example-sg"
  description = "Security group for EC2 instance"
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
# ---- more ---------------------

 
The example file represents a Terraform configuration to set up an EC2 with a load balancer.

Open the File in Golang

Before we can read a file line by line, we need to open the file. Luckily, Go provides us with a simple and intuitive method of opening the files using the OS package.

An example code is as follows:

package main
import (
    "fmt"
    "os"
)

func main() {
    fp := "./aws.tf"
    // Open the file for reading
    file, err := os.Open(fp)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer file.Close()
    // File is now open and ready for reading
}

 
In the given example, we start by importing the packages that we require. In our case, we just need the fmt and OS packages.

Next, we specify the path to the file that we want to open and save it to the “fp” variable. In our case, we want to open the “aws.tf” file.

Finally, we use the os.Open() function to open the file and check any error that might occur when attempting to open the file.

Read the File Line by Line in Golang

Now that we have the target file open and ready to read, we can proceed and discuss the methods and techniques of reading the file line by line.

Using the Bufio Package

One of the most powerful packages in the Go ecosystem is the “bufio” package. This package implements the buffered I/O operations. It wraps the “io.Reader” and “io.Writer” object, creating another object that implements the interface but with buffer support.

Method 1: Using Bufio.Scanner

We can use the scanner type from this package to read a file line by line as demonstrated in the following example code:

package main
import (
    "bufio"
    "fmt"
    "os"
)
func main() {
    fp := "./aws.tf"
    file, err := os.Open(fp)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer file.Close()
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        line := scanner.Text()
        fmt.Println(line)
    }
    if err := scanner.Err(); err != nil {
        fmt.Println("Error:", err)
    }
}

 
To use the scanner type, we start by creating a Scanner using the bufio.NewScanner(file) method where the file refers to the file that we opened for reading.

We then use a “for” loop to iterate over the lines of the file using the scanner.Scan() method.

Finally, we extract the text of each line using the scanner.Text() method and print it to the console.

Method 2: Using Bufio.NewReader

Another method that we can use is the NewReader() method from the “bufio” package. This allows us to read the lines of the file using the ReadString() function.

An example is as follows:

package main
import (
    "bufio"
    "fmt"
    "os"
)
func main() {
    fp := "./aws.tf"
    file, err := os.Open(fp)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer file.Close()
    reader := bufio.NewReader(file)
    for {
        line, err := reader.ReadString('\n')
        if err != nil {
            break
        }
        fmt.Print(line)
    }
}

 
In this approach, we create a “bufio.Reader” using the bufio.NewReader(file).

We then use a “for” loop to read the lines using the reader.ReadString(‘\n’). The loop continues to read the file until an error occurs. For example, when we reach the end of the file.

Using the Io Package

Another package that comes in handy when dealing with I/O operations in Go is the “io” package. This package provides basic interfaces for I/O primitives. The primary task of the package is to wrap the existing implementations such as primitives from the OS package into shared public interfaces that abstract the functionality.

 

Method 1: Using ReadString

In the “io” package, we can use the ReadString() function to read a file line by line as shown in the following example:

package main
import (
    "fmt"
    "io"
    "os"
)
func main() {
    fp := "./aws.tf"
    file, err := os.Open(fp)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer file.Close()
    buf := make([]byte, 1024)
    for {
        n, err := file.Read(buf)
        if err != nil {
            if err == io.EOF {
                break
            }
            fmt.Println("Error:", err)
            return
        }
        data := buf[:n]
        fmt.Print(string(data))
    }
}

 
In this implementation, we start by opening a file for reading as we did in the previous examples.

We then create a buffer called “buf” to read a chunk of data from the file.

Lastly, we use a “for” loop with the file.Read() function to read the data into the buffer. We also check for “io.EOF” to determine when we reached the end of the file and terminate the loop.

Method 2: Using ReadLine

We also have access to the “ReadLine” function from the “io” package which can come in handy when we need to read a file line by line.

An example is as follows:

package main
import (
    "fmt"
    "io"
    "os"
)
func splitLines(data []byte) [][]byte {
    var lines [][]byte
    for _, b := range data {
        if b == '\n' {
            lines = append(lines, []byte{})
        } else {
            if len(lines) == 0 {
                lines = append(lines, []byte{})
            }
            lines[len(lines)-1] = append(lines[len(lines)-1], b)
        }
    }
    return lines
}

func main() {
    fp := "./aws.tf"
    file, err := os.Open(fp)
    if err != nil {
        fmt.Println("Error:", err)
        return
    }
    defer file.Close()
    reader := io.Reader(file)
    buf := make([]byte, 1024)
    for {
        n, err := reader.Read(buf)
        if err != nil {
            if err == io.EOF {
                break
            }
            fmt.Println("Error:", err)
            return
        }
        data := buf[:n]
        lines := splitLines(data)

        for _, line := range lines {
            fmt.Println(string(line))
        }
    }
}

 
In this case, we create a buffer and read the chunks of data from the file using the reader.Read() function.

We then define a splitLines() function whose purpose is to split the data into lines by detecting when a new character is in the buff.

Finally, we print the contents of the file.

Conclusion

In this tutorial, we learned a lot about file operations in Go. We learned how to use the “bufio” package, the Scanner type, and the NewReader. We also learned how to use the methods such as the “ReadString” and “ReadLine” functions to read a file line by line.

View the full article