Terraform delivers launch-day support for Amazon S3 Express

[Hashicorp]

In partnership with Amazon Web Services (AWS), we are excited to announce launch-day support for Amazon S3 Express One Zone storage class in the HashiCorp Terraform AWS provider. Launched at AWS re:Invent 2023, Amazon S3 Express is a high-performance storage solution for users requiring ultra-low latency and large storage volumes for their applications. Applications can easily access S3 Express through the existing S3 API, allowing for quick and seamless integration with many other AWS services that are supported by the Terraform AWS provider.

High-performance storage

S3 Express is a new bucket type, built from the ground up to deliver single-digit millisecond

response times for the most frequently accessed datasets. Organizations with compute-intensive big data workloads such as autonomous vehicle data, financial risk modeling, real-time online advertising, and machine-learning training and inference can easily provision the new bucket type using Terraform.

Three key features support S3 Express’ performance goals:

A low-latency zonal storage class. S3 Express optimizes for speed by replicating and storing data within the same Availability Zone as your compute workloads.

A new bucket type with a hierarchical namespace. This new bucket type has a hierarchical namespace and stores object key names in a directory-like manner, as opposed to the flat key structure of traditional S3 buckets.

A new fast-authorization API. S3 Express introduces a new session-based authorization capability that reduces the latency associated with S3 request authorizations. This new capability can be used to create and periodically refresh your connection sessions to the new bucket type.

Configuring S3 Express in the Terraform AWS provider

To set up S3 Express in the Terraform AWS provider, use the new aws_s3_directory_bucket resource. You also need to use the existing resources to manage the new S3 Express buckets:

•  aws_s3_bucket_policy
•  aws_s3_object

To try out this feature, you need:

• Terraform 0.12 or greater installed
• Version 5.28 of the Terraform AWS provider

To create an S3 Express bucket, apply the following configuration:

resource "aws_s3_directory_bucket" "example" {
  # S3 directory bucket names must follow the format
  # ----x-s3
  # where  is the Availability Zone ID
  # 	https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#az-ids
  # You can use the aws_availability_zone data source to obtain the AZ ID.
  bucket = "example--usw2-az2--x-s3"

  location {
    name = "usw2-az2"
  }

  # All objects should be deleted from the bucket when the bucket is destroyed
  # so that the bucket can be destroyed without error.
  force_destroy = true
}

Here is an example configuration of an S3 Express bucket policy and object:

data "aws_partition" "current" {}
data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "example" {
  statement {
    effect = "Allow"

    actions = [
      "s3express:*",
    ]

    resources = [
      aws_s3_directory_bucket.example.arn,
    ]

    principals {
      type        = "AWS"
      identifiers = ["arn:${data.aws_partition.current.partition}:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
  }
}

resource "aws_s3_bucket_policy" "example" {
  bucket = aws_s3_directory_bucket.example.bucket
  policy = data.aws_iam_policy_document.example.json
}

# aws_s3_object is used with directory buckets just like general purpose buckets.
# Note: tags are not supported for objects in directory buckets.
resource "aws_s3_object" "example" {
  bucket = aws_s3_directory_bucket.example.bucket
  key    = "example"
  source = "path/to/file"
}

Key benefits of Terraform support for Amazon S3 Express

As the Terraform AWS provider download count tops 2 billion, AWS and HashiCorp continue to develop new integrations to help customers work faster, use more services and features, and enjoy developer-friendly ways to provision cloud infrastructure. Launch-day support of the Amazon S3 Express One Zone storage class in the Terraform AWS provider allows practitioners to immediately begin managing this new offering in their existing Terraform workflows. Here are two main benefits of Terraform’s launch-day support for S3 Express:

1. Increased productivity: Developers can leverage their existing Terraform knowledge and familiar syntax to manage Amazon S3 Express buckets using the Terraform AWS provider.
2. Cost efficiency: Platform teams can immediately take advantage of this new offering to accelerate AI/ML and other latency-sensitive compute workloads in AWS while providing a seamless user experience to downstream developers.

Learn more about AWS and HashiCorp

To learn more about Amazon S3 Express One Zone storage class support in Terraform, please refer to the documentation. To learn the basics of Terraform using the AWS provider, follow the hands-on tutorials for getting started with Terraform on AWS on our developer education platform.

Please share any bugs or enhancement requests with us via the Terraform AWS provider repository on GitHub. We look forward to your feedback and want to thank you for being such a great community!

If you are completely new to Terraform, sign up for Terraform Cloud and get started using the Free offering today.

View the full article