Amazon S3 Bucket Cross-Region and Cross-Account Replication Using Terraform

Overview

Ensuring disaster recovery and data redundancy is essential in today’s digital environment. Through object replication between various AWS regions, Amazon S3 cross-region replication improves data availability and durability. Managing this process across several AWS accounts can be difficult but Terraform makes it easier by automating resource provisioning and maintenance.
This blog post explains how Terraform can be used to build up Amazon S3 cross-region and cross-account replication. We will walk you through setting up replication rules, configuring AWS IAM roles for cross-account access, and generating Amazon S3 buckets in several locations. This blog will cover how to use Terraform to achieve Amazon S3 replication.

Introduction

The robust feature of Amazon S3’s cross-region replication lets you automatically duplicate Amazon S3 objects across various AWS regions. This is especially helpful for guaranteeing disaster recovery, fulfilling compliance requirements, and enhancing data availability.

You may safely duplicate data between Amazon S3 buckets in various AWS accounts when you combine this with cross-account replication, increasing the security and flexibility of your cloud infrastructure.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Step-by Step Guide

Cross-account role: First, we need a cross-account role allowing our source account to modify and create objects and buckets in the destination account. For that, we will run terraform in our destination account, creating a role for our source account. You must set up a provider to set up the region in the destination account.

provider "aws" {
  region = "us-east-1"
}

provider "aws" {

region = "us-east-1"

}

Now we will create a role.

resource "aws_iam_role" "destination_assume_role" {
  name = "destination-account-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          AWS = "111111111111"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy" "destination_policy" {
  name = "destination_policy"
  role = aws_iam_role.destination_assume_role.id
  policy = jsonencode(
    {
    Version = "2012-10-17",
    Statement = [
      {
        Effect   = "Allow",
        Action   = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:ListBucket",
          "s3:GetBucketLocation",
          "s3:ListAllMyBuckets",
          "kms:CreateKey",
          "s3:CreateBucket",
          "S3:PutBucketVersioning",
          "S3:GetBucketPolicy",
          "S3:GetBucketAcl",
          "s3:*"
        ],
        Resource = [
          "*"
        ]
      }
    ]
  })
}

resource "aws_iam_role" "destination_assume_role" {

name = "destination-account-role"

assume_role_policy = jsonencode({

Version = "2012-10-17"

Statement = [

{

Action = "sts:AssumeRole"

Effect = "Allow"

Principal = {

AWS = "111111111111"

}

]

})

}

resource "aws_iam_role_policy" "destination_policy" {

name = "destination_policy"

role = aws_iam_role.destination_assume_role.id

policy = jsonencode(

{

Version = "2012-10-17",

Statement = [

{

Effect = "Allow",

Action = [

"s3:GetObject",

"s3:PutObject",

"s3:DeleteObject",

"s3:ListBucket",

"s3:GetBucketLocation",

"s3:ListAllMyBuckets",

"kms:CreateKey",

"s3:CreateBucket",

"S3:PutBucketVersioning",

"S3:GetBucketPolicy",

"S3:GetBucketAcl",

"s3:*"

Resource = [

"*"

]

}

]

})

}

You can do “terraform apply” here to create this role in the destination account. Make sure to replace 11111111 with your source account ID and run this terraform script using destination account credentials.

Now, we will start the process of replication in our source account and destination account

2. Provider conf.

provider "aws" {
  alias  = "destination"
  region = "us-west-2"

  assume_role {
    role_arn = aws_iam_role.destination_assume_role.arn
 }
}
data "aws_caller_identity" "source" {}

data "aws_caller_identity" "destination" {
  provider = aws.destination
}

provider "aws" {

alias = "destination"

region = "us-west-2"

assume_role {

role_arn = aws_iam_role.destination_assume_role.arn

}

data "aws_caller_identity" "source" {}

data "aws_caller_identity" "destination" {

provider = aws.destination

}

3. AWS KMS Keys: We need some keys for both accounts. Our destination key will be a bit special because there will be a policy that will allow the source account to access it.

resource "aws_kms_key" "source" {
  deletion_window_in_days = 30
  enable_key_rotation     = true
}

resource "aws_kms_key" "destination" {
  provider = aws.destination

  deletion_window_in_days = 30
  enable_key_rotation     = true

  policy = data.aws_iam_policy_document.destination_kms_key.json
}

data "aws_iam_policy_document" "destination_kms_key" {
  statement {
    principals {
      type = "AWS"
      identifiers = [
        data.aws_caller_identity.source.account_id
      ]
    }

    actions = [
      "kms:Encrypt"
    ]

    resources = ["*"]
  }

  statement {
    principals {
      type = "AWS"
      identifiers = [
        data.aws_caller_identity.destination.account_id
      ]
    }

    actions = [
      "kms:*"
    ]

    resources = ["*"]
  }
}

resource "aws_kms_key" "source" {

deletion_window_in_days = 30

enable_key_rotation = true

}

resource "aws_kms_key" "destination" {

provider = aws.destination

deletion_window_in_days = 30

enable_key_rotation = true

policy = data.aws_iam_policy_document.destination_kms_key.json

}

data "aws_iam_policy_document" "destination_kms_key" {

statement {

principals {

type = "AWS"

identifiers = [

data.aws_caller_identity.source.account_id

]

}

actions = [

"kms:Encrypt"

]

resources = ["*"]

}

statement {

principals {

type = "AWS"

identifiers = [

data.aws_caller_identity.destination.account_id

]

}

actions = [

"kms:*"

]

resources = ["*"]

}

4. Amazon S3 buckets: It is time to create buckets with new keys.

resource "aws_s3_bucket" "source" {
  bucket = "replication-test-source-buckettt"
  acl    = "private"

  versioning {
    enabled = true
  }

  server_side_encryption_configuration {
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.source.arn
      sse_algorithm     = "aws:kms"
    }
  }
}

replication_configuration {
  role = aws_iam_role.replication.arn

  rules {
    id     = "replicate"
    status = "Enabled"

    source_selection_criteria {
      sse_kms_encrypted_objects {
        enabled = true
      }
    }

    destination {
      account_id         = data.aws_caller_identity.destination.account_id
      bucket             = aws_s3_bucket.destination.arn
      storage_class      = "STANDARD_IA"
      replica_kms_key_id = aws_kms_key.destination.arn
    }
  }
}
}

resource "aws_s3_bucket" "destination" {
  provider = aws.destination

  bucket = "replication-test-destination-buckettt"
  acl    = "private"

  versioning {
    enabled = true
  }

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = aws_kms_key.destination.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

resource "aws_s3_bucket" "source" {

bucket = "replication-test-source-buckettt"

acl = "private"

versioning {

enabled = true

}

server_side_encryption_configuration {

rule {

apply_server_side_encryption_by_default {

kms_master_key_id = aws_kms_key.source.arn

sse_algorithm = "aws:kms"

}

replication_configuration {

role = aws_iam_role.replication.arn

rules {

id = "replicate"

status = "Enabled"

source_selection_criteria {

sse_kms_encrypted_objects {

enabled = true

}

destination {

account_id = data.aws_caller_identity.destination.account_id

bucket = aws_s3_bucket.destination.arn

storage_class = "STANDARD_IA"

replica_kms_key_id = aws_kms_key.destination.arn

}

resource "aws_s3_bucket" "destination" {

provider = aws.destination

bucket = "replication-test-destination-buckettt"

acl = "private"

versioning {

enabled = true

}

server_side_encryption_configuration {

rule {

apply_server_side_encryption_by_default {

kms_master_key_id = aws_kms_key.destination.arn

sse_algorithm = "aws:kms"

}

5. AWS IAM Roles: An AWS IAM role in our source account is required so that Amazon S3 can access the destination bucket.

resource "aws_iam_role" "replication" {
  name               = "replication-test-role"
  assume_role_policy = data.aws_iam_policy_document.replication_role.json
}

resource "aws_iam_role_policy" "replication" {
  name   = "replication"
  role   = aws_iam_role.replication.id
  policy = data.aws_iam_policy_document.replication_policy.json
}

data "aws_iam_policy_document" "replication_role" {
  statement {
    principals {
      type        = "Service"
      identifiers = ["s3.amazonaws.com"]
    }

    actions = [
      "sts:AssumeRole"
    ]
  }
}

data "aws_iam_policy_document" "replication_policy" {
  statement {
    actions = [
      "s3:GetReplicationConfiguration",
      "s3:ListBucket"
    ]

    resources = [
      aws_s3_bucket.source.arn
    ]
  }

  statement {
    actions = [
      "s3:GetObjectVersionForReplication",
      "s3:GetObjectVersionAcl",
      "s3:GetObjectVersionTagging"
    ]

    resources = [
      "${aws_s3_bucket.source.arn}/*"
    ]
  }

  statement {
    actions = [
      "s3:ReplicateObject",
      "s3:ReplicateDelete",
      "s3:ReplicateTags"
    ]

    resources = [
      "${aws_s3_bucket.destination.arn}/*"
    ]
  }

  statement {
    actions = [
      "kms:Decrypt"
    ]

    resources = [
      aws_kms_key.source.arn
    ]
  }

  statement {
    actions = [
      "kms:Encrypt"
    ]

    resources = [
      aws_kms_key.destination.arn
    ]
  }
}

resource "aws_iam_role" "replication" {

name = "replication-test-role"

assume_role_policy = data.aws_iam_policy_document.replication_role.json

}

resource "aws_iam_role_policy" "replication" {

name = "replication"

role = aws_iam_role.replication.id

policy = data.aws_iam_policy_document.replication_policy.json

}

data "aws_iam_policy_document" "replication_role" {

statement {

principals {

type = "Service"

identifiers = ["s3.amazonaws.com"]

}

actions = [

"sts:AssumeRole"

]

}

data "aws_iam_policy_document" "replication_policy" {

statement {

actions = [

"s3:GetReplicationConfiguration",

"s3:ListBucket"

]

resources = [

aws_s3_bucket.source.arn

]

}

statement {

actions = [

"s3:GetObjectVersionForReplication",

"s3:GetObjectVersionAcl",

"s3:GetObjectVersionTagging"

]

resources = [

"${aws_s3_bucket.source.arn}/*"

]

}

statement {

actions = [

"s3:ReplicateObject",

"s3:ReplicateDelete",

"s3:ReplicateTags"

]

resources = [

"${aws_s3_bucket.destination.arn}/*"

]

}

statement {

actions = [

"kms:Decrypt"

]

resources = [

aws_kms_key.source.arn

]

}

statement {

actions = [

"kms:Encrypt"

]

resources = [

aws_kms_key.destination.arn

]

}

6. Destination bucket policy: We need to allow our new AWS IAM role to be replicated in our destination bucket.

resource "aws_s3_bucket_policy" "destination" {
  provider = aws.destination

  bucket = aws_s3_bucket.destination.id
  policy = data.aws_iam_policy_document.destination_bucket_policy.json
}

data "aws_iam_policy_document" "destination_bucket_policy" {
  statement {
    principals {
      type = "AWS"
      identifiers = [
        aws_iam_role.replication.arn
      ]
    }

    actions = [
      "s3:ReplicateDelete",
      "s3:ReplicateObject"
    ]

    resources = [
      "${aws_s3_bucket.destination.arn}/*"
    ]
  }

  statement {
    principals {
      type = "AWS"
      identifiers = [
        aws_iam_role.replication.arn
      ]
    }

    actions = [
      "s3:List*",
      "s3:GetBucketVersioning",
      "s3:PutBucketVersioning"
    ]

    resources = [
      aws_s3_bucket.destination.arn
    ]
  }
}

resource "aws_s3_bucket_policy" "destination" {

provider = aws.destination

bucket = aws_s3_bucket.destination.id

policy = data.aws_iam_policy_document.destination_bucket_policy.json

}

data "aws_iam_policy_document" "destination_bucket_policy" {

statement {

principals {

type = "AWS"

identifiers = [

aws_iam_role.replication.arn

]

}

actions = [

"s3:ReplicateDelete",

"s3:ReplicateObject"

]

resources = [

"${aws_s3_bucket.destination.arn}/*"

]

}

statement {

principals {

type = "AWS"

identifiers = [

aws_iam_role.replication.arn

]

}

actions = [

"s3:List*",

"s3:GetBucketVersioning",

"s3:PutBucketVersioning"

]

resources = [

aws_s3_bucket.destination.arn

]

}

We are good to go now, and you can do the final terraform apply to set up your cross-region and cross-account replication. After applying, you can test this setup by putting objects into your source bucket, and they will automatically get replicated to your destination bucket.

Conclusion

Using Terraform to set up Amazon S3 cross-region, cross-account replication improves compliance, disaster recovery, and data redundancy. You have now successfully created Amazon S3 buckets, configured AWS IAM roles, and defined replication rules using Terraform with this article.

You may improve availability and durability by automating this process and ensuring consistent data replication across AWS regions and accounts. Maintain a close eye on your replication status and expenses, and for best results, follow security best practices.

Drop a query if you have any questions regarding Amazon S3 cross-region, cross-account and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. Can I replicate objects between Amazon S3 buckets in different AWS accounts?

ANS: – Yes, cross-account replication allows you to duplicate objects between Amazon S3 buckets in several AWS accounts. Enabling rights for replication across accounts entails configuring IAM roles and rules. Terraform streamlines this procedure by automating the development and setup of AWS IAM roles and policies.

2. How does Amazon S3 cross-region replication impact data transfer costs?

ANS: – Data transport expenses may be incurred when sending data between AWS regions for Amazon S3 cross-region replication. These expenses vary depending on the AWS regions and volume of data copied, among other things. When planning your replication arrangement, you must consider these costs and keep a close eye on them to save on expenditures.

3. Can I use encryption with Amazon S3 cross-region replication?

ANS: – Yes, you can use encryption to protect your replicated data while using Amazon S3 cross-region replication. Both client-side encryption, which encrypts data before uploading it to Amazon S3, and server-side encryption (SSE) are supported by Amazon S3. To ensure that duplicated objects are encrypted per your security standards, you can provide encryption settings while configuring replication rules with Terraform.

WRITTEN BY Shakti Singh Chouhan

Shakti Singh is a Research Associate (Infra, Migration, and Security) at CloudThat. He is a passionate learner committed to learning new things every day. Shakti enjoys sharing his knowledge with others. He likes singing and listening to music in his leisure time.