EASY STEPS FOR AWS S3 CROSS-ACCOUNT REPLICATION

Introduction

Amazon provides a way of storing an unlimited number of objects and a reliable storage platform in serving the need of the Internet workloads through a service called Amazon Simple Storage Service (S3). Amazon S3 allows the storage of large sets of unstructured data.

AWS S3 provides the most durable and reliable storage service, which is highly economical. Also, Amazon states, “if you store 10,000,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000 years.” One of the unique features of S3 is that it also offers different storage classes according to the changing or unchanging, known or unknown, predictable or unpredictable access patterns. The other S3 storage classes are S3 Intelligent-Tiering, S3 Standard, S3 Standard-Infrequent Access, S3 One  Zone-Infrequent Access, and S3 Glacier. With this, S3 also provides S3 Same-Region Replication(replicate objects between the buckets in the same AWS region) and Cross-Region Replication(replicates objects from a source S3 bucket to destination buckets in different AWS regions). We can copy all or particular required objects from one S3 bucket to another bucket in the same account or other accounts.

Prerequisites

• Source and Destination Account
• Objects to be copied from the source bucket
• AWS CLI (If not, it will be informed how to install CLI in the later section)
• Appropriate Bucket Policy for Account A
• IAM Policy for IAM user in Account B

Guide to copy S3 objects across AWS Accounts

Copying objects from one bucket to another in the same account is simple. You must log in to AWS CLI using the Account credentials and create two buckets in the same account. Upload objects in one bucket, and for copying them to the other target bucket in the same account, run the following command:

However, copying objects across S3 from one account to another is quite tedious. The entire process is described in the below diagram.

Step by Step Guide

Step 1- Create a Source Bucket in Account A and upload files that need to be replicated in another account.

Step 2- Create a Destination Bucket in Account B where you want the objects to be replicated.

Step 3- Assigning appropriate Bucket Policy to the bucket in Account A from where the data is being replicated.

Bucket Policy Account A: Allows the Destination Bucket in Account B to list and get objects from the source bucket in Account A.

Step 4- Create an IAM user in the Destination Bucket. IAM user is a specific entity that      receives its credentials (Access Key ID and Secret Key)

Step 5- Provide the IAM user with an appropriate IAM Policy that defines the Permissions granted to the IAM user. Allows to list and get objects from source and destination buckets.

Step 6- Install and Configure AWS CLI using the IAM user credentials in Account B.

Step 7- Use the Sync Command to copy that source bucket objects from Account A to the destination bucket in Account B using the command below. Sync command syncs things in a specified bucket under another bucket by copying S3 objects.

In CLI, to verify if the files got copied successfully, run the following command:

Also, you can check in the console that in Account B, the destination bucket contains the data you copied from account A.

Installation of CLI in Windows

AWS CLI is a tool that manages all the AWS services from a terminal. AWS CLI helps create, manage, and destroy AWS resources; thus, it is a very powerful tool and saves a lot of time when managing AWS resources. We can install CLI for all Windows, Linux, and MACOS users, but here I will be telling only CLI installation in Windows.

1. Install 64-bit Windows Installer

2. After the installation is complete, open the Command prompt and give the following command to check if the CLI is properly installed.
aws –version

3. How to login into AWS CLI- Previously, while creating Identity and Access Management user, we had to choose “Access Key- Programmatic Access,” which helps to generate Access Key ID and Secret Access Key for CLI. (You can generate new credentials within AWS IAM if you do not have one)

4. While creating a User, you will get the Access Key and Secret Access Key ID. Note the Access Key ID and Secret Access Key

5. To configure AWS CLI, give the following command in the command prompt:  AWS configureAWS CLI will ask you for the below four information. Provide access Key ID, and Secret Access Key copied from above and also the region of the bucket (e.g. ap-south-1 //for Mumbai region)

Takeaway

In this blog, we have seen how to copy objects from one account’s S3 Source bucket to another account’s S3 Destination bucket using AWS CLI. It is easiest to copy objects when we have our target bucket in a different account. Using AWS CLI, we can copy multiple objects from multiple buckets, whether the destination bucket is in the same account or a different one

About CloudThat

CloudThat is the official AWS (Amazon Web Services) Advanced Consulting Partner, Microsoft Gold Partner, Google Cloud Partner, and Training Partner helping people develop knowledge of the cloud and help their businesses aim for higher goals using best-in-industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Explore our consulting here.

If you have any queries regarding AWS S3, S3 cross account replication, and any other AWS services, drop a line in the below comments section. I will get back to you at the earliest.