Transferring File Data Across AWS Regions Using AWS DataSync

Overview

To maintain business continuity and provide low-latency access for applications, many Amazon EFS customers in other AWS Regions must synchronize their file systems across AWS Regions. AWS DataSync is an effective technology that allows private, secure data replication across Amazon EFS file systems without requiring data to travel over the open internet.

AWS DataSync makes it simple to set up a data replication procedure that uses private IP addresses inside your Amazon VPC, guaranteeing safe and effective data transport between regions. Organizations looking to lower latency for applications running in geographically dispersed settings and maintain strong disaster recovery plans will find this feature useful.

Introduction

Because businesses depend increasingly on data resilience and accessibility, it is now crucial to replicate Amazon EFS file systems across AWS Regions for low-latency access and business continuity. By facilitating safe and effective data transfers without using the public internet, AWS DataSync streamlines this procedure. AWS DataSync guarantees secure and efficient data replication between Amazon EFS file systems by utilizing private IP addresses found in Amazon VPCs. In this post, we will walk you through setting up AWS DataSync for cross-region replication, giving you a reliable answer to your data management requirements.

Need for this method

We clone two filesystems using utilities like cp and rsync, which function well. However, they are single-threaded, which may be fixed with programs like fpsync, MCP, and GNU parallel, as well as a combination of these programs that speed up transfers.

However, we must use Amazon EC2 machines or servers to transport the data for this operation.

AWS has provided a solution for this called AWS Data Sync, which employs a data sync agent to transport data over some time. There are still a few limitations with this solution, such as the amount of data that may sync in a day relies on your consumption and burst limit, and you must contact AWS Support to have your restrictions raised. The methods that worked for encrypted and unencrypted Amazon EFS file sync between AWS regions using sample data are listed below.

Prerequisites

• Suppose I want to synchronize data from the Mumbai region to the Ohio region. So, two Amazon VPCs have to be created in source and destination regions with different CIDRs.
• Spin up another instance in the Ohio(Destination) region.

Step-by-Step Guide

Creating Source and Destination Amazon VPC

Each region requires its Amazon Virtual Private Cloud (VPC) with unique CIDR ranges for network isolation to prevent clashes.

• Create Amazon VPC in Mumbai Region (ap-south-1) with CIDR range 0.0.0/16.

• Follow the same process to create another Amazon VPC in the Ohio Region (us-east-2) with a different CIDR block (20.0.0.0/16).

Amazon VPC Peering

Amazon VPC Peering is required to establish a network connection between the two Amazon VPCs, allowing them to communicate.

Amazon EFS Configuration:

Create an Amazon EFS file system in the source region (Mumbai).

Once the Amazon EFS is created, it must be mounted on the Amazon EC2 instance.

Install NFS client.

• Connect to the Amazon EC2 instance.
• Update and restart.

Reconnect to your Amazon EC2 instance after the reboot.

• Bring up the NFS client.

Test by mounting the file system on your Amazon EC2 instance.

• Create a directory called “efs-mount-point“.

• Get the Amazon EFS file system mounted.

• We can start creating files after your Amazon EC2 instance has installed the Amazon EFS file system.

1. Switch the directory.

2. List the contents of the directory.

3. Make a text file.

Note: Reboots do not preserve the file system you mounted. The fstab file may be used to remount the directory automatically.

Configure the Security group in the source and destination Amazon EFS:

The security group rules for the Amazon EC2 instances and the Amazon EFS mount targets must be configured for the Amazon EC2 instances and the Amazon EFS file system to interact.

Source Amazon EFS:

Destination Amazon EFS:

Creating locations in source and destination

Choose Amazon EFS as the location type for the source location, select the Amazon EFS file system set up in the Mumbai area, and provide the correct mount path to sync the data.

When the final destination, the goal is Amazon EFS, choose Amazon EFS and direct it to the Ohio-created Amazon EFS file system. This completes the AWS DataSync configuration for both the source and destination locations.

Create a task in the Source region

We can build a task to move data from the Mumbai EFS to the destination (either Ohio EFS) once the source and destination locations have been established in AWS DataSync. Choose the source location (Mumbai EFS) and the destination location (Ohio) when creating a new job in the AWS DataSync console. Task scheduling, file filters to include or exclude particular files, and transfer features like bandwidth limitations, file ownership, and data verification are all configurable. Once set up, you can execute the task to begin synchronization immediately or schedule it to run regularly for ongoing syncing.

Verify that the file has been sent

Conclusion

To sum up, AWS DataSync offers a smooth and effective means of synchronizing data across geographical boundaries, such as Ohio and Mumbai.

Drop a query if you have any questions regarding AWS DataSync and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, AWS GenAI Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner,  AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, Amazon ECS Service Delivery Partner, AWS Glue Service Delivery Partner, Amazon Redshift Service Delivery Partner, AWS Control Tower Service Delivery Partner, AWS WAF Service Delivery Partner and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.