Simplify Cloud Data Migration with AWS DataSync – Amazon’s Online Data Transfer Service

Introduction to AWS DataSync

There has been increased demand to back up our files to a cloud environment for long-term storage to cover any disaster-related incidents. Customers want to securely migrate their data with the help of reliable utility tools which help them to do so. Moreover, they want to automate these tasks through a reliable mechanism to transfer their data to the cloud-like AWS in S3, EFS, Windows file server, etc.

AWS DataSync helps connect your on-prem storage to S3 and much more with a reliable automation architecture. File share systems like NFS (network file system) and SMB (Server Message Block) can now be integrated into AWS DataSync to transfer your required files.

DataSync allows you to transfer all the files or only the changed data the next time you start transferring; it does so by using metadata related to previous data captured, which helps decrease the transfer size and the related time to transfer. AWS DataSync uses a DataSync agent, either installed on on-premises hardware such as VMware, Hyper-V, or an EC2 machine using AWS-provided AMI. This server helps in connecting the source endpoint of the local server to the target endpoint on S3 EFS, etc.

I will be showing a small but powerful setup where you can transfer the files into the S3 from SMB (Samba) server using AWS data DataSync.

Step to Covert your Linux machine into a Samba Server

Here I am using an Amazon Linux AMI in another VPC to act as a remote Samba server

1. Launch Linux EC2 of your choice here; I am using Amazon Linux to install the SMB server.
2. In the security group, open port 22and port 445 to anywhere, i.e., 0.0.0.0/0 range (later, we can change this to a dedicated IP)
3. It is an Amazon Linux 2 AMI; we need to follow the below steps to change the hostname to a user-friendly name
   
   sudo vi /etc/cloud/cloud.cfg

   1	sudo vi /etc/cloud/cloud.cfg

4. At the end of the file, add the below line and save
   
   preserve_hostname: true

   1	preserve_hostname: true

5. Change Hostname
   
   $ sudo hostnamectl set-hostname samba-server $ sudo yum update -y $ sudo reboot

   1 2 3

   $ sudo hostnamectl set-hostname samba-server $ sudo yum update -y $ sudo reboot

6. Install Samba, Samba-client, and cifs-utils
   
   $ sudo su - # yum install -y samba samba-client cifs-utils

   1 2	$ sudo su - # yum install -y samba samba-client cifs-utils

7. Configuration file changes
   
   # vim /etc/samba/smb.conf

   1	# vim /etc/samba/smb.conf

   
   
   security = user hosts allow = ip address of Data DataSync agent VM *** Add the loopback IP and the VPC starting ip as shown above interfaces = lo eth0 passdb backend = smbpasswd:/etc/samba/sambapasswd.txt printing = cups printcap name = cups load printers = yes cups options = raw [homes] comment = Home Directories valid users = %S, %D%w%S browseable = No read only = No inherit acls = Yes printers] comment = All Printers path = /var/tmp printable = Yes create mask = 0600 browseable = No [print$] comment = Printer Drivers path = /var/lib/samba/drivers write list = @printadmin root force group = @printadmin create mask = 0664 directory mask = 0775 [samba] comment = Development documentation read only = no available = yes path = /smbfolder public = yes valid users = sambauser write list = sambauser writable = yes browseable = yes

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

   security = user         hosts allow = ip address of Data DataSync agent VM   *** Add the loopback IP and the VPC starting ip as shown above         interfaces = lo eth0        passdb backend = smbpasswd:/etc/samba/sambapasswd.txt        printing = cups         printcap name = cups         load printers = yes         cups options = raw [homes]         comment = Home Directories         valid users = %S, %D%w%S         browseable = No         read only = No         inherit acls = Yes printers]         comment = All Printers         path = /var/tmp         printable = Yes         create mask = 0600         browseable = No [print$]         comment = Printer Drivers         path = /var/lib/samba/drivers         write list = @printadmin root         force group = @printadmin         create mask = 0664         directory mask = 0775 [samba]         comment = Development documentation         read only = no         available = yes         path = /smbfolder         public = yes         valid users = sambauser         write list = sambauser         writable = yes         browseable = yes

   
   Make sure to edit the config as above and change the host allow to your data DataSync agent to connect
8. We will now create a samba user to access the samba folder directory
   
   # useradd sambauser # passwd sambauser # smbpasswd -a sambauser # service smb restart

   1 2 3 4

   # useradd sambauser # passwd sambauser # smbpasswd -a sambauser # service smb restart

9. Create a directory for file share and give permissions
   
   # mkdir /smbfolder # chmod 777 /smbfolder

   1 2	# mkdir /smbfolder # chmod 777 /smbfolder

10. Restart and test with the data DataSync agent
    
    # service smb restart # testparm

    1 2	# service smb restart # testparm

Create a DataSync agent in EC2 in a VPC

You can choose your DataSync AMI using the below command

Create the agent in the DataSync in the AWS Portal by navigating to DataSync in the AWS Console

Place the Data DataSync agent ID in the agent address

Create a Task

• Choose location type as SMB server
• Choose the Data DataSync agent you configured
• Put IP of SMB server launched up earlier
• Share the name for the folder
• User and Password for SMB server user
• Choose destination location type as S3 / EFS/ NFS as per requirement
• Give the task a name and verify the options selected in the review screen

Start the transfer

Once the task is created you can start transferring files using the start tab above

Navigate to the S3 bucket and now you can find the files transferred

Summary

Data DataSync helps create a reliable connection between your on-premises storage to AWS S3 / EFS/ NFS servers to transfer data. You can also change the storage classes to store the data when it comes to S3 as the destination. By clicking start on tasks, you can start transferring all the files or files that are not yet transferred. You can also delete the files in AWS storage by choosing the delete option to DataSynchronize your on-premises drives completely.

About CloudThat

Here at CloudThat are the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Feel free to drop a comment or any queries that you have regarding AWS services, cloud adoption, or consulting and we will get back to you quickly. To get started, go through our Expert Advisory page and Managed Services Package that is CloudThat’s offerings.

FAQs

1. Which file system is from where we can replicate our data to AWS?

Ans: File systems such as NFS, SMB, and HDFS can be set up for on-premises storage locations. Moreover, Amazon EFS, Amazon FSx, and AWS S3 can also be made as source points for data capture.

2. Can we change the storage class when choosing the destination location to AWS S3?

Ans: Yes, at any point in time, once the AWS DataSync agent is set up, you can change the storage class for a new task where the destination is S3, such as Standard, Glacier, or Deep-archive.

3. Can we use CloudWatch to monitor Data DataSync tasks?

Ans: Yes, you can monitor the files which are copied using AWSDataSync through AWS CloudWatch metrics.

WRITTEN BY Akshay Mishra