An E-Commerce Platform Achieves Seamless AWS Data Migration to Enhance Availability, Lower Costs, and Meet Evolving User Needs

Solutions

• We used Amazon EMR clusters to migrate 150 TB of data from GCP BigQuery to Amazon S3. 
• The Amazon EMR cluster was set up on a VPC with spot instances for cost savings. 
• Spot Amazon EC2 instances reduced costs by up to 90% compared to on-demand instances. 
• We migrated 82 GCS Coldline data buckets to a single Amazon S3 bucket as per client requirements. 
• Custom Python scripts on Amazon EMR Notebooks validated table data before and after migration. 
• AWS Glue was created with Terraform to auto-configure databases and tables for Amazon Athena. 
• Amazon S3 was used for uploading sales orders, inventory, and trends data. 
• Amazon Glue ensured availability, durability, and scalability for ETL. 
• Processed data from AWS Glue jobs were loaded into Amazon Redshift for reporting and business intelligence. 
• Created a custom PySpark file for data migration with configurable source and destination. 
• Wrote a Python script to assign PySpark jobs to the Amazon EMR cluster based on CSV values. 
• Used DistCp for GCS to Amazon S3 data migration, automated through Python scripting.