Dynamic Sales Dashboard Automating Multi-Source Data Integration for Enhanced Insights

Solutions

• The goal of achieving a centralized data store for analytics is achieved using AWS Glue, AWS Lambda, and Amazon S3.
• AWS Lambda functions are triggered once every day to fetch sales data from Shopify, Zoho, Unicommerce, and Google Drive and store it in an Amazon S3 bucket as raw data in CSV format in different partitions.
• Historical data, which is in CSV format, is uploaded to Amazon S3 manually.
• Once raw sales data is available in the Amazon S3 bucket, then another lambda function is triggered to preprocess and store the data back in Amazon S3.
• AWS Glue job is created to process the raw CSV data from the 9 partitions and to create a sales register from the last month of sales data. This job is scheduled and runs five times every month.
• This AWS Glue job maps the fields among the files from each partition and appends the data to create one single file and store it in Amazon S3.
• Another AWS Glue job is also created to extract the dates and details of the canceled orders and store them in Amazon S3. This job is scheduled to run daily.
• Generated dashboards in Amazon QuickSight by using Amazon S3 as a data source.