Seamlessly Transfer Data Across Applications with Amazon AppFlow

Introduction

Amazon AppFlow is a service that allows you to automate data transfer between different software-as-a-service (SaaS) and AWS services. It’s a fully managed integration service that enables you to transfer large amounts of data without providing system resources. With AppFlow, you can easily catalog your data, making it simpler to discover and share across AWS analytics and machine learning services.

Underlying Principle

Amazon AppFlow enables you to automate bi-directional data flows between various Software-as-a-Service (SaaS) applications and Amazon Web Services (AWS) with just a few clicks. You can run data flows at your preferred frequency on a schedule, in response to a business event, or on demand. With Amazon AppFlow, you can simplify the data preparation process by using transformations, partitioning, and aggregation. The tool also automates the preparation and registration of your schema with the AWS Glue Data Catalog, making it easier for you to discover and share data with AWS analytics and machine learning services.

Why Amazon Appflow?

• Simplified Data Integration: AppFlow eliminates the need for custom scripts and manual data transfers, making it easy to integrate your SaaS applications and AWS services without needing to write any code.
• Built-in Security: AppFlow ensures secure data transfer by using encryption and tokenization. It also provides full control over the data flow and access, enabling you to manage access permissions, roles, and security policies.
• Flexible Mapping and Transformation: With AppFlow, you can map and transform data to match the target format, schema, and structure. This flexibility allows you to tailor the data integration process to your needs.
• Monitoring and Alerting: AppFlow provides a centralized dashboard for monitoring data flows and detecting errors. It also sends alerts and notifications when specific events or thresholds are triggered, allowing you to take corrective action quickly.
• Cost-Effective: AppFlow provides pay-as-you-go pricing, enabling you to only pay for what you use. This cost-effective approach eliminates the need for upfront investments or long-term commitments, easily integrating data across multiple applications and services without breaking the bank.

Steps to Move data from one application to another using Amazon AppFlow

Prerequisites
IAM Access: you must need an IAM access with AWS managed policy AmazonAppFlowFullAccess to the user, group, or role.

Step 1: Upload the data to Amazon S3.

• Create a unique bucket name and folder in the AWS management console.
• Download the sample data from AWS official website or use the link provided.

https://docs.aws.amazon.com/appflow/latest/userguide/samples/tutorial-account-data.zip

• Select your Amazon S3 bucket, go to the Objects tab, and click Upload.
• Choose the file you want to upload or select the tutorial-account-data.csv file if using the sample dataset.
• Click Upload to start the upload process

Step 2: Integrate the SAAS application with Amazon AppFlow.

Here we are using Salesforce as a SAAS connection.

To establish a link between Salesforce and Amazon AppFlow, follow these steps:

1. Go to the Amazon AppFlow console.
2. Click “Connections” in the navigation pane on the left-hand side of the console page.
3. Select “Salesforce” under the “Connectors” option.
4. Create a new connection by clicking on “Create connection”.
5. Provide a name for the connection.
6. Click on “Continue” after selecting the default options.
7. You will be prompted to log in if you’re not already in Salesforce.
8. Click on “Allow” to grant Amazon AppFlow access to your Salesforce account.

Step 3: move data from Amazon S3 to a Software-as-a-Service (SaaS)

1. Creating an AppFlow

Search AppFlow in AWS Console management. Under AppFlow, choose “Flows” and click “Create Flow” provide a Flow Name and its description.

2. Configure flow

To transfer data from S3 to Salesforce, select the Amazon S3 bucket with the prefix “source” and format set to CSV as the source. Then, choose Salesforce as the destination with the previously created connection. If using sample data, select the Salesforce Account object as the destination.

The method will vary based on the destination application to map data fields for transfer. For Salesforce, use default mapping options and map all fields directly. For other apps, manually map fields or upload a .csv file with mappings. Specify how to handle invalid data and move on to Add Filters.

Select the desired flow from the list to execute a demand flow and click the “Run flow” button. This will initiate the transfer of data and start the flow.

To add a data field to a flow, select “Flows”, select the relevant app flow, and click “Edit”. In the “Edit Data Fields” section, select “Add Validation”, choose the required fields, and save the changes.

Step 4: Transfer data from Salesforce to Amazon S3

1. To create an AppFlow, go to the AWS Console management and search for AppFlow. Then, select “Flows” and click “Create Flow.” Provide a name and description for the flow.

2. To configure a flow from Salesforce to Amazon S3 using Amazon AppFlow:

• Choose Salesforce as the source and select the previously created connection.
• Select “AccountChangeEvent” as the Salesforce object.
• Choose S3 as the destination and select the same bucket as the source with the “destination” prefix.
• Use the default mapping method to map fields and map all fields directly manually.
• Skip validation and filter sections.
• Review and create the flow, then activate it from the AppFlow console.

To confirm a successful data transfer from Amazon S3 to Salesforce:

• Edit a record in Salesforce and wait for about a minute.
• Refresh the AppFlow page.
• Check the timestamp to confirm the flow ran successfully.

For example, change the Rating of a record in the provided sample data from cold to hot.

• In the salesforce account – go to accounts
• Choose any of the sample data (Example3)

Step 5: To View transferred data.

• Go to Amazon S3, and select your bucket from the list.
• Next, select the destination folder and go to the flow folder.
• In this folder, there should be a file that you can download.
• After downloading the file, rename it with a descriptive name and locate it in your Downloads folder.
• Open the file to view the updated record.

Conclusion

Amazon AppFlow is highly scalable and can handle large volumes of data with ease. It is also highly secure and offers advanced encryption and authentication features to ensure the privacy and security of your data. With AppFlow, you can quickly set up data flows between different sources and destinations, including Salesforce, Marketo, Zendesk, Slack, and Amazon S3. You can also use AppFlow to transform data during the transfer process, map fields, and apply filters to ensure that only the required data is transferred. In conclusion, AWS AppFlow is an excellent choice for businesses that need to integrate data across different systems quickly and easily. Its powerful features, ease of use, and security make it an excellent choice for organizations of all sizes.

About CloudThat

CloudThat is also the official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft gold partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Drop a query if you have any questions regarding Amazon AppFlow and I will get back to you quickly.

To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.