Real-time Data Streaming: Amazon RDS MySQL CDC with Apache Kafka on Amazon EC2, Debezium, and AWS Lambda – Part 1

Overview

Traditionally, users had to establish connections through a Bastion host with a public IP address, which was set up by their administrators using an Internet Gateway (IGW) within their VPC. This involved using port forwarding to reach the intended destination. However, introducing the EIC Endpoint eliminates the need for an IGW, a public IP address on the resource, a bastion host, or any additional agent for establishing connections. 

This solution is explained in 3 parts, and this is the 1^st part which contains the creation of VPC and installation of an Amazon EC2 machine (private) that can be SSH without using Bastion host that is with using Amazon EC2 Instance Connect (EIC) Endpoint.

Pre-requisites for Amazon RDS MySQL CDC with Apache Kafka on Amazon EC2, Debezium, and AWS Lambda

• Amazon EC2 – Apache Kafka and Debezium
• RDS MySQL – Database
• AWS Lambda – Apache Kafka as a source event and sending CDC event to RDS MySQL with Dynamic attribute creation.

Amazon EC2

Amazon EC2 (Elastic Compute Cloud) is a highly flexible and scalable cloud computing service provided by Amazon Web Services (AWS).

Amazon EC2 provides a reliable and secure infrastructure for deploying and managing applications in the cloud, making it a popular choice for organizations of all sizes. We will install Apache Kafka and Debezium on Amazon EC2.

Steps to Create an Amazon VPC

Step 1: Create Amazon VPC with its subnets, route tables, Internet Gateway, and NAT Gateways (for installation of resources on Private Instance, later will be removed)

Go to Amazon VPC on AWS, click on Create VPC, select VPC and more, enter the name of VPC, Number of Availability Zones (AZs) 2, Number of public subnets 2, Number of private subnets 2, NAT gateways ($) 1, and VPC endpoints None.

Steps to Launch Amazon EC2

Step 1: Connect Amazon Private EC2 instance without using Bastion host that is with using Amazon EC2 EndPoint

• Let’s launch the Amazon EC2 instance
• Select Ubuntu Latest Version

Step 2: Select Instance Type t2.large(depends on workload), the key pair (if you don’t have to create one), the VPC, created earlier step, a private subnet, public IP disable, and security group set it to all traffic anywhere.

Step 3: Let’s create Amazon EC2 Endpoint to SSH into Amazon EC2 Private Instance

Use of Amazon EC2 Instance Connect (EIC) Endpoint

AWS recently unveiled a groundbreaking feature called the Amazon EC2 Instance Connect (EIC) Endpoint, designed to provide users with secure connectivity to their instances and other Amazon Virtual Private Cloud (Amazon VPC) resources via the internet.

Traditionally, users had to establish connections through a bastion host with a public IP address, which was set up by their administrators using an Internet Gateway (IGW) within their VPC. This involved using port forwarding to reach the intended destination. However, introducing the EIC Endpoint eliminates the need for an IGW, a public IP address on the resource, a bastion host, or any additional agent for establishing connections.

By combining identity-based and network-based access controls, the EIC Endpoint ensures comprehensive security measures, including isolation, control, and detailed logging, to meet the organization’s specific security requirements. Additionally, it relieves administrators of the operational burden associated with maintaining and patching bastion hosts for connectivity purposes. Users can still leverage familiar tools like PuTTY and OpenSSH while seamlessly accessing resources via the AWS Management Console and AWS Command Line Interface (CLI).

The EIC Endpoint is an identity-aware TCP proxy and offers two operational modes. In the first mode, it establishes a secure WebSocket tunnel from the user’s workstation to the endpoint using AWS Identity and Access Management (IAM) credentials, enabling users to connect to resources as they would normally. The second mode, accessible through the Console when not using the AWS CLI, provides secure access to VPC resources by evaluating authentication and authorization before allowing traffic into the VPC.

Steps to Create Amazon EC2 Instance Connect (EIC) Endpoint and use it to SSH into Amazon Private EC2

Step 1: Select the Amazon EC2 Instance Connect Endpoint, the VPC, the Security Group, and Subnet, which was created earlier.

Step 2: Goto Instance and click on Connect. Select Connect using Amazon EC2 Instance Connect Endpoint.

Step 3: The SSH will look like this once we click Connect.

We will see the further process in part 2. We will launch Amazon RDS, install Apache Kafka, and configure Debezium on Private Amazon EC2.

Conclusion

Amazon EC2 Instance Connect Endpoint revolutionizes secure connectivity to private Amazon EC2 instances within Amazon VPCs. It simplifies management by eliminating bastion hosts, public IP addresses, and complex network configurations. EIC Endpoint provides enhanced security controls through IAM-based authentication, network-perimeter controls, and auditability. This solution ensures secure remote access to private resources, offering a streamlined and secure connectivity option in AWS environments.

Click here for the Part 2.

Drop a query if you have any questions regarding Amazon EC2 Instance Connect Endpoint and we will get back to you quickly.

About CloudThat

CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft Gold Partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.