OpenSearch Service Zero-ETL integration with Amazon S3

Overview

During AWS reInvent 2023, AWS released a preview of Amazon OpenSearch Service zero-ETL integration with Amazon S3, providing a new approach to query operational logs in Amazon S3 and S3-based data lakes without switching between services. You may now analyze infrequently queried data in cloud object stores while utilizing OpenSearch Service’s operational analytics and visualization capabilities.

Zero ETL

Zero-ETL is a collection of connectors that eliminates or reduces ETL data pipeline requirements. ETL combines, cleans, and normalizes data from many sources to prepare it for analytics, artificial intelligence (AI), and machine learning (ML) applications. Traditional ETL methods take time and are difficult to design, maintain, and scale. On the other hand, zero-ETL connectors allow for point-to-point data flow without the requirement for ETL data pipelines. Zero-ETL can also enable cross-data-silos querying without the requirement for data migration.

Different use cases for Zero-ETL

Federated querying

Using federated querying technologies, you can query multiple data sources without worrying about data movement. Using well-known SQL commands, you can run queries and join data from multiple sources, including operational databases, data warehouses, and data lakes. With In-Memory Data Grids (IMDG), you can benefit from instantaneous analysis and query response times by storing data in memory for caching and processing. The join results can then be kept for later use and analysis in a data store.

Streaming ingestion

Platforms for message queuing and data streaming provide real-time data streaming from multiple sources. You can almost instantly present data for analytics after ingesting it from several of these streams using a zero-ETL integration with a data warehouse. The streaming data does not need to be staged for transformation on any other storage service.

Instant replication

Traditionally, an intricate ETL solution was always needed to transfer data from a transactional database to a central data warehouse. These days, data can be instantaneously replicated from the transactional database to the data warehouse using zero-ETL as a data replication tool. The duplication mechanism may be integrated into the data warehouse, and change data capture (CDC) techniques may be used. Users are unaware of the duplication because analysts can easily query data from the warehouse, and applications store data in the transactional database.

OpenSearch Service Zero-ETL integration with Amazon S3

By allowing users to query their operational data directly, Amazon OpenSearch Service direct queries with Amazon S3 offer a zero-ETL integration that lowers the operational complexity of duplicating data or managing multiple analytics tools, saving money and time to action. OpenSearch Service will offer a configurable zero-ETL integration. From there, you can utilize different log type templates, including pre-made dashboards, and set up data accelerations specific to that type. Skipping indexes, materialized views, and covered indexes are examples of accelerations; templates include VPC Flow Logs, Elastic Load Balancing Logs, and NGINX Logs.

Direct queries with Amazon S3 enable you to run intricate queries essential for threat and security forensic analysis. These queries correlate data from various sources, assisting teams in investigating security events and service outages. Once you’ve created an integration, you can begin directly querying their data from the OpenSearch Dashboards or OpenSearch API. Connections can be easily audited to make sure they are configured in a secure, scalable, and economical manner.

Limitations

Direct queries using Amazon S3 through OpenSearch Service are subject to the following restrictions.

• To support OpenSearch Service direct queries, your OpenSearch domain needs to be 2.11 or later.
• Only Spark tables in the AWS Glue Data Catalogue are supported by OpenSearch Service direct queries with Amazon S3. Index updates depend on Spark streaming, which is not supported by Hive tables.
• Certain data types are not compatible. The only supported data formats are Parquet, CSV, and JSON.
• The direct query preview release does not support AWS CloudFormation templates.
• The AWS Glue Data Catalogue and your OpenSearch domain must be in the same AWS account. Although they must be in the same AWS Region as your domain, your Amazon S3 tables may be in a different account.
• There is no support for nested Spark structures. If they are present, you must explode any nested structures in your source data to rows. There is no support for nested Spark structures. If they are present, you must explode any nested structures in your source data to rows.

Conclusion

OpenSearch Service users also use Amazon S3 as an affordable means of storing operational log data that is not frequently accessed. Customers had to copy data from Amazon S3 into OpenSearch Service to use its rich analytics and visualization features, which aid in understanding data, spotting anomalies, and spotting possible threats. This required customers to analyze Amazon S3 data and correlate data from multiple sources. On the other hand, constantly maintaining and replicating data between services can be costly.

Drop a query if you have any questions regarding Amazon S3 and we will get back to you quickly.

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.