How to Quickly Power Data with AWS LakeFormation

Introduction

Our data’s volume, variety, and velocity continue to increase dramatically as we live in a data-driven world. We are talking about zettabytes of data collected from click streams, machine-generated log data, IoT sensors, and social media, and no people are accessing this huge data for different causes.

Therefore, regardless of how we acquire and store the data, we need a system that can store, manage and access structured and unstructured data in various formats. It should be able to analyze the data and often discover useful, meaningful insights in real-time.

So, a data lake gives you access to a central platform that allows structured, unstructured, and semi-structured data storage in a wide range of data formats without having to relocate your data, and it supports various use cases for analytics and machine learning.

Image source: AWS Lake Formation – Now Generally Available | AWS News Blog (amazon.com)

Why are Customers Moving to a Data Lake?

Data lakes offer a mechanism for users to store both relational and non-relational data on a massive scale, along with support that provides a wide range of tools that can be used to analyze this data and derive deeper insights. You have a central data catalog that can show you the data you hold and its characteristics, often termed meta-data. Then one can use AWS services like Amazon Athena for running your ad-hoc queries for real-time interactive analysis or Amazon EMR for executing your big data applications, or one can use Amazon Redshift as your data warehouse and redshift spectrum to run scale-out exabyte queries across data that is stored in your data lake in S3 as well in your data warehouse in Redshift. One can also use Amazon QuickSight to analyze this data, and then you can run your machine learning workloads using services like Amazon Sagemaker and others.

Typical Steps in Building a Data Lake

Typical steps that customers take in building a data lake.

• The first step is to set up your storage, and Amazon S3, with its 11 9’s of durability, provides a great storage layer for your data lake.
• Then one needs to move data from various sources into your data lake in its raw format. Sources can be your real-time sources, sources that are on AWS, or on-premises sources.
• The next step involves the cleaning and preparation phase, where we catalog this raw data to be discoverable and readily available for analytics. One has to take care of the security policies that need to be specified on this data and the encryption to make this information accessible to the appropriate people within your company.
• And finally, you need to ensure that this data is available across a wide range of use cases within your organization and a wide range of teams and users.

Image Source: Cleaning Big Data: Most Time-Consuming, Least Enjoyable Data Science Task, Survey Says (forbes.com)

So, Building a data lake can take months.

Although many customers are building data lakes on AWS today, we still find that customers can take up to months to build a data lake and go through the various steps required to operationalize a data lake. We see that customers spend most of their time preparing their data and cataloging it so that it is available for analytics. The study was done across 80 different data scientists, and what they found is that data scientists spend 60% of their time cleaning and organizing data, and then collecting data comes second at 19% of their time, meaning that the data scientist spends around 80% of their time on preparing and managing data for analytics.

Source: https://aws.amazon.com/lake-formation/

So, this process tends to be very manual and time-consuming, so we build AWS Lake Formation. AWS Lake Formation does three things for you. It helps identify, ingest and clean and transform data in a way that is readily available for your downstream analytics and machine learning workloads. It provides a central place for you to define and enforce security policies across multiple services and workloads and to be able to audit all data access. Then it provides a central metadata catalog that allows you to easily share data, search for data, and to collaborate across various business use cases to get new insights.

How Does it Work?

You can set up your ingest process, and you can register the data that you have. If you already have existing data in Amazon S3, you can set up ingest process to bring new data into your data lake. You can define permissions at the table level. You can also define and view permissions peruse. You can do a text-based search and filter on certain facets to find the right data sets you want to work with. You can also enrich the metadata available in the catalog and add more business context to this data. Finally, you can monitor all activities and audit all data access in one place.

Features

Fully managed data lake: AWS Lake Formation integrates with underlying AWS security, storage, analysis, and machine learning services and automatically configures them to comply with your centrally defined access controls using an easy-to-use web console.

Powerful data cleansing AWS Lake Formation can also optionally cleanse your data to remove duplicates, fill in missing values, link records across data sets, and remove data errors.

Automated data ingestion: AWS Lake Formation supports batch and streaming data ingestion into Amazon S3 and can convert the data into open data formats.

Centralized access control: Define access permissions, including table, row, and column level control and encryption policies for data at rest and in motion. You can then access your data lake using various AWS analytic and machine learning services. All access is secured, governed, and auditable.

Prominent Customers of AWS LakeFormation

1] BookMyShow

Image Source: BookMyShow | Amazon Web Services

2] Onefootball

Image Source: OneFootball AWS Lake Formation Case Study (amazon.com)

3] KOHO

Image Source: Lower Personal Finance Costs and Access to Same-Day Pay with KOHO | KOHO Case Study | AWS (amazon.com)

Conclusion

From this blog, we all will understand the benefits and features we get from AWS Lake Formation for building our data lake.

About CloudThat

CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner & Training partner and Microsoft Solutions Partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best-in-industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

To get started, go through our Consultancy page and AWS Training Page to learn about our salient consulting and training offerings.