Open-Source File System for Amazon S3: Better Mountpoint Solution

Introduction

Amazon Simple Storage Service (Amazon S3) is an object storage service widely used as a data lake due to its high availability, durability, scalability, and security. A data lake built on S3 stores large amounts of structured and unstructured data, which can be used by machine learning, data analytics, business intelligence, and many more. Most of these tasks are built on open-source analytics engines such as Apache Spark, but some data lake customers use more domain-specific tools that need to read and write from a file in a local file system as it doesn’t support S3 object APIs.

Mountpoint for Amazon S3 is an open-source file client for Linux-based applications to access objects using file APIs directly connecting to its buckets. With mounting, your applications can access the Amazon S3 bucket as a file system by converting local API calls like read and open to S3 API calls like GET and LIST. It is ideal for large-scale analytics applications that analyze data stored in a data lake built on it. Mountpoint maps S3 Buckets and prefixes into file system namespaces and accesses objects from S3 as local files. It delivers high throughput access, reducing compute costs for data lakes on Amazon S3.

Benefits of Mountpoint for Amazon S3

File API access

Mountpoint for Amazon S3 converts local file system API calls to REST API calls on S3 objects and displays S3 objects as files in the local file system. Sequential and random read operations and sequential write operations for creating new files are supported.

High throughput

The AWS CRT library, which applies best practice performance design patterns for S3 clients, is integrated into Mountpoint for its applications.

High reliability

Petabytes of data may be processed in Amazon S3 data lakes, and you can dependably scale up and down over thousands of instances.

Open source

The development of mountpoint for Amazon S3 is driven by contributions and user feedback. Go to GitHub to contribute and offer comments.

Installation of mountpoint on Amazon Linux instance

1. For the x86_64 architecture, copy the following download URL.

2. Install the package by using the following command:

3. Verify that Mountpoint is successfully installed by entering the following command:

Configure Mountpoint for Amazon S3

Step 1: Create Amazon S3 bucket and upload an object in it.

Step 2: Create an Amazon Linux EC2 instance and connect to it. Create a directory using the ‘mkdir Directory_name’ command.

Run the following command to mount the S3 bucket.

Change the directory and run the ‘ls’ command. It will list all the objects present in the Amazon S3 bucket.

Step 3: create sample file ‘first.txt’. You can see the file is uploaded in the S3 bucket.

You can also create prefixes using the mkdir command.

Step 4: Change the directory to root and unmount the S3 bucket.

Conclusion

Without requiring code modifications, data lake apps can easily convert file operations into S3 API calls using Mountpoint. This enables them to operate on your S3 data.

References

Open Source File Client – Mountpoint for Amazon S3 – AWS

About CloudThat

CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, and Microsoft Gold Partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best-in-industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.