Simplifying Large File Uploads to Amazon S3 with Presigned URLs

Overview

In this blog article, we’ll look at using presigned URLs to upload big files to Amazon S3. Amazon API Gateway initially managed our file upload process, which can only handle files up to 10 MB. We’ve developed a method utilizing Amazon S3 presigned URLs to support larger files, enabling direct file uploads from the client interface without being constrained by Amazon API Gateway size restrictions.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Introduction

The 10MB payload size restriction for the Amazon API Gateway was set in response to user input and load testing conducted during the service launch. Although this limit cannot currently be configured, AWS may think about improving this capability in the future for users who need further restrictions. The Amazon API Gateway will accept a compressed payload supplied in less than 10MB. The Amazon API Gateway will automatically decompress the payload and may reject the request with an “Error 413 Request entity too large” if the uncompressed payload is larger than 10MB. One suggestion is using Amazon S3 for large file transfers to get around this limit. This may be done by having the Amazon API Gateway answer provide an Amazon S3 Pre-signed URL, which enables direct uploads to S3. With this method, uploads to Amazon S3, which supports multipart uploads and bigger file sizes up to 5TB, can be made without using the Amazon API Gateway’s 10MB limit.

How to proceed?

To go ahead, it involves two main steps:

Use an Amazon API Gateway to retrieve the presigned URL from Amazon S3 and set up a Lambda.
Upload the file straight from the client using the URL you were given.

Benefits and Considerations

Scalability: Amazon S3 is designed to handle massive amounts of data and concurrent uploads, ensuring scalability for your application.
Reduced Latency: Direct uploads to Amazon S3 minimize latency by bypassing unnecessary intermediate steps.
Cost-Effectiveness: By offloading file uploads to Amazon S3, you can potentially reduce data transfer costs associated with Amazon API Gateway.

Process Architecture Diagram

AWS Lambda Function to Generate Presigned URLS

Let’s start by examining the AWS Lambda function, which creates the presigned URL. Node.js will be utilized for the AWS Lambda function:

const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');
const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');

const client = new S3Client({
    region: 'eu-west-2',
    signatureVersion: 'v4'
});

exports.handler = async (event, context) => {
    // Define the parameters for the S3 bucket and the object key
    const params = {
        Bucket: 'YOUR_BUCKET_NAME',
        Key: 'OBJECT_KEY'
    }
    const expires = 3600; // Set expiration time for the signed URL (in seconds)

    try {
        const url = await getSignedUrl(client, new PutObjectCommand(params), {
            expiresIn: expires
        });

        return {
            headers: {
                'Access-Control-Allow-Origin': '*'
            },
            body: JSON.stringify({ url })
        }
    } catch (error) {
        return {
            headers: {
                'Access-Control-Allow-Origin': '*'
            },
            body: JSON.stringify({ error: error.message })
        }
    }
};

const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');

const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');

const client = new S3Client({

region: 'eu-west-2',

signatureVersion: 'v4'

});

exports.handler = async (event, context) => {

// Define the parameters for the S3 bucket and the object key

const params = {

Bucket: 'YOUR_BUCKET_NAME',

Key: 'OBJECT_KEY'

}

const expires = 3600; // Set expiration time for the signed URL (in seconds)

try {

const url = await getSignedUrl(client, new PutObjectCommand(params), {

expiresIn: expires

});

return {

headers: {

'Access-Control-Allow-Origin': '*'

body: JSON.stringify({ url })

}

} catch (error) {

return {

headers: {

'Access-Control-Allow-Origin': '*'

body: JSON.stringify({ error: error.message })

}

};

Uploading Client-Side using React

Let’s now talk about using this presigned URL to upload a file from the client side. Here, React is utilized:

import axios from 'axios';

// Function to upload the file using the presigned URL
export async function upload(file) {
    const getUrlResponse = await getSignedUrl();
    const url = getUrlResponse.body;

    // Handle cases where the URL is not obtained
    if (!url) return;

    await uploadFile(url, file);
}

// Function to retrieve the signed URL
export async function getSignedUrl() {
    const result = await axios.get('API_GATEWAY_URL')
        .then(response => {
            return { body: response.data.url }
        })
        .catch(err => {
            return { body: null }
        });

    return result;
}

// Function to perform the file upload
export async function uploadFile(url, file) {
    const result = await axios.put(url, file, {
        headers: { 'Content-Type': file.type }
    }).then(response => {
        return { body: response.data }
    }).catch(err => {
        return { body: null }
    });

    return result;
}

import axios from 'axios';

// Function to upload the file using the presigned URL

export async function upload(file) {

const getUrlResponse = await getSignedUrl();

const url = getUrlResponse.body;

// Handle cases where the URL is not obtained

if (!url) return;

await uploadFile(url, file);

}

// Function to retrieve the signed URL

export async function getSignedUrl() {

const result = await axios.get('API_GATEWAY_URL')

.then(response => {

return { body: response.data.url }

})

.catch(err => {

return { body: null }

});

return result;

}

// Function to perform the file upload

export async function uploadFile(url, file) {

const result = await axios.put(url, file, {

headers: { 'Content-Type': file.type }

}).then(response => {

return { body: response.data }

}).catch(err => {

return { body: null }

});

return result;

}

CORS Configuration on Amazon S3

CORS failures are one frequent problem during implementation. Here’s how to set up CORS in the settings of your Amazon S3 bucket:

{
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["PUT"], // Ensure 'PUT' is allowed for file uploads
    "AllowedOrigins": ["http://localhost:3000"] // Adjust this according to your client's URL
}

{

"AllowedHeaders": ["*"],

"AllowedMethods": ["PUT"], // Ensure 'PUT' is allowed for file uploads

"AllowedOrigins": ["http://localhost:3000"] // Adjust this according to your client's URL

}

Conclusion

We can get around Amazon API Gateway’s restrictions on large file uploads by using Amazon S3 presigned URLs, which provides a safe and scalable alternative that can be used straight from client apps. This strategy protects your data while in transit and simplifies the process.

Drop a query if you have any questions regarding Amazon S3 presigned URLs and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. How do I handle security concerns when using presigned URLs?

ANS: – Implementing appropriate access controls and expiration times when generating presigned URLs is crucial to prevent unauthorized access to Amazon S3 resources. Ensure that presigned URLs are securely generated and shared only with authorized users.

2. Can presigned URLs be used for other Amazon S3 operations besides uploads?

ANS: – Yes, presigned URLs can be generated for other Amazon S3 operations, such as downloading, copying, or deleting objects. They provide temporary access to perform specific actions on Amazon S3 objects within a limited timeframe.