Voiced by Amazon Polly |
Overview
In this blog article, we’ll look at using presigned URLs to upload big files to Amazon S3. Amazon API Gateway initially managed our file upload process, which can only handle files up to 10 MB. We’ve developed a method utilizing Amazon S3 presigned URLs to support larger files, enabling direct file uploads from the client interface without being constrained by Amazon API Gateway size restrictions.
Introduction
The 10MB payload size restriction for the Amazon API Gateway was set in response to user input and load testing conducted during the service launch. Although this limit cannot currently be configured, AWS may think about improving this capability in the future for users who need further restrictions. The Amazon API Gateway will accept a compressed payload supplied in less than 10MB. The Amazon API Gateway will automatically decompress the payload and may reject the request with an “Error 413 Request entity too large” if the uncompressed payload is larger than 10MB. One suggestion is using Amazon S3 for large file transfers to get around this limit. This may be done by having the Amazon API Gateway answer provide an Amazon S3 Pre-signed URL, which enables direct uploads to S3. With this method, uploads to Amazon S3, which supports multipart uploads and bigger file sizes up to 5TB, can be made without using the Amazon API Gateway’s 10MB limit.
Pioneers in Cloud Consulting & Migration Services
- Reduced infrastructural costs
- Accelerated application deployment
How to proceed?
To go ahead, it involves two main steps:
- Use an Amazon API Gateway to retrieve the presigned URL from Amazon S3 and set up a Lambda.
- Upload the file straight from the client using the URL you were given.
Benefits and Considerations
- Scalability: Amazon S3 is designed to handle massive amounts of data and concurrent uploads, ensuring scalability for your application.
- Reduced Latency: Direct uploads to Amazon S3 minimize latency by bypassing unnecessary intermediate steps.
- Cost-Effectiveness: By offloading file uploads to Amazon S3, you can potentially reduce data transfer costs associated with Amazon API Gateway.
Process Architecture Diagram
AWS Lambda Function to Generate Presigned URLS
Let’s start by examining the AWS Lambda function, which creates the presigned URL. Node.js will be utilized for the AWS Lambda function:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3'); const { getSignedUrl } = require('@aws-sdk/s3-request-presigner'); const client = new S3Client({ region: 'eu-west-2', signatureVersion: 'v4' }); exports.handler = async (event, context) => { // Define the parameters for the S3 bucket and the object key const params = { Bucket: 'YOUR_BUCKET_NAME', Key: 'OBJECT_KEY' } const expires = 3600; // Set expiration time for the signed URL (in seconds) try { const url = await getSignedUrl(client, new PutObjectCommand(params), { expiresIn: expires }); return { headers: { 'Access-Control-Allow-Origin': '*' }, body: JSON.stringify({ url }) } } catch (error) { return { headers: { 'Access-Control-Allow-Origin': '*' }, body: JSON.stringify({ error: error.message }) } } }; |
Uploading Client-Side using React
Let’s now talk about using this presigned URL to upload a file from the client side. Here, React is utilized:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
import axios from 'axios'; // Function to upload the file using the presigned URL export async function upload(file) { const getUrlResponse = await getSignedUrl(); const url = getUrlResponse.body; // Handle cases where the URL is not obtained if (!url) return; await uploadFile(url, file); } // Function to retrieve the signed URL export async function getSignedUrl() { const result = await axios.get('API_GATEWAY_URL') .then(response => { return { body: response.data.url } }) .catch(err => { return { body: null } }); return result; } // Function to perform the file upload export async function uploadFile(url, file) { const result = await axios.put(url, file, { headers: { 'Content-Type': file.type } }).then(response => { return { body: response.data } }).catch(err => { return { body: null } }); return result; } |
CORS Configuration on Amazon S3
CORS failures are one frequent problem during implementation. Here’s how to set up CORS in the settings of your Amazon S3 bucket:
1 2 3 4 5 |
{ "AllowedHeaders": ["*"], "AllowedMethods": ["PUT"], // Ensure 'PUT' is allowed for file uploads "AllowedOrigins": ["http://localhost:3000"] // Adjust this according to your client's URL } |
Conclusion
Drop a query if you have any questions regarding Amazon S3 presigned URLs and we will get back to you quickly.
Making IT Networks Enterprise-ready – Cloud Management Services
- Accelerated cloud migration
- End-to-end view of the cloud environment
About CloudThat
CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.
CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.
To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.
FAQs
1. How do I handle security concerns when using presigned URLs?
ANS: – Implementing appropriate access controls and expiration times when generating presigned URLs is crucial to prevent unauthorized access to Amazon S3 resources. Ensure that presigned URLs are securely generated and shared only with authorized users.
2. Can presigned URLs be used for other Amazon S3 operations besides uploads?
ANS: – Yes, presigned URLs can be generated for other Amazon S3 operations, such as downloading, copying, or deleting objects. They provide temporary access to perform specific actions on Amazon S3 objects within a limited timeframe.
WRITTEN BY Garima Pandey
Comments