Direct S3 presigned uploads: avoiding the API backend bottleneck

A common anti-pattern in early-stage web architectures is proxying user file uploads directly through an application backend server (such as a Node.js, Django, or Rails API container).

When thousands of users simultaneously upload multi-megabyte images or gigabyte video files, your application server runs out of worker threads, exhausts RAM buffers, and crashes under out-of-memory (OOM) conditions.

ANTI-PATTERN (Proxying Uploads):
Browser ──[500MB Video Stream]──▶ API Server ──[500MB Upload]──▶ S3 Bucket
                               (Locks CPU & RAM)

SCALABLE ARCHITECTURE (S3 Presigned URLs):
Browser ──[1. Get Presigned URL (20ms)]──▶ API Server
Browser ──[2. Direct HTTP PUT 500MB]───────────────────────────▶ S3 Bucket
                                                          (Zero API Server Load)

---

1. How Direct S3 Presigned Uploads Work

An S3 Presigned URL gives the client temporary, cryptographic authorization (via AWS SigV4) to perform a specific action (like PUT an object) directly against an S3 bucket without exposing AWS credentials to the frontend.

Step-by-Step Flow:

1. Metadata Request: The frontend browser sends a lightweight JSON payload to your backend (POST /api/upload-url with { filename: "video.mp4", contentType: "video/mp4", size: 104857600 }).
2. Backend Authentication & URL Signing: The backend authenticates the user session, validates file constraints, and calls the AWS SDK:

   import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
   import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
   
   const command = new PutObjectCommand({
     Bucket: process.env.AWS_S3_BUCKET,
     Key: `uploads/${userId}/${fileId}-${filename}`,
     ContentType: contentType,
   });
   
   const presignedUrl = await getSignedUrl(s3Client, command, { expiresIn: 300 });

3. Instant Response: The backend returns the presignedUrl to the browser in under 20 milliseconds.
4. Direct Browser Upload: The browser executes a standard fetch or XMLHttpRequest with PUT directly to the Amazon S3 endpoint:

   await fetch(presignedUrl, {
     method: "PUT",
     headers: { "Content-Type": file.type },
     body: file,
   });

---

2. Asynchronous Post-Processing with S3 Events

Once the file is securely stored in S3, how does your backend know the upload finished?

Instead of relying on the client to send a follow-up “I’m done” HTTP request (which fails if the user closes their tab), leverage S3 Event Notifications:

1. S3 Event Trigger: S3 automatically publishes an s3:ObjectCreated:Put event to AWS Lambda, SQS, or EventBridge.
2. Serverless Worker Execution: An AWS Lambda worker parses the event, validates the file checksum, resizes images, or enqueues video transcoding jobs.
3. Database State Mutation: The worker updates the database record from status = 'pending' to status = 'ready'.

┌─────────────┐      ObjectCreated      ┌────────────────┐      Update DB      ┌────────────┐
│  AWS S3     │ ──────────────────────▶ │ AWS Lambda     │ ──────────────────▶ │ PostgreSQL │
│  Bucket     │                         │ Worker Handler │                     │ Database   │
└─────────────┘                         └────────────────┘                     └────────────┘

---

3. Production Hardening Checklist

1. Strict Expiration Windows: Set the presigned URL expiration (expiresIn) to 3–5 minutes.
2. Enforce Allowed Content Types: Lock the ContentType header during signing so malicious clients cannot upload arbitrary .exe or executable scripts under a .png disguise.
3. CORS Configuration: Configure your S3 Bucket CORS policy to allow PUT requests only from your verified production web origin domains.
4. Multipart Uploads for Large Files: For files larger than 100MB, use the S3 Multipart Upload API to allow parallel chunk uploads and resume-on-failure capabilities.