Build a Production-Grade Node.js File Upload Pipeline with S3, Sharp, and Presigned URLs

I remember the first time I tried to handle file uploads in a Node.js application. I used a simple library that stored everything in a local uploads folder, and everything worked fine in development. But when I put it into production, the server started running out of memory. Files were buffered entirely in RAM, and large images brought the entire process to a crawl. I had to stay up late, refactoring the whole pipeline. Since then, I have become obsessed with building upload systems that are safe, fast, and scalable. Today I want to share a production‑grade pipeline that streams multipart files directly to AWS S3, processes images in flight with Sharp, and delivers them through time‑limited presigned URLs. This is not just another tutorial; it is a blueprint you can drop into your own projects.

Have you ever wondered why so many upload tutorials store files on disk first, then upload them later? That old pattern doubles your I/O and can crash your server under concurrent load. The right way is to stream the incoming data directly to S3 using a multipart parser like busboy, pipe it through Sharp for real‑time transformation, and never write a byte to local storage. Let me show you how.

First, we set up the environment and validate everything upfront. Using Zod to parse environment variables ensures we fail fast if something is missing. The S3 client is a singleton, reused across all requests.

// src/config/s3.config.ts
import { S3Client } from '@aws-sdk/client-s3';
import { z } from 'zod';
import 'dotenv/config';

const envSchema = z.object({
  AWS_REGION: z.string().min(1),
  AWS_ACCESS_KEY_ID: z.string().min(1),
  AWS_SECRET_ACCESS_KEY: z.string().min(1),
  S3_BUCKET_NAME: z.string().min(1),
  S3_PRESIGN_EXPIRY_SECONDS: z.coerce.number().default(3600),
  MAX_FILE_SIZE_BYTES: z.coerce.number().default(50 * 1024 * 1024),
  ALLOWED_MIME_TYPES: z
    .string()
    .default('image/jpeg,image/png,image/webp,image/gif,application/pdf'),
});

const env = envSchema.parse(process.env);

export const s3Config = {
  region: env.AWS_REGION,
  bucketName: env.S3_BUCKET_NAME,
  presignExpirySeconds: env.S3_PRESIGN_EXPIRY_SECONDS,
  maxFileSizeBytes: env.MAX_FILE_SIZE_BYTES,
  allowedMimeTypes: env.ALLOWED_MIME_TYPES.split(',').map((t) => t.trim()),
} as const;

export const s3Client = new S3Client({
  region: env.AWS_REGION,
  credentials: {
    accessKeyId: env.AWS_ACCESS_KEY_ID,
    secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
  },
  maxAttempts: 3,
});

Now let’s talk about security. The Content-Type header sent by a browser can be easily faked. A dedicated attacker could send a JavaScript file claiming to be an image. To prevent that, we validate the actual magic bytes of the file – the first few bytes that identify the real format. I wrote a simple guard that reads the first chunk of the stream, checks against a list of known signatures, and only then passes the stream onward.

// src/middleware/magic-bytes.guard.ts
import { Readable } from 'stream';

const MAGIC_BYTES: Record<string, Uint8Array> = {
  'image/jpeg': new Uint8Array([0xFF, 0xD8, 0xFF]),
  'image/png': new Uint8Array([0x89, 0x50, 0x4E, 0x47]),
  'image/webp': new Uint8Array([0x52, 0x49, 0x46, 0x46]),
  'image/gif': new Uint8Array([0x47, 0x49, 0x46, 0x38]),
  'application/pdf': new Uint8Array([0x25, 0x50, 0x44, 0x46]),
};

export async function validateMagicBytes(
  stream: Readable,
  expectedMime: string
): Promise<boolean> {
  return new Promise((resolve, reject) => {
    stream.once('readable', () => {
      const chunk = stream.read(4) as Buffer | null;
      if (!chunk) return resolve(false);
      const signature = MAGIC_BYTES[expectedMime];
      if (!signature) return resolve(true); // allow unknown types? In a real app, reject
      const matched = chunk.slice(0, signature.length).every(
        (byte, idx) => byte === signature[idx]
      );
      // Push the peeked chunk back so the stream remains intact
      stream.unshift(chunk);
      resolve(matched);
    });
    stream.on('error', reject);
  });
}

But wait – what about large files? If you try to upload a 500 MB video, buffering it all in memory will crash your Node process. That’s why we stream straight to S3 using Upload from @aws-sdk/lib-storage. This uses multipart upload under the hood, splitting the stream into parts and sending them concurrently.

// src/services/s3.service.ts
import { PutObjectCommand } from '@aws-sdk/client-s3';
import { Upload } from '@aws-sdk/lib-storage';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { GetObjectCommand } from '@aws-sdk/client-s3';
import { s3Client, s3Config } from '../config/s3.config';
import { Readable } from 'stream';

export async function uploadStream(
  key: string,
  stream: Readable,
  contentType: string
): Promise<string> {
  const upload = new Upload({
    client: s3Client,
    params: {
      Bucket: s3Config.bucketName,
      Key: key,
      Body: stream,
      ContentType: contentType,
    },
  });
  await upload.done();
  return key;
}

export async function generatePresignedUrl(
  key: string
): Promise<string> {
  const command = new GetObjectCommand({
    Bucket: s3Config.bucketName,
    Key: key,
  });
  return getSignedUrl(s3Client, command, {
    expiresIn: s3Config.presignExpirySeconds,
  });
}

Now the real magic happens when we combine streaming with image processing. I love using Sharp because it works with Node.js streams natively. You can pipe a JPEG stream into Sharp, resize it to different thumbnail sizes, convert it to WebP (which can cut file size by 30% or more), and then pipe the output into the S3 upload. All of this happens in the same event loop without writing a temporary file.

// src/services/image.service.ts
import sharp from 'sharp';
import { Readable, Transform } from 'stream';

export function createResizeTransform(width: number, height: number): Transform {
  return sharp()
    .resize(width, height, { fit: 'cover', position: 'centre' })
    .webp({ quality: 80 });
}

I use this inside the upload route. The busboy parser emits a file stream for each uploaded file. I pipe that stream first through magic byte validation, then through the resize transform, and finally into the S3 upload. And because I want the user to know what’s happening, I send progress events via Server‑Sent Events (SSE). The client simply opens an EventSource to /upload/progress?fileId=xxx and receives { progress: 45 } messages as chunks are uploaded.

Here is the Express route that ties everything together. Notice how we never store the file on disk and never buffer it entirely in memory.

// src/routes/upload.routes.ts
import { Router, Request, Response } from 'express';
import Busboy from 'busboy';
import { v4 as uuid } from 'uuid';
import { s3Config } from '../config/s3.config';
import { uploadStream } from '../services/s3.service';
import { createResizeTransform } from '../services/image.service';
import { validateMagicBytes } from '../middleware/magic-bytes.guard';

const router = Router();

router.post('/upload', (req: Request, res: Response) => {
  const busboy = Busboy({ headers: req.headers, limits: { fileSize: s3Config.maxFileSizeBytes } });

  const fileId = uuid();

  busboy.on('file', async (fieldname, fileStream, filename, encoding, mimetype) => {
    // Validate magic bytes before processing
    const isValid = await validateMagicBytes(fileStream, mimetype);
    if (!isValid) {
      fileStream.resume(); // drain the stream
      res.status(400).json({ error: 'Invalid file type' });
      return;
    }

    // For images, apply a resize transform and convert to WebP
    const key = `uploads/${fileId}.webp`;
    const transform = createResizeTransform(800, 800);
    const pipeline = fileStream.pipe(transform);
    const result = await uploadStream(key, pipeline, 'image/webp');
    const url = await generatePresignedUrl(key);

    res.json({ fileId, key, presignedUrl: url });
  });

  busboy.on('error', (err) => {
    res.status(500).json({ error: err.message });
  });

  busboy.on('limit', () => {
    res.status(413).json({ error: 'File too large' });
  });

  req.pipe(busboy);
});

You might ask: Isn’t it risky to pipe user input directly through Sharp? Last year, a colleague argued that Sharp could crash if given malformed data. I countered by adding a try/catch around the pipeline and a timeout. If Sharp takes more than 10 seconds, we abort and return a helpful error. That extra guard gives me peace of mind.

const timeout = setTimeout(() => {
  transform.destroy(new Error('Image processing timed out'));
}, 10000);

try {
  const result = await uploadStream(key, pipeline, 'image/webp');
  // ...
} catch (err) {
  // log and handle
} finally {
  clearTimeout(timeout);
}

Now, let’s talk about the big picture. Why go through all this trouble? Because every millisecond counts, every byte saved matters, and every security hole you avoid is a future crisis averted. The streaming architecture I’ve described can handle dozens of concurrent uploads without spiking memory usage. It integrates well with any frontend, from React to Svelte, and the presigned URLs keep your S3 bucket private while allowing direct, time‑limited downloads.

I still remember that late‑night refactor. I learned the hard way that memory is not infinite. But you don’t have to. I hope this guide saves you those headaches.

Before you leave, I want to ask: Have you ever discovered a bug in your upload pipeline after deploying? Or have you tried streaming directly to cloud storage? I’d love to hear your war stories in the comments. And if this helped, give it a like and share it with your team. They’ll thank you the next time a client says “Can we handle 50‑megabyte files?” – and you can say yes, easily.

---

As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!

---

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

---

📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!

---

Our Creations

Be sure to check out our creations:

Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools

---

We are on Medium

Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva