Complete Node.js File Upload Pipeline with Multer, Sharp, and S3

js

Complete Node.js File Upload Pipeline with Multer, Sharp, and S3

Learn secure Node.js file uploads with Multer, Sharp, and AWS S3. Validate, process, store, and serve files safely in production.

Mar 31, 2026

Complete Node.js File Upload Pipeline with Multer, Sharp, and S3

You know what’s funny? I’ve lost count of the number of times I’ve seen a file upload tutorial that just stops after saving a file to a folder. It’s like learning to drive in a parking lot but never being told about highways, traffic signs, or what to do in the rain. That’s why I’m writing this. Because in the real world, handling a file upload is a multi-step journey. You need to check the file, process it, store it securely, and then give people a safe way to get it back. Miss one step, and you’ve got problems.

Let’s build that complete journey.

First, we set the stage. We’ll use Node.js with Express as our base. For handling the raw file data from a form, we use a tool called Multer. Think of Multer as a dedicated receptionist for file uploads. It takes the incoming data, checks it, and hands it to us in a way we can work with. But here’s the first crucial step: we don’t save it directly to disk. We use memory storage. This keeps the file in our server’s memory so we can inspect and process it before deciding its final destination.

const multer = require('multer');
const upload = multer({
  storage: multer.memoryStorage(),
  limits: { fileSize: 10 * 1024 * 1024 } // 10MB limit
});

See that limit? That’s our first line of defense. Without it, someone could try to send a massive file and crash our server. But what if someone uploads an executable file when we only want images? We need to check the file type. However, we can’t just trust the filename. A malicious user can rename a virus to ‘cat.jpg’. We need to check the file’s “magic number” or MIME type from its actual content. Libraries like file-type can help with this, but for simplicity here, we’ll validate the extension and later use Sharp, which itself is strict about image formats.

Now, what if the file is an image? We often need to resize it or convert it to a more efficient format like WebP. This is where Sharp comes in. It’s incredibly fast. We can take the file buffer from Multer, pass it to Sharp, and get back an optimized image without ever saving an intermediate file to our server’s hard drive.

const sharp = require('sharp');

async function processImage(buffer) {
  return await sharp(buffer)
    .resize(800, 800, { fit: 'inside' }) // Fit within 800x800
    .webp({ quality: 80 }) // Convert to WebP at 80% quality
    .toBuffer();
}

This step alone saves storage space and makes websites load faster. Ever wondered why some sites feel snappy while others lag on images? This processing is often why.

Okay, we have a clean, optimized file. Now we need to put it somewhere reliable and scalable. This is where cloud storage like AWS S3 comes in. Storing files on your own server is risky; it fills up space and doesn’t scale well. S3 is built for this. We upload our processed file buffer directly to an S3 “bucket.”

But we must be smart about it. We should never use the original filename. Two users might both upload resume.pdf. We generate a unique identifier (like a UUID) for the file and use that as its name in the bucket. We can also prefix it with a folder path, like users/1234/profile-picture/.

const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');
const { v4: uuidv4 } = require('uuid');

const s3Client = new S3Client({ region: 'us-east-1' });

async function uploadToS3(buffer, mimeType) {
  const key = `uploads/${uuidv4()}`; // Unique key

  const command = new PutObjectCommand({
    Bucket: 'my-app-bucket',
    Key: key,
    Body: buffer,
    ContentType: mimeType,
  });

  await s3Client.send(command);
  return key; // We save this key in our database
}

We store that unique S3 key in our application’s database, linked to the user or post it belongs to. The file itself lives safely in S3. But how does a user’s browser get it back? We can’t just give everyone full access to the bucket.

This leads to a critical question: is the file public or private? A profile picture might be public. A user’s invoice PDF must be private. For public files, we can use a direct S3 URL or, better yet, a Content Delivery Network (CDN) like CloudFront in front of S3 for faster global delivery. For private files, we cannot give out a permanent direct link. Instead, we generate a temporary, time-limited URL called a “pre-signed URL.”

const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');
const { GetObjectCommand } = require('@aws-sdk/client-s3');

async function getPrivateFileUrl(s3Key) {
  const command = new GetObjectCommand({
    Bucket: 'my-app-bucket',
    Key: s3Key,
  });

  // URL expires in 1 hour (3600 seconds)
  const url = await getSignedUrl(s3Client, command, { expiresIn: 3600 });
  return url;
}

When your front-end app needs to display a private file, it asks your backend for a URL. The backend checks if the user is allowed, generates this short-lived pre-signed URL, and sends it back. The front-end can then use it to fetch the file directly from S3 for the next hour. The link then becomes useless. This keeps our data secure without our server having to act as a middleman for downloading every single file.

Putting it all together in an Express route looks like this:

app.post('/upload', upload.single('file'), async (req, res) => {
  try {
    // 1. Validate file exists and is an image
    if (!req.file) throw new Error('No file uploaded');
    if (!req.file.mimetype.startsWith('image/')) throw new Error('File must be an image');

    // 2. Process with Sharp
    const processedBuffer = await processImage(req.file.buffer);

    // 3. Upload to S3
    const s3Key = await uploadToS3(processedBuffer, 'image/webp');

    // 4. Save `s3Key` to your database here...

    // 5. Send response
    res.json({
      success: true,
      key: s3Key,
      message: 'File uploaded and processed successfully.'
    });

  } catch (error) {
    res.status(400).json({ success: false, error: error.message });
  }
});

This flow—validate, process, store, track, and serve securely—is what separates a basic demo from a robust feature. It handles security, performance, and cost. You’re not just accepting a file; you’re managing a digital asset throughout its lifecycle.

I hope this walkthrough connects the dots and gives you a clear path forward. Building it piece by piece makes it manageable. What part of this pipeline do you think is most often overlooked by developers? Share your thoughts in the comments below—I’d love to hear about your experiences. If you found this guide helpful, please like and share it so other builders can find it too. Happy coding

As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!

Our Creations

Be sure to check out our creations:

We are on Medium

Share: Facebook Twitter Reddit LinkedIn WhatsApp Telegram Pinterest Email Instagram

js