I’ve been building web applications for a while now, and one task that consistently trips up developers—myself included—is handling file uploads. It seems simple: a user picks a file, you accept it. But the gap between a basic demo and a system ready for real-world use is vast. Today, I want to walk you through building a reliable file upload pipeline. This isn’t just about getting bytes from point A to point B. It’s about doing it safely, efficiently, and in a way that won’t keep you up at night worrying about security or bills.
Let’s start with the gateway: the point where files enter your system. In Node.js with Express, this often means using middleware to parse multipart form data. This is where Multer comes in. It’s robust, but out of the box, it might not fit our specific needs. We need control. For instance, what if a user tries to upload a file that’s far too large? We shouldn’t wait for the entire upload to finish before rejecting it.
const multer = require('multer');
const upload = multer({
limits: { fileSize: 50 * 1024 * 1024 }, // 50MB limit
fileFilter: (req, file, cb) => {
const allowedTypes = /jpeg|jpg|png|pdf/;
const isValid = allowedTypes.test(file.mimetype);
cb(isValid ? null : new Error('Invalid file type'), isValid);
}
});This sets a hard ceiling on size and checks the file’s MIME type early. But we can push this further. Have you ever considered what metadata is coming in with that file? The filename, the size, the MIME type—this is all user-provided data. And like any user input, it shouldn’t be trusted blindly.
This is where adding a validation layer becomes non-negotiable. I like using Zod for this. It lets me define exactly what a valid file object should look like, right down to the byte. It turns vague assumptions into explicit rules.
const z = require('zod');
const FileSchema = z.object({
originalname: z.string().min(1).max(255),
mimetype: z.string().refine(val => val.startsWith('image/') || val === 'application/pdf'),
size: z.number().positive().max(50 * 1024 * 1024),
});Now, before any business logic runs, I can pass the file info through this schema. If it doesn’t match, the request fails fast with a clear error. This moves us from hoping the data is correct to knowing it is. But what about the file’s contents? A valid-looking PDF could still be malicious.
So, what’s the next line of defense? For many production systems, it’s virus scanning. The idea is straightforward: before a file touches permanent storage, it should be checked. I can integrate a scan using a tool like ClamAV. The process involves buffering the file in memory temporarily, passing it to the scanner, and only proceeding if it’s clean.
This scan acts as a crucial checkpoint. But it introduces a new question: where do we store the file after it passes inspection? Saving files directly on your server’s disk can cause problems—it fills up space, complicates backups, and doesn’t scale well. This is where cloud object storage, like Amazon S3, shines.
The goal is to stream the file directly to S3. This avoids saving the full file to our server’s disk or memory, which is much more efficient. We create a writable stream to S3 and pipe the incoming file data to it. This method handles large files gracefully.
const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');
const s3Client = new S3Client({ region: 'us-east-1' });
const uploadStreamToS3 = async (fileBuffer, key, mimetype) => {
const command = new PutObjectCommand({
Bucket: process.env.S3_BUCKET,
Key: key,
Body: fileBuffer,
ContentType: mimetype,
});
await s3Client.send(command);
return key; // Return the unique file identifier
};Once the file is safely in S3, we have a new challenge. How do we let users access it? We could proxy the file through our server, but that adds unnecessary load. A better way is to generate a pre-signed URL. This is a special URL that grants temporary access to a private file in S3.
Think about it: you can give a frontend application a URL that works for only 5 minutes, perfect for displaying an uploaded image. After that time, the link expires. This keeps your files private by default and offloads the bandwidth cost to AWS.
const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');
const { GetObjectCommand } = require('@aws-sdk/client-s3');
const getFileUrl = async (fileKey) => {
const command = new GetObjectCommand({
Bucket: process.env.S3_BUCKET,
Key: fileKey,
});
const url = await getSignedUrl(s3Client, command, { expiresIn: 300 }); // 5 minutes
return url;
};Finally, we need to consider who can upload and who can view. Not every user should be able to upload executables, and some uploaded files might only be viewable by managers. This is about applying access rules. We can check a user’s role or permissions at the upload endpoint and again when generating a view link.
Pulling all this together—validation, scanning, streaming storage, and secure access—creates a pipeline you can trust. It’s a system that respects user input but verifies it, stores data efficiently, and controls access tightly.
Building this changed how I view features that seem simple on the surface. It’s about layering thoughtful checks and balances to create something resilient. I hope this guide helps you build something robust. If you found this walk-through useful, please share it with a colleague or leave a comment below with your own experiences. What’s the biggest challenge you’ve faced with file uploads?
As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!
101 Books
101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.
Check out our book Golang Clean Code available on Amazon.
Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!
📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!
Our Creations
Be sure to check out our creations:
Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools
We are on Medium
Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva
