I’ve been thinking about file uploads lately because they’re such a common pain point in web development. Every application needs them, but building a robust solution requires juggling multiple concerns - security, performance, and reliability. Today I’ll share how I created a production-ready upload system using Node.js, Multer, Sharp, and AWS S3. Stick around because this approach handles everything from image optimization to resumable uploads.

First, let’s set up our project. Why start from scratch when we can use proven tools? I chose Express for routing, Multer for file handling, Sharp for image processing, and AWS S3 for cloud storage. Here’s how I initialized the project:

npm init -y
npm install express multer sharp @aws-sdk/client-s3

For TypeScript support, I added these dev dependencies:

npm install -D typescript @types/node @types/express

My folder structure organizes functionality logically:

src/
├── controllers/  # Request handlers
├── middleware/   # Upload validation
├── services/     # Core logic
└── app.ts        # Entry point

Configuration happens through environment variables. I use a .env file for settings like:

AWS_REGION=us-east-1
S3_BUCKET_NAME=my-upload-bucket
MAX_FILE_SIZE=10485760  # 10MB

Multer handles the initial upload processing. I created a reusable middleware that validates files before they hit our server. Notice how it checks file types and sizes:

// uploadMiddleware.ts
import multer from 'multer';
import { Request } from 'express';

const upload = multer({
  limits: { fileSize: parseInt(process.env.MAX_FILE_SIZE!) },
  fileFilter: (req: Request, file, cb) => {
    const allowedTypes = ['image/jpeg', 'application/pdf'];
    if (!allowedTypes.includes(file.mimetype)) {
      return cb(new Error('Invalid file type'));
    }
    cb(null, true);
  }
});

export const singleUpload = upload.single('file');

What happens when users upload massive files? We prevent server overload by limiting sizes upfront. But what about partial uploads when connections drop? Later I’ll show how to implement resumable uploads.

For images, Sharp is my go-to processor. It’s lightning-fast for resizing and optimization. Here’s how I convert images to efficient WebP format:

// imageService.ts
import sharp from 'sharp';

export async function processImage(buffer: Buffer) {
  return sharp(buffer)
    .resize(1200, 800, { fit: 'inside' })
    .webp({ quality: 80 })
    .toBuffer();
}

Notice how this maintains aspect ratio while reducing file size. For PDFs, I skip processing and upload directly. Ever wonder why some sites load images faster? This optimization is their secret.

Now to cloud storage. AWS S3 provides durability, but direct uploads from client to S3 are more scalable. I use presigned URLs to securely bypass our server:

// s3Service.ts
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';

const s3 = new S3Client({ region: process.env.AWS_REGION });

export async function getPresignedUrl(key: string) {
  const command = new PutObjectCommand({
    Bucket: process.env.S3_BUCKET_NAME,
    Key: key
  });
  return await getSignedUrl(s3, command, { expiresIn: 3600 });
}

Security can’t be an afterthought. I implemented multiple layers of protection:

1. File type whitelisting
2. Size restrictions
3. Virus scanning integration
4. Rate limiting with express-rate-limit
5. Content-Disposition headers to prevent execution

For large files, resumable uploads are essential. I used the AWS SDK’s upload method with progress tracking:

const upload = new Upload({
  client: s3,
  params: { Bucket: 'my-bucket', Key: 'file.zip' },
  leavePartsOnError: false
});

upload.on('httpUploadProgress', (progress) => {
  console.log(`Uploaded ${progress.loaded} of ${progress.total}`);
});

Testing revealed interesting edge cases. What happens when someone uploads a file disguised as an image? Sharp validates images during processing, rejecting invalid ones. For load testing, I used Artillery to simulate 100 concurrent users.

Performance optimizations that made a difference:

• Stream processing instead of loading full files into memory
• Connection pooling for S3
• CDN integration for downloads
• Worker threads for CPU-intensive tasks

Common pitfalls I encountered:

• Forgetting to delete temp files after upload
• Not setting proper S3 CORS policies
• Overlooking client-side validation
• Miscalculating file size limits

The complete flow looks like:

1. Client requests presigned URL from server
2. Server validates permissions and returns URL
3. Client uploads directly to S3
4. On completion, server processes if needed
5. File metadata stored in database

This architecture handles 90% of use cases. For specialized needs like video processing, consider dedicated services like AWS MediaConvert.

Building this taught me that file uploads seem simple but require careful design. Each component must excel at its specific task while playing nicely with others. What optimization tricks have you discovered in your projects?

If you found this useful, share it with others facing similar challenges. I’d love to hear about your implementation experiences in the comments!