I’ve been building web applications for years, and one challenge that consistently arises is handling file uploads efficiently. Whether it’s user avatars, document submissions, or media libraries, file uploads can quickly become a bottleneck in any system. I recently tackled this problem head-on while developing a content management platform, and I want to share my approach to creating a robust, scalable file upload system using NestJS, Bull Queue, and AWS S3. If you’ve ever struggled with slow uploads, memory issues, or unreliable file processing, this guide might change how you handle files forever.

Why did I choose this specific tech stack? NestJS provides a solid foundation with its modular architecture and excellent TypeScript support. Bull Queue handles background processing reliably, while AWS S3 offers virtually limitless storage scalability. Together, they create a powerful combination that can grow with your application’s needs. Have you ever wondered how large platforms handle thousands of simultaneous uploads without crashing?

Let’s start with the basic setup. You’ll need Node.js 18 or higher, a Redis server for queue management, PostgreSQL for metadata storage, and an AWS account with S3 access. Here’s how I typically initialize the project:

nest new file-upload-system
cd file-upload-system
npm install @nestjs/bull bull redis @nestjs/websockets @nestjs/platform-socket.io @nestjs/typeorm typeorm pg aws-sdk @aws-sdk/client-s3 multer sharp @nestjs/config @nestjs/throttler

Configuration is crucial for maintaining different environments. I create a dedicated configuration module that handles all environment variables securely:

// src/config/configuration.ts
export default () => ({
  database: {
    host: process.env.DATABASE_HOST,
    port: parseInt(process.env.DATABASE_PORT, 10) || 5432,
    username: process.env.DATABASE_USERNAME,
    password: process.env.DATABASE_PASSWORD,
    database: process.env.DATABASE_NAME,
  },
  redis: {
    host: process.env.REDIS_HOST,
    port: parseInt(process.env.REDIS_PORT, 10) || 6379,
  },
  aws: {
    region: process.env.AWS_REGION,
    s3Bucket: process.env.AWS_S3_BUCKET,
  },
});

The architecture follows a clear flow: when a user uploads a file, it first goes through validation, then gets queued for processing, and finally moves to S3 storage. But what happens when multiple users upload large files simultaneously? That’s where Bull Queue shines by managing the workload efficiently.

Implementing the upload endpoint requires careful consideration of security and performance. I use Multer for handling multipart/form-data and implement strict validation rules:

// src/file-upload/file-upload.controller.ts
@Post('upload')
@UseInterceptors(FileInterceptor('file'))
async uploadFile(@UploadedFile() file: Express.Multer.File) {
  const job = await this.fileQueue.add('process-file', {
    file: file,
    userId: 'current-user-id',
  });
  return { jobId: job.id, status: 'queued' };
}

Bull Queue configuration is straightforward but powerful. I set up separate queues for different file types and priorities:

// src/file-upload/file.processor.ts
@Processor('file-queue')
export class FileProcessor {
  constructor(private s3Service: S3Service) {}

  @Process('process-file')
  async handleFileProcessing(job: Job) {
    const { file, userId } = job.data;
    // Process and upload to S3
    await this.s3Service.uploadFile(file);
    await job.progress(100);
  }
}

AWS S3 integration becomes simple with the AWS SDK v3. I create a dedicated service that handles all S3 operations:

// src/s3/s3.service.ts
async uploadFile(file: Express.Multer.File) {
  const uploadResult = await this.s3Client.send(
    new PutObjectCommand({
      Bucket: this.bucketName,
      Key: `uploads/${Date.now()}-${file.originalname}`,
      Body: file.buffer,
      ContentType: file.mimetype,
    })
  );
  return uploadResult;
}

Image processing is where Sharp library demonstrates its value. It efficiently handles resizing and optimization without consuming excessive memory:

// src/image-processor/image-processor.service.ts
async resizeImage(buffer: Buffer, width: number, height: number) {
  return sharp(buffer)
    .resize(width, height)
    .jpeg({ quality: 80 })
    .toBuffer();
}

Real-time progress tracking through WebSockets significantly improves user experience. I implement Socket.io to send progress updates back to the client:

// src/progress/progress.gateway.ts
@WebSocketGateway()
export class ProgressGateway {
  @WebSocketServer()
  server: Server;

  async sendProgress(jobId: string, progress: number) {
    this.server.emit(`progress-${jobId}`, { progress });
  }
}

Database models store essential file metadata, making retrieval and management efficient. I use TypeORM for database interactions:

// src/file-upload/entities/file.entity.ts
@Entity()
export class FileEntity {
  @PrimaryGeneratedColumn('uuid')
  id: string;

  @Column()
  originalName: string;

  @Column()
  s3Key: string;

  @Column()
  mimeType: string;

  @Column()
  size: number;

  @CreateDateColumn()
  createdAt: Date;
}

Error handling deserves special attention. I implement retry mechanisms for failed uploads and comprehensive logging:

// src/file-upload/file.processor.ts
@Process('process-file')
async handleFileProcessing(job: Job) {
  try {
    await this.processFile(job.data.file);
  } catch (error) {
    if (job.attemptsMade < 3) {
      throw error; // Bull will retry
    }
    await this.handlePermanentFailure(job, error);
  }
}

Security measures include file type validation, size limits, and sanitization. I always validate files before processing and use throttling to prevent abuse:

// src/file-upload/file-upload.controller.ts
@UseInterceptors(FileInterceptor('file', {
  limits: {
    fileSize: 10 * 1024 * 1024, // 10MB
  },
  fileFilter: (req, file, callback) => {
    if (!file.originalname.match(/\.(jpg|jpeg|png|pdf)$/)) {
      return callback(new Error('Invalid file type'), false);
    }
    callback(null, true);
  },
}))

Testing is essential for reliability. I write comprehensive unit and integration tests using Jest:

// test/file-upload.e2e-spec.ts
describe('File Upload', () => {
  it('should upload a file successfully', async () => {
    const response = await request(app.getHttpServer())
      .post('/upload')
      .attach('file', './test/fixtures/sample.jpg')
      .expect(201);
    expect(response.body.jobId).toBeDefined();
  });
});

Performance optimization includes implementing streaming for large files and connection pooling for database operations. Monitoring with proper logging helps identify bottlenecks early. Did you know that proper connection management can improve throughput by up to 40%?

Common pitfalls include not handling memory properly with large files and missing edge cases in file validation. I always test with various file types and sizes to ensure stability. Another frequent mistake is not implementing proper cleanup procedures for failed uploads.

This system has served me well in production environments, handling thousands of uploads daily. The modular design makes it easy to extend functionality, such as adding video processing or integrating with CDNs. What other features would you add to make this system even more powerful?

I hope this guide helps you build better file upload systems. If you found this useful, please share it with others who might benefit. I’d love to hear about your experiences and any improvements you’ve implemented—leave a comment below with your thoughts and questions!