I’ve been thinking about how modern applications handle heavy workloads without slowing down user experiences. That’s what led me to explore distributed task queues – a system that can process background jobs efficiently while keeping your application responsive. Today, I want to share how you can build one using BullMQ, Redis, and TypeScript.

Have you ever wondered how applications send thousands of emails without crashing or process large files while remaining responsive? The secret often lies in distributed task queues.

Let me show you how to set up a production-ready system. First, we need Redis running. I prefer using Docker for consistency across environments.

docker run -d -p 6379:6379 redis:7-alpine

Now, let’s create our core queue setup with TypeScript. The type safety here is crucial – it prevents runtime errors and makes our code self-documenting.

interface EmailJob {
  to: string;
  subject: string;
  body: string;
}

const emailQueue = new Queue<EmailJob>('email', {
  connection: { host: 'localhost', port: 6379 }
});

What happens when a job fails? BullMQ provides sophisticated retry mechanisms. Here’s how I configure mine:

const queue = new Queue('processing', {
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 1000
    }
  }
});

The real power comes when we separate job producers from consumers. This allows horizontal scaling – you can run multiple workers across different machines.

Here’s a simple worker implementation:

const worker = new Worker<EmailJob>('email', async job => {
  await sendEmail(job.data);
}, { connection: redis });

But what about monitoring? I always include queue metrics to track performance. BullMQ’s built-in events make this straightforward.

queue.on('completed', job => {
  console.log(`Job ${job.id} completed`);
});

queue.on('failed', (job, err) => {
  console.error(`Job ${job?.id} failed:`, err);
});

For high-priority tasks, I use job prioritization. This ensures critical jobs get processed first.

await queue.add('urgent-email', data, {
  priority: 1, // Higher priority
  delay: 5000  // Process after 5 seconds
});

One question I often get: how do you handle different job types in the same system? I create separate queues for different workloads.

const queues = {
  email: new Queue<EmailJob>('email', { connection: redis }),
  image: new Queue<ImageJob>('image-processing', { connection: redis }),
  report: new Queue<ReportJob>('reports', { connection: redis })
};

Error handling deserves special attention. I implement dead letter queues for jobs that repeatedly fail.

const worker = new Worker('email', async job => {
  try {
    await processJob(job);
  } catch (error) {
    await deadLetterQueue.add('failed-email', job.data);
    throw error;
  }
});

Did you know you can schedule recurring jobs? This is perfect for maintenance tasks or regular reports.

await queue.add('daily-report', data, {
  repeat: {
    pattern: '0 2 * * *' // 2 AM daily
  }
});

The combination of BullMQ and TypeScript creates a robust foundation. The type definitions catch errors during development, while Redis ensures persistence.

Here’s how I structure my job data interfaces:

interface BaseJob {
  id: string;
  timestamp: number;
  userId: string;
}

interface EmailJob extends BaseJob {
  type: 'email';
  template: 'welcome' | 'notification';
  data: Record<string, any>;
}

Scaling workers is straightforward. I run multiple instances behind a load balancer, all connected to the same Redis instance.

# Start multiple workers
node worker.js & node worker.js & node worker.js

What monitoring tools do I recommend? I combine BullMQ’s built-in events with custom metrics. This gives me real-time insights into queue health.

Performance optimization often involves tuning Redis configuration and worker concurrency. I typically start with 5-10 concurrent jobs per worker.

const worker = new Worker('email', processor, {
  concurrency: 5,
  connection: redis
});

Testing is crucial. I use BullMQ’s test utilities to verify job processing without needing a real Redis instance.

import { Queue, Worker } from 'bullmq';
import { describe, it } from '@jest/globals';

describe('Email Queue', () => {
  it('processes jobs correctly', async () => {
    const queue = new Queue('test-email');
    const worker = new Worker('test-email', processor);
    
    await queue.add('test', { to: '[email protected]' });
    // Add assertions here
  });
});

The beauty of this system is its resilience. Even if your application restarts, Redis preserves all pending jobs. They’ll resume processing once workers are back online.

I hope this gives you a solid foundation for building your own distributed task queue system. The patterns I’ve shared have served me well in production environments handling millions of jobs.

What challenges have you faced with background job processing? I’d love to hear your experiences in the comments below. If you found this helpful, please share it with other developers who might benefit from these insights.