I’ve been thinking a lot lately about how modern applications handle heavy workloads without slowing down. The answer often lies in distributed task queues – systems that manage background jobs efficiently across multiple workers. Today, I want to share my approach to building a robust queue system using BullMQ, Redis, and TypeScript.

Have you ever wondered how platforms process thousands of emails or generate complex reports without affecting user experience?

Let me show you how to set up a production-ready system. First, we need to establish our Redis connection – the backbone of our queue system. Here’s how I typically configure it:

import { Redis } from 'ioredis';

const redis = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  maxRetriesPerRequest: null
});

Now, let’s create our first queue. I prefer defining queues with TypeScript interfaces for type safety:

interface EmailJobData {
  to: string;
  subject: string;
  template: string;
}

const emailQueue = new Queue<EmailJobData>('emails', { connection: redis });

What happens when a job fails? BullMQ provides excellent retry mechanisms. Here’s my approach to handling failures:

emailQueue.add('welcome-email', {
  to: '[email protected]',
  subject: 'Welcome!',
  template: 'welcome'
}, {
  attempts: 3,
  backoff: {
    type: 'exponential',
    delay: 2000
  }
});

Now let’s create a worker to process these jobs. Notice how TypeScript helps us catch errors early:

new Worker<EmailJobData>('emails', async (job) => {
  const { to, subject, template } = job.data;
  
  // Simulate email sending
  await sendEmail({ to, subject, template });
  
  // Update progress
  await job.updateProgress(100);
}, { connection: redis });

But how do we monitor what’s happening in our queues? BullMQ provides built-in methods for tracking:

// Get job counts
const counts = await emailQueue.getJobCounts('waiting', 'active', 'completed');

// Listen for events
emailQueue.on('completed', (job) => {
  console.log(`Job ${job.id} completed successfully`);
});

emailQueue.on('failed', (job, error) => {
  console.error(`Job ${job.id} failed:`, error.message);
});

Scaling becomes crucial as your application grows. I often deploy multiple worker instances:

// In your worker setup
const worker = new Worker('emails', processor, {
  connection: redis,
  concurrency: 10, // Process 10 jobs simultaneously
  limiter: {
    max: 1000, // Max jobs per interval
    duration: 5000 // 5 seconds
  }
});

What about prioritizing urgent jobs? BullMQ makes this straightforward:

// High priority email
await emailQueue.add('urgent-email', data, {
  priority: 1, // Higher priority
  delay: 0 // Process immediately
});

// Regular email
await emailQueue.add('regular-email', data, {
  priority: 10, // Lower priority
  delay: 60000 // Wait 1 minute
});

Error handling is where TypeScript really shines. I create custom error types for different failure scenarios:

class QueueError extends Error {
  constructor(
    message: string,
    public readonly jobId?: string,
    public readonly retryable: boolean = true
  ) {
    super(message);
  }
}

// In your worker
try {
  await processJob(job.data);
} catch (error) {
  if (error instanceof NetworkError) {
    throw error; // Retry network issues
  } else {
    throw new QueueError('Processing failed', job.id, false);
  }
}

Monitoring and metrics are essential for production systems. Here’s how I track performance:

const metrics = {
  completed: 0,
  failed: 0,
  duration: 0
};

worker.on('completed', (job) => {
  metrics.completed++;
  metrics.duration += job.processedOn! - job.timestamp;
});

worker.on('failed', (job) => {
  metrics.failed++;
});

Remember to always clean up completed jobs to prevent Redis memory issues:

// Clean old completed jobs
await emailQueue.clean(1000, 1000, 'completed');
await emailQueue.clean(1000, 1000, 'failed');

Building with BullMQ has transformed how I handle background processing. The combination of Redis persistence, TypeScript type safety, and BullMQ’s robust features creates a system that’s both reliable and scalable.

What challenges have you faced with background job processing? I’d love to hear your experiences and solutions. If you found this useful, please share it with others who might benefit from these patterns. Your comments and feedback help me create better content for our community.