Ever had an API request timeout because it was processing a massive image upload? I faced this exact challenge last month while optimizing our notification system. That moment sparked my journey into distributed task queues - systems that handle background jobs without blocking user interactions. Today, I’ll walk you through building one with BullMQ, Redis, and TypeScript.

First, why should you care about task queues? Imagine 10,000 users simultaneously requesting email notifications. Without queues, your server would crash. With queues, jobs get processed sequentially in the background while users receive instant responses. See the difference:

// Blocking approach - avoid this!
app.post('/send-email', async (req, res) => {
  await sendEmail(req.body); // User waits for completion
  res.sendStatus(200);
});

// Queue-powered solution - recommended
app.post('/send-email', async (req, res) => {
  await emailQueue.add('notification', req.body); // Immediate response
  res.json({ queued: true, message: "Processing started" });
});

Setting up is straightforward. We need Redis as our job storage backbone - install it locally or use Docker. For our TypeScript project:

npm install bullmq redis ioredis @types/node typescript

Now, configure Redis properly. Notice the retry settings - they’re crucial for production resilience:

// redis.config.ts
export const redisConfig = {
  host: process.env.REDIS_HOST || 'localhost',
  port: 6379,
  password: process.env.REDIS_PASSWORD,
  retryDelayOnFailover: 100,
  maxRetriesPerRequest: 3
};

export const queueConfig = {
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 5000 }
  }
};

The real magic happens in our queue manager. We’ll create a reusable BaseQueue class that handles logging, events, and errors. Why reinvent the wheel when you can build an extensible foundation?

// base-queue.ts
export abstract class BaseQueue<T> {
  protected queue: Queue;

  constructor(queueName: string) {
    this.queue = new Queue(queueName, { 
      connection: redisConfig,
      ...queueConfig
    });
    this.setupEventHandlers();
  }

  private setupEventHandlers(): void {
    this.queue.on('failed', (job, err) => {
      console.error(`Job ${job?.id} failed:`, err);
    });
    
    // More event listeners for 'completed', 'waiting', etc.
  }

  async addJob(jobType: string, data: T): Promise<void> {
    await this.queue.add(jobType, data);
  }
}

Now, for a concrete implementation - our EmailQueue. Notice how we extend BaseQueue for type safety:

// email-queue.ts
interface EmailData {
  to: string;
  subject: string;
  template: string;
}

class EmailQueue extends BaseQueue<EmailData> {
  constructor() {
    super('email-queue');
  }

  async sendNotification(emailData: EmailData): Promise<void> {
    await this.addJob('send-email', emailData);
  }
}

Workers bring our queued jobs to life. They’re separate processes that listen for jobs and execute logic. Here’s a pattern I’ve found invaluable - wrapping processors in error handlers:

// email-worker.ts
const worker = new Worker('email-queue', async job => {
  try {
    await sendActualEmail(job.data); // Your mail service integration
  } catch (error) {
    console.error(`Delivery failed for ${job.data.to}`);
    throw error; // Triggers BullMQ's retry mechanism
  }
}, { connection: redisConfig });

What happens when jobs fail? Our configuration automatically retries with exponential backoff. After 3 failures, jobs move to the dead-letter queue for inspection. Ever wondered how platforms retry failed payments? This is their secret sauce.

Monitoring is non-negotiable in production. BullMQ’s QueueMetrics gives real-time insights:

// monitoring.ts
const metrics = await emailQueue.getJobCounts();
console.log(`
  Active: ${metrics.active}
  Completed: ${metrics.completed}
  Failed: ${metrics.failed}
  Waiting: ${metrics.waiting}
`);

For deployment, run workers in Kubernetes pods or PM2 clusters. Scale horizontally by increasing worker instances - Redis handles coordination automatically. Remember to set memory limits though; I learned this the hard way when a memory leak crashed our servers!

Common pitfalls? Always:

1. Gracefully shut down workers (worker.close())
2. Limit concurrent jobs per worker
3. Use unique job IDs for idempotency
4. Monitor Redis memory usage

Want to handle scheduled jobs? Try:

// Daily digest scheduler
await emailQueue.add('daily-digest', {}, { 
  repeat: { pattern: '0 9 * * *' } // 9 AM daily
});

Building this transformed our system’s reliability - we now process 500K jobs daily with zero downtime. The best part? You can implement this in a weekend.

Found this useful? Share it with your team! Have questions or war stories about task queues? Drop them in the comments - let’s learn from each other’s experiences. If this saved you hours of debugging, consider liking this post to help others discover it too.