Have you ever struggled to trace a critical bug across distributed microservices? I faced this challenge recently in a production outage, which inspired me to build a robust logging solution. The system combines Winston for structured logging, OpenTelemetry for distributed tracing, and ELK Stack for visualization - a powerful trio that transformed our debugging capabilities. Let me share how you can implement this.

First, ensure you have Node.js 16+ installed. You’ll need Docker for running ELK containers and basic Express.js knowledge. We’ll use TypeScript for type safety - consider how much time you could save with compile-time checks?

Initialize your project:

npm init -y
npm install express winston @opentelemetry/api @elastic/elasticsearch

Our Docker setup runs the entire ELK stack and Jaeger for tracing. This docker-compose file gets everything running:

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    ports: ["9200:9200"]
  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports: ["5601:5601"]
  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    ports: ["5044:5044"]

For Winston, we create a logger with custom formatting. Notice how we include OpenTelemetry trace IDs for correlation:

// logger.ts
import winston from 'winston';
import { context } from '@opentelemetry/api';

const logger = winston.createLogger({
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format((info) => {
      const span = context.active();
      if (span) {
        info.traceId = span.spanContext().traceId;
        info.spanId = span.spanContext().spanId;
      }
      return info;
    })(),
    winston.format.json()
  ),
  transports: [new winston.transports.Console()]
});

Integrating OpenTelemetry gives us distributed tracing superpowers. This setup instruments Express and HTTP calls automatically:

// tracing.ts
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const provider = new NodeTracerProvider();
provider.register();

provider.addSpanProcessor(
  new SimpleSpanProcessor(
    new OTLPTraceExporter({ url: 'http://jaeger:4318/v1/traces' })
  )
);

When connecting to ELK, we use Winston’s Elasticsearch transport. Notice the index pattern that rotates daily:

// elastic-transport.ts
import { Client } from '@elastic/elasticsearch';
import { ElasticsearchTransport } from 'winston-elasticsearch';

const esClient = new Client({ node: 'http://elasticsearch:9200' });

const esTransport = new ElasticsearchTransport({
  client: esClient,
  indexPrefix: 'logs-app',
  indexSuffixPattern: 'YYYY.MM.DD'
});

logger.add(esTransport);

For error tracking, we implement proactive alerting. This Slack notifier triggers on critical errors:

// alerts.ts
import { WebClient } from '@slack/web-api';

logger.on('error', async (error) => {
  if (error.level === 'error') {
    const slack = new WebClient(process.env.SLACK_TOKEN);
    await slack.chat.postMessage({
      channel: '#alerts',
      text: `Critical error: ${error.message}`
    });
  }
});

In production, remember to:

• Set log retention policies in Elasticsearch
• Enable gzip compression for Logstash
• Use bulk writes for better performance
• Secure your endpoints with TLS

During testing, verify trace propagation with this simple middleware:

app.use((req, res, next) => {
  logger.info(`Request started: ${req.method} ${req.path}`);
  next();
});

What happens when you forget to propagate context between async operations? We use AsyncLocalStorage to maintain context:

import { AsyncLocalStorage } from 'async_hooks';
const contextStore = new AsyncLocalStorage();

app.use((req, res, next) => {
  contextStore.run(new Map(), () => {
    const traceId = generateId();
    contextStore.getStore()?.set('traceId', traceId);
    next();
  });
});

While this stack works well, consider Loki for simpler setups or Datadog for managed solutions. But for control and customization, this combination is hard to beat.

After implementing this, our mean time to resolve production issues dropped by 70%. The trace correlation between Winston logs and OpenTelemetry spans became invaluable during complex debugging sessions. What frustrating debugging scenario could this solve for you?

If you found this guide helpful, share it with your team! I’d love to hear about your logging challenges in the comments. What logging pain points keep you up at night?