Apache Kafka — Node.js

Kafka with Node.js

Apache Kafka is a distributed event-streaming platform built around a persistent, ordered, partitioned log. Unlike RabbitMQ, where messages are deleted after acknowledgement, Kafka retains messages for a configurable retention period — any consumer can replay the log from any offset. This makes Kafka the right choice for event sourcing, audit logs, stream processing, high-throughput pipelines, and scenarios where multiple independent consumer groups need to process the same events at their own pace. This page explains the Kafka model (brokers, topics, partitions, offsets, consumer groups) and then covers the KafkaJS Node client.

Core model: topics, partitions, offsets

Text

Topic: "orders"   (retention: 7 days — messages are NOT deleted on consume)
  │
  ├─ Partition 0:  offset 0 → offset 1 → offset 2 → offset 3 …
  ├─ Partition 1:  offset 0 → offset 1 → offset 2 …
  └─ Partition 2:  offset 0 → offset 1 …

Consumer Group A (email service):
  worker-1 ← reads Partition 0   worker-2 ← reads Partition 1   worker-3 ← reads Partition 2
  (each worker reads its partitions at its OWN offset — independent of other groups)

Consumer Group B (analytics service):
  worker-1 ← reads Partition 0, 1, 2   (can lag behind Group A independently)

Kafka is a persistent log, not a queue — messages stay after consumption; consumer groups each track their own offset and can replay independently

The partition is the unit of parallelism and ordering in Kafka. Within a partition, messages are **strictly ordered** and assigned monotonically increasing **offsets**. Messages are NOT deleted when consumed — they age out based on retention time or size. A **consumer group** is a set of workers that cooperate to consume a topic: Kafka assigns partitions to workers such that each partition is handled by exactly one worker in the group. If you have 3 partitions and 3 workers, each worker gets one partition. If you have 3 partitions and 5 workers, 2 workers sit idle (you can never have more active workers than partitions). Multiple consumer groups are fully independent — an email service group and an analytics service group both read the same topic with no coordination. This is what makes Kafka fundamentally different from a work queue: each consumer group gets every message, at its own pace, with the ability to replay.

Kafka vs RabbitMQ

Dimension	RabbitMQ	Kafka
Message retention	Deleted after ack	Retained for configurable duration (days/weeks)
Delivery model	Push to consumers	Consumers pull at their own pace
Ordering	Per-queue FIFO	Per-partition strict order
Throughput	Hundreds of thousands/sec	Millions of messages/sec
Consumer groups	Competing consumers share a queue	Each group reads all messages independently
Replay	Not possible (message gone after ack)	Yes — seek to any offset
Routing	Exchanges, bindings, routing keys	Topic + partition key
Operational complexity	Moderate (single broker for dev)	High (ZooKeeper/KRaft, replication, retention config)
Best for	Task queues, RPC-style work, flexible routing	Event streams, audit logs, high throughput, replay

Installing KafkaJS

Bash

npm install kafkajs

Connecting and creating a topic

import { Kafka } from 'kafkajs'

// One Kafka instance per application — reuse it across producers and consumers
export const kafka = new Kafka({
  clientId: 'my-app',
  brokers: [process.env.KAFKA_BROKER ?? 'localhost:9092'],
  // In production: SSL + SASL credentials:
  // ssl: true,
  // sasl: { mechanism: 'plain', username: process.env.KAFKA_USER, password: process.env.KAFKA_PASS },
})

// Create topics (usually done by ops/Terraform, but useful for dev setup):
async function createTopics() {
  const admin = kafka.admin()
  await admin.connect()
  await admin.createTopics({
    topics: [
      { topic: 'orders', numPartitions: 3, replicationFactor: 1 },
    ],
  })
  await admin.disconnect()
}

Producer — publishing events

import { kafka } from './kafka'

const producer = kafka.producer()

export async function publishOrderEvent(order: { id: string; userId: string; total: number }) {
  await producer.connect()

  await producer.send({
    topic: 'orders',
    messages: [
      {
        // key determines partition — same userId always goes to same partition,
        // guaranteeing order of events for that user
        key: order.userId,
        value: JSON.stringify({ type: 'order.created', data: order }),
        // optional headers for metadata:
        headers: { 'content-type': 'application/json' },
      },
    ],
  })
}

// In a real app, connect the producer once at startup, not per message:
async function startProducer() {
  await producer.connect()
  // producer is now ready; publishOrderEvent() can be called without reconnecting
}

Use a consistent partition key for related events — same key → same partition → strict ordering guaranteed for that entity's events

**Partition keys** are the mechanism for ordering guarantees. Kafka hashes the key to a partition number, so all messages with the same key go to the same partition and are read in order. For an orders service, keying on `userId` ensures all events for a given user (order created, order updated, order shipped) arrive in order at the consumer. Without a key, Kafka distributes round-robin across partitions — fine for parallelism, but no ordering. The tradeoff: a hot user (many events) can create a hot partition that becomes a bottleneck while others sit idle. For hotspot scenarios, add a suffix to spread load while preserving logical grouping (e.g. `userId:shard`).

Consumer — reading events

import { kafka } from './kafka'

const consumer = kafka.consumer({ groupId: 'email-service' })

export async function startConsumer() {
  await consumer.connect()
  await consumer.subscribe({ topic: 'orders', fromBeginning: false })
  // fromBeginning: true → replay all retained messages from offset 0
  // fromBeginning: false → start from latest (new messages only)

  await consumer.run({
    // eachMessage is called once per message, in partition order
    eachMessage: async ({ topic, partition, message }) => {
      const value = message.value?.toString()
      if (!value) return

      try {
        const event = JSON.parse(value)
        await handleOrderEvent(event)
        // KafkaJS auto-commits the offset after eachMessage resolves successfully
      } catch (err) {
        console.error('Failed to process message:', { partition, offset: message.offset, err })
        // Don't throw — KafkaJS will stop processing and retry from this offset on throw.
        // Log it and move on, or route to a dead-letter topic.
      }
    },
  })
}

async function handleOrderEvent(event: { type: string; data: unknown }) {
  if (event.type === 'order.created') {
    // send confirmation email...
  }
}

Throwing inside eachMessage causes KafkaJS to retry the same message in a loop — handle errors explicitly and route permanently failing messages to a dead-letter topic

Unlike RabbitMQ where you explicitly `nack` a message, KafkaJS ties retry behavior to whether the `eachMessage` function throws. If it throws, KafkaJS re-processes the same message — indefinitely by default. This is the right behavior for **transient errors** (network blip, temporary DB unavailability) but catastrophic for **permanent errors** (malformed message, logic bug): you'll block that partition forever and never move to the next message. The recommended pattern: wrap the handler in try/catch, log the error with the partition and offset for manual investigation, optionally publish the failing message to a **dead-letter topic** (`orders.dlq`), and return normally so Kafka commits the offset and continues. Use a circuit-breaker or alerting on your DLQ rather than letting bad messages halt processing.

Consumer with manual offset commit

await consumer.run({
  autoCommit: false,           // take control of offset commits
  eachMessage: async ({ topic, partition, message, heartbeat, commitOffsets }) => {
    await processMessage(message)

    // Commit after successful processing — ensures at-least-once delivery
    await commitOffsets([{ topic, partition, offset: (Number(message.offset) + 1).toString() }])
    // offset to commit = current offset + 1 (next message to read)
  },
})

Auto-commit is convenient but can cause message loss — if auto-commit fires before processing completes and the consumer crashes, that message is skipped

KafkaJS defaults to **auto-committing** offsets at an interval (5 seconds). The risk: auto-commit fires while you're still processing a batch, then the consumer crashes — the committed offset means Kafka thinks those messages were processed, but they weren't. **Manual commit** after processing gives you true at-least-once guarantees at the cost of more explicit code. The offset you commit is always `current + 1` — the offset of the **next message you want to receive**, not the one you just processed. For high-throughput scenarios, commit once per batch rather than per message to reduce broker round-trips.

Seeking and replaying

// Seek to a specific offset — useful for replay or recovery:
consumer.on(consumer.events.GROUP_JOIN, async () => {
  // Seek to offset 0 on all partitions to replay the entire topic:
  const topicMetadata = await kafka.admin().fetchTopicMetadata({ topics: ['orders'] })
  const partitions = topicMetadata.topics[0].partitions.map(p => p.partitionId)

  for (const partition of partitions) {
    consumer.seek({ topic: 'orders', partition, offset: '0' })
  }
})

Graceful shutdown

process.on('SIGTERM', async () => {
  console.log('Shutting down Kafka consumer...')
  await consumer.disconnect()   // commits pending offsets, closes gracefully
  await producer.disconnect()
  process.exit(0)
})

For synchronous service-to-service communication with binary efficiency, see [gRPC](/nodejs/grpc).