Message Queues

A message queue decouples a producer (the service that generates work) from a consumer (the service that processes it) by placing a durable buffer between them. Instead of service A calling service B directly and waiting for a response, A drops a message into the queue and returns immediately; B picks it up when ready. This pattern enables async processing, load leveling (the queue absorbs traffic spikes), retry on failure, and loose coupling — B can go down and restart without A caring. Message queues are fundamental to resilient microservices and any system that needs to offload slow, heavy, or unreliable work. This page covers the core concepts before diving into RabbitMQ and Kafka.

Queues vs direct calls

Text

SYNCHRONOUS (direct call)             ASYNC (message queue)
   A ──HTTP/gRPC──► B                   A ──publish──► [Queue] ──consume──► B
       waits for reply                  A returns immediately  B processes when ready

   Problems with direct:                Benefits of queue:
   • B slow → A blocks                  • A never blocks on B's speed
   • B down → A errors immediately      • B can restart; messages wait
   • Traffic spike → B overwhelmed      • Queue absorbs spikes; B processes at pace
   • A must know B's address            • A doesn't know B exists (loose coupling)

A queue buffers work between producer and consumer — A sends and returns immediately; B processes at its own pace, surviving restarts

The fundamental shift is temporal decoupling: the producer and consumer **don't need to be available at the same time**. The producer puts a message on the queue and continues; the consumer reads from the queue when it's ready. If the consumer is slow, the queue grows (a visible backlog you can monitor and alert on) rather than backing up the producer. If the consumer crashes and restarts, the broker holds the unacknowledged messages and redelivers them. This makes the system resilient to consumer downtime, traffic spikes, and processing variability — problems that synchronous direct calls simply cannot absorb. The cost is that you now have **eventual consistency**: the producer can't know immediately whether the work succeeded, so workflows that need a synchronous answer still need HTTP or gRPC.

Key concepts

Concept	Meaning
Producer	Publishes messages to a queue or topic
Consumer	Reads and processes messages, then acknowledges them
Acknowledgement (ack)	Consumer signals "I processed this" — broker removes it from the queue
Negative ack / nack	Consumer signals failure — broker requeues or dead-letters the message
Dead-letter queue (DLQ)	Holds messages that failed too many times for manual inspection
At-least-once delivery	The broker may redeliver — consumers must be idempotent
Exactly-once	Hard to guarantee end-to-end; most systems aim for at-least-once + idempotency

Ack when work is done, not when received — nack on failure so the broker redelivers, and use a DLQ for permanently failing messages

**Acknowledgement** is the most important operational detail. A consumer should **ack** a message only after it has **successfully processed and persisted** the result — not when it received the message. If the consumer acks on receipt and then crashes mid-processing, the work is lost (the broker won't redeliver an acked message). If it crashes before acking, the broker sees the message as unacknowledged and redelivers to another consumer. The consequence of at-least-once delivery is that consumers must be **idempotent** — processing the same message twice produces the same result as processing it once (de-duplicate on a message id, use database upserts). When a message fails repeatedly (bad data, persistent downstream error), **nack with no requeue** and let it land in the **dead-letter queue** for human inspection, rather than letting it loop forever and block other messages.

Queue vs pub/sub vs stream

Pattern	Semantics	Use when
Queue	Each message consumed by ONE consumer (competing consumers)	Distribute work across worker pool (email sending, jobs)
Pub/sub	Each message delivered to ALL subscribers (fan-out)	Notify multiple services of an event (user created → email + analytics)
Stream / log	Ordered, persistent log; consumers read at their own position	Event sourcing, replay, audit log (Kafka)

Queues distribute work; pub/sub broadcasts events; streams are persistent ordered logs — pick by whether you want one consumer or many

These three patterns differ in **delivery semantics**, not just implementation. A **work queue** distributes a message to exactly one consumer out of a pool — ideal for processing jobs where each job needs to be done once. **Pub/sub** (publish/subscribe) delivers every message to every subscriber — ideal for broadcasting domain events to multiple independent services (an order created event goes to email *and* analytics *and* inventory). A **stream** (Kafka's model) is a persistent ordered log where each consumer group tracks its own read position — messages aren't deleted on consumption, enabling replay, audit, and multiple independent consumers reading at different speeds. [RabbitMQ](/nodejs/rabbitmq) is the classic queue/pub-sub broker; [Kafka](/nodejs/kafka) is the canonical stream.

The most popular queue broker for Node: [RabbitMQ](/nodejs/rabbitmq).