Service Communication Patterns
In a microservices system, services communicate in two fundamental modes: synchronous (the caller waits for a response) and asynchronous (the caller publishes an event and continues). Neither is universally better — they're tools with different trade-offs, and most real systems use both. This page maps out the decision: when to use REST, gRPC, message queues, or event streaming, and how the choice cascades into system behavior around failure, latency, and coupling.
Synchronous vs asynchronous
SYNCHRONOUS (request/response) ASYNCHRONOUS (event-driven)
Service A ──► Service B Service A ──► [Queue/Stream] ──► Service B
waits for reply returns immediately
A's latency = sum of all downstream A's latency is independent of B
A fails if B is down A succeeds even if B is down
Tight coupling (A knows B's address) Loose coupling (A knows the event, not who handles it)
Simple to reason about for one call Harder to trace end-to-end
Needs circuit breakers + timeouts Needs DLQs + idempotency + eventual consistencyCommunication pattern decision guide
Pattern | Protocol | When to use |
|---|---|---|
REST/HTTP | HTTP/1.1 + JSON | Public APIs, browser clients, external integrations, when simplicity beats performance |
gRPC | HTTP/2 + Protobuf | Internal service calls needing speed/type safety, streaming, polyglot services |
Message queue | AMQP (RabbitMQ) | Task distribution, async jobs, retry on failure, rate leveling |
Event stream | Kafka | High throughput, audit logs, multiple consumers, event sourcing, replay |
WebSocket | TCP upgrade | Real-time push to browser clients (notifications, live data) |
The synchronous call chain problem
API Gateway ──► Orders ──► Inventory ──► Payments ──► Notifications
100ms + 80ms + 200ms + 150ms + 50ms
= 580ms total
If Payments is down: API Gateway returns 500 to the user
If Inventory is slow: Every request to Orders is slow
If Notifications is slow: Payments blocks waiting for it
FIX: Make side-effect calls async:
API Gateway ──► Orders ──► Inventory ──► Payments
│ │
└──►[queue] └──►[queue]
│ │
Notifications Receipt email
Response time: 430ms (not 580ms)
Notifications down → payment still succeeds; event is redelivered when it comes back upResilience patterns for synchronous calls
import axios from 'axios'
// 1. Always set a timeout — without it, a slow service hangs your request forever:
const inventoryClient = axios.create({
baseURL: process.env.INVENTORY_SERVICE_URL,
timeout: 3000, // fail fast after 3s
})
// 2. Retry with exponential backoff — for transient errors only:
async function callWithRetry<T>(fn: () => Promise<T>, maxRetries = 3): Promise<T> {
for (let attempt = 0; attempt <= maxRetries; attempt++) {
try {
return await fn()
} catch (err: any) {
const isTransient = err.response?.status >= 500 || err.code === 'ECONNRESET'
if (!isTransient || attempt === maxRetries) throw err
const delay = 100 * 2 ** attempt + Math.random() * 100 // jitter prevents thundering herd
await new Promise(r => setTimeout(r, delay))
}
}
throw new Error('unreachable')
}
// 3. Circuit breaker — stop calling a failing service to let it recover:
// Use a library like 'opossum' for production:
// import CircuitBreaker from 'opossum'
// const breaker = new CircuitBreaker(callInventory, { timeout: 3000, errorThresholdPercentage: 50 })Idempotency keys for safe retries
import { randomUUID } from 'node:crypto'
// Producer: generate a stable key for each logical operation
const idempotencyKey = randomUUID() // or derive from request ID
await axios.post('/payments', { amount: 100, currency: 'USD' }, {
headers: { 'Idempotency-Key': idempotencyKey },
})
// Receiver (payments service): store the key and result; return cached result on duplicate:
async function processPayment(req: Request, res: Response) {
const key = req.headers['idempotency-key'] as string
if (key) {
const cached = await redis.get(`idem:${key}`)
if (cached) return res.json(JSON.parse(cached)) // return same result as first call
}
const result = await chargeCard(req.body)
if (key) await redis.setex(`idem:${key}`, 86400, JSON.stringify(result))
res.json(result)
}Choosing the right pattern — a decision tree
Does the caller need a response before it can proceed? → synchronous (REST or gRPC).
Is this a side effect the caller doesn't need to wait for? → async message queue or event.
Is this internal, performance-critical, and typed between two services you control? → gRPC over REST.
Will multiple independent services need to react to this event? → event stream (Kafka fanout) or pub/sub exchange (RabbitMQ fanout).
Does order of events for a given entity matter? → Kafka with a partition key (not a work queue).
Do you need replay or audit? → Kafka (retained log) over RabbitMQ (messages deleted on ack).
Is this a browser or mobile client? → REST; gRPC requires a proxy.
The saga pattern for distributed transactions
Problem: creating an order requires atomically updating Orders + Inventory + Payments.
No single ACID transaction spans three services.
Choreography-based saga (event-driven):
Orders publishes "order.created"
→ Inventory consumes, reserves stock, publishes "inventory.reserved"
→ Payments consumes, charges card, publishes "payment.completed"
→ Orders consumes, marks order CONFIRMED
If Payments fails → publishes "payment.failed"
→ Inventory consumes, releases reserved stock (compensating transaction)
→ Orders consumes, marks order CANCELLED
Orchestration-based saga (central coordinator):
Saga orchestrator calls each service in sequence,
tracks state, and issues compensating calls on failure.