Health Checks & Metrics

A health check is an HTTP endpoint that reports whether your service is working, designed to be probed automatically by load balancers, container orchestrators (Kubernetes), and uptime monitors. Get it right and a sick instance is pulled out of rotation or restarted before users notice; get it wrong and you either route traffic to broken pods or restart healthy ones in a crash loop. This page covers the crucial liveness vs readiness distinction, dependency checks, the startup probe, what to expose (and what not to), and tying health checks into graceful shutdown.

Liveness vs readiness — the distinction that matters most

Probe	Question	On failure	Checks
Liveness	Is the process alive and not deadlocked?	Orchestrator restarts the container	Cheap, self-only — is the event loop responsive?
Readiness	Can it serve traffic right now?	Orchestrator stops routing traffic (no restart)	Dependencies — DB, cache, migrations done?
Startup	Has a slow-booting app finished starting?	Delays liveness/readiness until ready	One-time init, warm-up, migrations

Don't check dependencies in the LIVENESS probe — a DB blip will trigger a pointless restart loop

This is the single most common health-check mistake. If your **liveness** probe checks the database and the DB has a brief hiccup, the orchestrator concludes the *app* is dead and restarts it — but restarting doesn't fix the database, so it restarts again, and you get a crash loop that makes the outage worse. Liveness should answer only "is this process itself wedged?" — keep it cheap and dependency-free. Dependency checks belong in **readiness**: if the DB is down, readiness fails, traffic stops routing to this instance, but the process keeps running and recovers automatically when the DB returns. Separating these correctly is what makes self-healing actually heal.

A liveness endpoint — cheap and self-contained

// Liveness: just proves the process is up and the event loop responds.
// No DB, no external calls — must be fast and never fail for external reasons.
app.get('/healthz', (req, res) => {
  res.status(200).json({ status: 'ok', uptime: process.uptime() })
})

A readiness endpoint — verify dependencies

app.get('/readyz', async (req, res) => {
  const checks = {}
  let healthy = true

  // Each dependency check has a timeout so a hung DB can't hang the probe:
  try {
    await withTimeout(db.query('SELECT 1'), 2000)
    checks.database = 'ok'
  } catch {
    checks.database = 'failing'
    healthy = false
  }

  try {
    await withTimeout(redis.ping(), 1000)
    checks.cache = 'ok'
  } catch {
    checks.cache = 'degraded'   // cache optional? decide if it fails readiness
  }

  res.status(healthy ? 200 : 503).json({ status: healthy ? 'ready' : 'not_ready', checks })
})

function withTimeout(promise, ms) {
  return Promise.race([
    promise,
    new Promise((_, reject) => setTimeout(() => reject(new Error('timeout')), ms)),
  ])
}

# Healthy:
HTTP 200  {"status":"ready","checks":{"database":"ok","cache":"ok"}}

# DB down — readiness fails, orchestrator stops routing traffic here:
HTTP 503  {"status":"not_ready","checks":{"database":"failing","cache":"ok"}}

Return 503 (not 200) when unhealthy, and time-box every dependency check

A health check signals status through the **HTTP status code**, because that's what load balancers and orchestrators inspect: `200` means healthy/ready, `503 Service Unavailable` means not ready. Returning `200` with a JSON body saying `"unhealthy"` defeats the purpose — the probe only reads the code. Equally important: wrap every dependency check in a **timeout**. A health endpoint that hangs because the DB is hung is worse than one that fails fast — it can exhaust the probe's own timeout and cause cascading restarts. Distinguish *critical* dependencies (fail readiness) from *optional* ones (report degraded but stay ready).

Kubernetes probe configuration

deployment.yaml

YAML

livenessProbe:
  httpGet: { path: /healthz, port: 3000 }
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3          # 3 fails (~30s) before restart

readinessProbe:
  httpGet: { path: /readyz, port: 3000 }
  periodSeconds: 5
  failureThreshold: 2          # stop routing quickly when deps fail

startupProbe:
  httpGet: { path: /healthz, port: 3000 }
  failureThreshold: 30         # allow up to 30 x 10s = 5 min to start
  periodSeconds: 10

The startup probe protects slow-booting apps from being killed before they're ready

Without a **startup probe**, a slow-starting app (one that runs migrations or warms a cache on boot) can be killed by the liveness probe before it ever finishes starting — the liveness check fails during boot, the orchestrator restarts it, and it never gets the chance to come up. The startup probe holds off liveness and readiness checks until the app reports started (`failureThreshold * periodSeconds` gives the max allowed boot time). Tune `initialDelaySeconds`, `periodSeconds`, and `failureThreshold` so transient blips don't trigger restarts but genuine failures are caught reasonably fast.

Health checks and graceful shutdown

let shuttingDown = false

// On SIGTERM: fail readiness FIRST so traffic drains before we close:
process.on('SIGTERM', async () => {
  shuttingDown = true                 // /readyz now returns 503
  await sleep(5000)                   // give the LB time to stop routing
  server.close(() => db.close())      // then finish in-flight + close
})

app.get('/readyz', (req, res) => {
  if (shuttingDown) return res.status(503).json({ status: 'shutting_down' })
  // ...normal dependency checks...
})

Flip readiness to 503 BEFORE closing the server so the load balancer drains traffic

During a rolling deploy or scale-down, the orchestrator sends `SIGTERM` and *then* keeps routing traffic for a short window until its own checks notice the pod is gone. If you close the server immediately on `SIGTERM`, those in-flight requests get connection-refused errors. The correct sequence: on `SIGTERM`, first make `/readyz` return `503` so the load balancer stops sending new requests, wait a few seconds for it to drain, *then* stop accepting connections and finish in-flight work. This ties directly into [graceful shutdown](/nodejs/graceful-shutdown) — health checks are how the platform learns it's time to stop sending you traffic.

What to expose — and what to keep private

Keep health/metrics endpoints internal — /metrics and detailed /readyz output can leak versions, dependency names, and topology; don't expose them publicly.
Liveness: minimal — status and uptime only; no secrets, no dependency detail.
Readiness: dependency status — enough to diagnose, but avoid connection strings or internal hostnames.
Separate the metrics port/path — scrape /metrics (Prometheus) on an internal interface, not the public one.
Version/build info endpoint is useful for ops, but gate it behind internal networking or auth.
Probe endpoints should not require auth from the orchestrator — restrict them by network instead.

When something does break, track it down systematically: [Debugging Node.js](/nodejs/debugging-intro).