Monitoring & APM

Logs tell you what happened on individual requests; monitoring tells you how the whole system is behaving right now and over time. APM — Application Performance Monitoring — goes further, tracing a single request across services, surfacing slow database queries, and pinpointing the line of code behind a latency spike. Together with logs and traces, metrics form the "three pillars of observability." This page covers metrics vs logs vs traces, the key signals to watch (the RED and USE methods), distributed tracing, APM tools, and how to monitor without drowning in noise.

The three pillars of observability

Pillar	Answers	Example
Logs	What happened on this specific event?	"payment declined for order 9981, code card_declined"
Metrics	How is the system behaving in aggregate?	Requests/sec, p95 latency, error rate, memory usage
Traces	Where did the time go across services?	Request spent 12ms in API, 340ms in DB, 80ms in payment svc

Logs, metrics, and traces answer different questions — you need all three

These pillars are complementary, not competing. **Metrics** are cheap, aggregated numbers (counters, gauges, histograms) ideal for dashboards and alerts — "error rate just crossed 5%". **Logs** are detailed per-event records you read once an alert fires — "*which* errors, and why". **Traces** stitch a single request's journey across multiple services and show the time spent in each span — essential in microservices where a slow response could originate anywhere. A mature setup uses metrics to *detect* a problem, traces to *locate* it, and logs to *explain* it.

What to measure — RED and USE

Text

RED method — for request-driven services (what users feel):
  Rate       → requests per second
  Errors     → failed requests per second (or error %)
  Duration   → latency distribution (p50 / p95 / p99)

USE method — for resources (what infrastructure shows):
  Utilization → % time the resource was busy (CPU, event loop)
  Saturation  → queued/waiting work (event-loop lag, conn pool waits)
  Errors      → error counts (failed allocations, dropped conns)

Track percentiles (p95/p99), not averages — averages hide the slow tail users actually hit

Latency is the metric most often measured wrong. An *average* response time of 80ms can hide that 1% of requests take 5 seconds — and that 1% is real users having a terrible experience. Track **percentiles**: p50 (median, the typical experience), p95, and p99 (the slow tail). A rising p99 with a flat average is the classic signature of a problem that averages mask. The RED method (Rate, Errors, Duration) gives you the three signals that matter most for any request-driven service; watch those before anything else.

Node-specific signals to watch

Event-loop lag — the #1 Node health metric. Rising lag means the single thread is blocked and every request slows down.
Heap usage & GC — climbing heap that never drops signals a memory leak; frequent long GC pauses hurt latency.
Active handles/requests — leaking sockets, timers, or DB connections.
Connection pool saturation — requests waiting for a free DB connection is a common hidden bottleneck.
Process restarts / crashes — a process manager restarting often points to uncaught exceptions or OOM kills.
External dependency latency — DB, cache, and third-party API response times, since they dominate most request durations.

Event-loop lag is the signal unique to Node — a blocked loop slows *every* request at once

Because Node runs your JavaScript on a [single thread](/nodejs/event-loop), any synchronous work that blocks the event loop — a heavy JSON parse, a sync crypto call, a tight CPU loop — delays *all* concurrent requests, not just one. **Event-loop lag** (the delay between when a timer should fire and when it actually does) is therefore the most important Node-specific health metric. Measure it (`perf_hooks.monitorEventLoopDelay()` or your APM agent) and alert on it: sustained lag above a few tens of milliseconds means users are waiting on a clogged loop, and the fix is usually to move CPU-bound work off-thread (worker threads) or make blocking calls async.

Distributed tracing with OpenTelemetry

Bash

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node

tracing.js — load this BEFORE your app (node --require ./tracing.js app.js)

import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }),
  // Auto-instruments http, express, pg, redis, etc. — no code changes:
  instrumentations: [getNodeAutoInstrumentations()],
})
sdk.start()

OpenTelemetry is the vendor-neutral standard — instrument once, export to any backend

**OpenTelemetry (OTel)** is the CNCF standard for generating traces, metrics, and logs in a vendor-neutral way. Its auto-instrumentation hooks into common libraries (HTTP, Express, Postgres, Redis) and produces **spans** automatically — so you get distributed traces showing where a request spent its time without manually editing handlers. Because the data format is standardized, you can export to Jaeger, Tempo, Datadog, Honeycomb, or any OTel-compatible backend and switch vendors without re-instrumenting. Loading the SDK *before* your application code (via `--require` or `--import`) is essential so it can patch libraries as they load.

APM tools

Category	Examples
Hosted APM	Datadog, New Relic, Dynatrace, Elastic APM, Sentry (errors + perf)
Metrics + dashboards	Prometheus (scrape `/metrics`) + Grafana
Tracing backends	Jaeger, Grafana Tempo, Zipkin, Honeycomb
Instrumentation	OpenTelemetry (vendor-neutral), `prom-client` (Prometheus metrics)

Exposing Prometheus metrics with prom-client

import client from 'prom-client'
client.collectDefaultMetrics()   // CPU, heap, event-loop lag, GC...

const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status'],
})

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType)
  res.end(await client.register.metrics())   // Prometheus scrapes this endpoint
})

Alerting without fatigue

Alert on symptoms users feel, not on every metric — noisy alerts get ignored

The fastest way to make monitoring useless is to alert on everything: when every minor blip pages someone, the team learns to ignore alerts, and the real outage gets missed too. Alert on **symptoms users actually experience** — elevated error rate, p99 latency breaching your SLO, the service being down — not on every CPU tick or single error. Make alerts *actionable* (each one should map to a clear response), set sensible thresholds and durations to avoid flapping, and route by severity (page for outages, ticket for slow degradations). Pair alerts with dashboards so an on-call engineer can immediately see context.

Give your monitors and orchestrators something to probe: [Health Checks & Metrics](/nodejs/health-checks).