Introduction to Streams

A stream is an abstraction for processing data piece by piece as it arrives, instead of loading the whole thing into memory at once. Read a 50 GB log file, proxy an HTTP response, compress a video on the fly — all of these work on a 50 GB file but only ever hold a few kilobytes in memory at a time. Streams are one of Node's most powerful (and most misunderstood) ideas, and they underpin fs, http, crypto, zlib, and more.

The problem streams solve

Compare reading a large file two ways. The buffered approach loads everything before you can touch a byte:

import { readFile } from 'node:fs/promises'

// ✗ Loads the ENTIRE file into RAM — a 2 GB file needs 2 GB of memory
const data = await readFile('huge.log', 'utf8')
process(data)

The streaming approach processes the file in small chunks, with near-constant memory regardless of file size:

import { createReadStream } from 'node:fs'

// ✓ Reads ~64 KB at a time — memory stays flat for any file size
const stream = createReadStream('huge.log', 'utf8')
for await (const chunk of stream) {
  process(chunk)        // handle each piece as it arrives
}

Streams trade peak memory for time-to-first-byte

Beyond saving memory, streams let you start working on data *before all of it has arrived* — you can begin sending an HTTP response, or parsing a CSV, while the source is still being read. This lower latency is why streaming is the backbone of servers and pipelines, not just a memory optimization.

The four stream types

Type	Direction	Examples
`Readable`	Source you read from	`fs.createReadStream`, HTTP request, `process.stdin`
`Writable`	Sink you write to	`fs.createWriteStream`, HTTP response, `process.stdout`
`Duplex`	Both, independently	TCP socket (`net.Socket`)
`Transform`	Duplex that modifies data	`zlib.createGzip`, `crypto` cipher

Each gets its own page: Readable Streams, Writable Streams, and Duplex & Transform Streams.

Chunks: the unit of flow

Data moves through a stream in chunks — usually Buffer objects (binary), or strings if an encoding is set, or arbitrary JavaScript values in object mode. You rarely choose the chunk size; the stream and OS decide based on the highWaterMark (default 64 KB for file/byte streams, 16 objects in object mode).

Chunk boundaries are arbitrary — never assume a chunk is a complete record

A chunk is just "however many bytes were available," not a line, a JSON object, or a UTF-8 character. A multi-byte character can be **split across two chunks**, and a single chunk may contain partial lines. Parsing per-chunk by hand is a classic bug source. Use a decoder/parser that buffers across chunks (e.g. `readline`, `StringDecoder`, or a Transform) instead of `chunk.toString().split('\n')`.

Piping: connecting streams

The real power is composition. pipe (or better, pipeline) connects a readable's output to a writable's input, automatically handling chunk flow and backpressure:

import { createReadStream, createWriteStream } from 'node:fs'
import { createGzip } from 'node:zlib'
import { pipeline } from 'node:stream/promises'

// Read → compress → write, chunk by chunk, with backpressure handled:
await pipeline(
  createReadStream('input.log'),
  createGzip(),
  createWriteStream('input.log.gz'),
)
console.log('compressed')

Prefer `pipeline` over `.pipe()`

`readable.pipe(writable)` works but does **not** forward errors or clean up the source if the destination fails — a leak waiting to happen. `stream.pipeline` (callback or promise form) wires up every stream, propagates errors, and destroys all of them on failure. Make `pipeline` your default; we cover both in [Piping Streams](/nodejs/piping-streams) and [stream.pipeline](/nodejs/stream-pipeline).

Backpressure: the core concept

A fast source can overwhelm a slow destination — that's backpressure

If you read from a fast disk and write to a slow network, data piles up in memory between them. **Backpressure** is the mechanism that tells the readable "stop, the writable is full" until the sink drains. `pipe`/`pipeline` handle it automatically; if you wire streams together manually and ignore the `write()` return value, you reintroduce the very memory blow-up streams exist to prevent. This is important enough to have its own page: [Stream Backpressure](/nodejs/stream-backpressure).

Streams are EventEmitters

Under the classic API, streams communicate via events — data, end, error, finish, drain. This is why understanding EventEmitter pays off here:

import { createReadStream } from 'node:fs'

const stream = createReadStream('file.txt')
stream.on('data',  (chunk) => console.log('got', chunk.length, 'bytes'))
stream.on('end',   () => console.log('done'))
stream.on('error', (err) => console.error('failed', err))

Modern code uses async iteration instead

The event API works, but `for await (const chunk of readable)` is cleaner, handles errors with normal `try/catch`, and applies backpressure for free. Reach for events only when you need fine-grained control (pause/resume, multiple listeners). We use the async-iterator style throughout the next pages.

When to use streams

Large or unbounded data — files, uploads, downloads bigger than you want in RAM.
Pipelines — read → transform → write, especially compress/encrypt/parse.
Low latency — start responding before all input has arrived (HTTP, proxies).
Not for small, in-memory data — if it comfortably fits in memory and you need it all at once, readFile/writeFile are simpler and fine.

Start with the source end: how to consume data from a [Readable Stream](/nodejs/readable-streams).