JavaScript Streams: From Beginner to Expert

Streams in JavaScript are an elegant pattern for handling data, allowing us to process data as if we were dealing with a flowing stream—bit by bit, rather than loading everything all at once. Imagine the water in a stream, continuously flowing; this is the approach streams take, enabling efficient handling of videos, large files, and real-time data, even in resource-constrained environments.

Do you remember the last time you tried to upload a large video file in a browser? The waiting, the freezing, and even the crashes can be quite frustrating. This is where the limitations of the traditional "load everything at once" approach become apparent.

When dealing with large files, the traditional method loads the entire file into memory at once, like pouring a bucket of water into a small cup—overflow is inevitable.

I encountered this issue in a project where users were uploading large files, causing the server's memory usage to spike instantly, eventually leading to a complete service outage. In such scenarios, stream programming is clearly the most appropriate solution. Let's compare the traditional method with stream processing:

// Traditional method - Load the entire file into memory
fs.readFile('huge-video.mp4', (err, data) => {
  if (err) 

After using stream programming, you'll notice that the server's memory usage drops significantly, and the freezing seems to never have occurred. This is the power of stream programming.

In fact, the strength of streams lies in "processing on demand"—you don't have to wait for all the data to start working. It's like sipping coffee while continuously refilling the cup, rather than waiting for a whole pot of coffee to brew before drinking. With streams, the memory footprint for a 1GB file might only be 64KB (default chunk size), a significant efficiency improvement compared to the traditional method's 1GB usage.

Even if the client has enough memory, processing large files all at once can cause noticeable performance issues. The main thread gets blocked, the user interface may freeze, and the overall responsiveness of the application is affected. Similarly, in a Node.js server environment, this blocking directly impacts the ability to handle other requests.

Moreover, modern applications often need to handle real-time data streams, such as live video streaming, real-time log analysis, or financial transaction data. Traditional data processing methods can no longer meet the demand for "processing as it is generated," leading to increased data processing delays and reduced real-time performance.

Therefore, what we need is a data processing mechanism that can quickly handle real-time data, has low memory costs, and is feature-rich. This mechanism is implemented in JavaScript as streams.

Note: Unless otherwise specified, the streams mentioned below refer to streams in the Node.js environment by default.

Streams provide a new paradigm for data processing, allowing us to split data into small chunks (chunks) and process them step by step, just like a continuous flow of water.

The best way to understand the concept of streams is through analogies from everyday life. Imagine water flowing through a pipe: water flows from the source (such as a reservoir) through the pipe system, eventually reaching the destination (such as a faucet). Throughout the process, the water is continuously flowing, not all delivered at once.

Technically, a stream is an abstract representation of an asynchronous sequence of data. It provides a standardized interface for handling continuously arriving data chunks. These data chunks can be:

• Binary data in files
• HTTP request bodies
• Terminal input
• Any segmentable continuous data

In Node.js, the stream module provides the core API for stream processing. Almost all I/O operations are built on streams, from the file system to HTTP requests and responses, streams are everywhere.

const { Readable, Writable, Transform, Duplex } = require('stream');

Stream processing enables a "pipeline" operation of data, where producers and consumers can work in parallel. Data can be processed immediately as it is generated, greatly improving the system's responsiveness and real-time performance. This is particularly prominent in scenarios such as video transcoding and real-time data analysis.

Performance Comparison Example:

The main types of streams in Node.js are (we will cover these in detail in the next chapter):

1. Readable Streams
2. Writable Streams
3. Duplex Streams
4. Transform Streams

Each type of stream is an instance of EventEmitter, meaning they communicate through events. Common stream events include:

• data - Triggered when there is data available to read
• end - Triggered when no more data is available to read
• error - Triggered when an error occurs
• finish - Triggered when all data has been flushed to the underlying system

In the following chapters, we will delve into the various types of streams in JavaScript, their usage methods, and best practices, helping you master this powerful data processing tool.

Readable streams are the source of data, like a reservoir or water source. File reading, HTTP requests, and user input are typical scenarios for readable streams. To describe it professionally, a readable stream is a producer of data, representing a source of data. Its core characteristics are:

• Data can only be read from the stream, not written to it
• Supports two data consumption modes: flowing mode and paused mode
• Automatically handles the backpressure mechanism
• Data can be piped to writable streams

Typical Use Cases:

• Reading data from files
• Receiving HTTP request bodies
• Reading database query results
• Any scenario that requires sequential reading of data

Here are some common implementations of readable streams:

const fs = require("fs");

// 1. File Read Stream
const fileStream = fs.createReadStream("./data.txt");

Writable streams are the consumers of data, representing the destination of data. Its characteristics are:

• Data can only be written to the stream, not read from it
• Supports buffering to handle differences in write speeds
• Provides a drain event to handle backpressure
• Can receive data piped from readable streams

Typical Use Cases:

• Writing to files
• Sending HTTP responses
• Writing to databases
• Any scenario that requires sequential writing of data

Here are some common implementations of writable streams:

// 1. File Write Stream
const fileWriter = fs.createWriteStream("./output.txt");

// 2. HTTP Response Stream
http.createServer((req,

Duplex streams are bidirectional streams, implementing both readable and writable interfaces, with both reading and writing capabilities. Its characteristics are:

• Can read and write
• The read and write ends are independent
• Commonly used in bidirectional communication scenarios

Here are some common implementations of duplex streams, such as TCP socket servers:

const net = require('net');

// Create a TCP server
const server = net.createServer((  


  

Transform streams are a special type of duplex stream, specifically used for data transformation. Its characteristics are:

• Can read and write simultaneously
• Data written to the write end is transformed and appears at the read end
• Commonly used in data format conversion, encryption/decryption, etc.

There are many types of transform stream implementations, such as compression/decompression streams, encryption streams, etc.:

const { Transform } = require("stream");
const zlib = require("zlib");