Writable Streams

[giscus[bot]]

Writable Streams

Understanding writable streams, implementing custom writable streams, and handling backpressure effectively.

https://www.thenodebook.com/streams/writable-streams

[Uriel29]

Hello!
You removed the anchors from the texts.
I used them to save my place in the browser and read later.
Without them, the reading flow changes for me — I need those pauses to better absorb the content!

I'm also not receiving the emails for the new chapters. What can I do?

Best regards.

[ishtms]

Hi @Uriel29, I migrated from Nextra to my own Markdown previewer, as nextra was very heavy, and tough to extend. I've fixed this now!

[SuperMo0]

when I use fs.createReadStream what is the default number bytes read from a source at each single call to _read() ?

by searching google on how can I control the chunk size, I got only answers saying : "you control the highWaterMark".
what I have just learned from you is that highWaterMark don't equal the chunk size

[ishtms]

highWaterMark is not a chunk size setting but a buffer threshold. It tells the stream "when your internal buffer hits this many bytes, stop asking the source for more data." It's actually about backpressure, and not how much data gets read in a single operation.

When the stream's internal _read(size) method gets called, size is passed as a hint, and that hint is usually the highWaterMark value. The file stream's implementation then allocates a buffer of that size and calls the low-level fs.read() with it. So for file streams, highWaterMark happens to control chunk size because of how it's implemented internally. The two concepts just happen to align in this case.

The default for fs.createReadStream() is 64KB (65536 bytes). So each read operation grabs up to 64KB from the file. Could be less if you're near the end of the file, but that's the target.

The base stream.Readable class defaults to 16KB. Same with stream.Writable. File streams just use a larger default because disk I/O benefits from bigger chunks, so fewer syscalls, better throughput.

So yeah, for file streams, setting highWaterMark does change chunk size. But that's an implementation detail of fs.createReadStream(), not a general stream rule. If you wrote your own custom readable stream, _read(size) gets that hint, but you can push whatever amount you want. The stream doesn't enforce it.

Hope this makes sense.

[adamakiva]

Hey,
Firstly I want to thank you for your work, been a while since I've seen this level of effort and diving into details.

In the section about The _final() Hook in Detail you've said, and I quote:
"...Or, since Node.js supports returning promises from stream methods, you can omit the callback:..."
What is your source for this statement?

I can't seem to find that with a google search or inside the nodejs documentation.
Typescript (which could very well be wrong) annotates the function(s) without a Promise return type.
Do you have a source (documentation/source code) to support this statement?

[adamakiva]

For reference, these are the typescript definitions for writable implementation methods: https://github.com/DefinitelyTyped/DefinitelyTyped/blob/a5064931f626650c0839e86b6a33f355d8c38260/types/node/v24/stream.d.ts#L813