stream: add auto-destroy mode #22795

mafintosh · 2018-09-10T15:44:15Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

This adds a new options to the stream constructor called autoDestroy that is default false.
When autoDestroy is enabled .destroy() will automatically be called when the stream has ended and/or finished depending on whether it's a readable/writable/duplex.

This simplifies error handling / resource management for users as you can then always do your teardown logic in the destroy method.

nodejs-github-bot · 2018-09-10T15:44:18Z

@mafintosh build started: https://ci.nodejs.org/blue/organizations/jenkins/node-test-pull-request-lite-pipeline/detail/node-test-pull-request-lite-pipeline/852/pipeline

mcollina · 2018-09-10T16:05:57Z

@mafintosh I think it needs to call .destroy(err) whenever there is a emit('error'), at least in the streams classes and the option is enabled.

I think this is the first step to implement #20096, right?

vsemozhetbyt · 2018-09-10T16:16:27Z

doc/api/stream.md

@@ -1492,6 +1492,8 @@ changes:
    [`stream._destroy()`][writable-_destroy] method.
  * `final` {Function} Implementation for the
    [`stream._final()`][stream-_final] method.
+  * `autoDestroy` {boolean} Whether this stream should automatically call
+    .destroy() on itself after ending.


.destroy() -> `.destroy()`?

Can you provide the default value explicitly here?

vsemozhetbyt · 2018-09-10T16:16:49Z

doc/api/stream.md

@@ -1741,6 +1743,8 @@ constructor and implement the `readable._read()` method.
    method.
  * `destroy` {Function} Implementation for the
    [`stream._destroy()`][readable-_destroy] method.
+  * `autoDestroy` {boolean} Whether this stream should automatically call
+    .destroy() on itself after ending.


mscdex · 2018-09-10T16:24:02Z

lib/_stream_readable.js

@@ -1075,6 +1078,10 @@ function endReadable(stream) {
  }
 }

+function writableAutoDestroy(wState) {
+  return !wState || (wState.autoDestroy && wState.finished);


Can't this just be inlined?

targos · 2018-09-10T16:25:03Z

It looks like the code in _stream_readable.js should be in _stream_writable.js and vice versa.

addaleax · 2018-09-11T10:45:42Z

doc/api/stream.md

@@ -1492,6 +1492,8 @@ changes:
    [`stream._destroy()`][writable-_destroy] method.
  * `final` {Function} Implementation for the
    [`stream._final()`][stream-_final] method.
+  * `autoDestroy` {boolean} Whether this stream should automatically call
+    .destroy() on itself after ending.


Can you provide the default value explicitly here?

addaleax · 2018-09-11T10:45:42Z

test/parallel/test-stream-auto-destroy.js

+  t.resume();
+  t.on('end', common.mustCall());
+  t.on('finish', common.mustCall());
+  t.on('close', common.mustCall());


Can we maybe check the order of the events/destroy callback as well?

addaleax · 2018-09-14T16:34:57Z

Gentle ping @mafintosh :)

@targos I think it’s this way around in order to support shutting down Duplex streams?

mafintosh · 2018-09-17T13:20:17Z

@addaleax i'll fix up the nits+comments

Fishrock123 · 2018-09-17T15:47:13Z

Seems like a good addition to me, minus others' comments.

tniessen · 2018-09-19T14:39:40Z

semver-minor commits should contain metadata in the YAML changes section of the affected API.

addaleax · 2018-09-22T21:17:42Z

lib/internal/http2/compat.js

@@ -251,6 +251,7 @@ function onStreamTimeout(kind) {

 class Http2ServerRequest extends Readable {
  constructor(stream, headers, options, rawHeaders) {
+    if (!options) options = {};


Just to make sure I’m not missing anything – how does this change relate to the others?

mcollina

LGTM

mafintosh · 2018-10-29T11:00:26Z

Fixed all the nits @addaleax @targos @vsemozhetbyt @Fishrock123

mafintosh · 2018-10-29T11:01:24Z

CI: https://ci.nodejs.org/view/Node.js/job/node-test-pull-request/18202/

mcollina

LGTM but I would like to see a CITGM run.

mcollina · 2018-10-29T11:33:33Z

doc/api/stream.md

@@ -1493,6 +1493,11 @@ changes:
    pr-url: https://github.com/nodejs/node/pull/18438
    description: >
      Add `emitClose` option to specify if `'close'` is emitted on destroy
+  - version: TBD


is the value we put here TBD? I never remember.

REPLACEME :)

Ah right! Thanks, will fix

mafintosh · 2018-10-29T12:20:29Z

CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1604/
CITGM master: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1603/

mafintosh · 2018-10-30T11:47:13Z

One more CITGM with readable-stream enabled

master: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1606/
pr: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1607/

mafintosh · 2018-10-30T14:15:29Z

The CITGM runs seem to report similar errors on master + this PR, landing ...

PR-URL: nodejs#22795 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

mafintosh · 2018-10-30T14:19:24Z

Landed in f24b070

PR-URL: #22795 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>

nodejs-github-bot added dont-land-on-v6.x http2 stream labels Sep 10, 2018

mcollina added semver-minor dont-land-on-v8.x labels Sep 10, 2018

vsemozhetbyt reviewed Sep 10, 2018

View changes

mscdex reviewed Sep 10, 2018

View changes

jasnell approved these changes Sep 10, 2018

View changes

addaleax reviewed Sep 11, 2018

View changes

addaleax reviewed Sep 22, 2018

View changes

mcollina mentioned this pull request Oct 5, 2018

pipeline does not wait for full stream destruction #23214

Closed

mafintosh force-pushed the auto-destroy-stream branch 2 times, most recently from bb2cb3c to 387516c Oct 26, 2018

stream: add auto-destroy mode

ccc94a9

mafintosh force-pushed the auto-destroy-stream branch from 387516c to ccc94a9 Oct 26, 2018

mafintosh added 2 commits Oct 26, 2018

inline duplex autodestroy logic

38077e6

move errorOrDestroy to destroyImpl

294e10c

mcollina approved these changes Oct 26, 2018

View changes

mafintosh added 3 commits Oct 26, 2018

test ordering of events

a6eb82e

fix docs

c40028a

add to changes: section

7eb4e64

mcollina approved these changes Oct 29, 2018

View changes

TDB -> REPLACEME

713a808

addaleax approved these changes Oct 29, 2018

View changes

Fishrock123 approved these changes Oct 29, 2018

View changes

mcollina mentioned this pull request Oct 30, 2018

fs: read/write streams don't emit close after error #23338

Closed

mafintosh closed this Oct 30, 2018

mafintosh deleted the auto-destroy-stream branch Oct 30, 2018

watilde mentioned this pull request Nov 6, 2018

test: add test for autoDestroy in stream #24127

Closed

3 tasks

BridgeAR mentioned this pull request Nov 14, 2018

Release proposal: v11.2 #24350

Merged

beevelop mentioned this pull request Nov 15, 2018

2018-11-15 Version 11.2.0 (Current) @BridgeAR beevelop/docker-nodejs#232

Closed

beevelop mentioned this pull request Nov 15, 2018

2018-11-15 Version 11.2.0 (Current) @BridgeAR beevelop/docker-android-nodejs#249

Closed

beevelop mentioned this pull request Nov 15, 2018

2018-11-15 Version 11.2.0 (Current) @BridgeAR beevelop/docker-nodejs#233

Closed

beevelop mentioned this pull request Nov 16, 2018

2018-11-15 Version 11.2.0 (Current) @BridgeAR beevelop/docker-nodejs#234

Closed

beevelop mentioned this pull request Nov 16, 2018

2018-11-15 Version 11.2.0 (Current) @BridgeAR beevelop/docker-nodejs#235

Closed

beevelop mentioned this pull request Nov 16, 2018

2018-11-15 Version 11.2.0 (Current) @BridgeAR beevelop/docker-nodejs#236

Closed

BethGriggs mentioned this pull request May 1, 2019

v10.16.0 proposal #27514

Merged

beevelop mentioned this pull request May 29, 2019

2019-05-28 Version 10.16.0 'Dubnium' (LTS) @BethGriggs beevelop/docker-android-nodejs#315

Closed

beevelop mentioned this pull request May 29, 2019

2019-05-28 Version 10.16.0 'Dubnium' (LTS) @BethGriggs beevelop/docker-android-nodejs#316

Closed

beevelop mentioned this pull request May 29, 2019

2019-05-28 Version 10.16.0 'Dubnium' (LTS) @BethGriggs beevelop/docker-android-nodejs#317

Closed

beevelop mentioned this pull request May 29, 2019

2019-05-28 Version 10.16.0 'Dubnium' (LTS) @BethGriggs beevelop/docker-android-nodejs#318

Closed

beevelop mentioned this pull request May 29, 2019

2019-05-28 Version 10.16.0 'Dubnium' (LTS) @BethGriggs beevelop/docker-android-nodejs#319

Closed

nodejs / node Public

stream: add auto-destroy mode #22795

stream: add auto-destroy mode #22795

mafintosh commented Sep 10, 2018

nodejs-github-bot commented Sep 10, 2018

mcollina commented Sep 10, 2018

vsemozhetbyt Sep 10, 2018

addaleax Sep 11, 2018

vsemozhetbyt Sep 10, 2018

mscdex Sep 10, 2018

targos commented Sep 10, 2018 •

edited

addaleax Sep 11, 2018

addaleax Sep 11, 2018

addaleax commented Sep 14, 2018

mafintosh commented Sep 17, 2018

Fishrock123 commented Sep 17, 2018

tniessen commented Sep 19, 2018

addaleax Sep 22, 2018

mcollina left a comment

mafintosh commented Oct 29, 2018

mafintosh commented Oct 29, 2018

mcollina left a comment

mcollina Oct 29, 2018

addaleax Oct 29, 2018

mafintosh Oct 29, 2018

mafintosh commented Oct 29, 2018

mafintosh commented Oct 30, 2018

mafintosh commented Oct 30, 2018

mafintosh commented Oct 30, 2018

nodejs / node Public

stream: add auto-destroy mode #22795

stream: add auto-destroy mode #22795

Conversation

mafintosh commented Sep 10, 2018

Checklist

nodejs-github-bot commented Sep 10, 2018

mcollina commented Sep 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

targos commented Sep 10, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

addaleax commented Sep 14, 2018

mafintosh commented Sep 17, 2018

Fishrock123 commented Sep 17, 2018

tniessen commented Sep 19, 2018

Choose a reason for hiding this comment

mcollina left a comment

mafintosh commented Oct 29, 2018

mafintosh commented Oct 29, 2018

mcollina left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mafintosh commented Oct 29, 2018

mafintosh commented Oct 30, 2018

mafintosh commented Oct 30, 2018

mafintosh commented Oct 30, 2018

targos commented Sep 10, 2018 •

edited