★ wanayoo — archive 1999 https://github.com/nodejs/node/pull/23785Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure async iteration works with destroyed streams #23785

Closed
wants to merge 1 commit into from

Conversation

@mcollina
Copy link
Member

mcollina commented Oct 20, 2018 •

This PR is twofold:

  1. makes sure that .destroy() works correctly on zlib streams (before it was not cleaning up the _handle) done in #23734
  2. fixes support for async-iterating destroyed streams.

Fixes #23730 .

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines

@mcollina mcollina requested review from mafintosh, jasnell, addaleax and devsnek Oct 20, 2018

@mafintosh

This comment has been minimized.

Copy link
Member

mafintosh commented Oct 20, 2018

@mcollina does this supersede my zlib destroy leak PR?

@Hakerh400

This comment has been minimized.

Copy link
Contributor

Hakerh400 commented Oct 20, 2018

// this is needed because if .destro(err) is called, the error
// will be emitted via nextTick

Nit: destroy(err)

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 20, 2018

@mafintosh which one? Very likely that should be merged first.

@devsnek
Copy link
Member

devsnek left a comment

It seems this relies on a race condition? I won't pretend to know anything about our crazy streams but I want to just verify what's going on here is what is intended.

if (this[kStream].destroyed) {
// this is needed because if .destro(err) is called, the error
// will be emitted via nextTick
return new Promise((resolve, rejects) => {

This comment has been minimized.

Copy link
@devsnek

devsnek Oct 20, 2018

Member

any reason for rejects rather than reject?

This comment has been minimized.

Copy link
@mcollina

mcollina Oct 20, 2018

Author Member

none, I’ll fix it.

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 20, 2018

@devsnek this is normalizing a race condition that we cannot get rid of inside our stream machinery :/.

@mafintosh

This comment has been minimized.

Copy link
Member

mafintosh commented Oct 20, 2018

@mcollina this one #23734

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 20, 2018

Ah I didn’t know we were working on essentially the same issue!

Can you pick up the change that I did inside processCallback? It might even be worth a test on its own. I’ll rebase this on top of yours.

lib/zlib.js Outdated

Zlib.prototype._destroy = function(err, cb) {
const handle = this._handle;
if (handle && handle.buffer !== null) {

This comment has been minimized.

Copy link
@addaleax

addaleax Oct 21, 2018

Member

Can you add a comment here describing what this condition means?

lib/zlib.js Outdated
_close(this, callback);
this.destroy();
Zlib.prototype.close = function(callback) {
this.destroy(null, callback);

This comment has been minimized.

Copy link
@addaleax

addaleax Oct 21, 2018

Member

This changes timing for the close callback… is that intentional?

@addaleax addaleax referenced this pull request Oct 21, 2018

Closed

zlib: do not leak on destroy #23734

2 of 4 tasks complete

@mcollina mcollina force-pushed the mcollina:fix-23730 branch from c13cc44 to 0d029f6 Oct 23, 2018

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 23, 2018

CI: https://ci.nodejs.org/job/node-test-pull-request/18090/

PTAL, this should be ready for review.

@mcollina mcollina force-pushed the mcollina:fix-23730 branch from 0d029f6 to 80487d0 Oct 24, 2018

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 24, 2018

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 24, 2018

@lpinca @addaleax is this ok to land for you?

@@ -158,6 +174,9 @@ const createReadableStreamAsyncIterator = (stream) => {
stream.on('readable', onReadable.bind(null, iterator));
stream.on('end', onEnd.bind(null, iterator));
stream.on('error', onError.bind(null, iterator));
// needed because some streams will not emit 'end', e.g. Zlib.
// https://github.com/nodejs/node/issues/23730
stream.on('close', onEnd.bind(null, iterator));

This comment has been minimized.

Copy link
@mafintosh

mafintosh Oct 24, 2018

Member

could use stream.finished to simplify in the future, but 👍

@lpinca

lpinca approved these changes Oct 24, 2018

const passthrough = new PassThrough();
const err = new Error('kaboom');
pipeline(readable, passthrough, common.mustCall((e) => {
assert.strictEqual(err, e);

This comment has been minimized.

Copy link
@lpinca

lpinca Oct 24, 2018

Member
Suggested change
assert.strictEqual(err, e);
assert.strictEqual(e, err);
try {
await readable[Symbol.asyncIterator]().next();
} catch (e) {
assert.strictEqual(err, e);

This comment has been minimized.

Copy link
@lpinca

lpinca Oct 24, 2018

Member
Suggested change
assert.strictEqual(err, e);
assert.strictEqual(e, err);

@addaleax addaleax removed the zlib label Oct 24, 2018

@addaleax

This comment has been minimized.

Copy link
Member

addaleax commented Oct 24, 2018

@mcollina There’s nothing speaking against landing this as far as I am concerned :)

@mcollina mcollina force-pushed the mcollina:fix-23730 branch from 80487d0 to 14586b0 Oct 24, 2018

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 24, 2018

CI: https://ci.nodejs.org/job/node-test-pull-request/18110/

PTAL, I've switched to use finished instead.

@lpinca

This comment has been minimized.

Copy link
Member

lpinca commented Oct 24, 2018

Still LGTM.

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Oct 24, 2018

Landed in 3ec8cec

@mcollina mcollina closed this Oct 24, 2018

@mcollina mcollina deleted the mcollina:fix-23730 branch Oct 24, 2018

mcollina added a commit that referenced this pull request Oct 24, 2018

stream: async iteration should work with destroyed stream
Fixes #23730.

PR-URL: #23785
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Matheus Marchini <mat@mmarchini.me>
@mafintosh

This comment has been minimized.

Copy link
Member

mafintosh commented Oct 24, 2018

(Also still LGTM for the record)

targos added a commit that referenced this pull request Oct 24, 2018

stream: async iteration should work with destroyed stream
Fixes #23730.

PR-URL: #23785
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Matheus Marchini <mat@mmarchini.me>

@targos targos referenced this pull request Oct 27, 2018

Merged

Release proposal: v11.1.0 #23922

@MylesBorins

This comment has been minimized.

Copy link
Member

MylesBorins commented Nov 26, 2018

Landed in 10.x with 398418d and b1e1fe4

Please lmk if it should be backed out

@mcollina

This comment has been minimized.

Copy link
Member Author

mcollina commented Nov 26, 2018

MylesBorins added a commit that referenced this pull request Nov 26, 2018

stream: async iteration should work with destroyed stream
Fixes #23730.

PR-URL: #23785
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Matheus Marchini <mat@mmarchini.me>

@codebytere codebytere referenced this pull request Nov 27, 2018

Closed

v10.13.1 proposal #24675

rvagg added a commit that referenced this pull request Nov 28, 2018

stream: async iteration should work with destroyed stream
Fixes #23730.

PR-URL: #23785
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Matheus Marchini <mat@mmarchini.me>

MylesBorins added a commit that referenced this pull request Nov 29, 2018

stream: async iteration should work with destroyed stream
Fixes #23730.

PR-URL: #23785
Reviewed-By: Gus Caplan <me@gus.host>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Matheus Marchini <mat@mmarchini.me>

@codebytere codebytere referenced this pull request Nov 29, 2018

Merged

v10.14.2 proposal #24727

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.