★ wanayoo — archive 1999 https://github.com/nodejs/node/pull/21968Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: reduce memory retention when streaming small files #21968

Closed
wants to merge 2 commits into from

Conversation

Projects
None yet
7 participants
@addaleax
Copy link
Member

addaleax commented Jul 25, 2018

Fixes: #21967

/cc @nodejs/fs

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • commit message follows commit guidelines

@addaleax addaleax requested a review from ChALkeR Jul 25, 2018

@ChALkeR ChALkeR added the memory label Jul 25, 2018

@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Jul 25, 2018

@ChALkeR
Copy link
Member

ChALkeR left a comment

Changes look good, but lint fails.
Also, this probably needs benchmarking.

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Jul 25, 2018 •

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Jul 25, 2018

Benchmark results:

                                                                                  confidence improvement accuracy (*)    (**)   (***)
 fs/read-stream-throughput.js size=1024 filesize=1048576000 encodingType='asc'                   -0.88 %       ±4.57%  ±6.09%  ±7.92%
 fs/read-stream-throughput.js size=1024 filesize=1048576000 encodingType='buf'                    3.41 %       ±4.63%  ±6.16%  ±8.02%
 fs/read-stream-throughput.js size=1024 filesize=1048576000 encodingType='utf'                    0.92 %       ±2.74%  ±3.66%  ±4.79%
 fs/read-stream-throughput.js size=1048576 filesize=1048576000 encodingType='asc'                 0.13 %       ±7.84% ±10.50% ±13.81%
 fs/read-stream-throughput.js size=1048576 filesize=1048576000 encodingType='buf'                 0.21 %       ±7.09%  ±9.45% ±12.35%
 fs/read-stream-throughput.js size=1048576 filesize=1048576000 encodingType='utf'          *    -12.78 %      ±10.99% ±14.66% ±19.15%
 fs/read-stream-throughput.js size=4096 filesize=1048576000 encodingType='asc'                    1.49 %       ±3.29%  ±4.38%  ±5.72%
 fs/read-stream-throughput.js size=4096 filesize=1048576000 encodingType='buf'                    0.86 %       ±4.67%  ±6.22%  ±8.12%
 fs/read-stream-throughput.js size=4096 filesize=1048576000 encodingType='utf'                   -7.13 %       ±8.20% ±10.91% ±14.22%
 fs/read-stream-throughput.js size=65535 filesize=1048576000 encodingType='asc'                   0.07 %       ±6.29%  ±8.38% ±10.91%
 fs/read-stream-throughput.js size=65535 filesize=1048576000 encodingType='buf'                  -1.41 %       ±7.39%  ±9.83% ±12.81%
 fs/read-stream-throughput.js size=65535 filesize=1048576000 encodingType='utf'                   4.10 %      ±12.59% ±16.75% ±21.81%

Since there really isn’t any reason why the encoding should matter here, and we should expect 0.6 false positives, I’d ignore the one * result. If somebody feels like we should make sure, feel free to run the benchmarks with more iterations.

@mcollina
Copy link
Member

mcollina left a comment

LGTM.

Could this be effecting also 'net'?

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Jul 25, 2018

@mcollina You mean, the same bug there? No, that’s not the case, because we shrink the buffer in native land before handing it over to JS. I’ve thought about implementing similar pooling there, though, as part of looking into nodejs-private/security#186 (for those who can’t see it: it’s the bug that was resolved in 3217e8e).

@mcollina

This comment has been minimized.

Copy link
Member

mcollina commented Jul 25, 2018

@addaleax Thx! there are a few cases in HTTP applications where I have seen very high RSS vs heap usage, without a leak. The effect looks very similar to this one.

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Jul 27, 2018

@addaleax

This comment has been minimized.

Copy link
Member Author

addaleax commented Jul 29, 2018

Landed in e3a4702

@addaleax addaleax closed this Jul 29, 2018

@addaleax addaleax deleted the addaleax:fs-pool-reuse branch Jul 29, 2018

addaleax added a commit that referenced this pull request Jul 29, 2018

fs: reduce memory retention when streaming small files
Fixes: #21967

PR-URL: #21968
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@ChALkeR

This comment has been minimized.

Copy link
Member

ChALkeR commented Jul 29, 2018

/cc @nodejs/lts ?

targos added a commit that referenced this pull request Jul 31, 2018

fs: reduce memory retention when streaming small files
Fixes: #21967

PR-URL: #21968
Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Trivikram Kamat <trivikr.dev@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

@targos targos referenced this pull request Jul 31, 2018

Merged

v10.8.0 proposal #22040

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.