fs: add promises API #18297

Closed
wants to merge 4 commits into
from

Conversation

Owner

jasnell commented Jan 22, 2018

fs.Promises, the actual PR (replaces #17739)

/cc @mcollina @addaleax

/cc @bmeurer and @nodejs/v8 @nodejs/chakracore ... it would be excellent if we could have y'all take a look at this and do some performance testing.

Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
Affected core subsystem(s)

fs

@jasnell jasnell changed the title from Fs promises to fs: add promises API Jan 22, 2018

Owner

jasnell commented Jan 22, 2018

@addaleax ... because I know you'll ask.. this intentionally does not do the util.promisify() customization yet for a couple reasons. 1) because there are folks who use that in production today, I'd like to make sure this implementation is solid before switching... and 2) there's a discrepancy between methods like fs.fstat() and fs.promises.fstat() in that the former takes the numeric file descriptor and the latter takes the new FileHandle object.

Owner

jasnell commented Jan 22, 2018

My intent would be to land this as an experimental feature

Owner

jasnell commented Jan 22, 2018

Just fyi.. benchmark comparison for bench-stat vs bench-stat-promises (running in a vm on a laptop...)

$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 20,714.827782562465
fs/bench-stat-promise.js statType="lstat" n=200000: 15,589.439105836866
fs/bench-stat-promise.js statType="stat" n=200000: 22,186.395158011303

$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 13,882.790153144382
fs/bench-stat-promise.js statType="lstat" n=200000: 15,744.878475049993
fs/bench-stat-promise.js statType="stat" n=200000: 17,180.365568781977

$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 20,064.173934086277
fs/bench-stat-promise.js statType="lstat" n=200000: 16,715.911703541507
fs/bench-stat-promise.js statType="stat" n=200000: 17,014.91350175299

$ ./node benchmark/fs/bench-stat.js
fs/bench-stat.js statType="fstat" n=200000: 19,148.535379755922
fs/bench-stat.js statType="lstat" n=200000: 17,326.004694161828
fs/bench-stat.js statType="stat" n=200000: 18,626.09603864018

$ ./node benchmark/fs/bench-stat.js
fs/bench-stat.js statType="fstat" n=200000: 14,399.220751034958
fs/bench-stat.js statType="lstat" n=200000: 14,749.28092887841
fs/bench-stat.js statType="stat" n=200000: 21,620.088712923043

$ ./node benchmark/fs/bench-stat.js
fs/bench-stat.js statType="fstat" n=200000: 19,873.273062278913
fs/bench-stat.js statType="lstat" n=200000: 23,040.86746115833
fs/bench-stat.js statType="stat" n=200000: 27,742.886612762228
doc/api/fs.md
+`fs.promises.open()` method.
+
+Unlike the callback-based `fs.fstat()`, `fs.fchown()`, `fs.fchmod()`,
+`fs.ftruncate()`, `fs.read()`, and `fs.write()`, operations -- all of which
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Is it supposed to be an exhaustive list? If so, maybe fs.appendFile(), fs.fdatasync(), fs.fsync(), fs.readFile(), fs.futimes(), and fs.writeFile() should be added.

@jasnell

jasnell Jan 23, 2018

Owner

Not exhaustive, will add some clarifying language

doc/api/fs.md
+-->
+
+* Returns: {Promise} A `Promise` that will be resolved once the underlying
+ file descriptor is closed, or will reject if an error occurs while closing.
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

will reject -> will be rejected?

doc/api/fs.md
+* `fs.constants.X_OK` - `path` can be executed by the calling process. This has
+no effect on Windows (will behave like `fs.constants.F_OK`).
+
+If the accessible check is successful, the `Promise` is resolved with no
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

accessible check -> access check / accessibility check ?

doc/api/fs.md
+ * `encoding` {string|null} **Default:** `'utf8'`
+ * `mode` {integer} **Default:** `0o666`
+ * `flag` {string} **Default:** `'a'`
+
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Missing "Returns: ..." and description of it.

doc/api/fs.md
+
+The `file` may be specified as a `FileHandle` that has been opened
+for appending (using `fs.promises.open()`). The `FileHandle` will
+not be closed automatically.
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Does this mean that any FileHandle opened for appending will not be closed automatically? If so, should this be added to FileHandle description as an exception?

doc/api/fs.md
+* Returns: {Promise}
+
+Changes the ownership of a file then resolves the `Promise` with no arguments
+upon success
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

missing period.

doc/api/fs.md
+
+Asynchronously copies `src` to `dest`. By default, `dest` is overwritten if it
+already exists. No arguments other than a possible exception are given to the
+callback function. Node.js makes no guarantees about the atomicity of the copy
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

An obsolete note about callback function?

doc/api/fs.md
+Truncates the file represented by `filehandle` then resolves the `Promise`
+with no arguments upon success.
+
+If the file referred to by the file descriptor was larger than `len` bytes, only
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

file descriptor -> `filehandle`?

doc/api/fs.md
+If the file referred to by the file descriptor was larger than `len` bytes, only
+the first `len` bytes will be retained in the file.
+
+For example, the following program retains only the first four bytes of the file
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Missing period or colon?

doc/api/fs.md
+* `mtime` {number|string|Date}
+* Returns: {Promise}
+
+Change the file system timestamps of the object referenced by the supplied file
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

file descriptor -> `filehandle`?

doc/api/fs.md
+* `path` {string|Buffer|URL}
+* Returns: {Promise}
+
+Asynchronous lstat(2). The `Promise` is resolved with the [`fs.Stats'][] object
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Mismatched qotes: [`fs.Stats'] -> [`fs.Stats`]

doc/api/fs.md
+
+*Note*: The behavior of `fs.promises.open()` is platform-specific for some
+flags. As such, opening a directory on macOS and Linux with the `'a+'` flag -
+see example below - will return an error. In contrast, on Windows and FreeBSD,
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Obsolete "see example below"?

doc/api/fs.md
+
+Following successful read, the `Promise` is resolved with an object with a
+`bytesRead` property specifying the number of bytes read, and a `buffer` property
+that is a referece to the passed in `buffer` argument.
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

referece -> reference

doc/api/fs.md
+ * `encoding` {string} **Default:** `'utf8'`
+* Returns: {Promise}
+
+Reads the contents of a directory then resolves with `Promise` with an array
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

resolves with Promise with an array -> resolves the Promise with an array?

doc/api/fs.md
+
+The optional `options` argument can be a string specifying an encoding, or an
+object with an `encoding` property specifying the character encoding to use for
+the filenames passed to the callback. If the `encoding` is set to `'buffer'`,
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Obsolete "passed to the callback".

doc/api/fs.md
+* `options` {Object|string}
+ * `encoding` {string|null} **Default:** `null`
+ * `flag` {string} **Default:** `'r'`
+
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Missing "Returns: ..."?

doc/api/fs.md
+returned.
+
+Any specified `FileHandle` has to support reading. If a `FileHandle` is
+specified as the `path`, it will not be closed automatically.
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

"it will not be closed automatically" — if this is true, is it worth to be mentioned as an exception in the FileHandle description?

doc/api/fs.md
+
+The optional `options` argument can be a string specifying an encoding, or an
+object with an `encoding` property specifying the character encoding to use for
+the link path passed to the callback. If the `encoding` is set to `'buffer'`,
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Obsolete "passed to the callback".

+* Returns: {Promise}
+
+Determines the actual location of `path` using the same semantics as the
+`fs.realpath.native()` function then resolves the `Promise` with the resolved
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Should it be fs.promises.realpath.native() then?

@jasnell

jasnell Jan 23, 2018

Owner

No, I don't think so. The current fs.realpath() function is a legacy artifact that we're pretty much stuck with thanks to existing userland code. Since this is a new API, we have the opportunity not to continue that forward.

doc/api/fs.md
+
+The optional `options` argument can be a string specifying an encoding, or an
+object with an `encoding` property specifying the character encoding to use for
+the path passed to the callback. If the `encoding` is set to `'buffer'`,
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Obsolete "passed to the callback".

doc/api/fs.md
+no arguments upon success.
+
+Using `fs.promises.rmdir()` on a file (not a directory) results in the
+`Promise` being reject with an `ENOENT` error on Windows and an `ENOTDIR` error
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

being reject -> being rejected?

doc/api/fs.md
+- Values can be either numbers representing Unix epoch time, `Date`s, or a
+ numeric string like `'123456789.0'`.
+- If the value can not be converted to a number, or is `NaN`, `Infinity` or
+ `-Infinity`, a `Error` will be thrown.
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

a Error -> an Error

doc/api/fs.md
+
+The `Promise` is resolved with an object containing a `bytesWritten` property
+identifying the number of bytes written, and a `buffer` property containing
+the a reference to the `buffer` written.
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Excessive "the a")

doc/api/fs.md
+should be written. If `typeof position !== 'number'`, the data will be written
+at the current position. See pwrite(2).
+
+The callback will be given three arguments `(err, bytesWritten, buffer)` where
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Obsolete note.

doc/api/fs.md
+The callback will be given three arguments `(err, bytesWritten, buffer)` where
+`bytesWritten` specifies how many _bytes_ were written from `buffer`.
+
+Note that it is unsafe to use `fs.write` multiple times on the same file
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

fs.write -> fs.promises.write()

doc/api/fs.md
+`bytesWritten` specifies how many _bytes_ were written from `buffer`.
+
+Note that it is unsafe to use `fs.write` multiple times on the same file
+without waiting for the `Promise` to resolve. For this scenario,
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

"to resolve" -> "to be resolved (or rejected)"?

doc/api/fs.md
+ * `encoding` {string|null} **Default:** `'utf8'`
+ * `mode` {integer} **Default:** `0o666`
+ * `flag` {string} **Default:** `'w'`
+
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Missing "Returns: ..."?

doc/api/fs.md
+The `encoding` option is ignored if `data` is a buffer. It defaults
+to `'utf8'`.
+
+If `options` is a string, then it specifies the encoding. Example:
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

Obsolete " Example:"

doc/api/fs.md
+
+Any specified `FileHandle` has to support writing.
+
+It is unsafe to use `fs.writeFile` multiple times on the same file without
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

fs.writeFile -> fs.promises.writeFile()

doc/api/fs.md
+waiting for the `Promise` to be resolved (or rejected).
+
+If a `FileHandle` is specified as the `file`, it will not be closed
+automatically.
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

"it will not be closed automatically" — if this is true, is it worth to be mentioned as an exception in the FileHandle description?

@jasnell

jasnell Jan 23, 2018

Owner

good point, will need to figure out the different wording here. The FileHandle will be closed if it is allowed to gc .... but if a FileHandle is provided to this function, the function itself will not call filehandle.close(), it's still up to the user to do that.

doc/api/fs.md
@@ -3446,6 +4203,7 @@ The following constants are meant for use with the [`fs.Stats`][] object's
[`AHAFS`]: https://www.ibm.com/developerworks/aix/library/au-aix_event_infrastructure/
[`Buffer.byteLength`]: buffer.html#buffer_class_method_buffer_bytelength_string_encoding
[`Buffer`]: buffer.html#buffer_buffer
+[`FileHandle`]: #fs_class_filehandle
@vsemozhetbyt

vsemozhetbyt Jan 23, 2018

Member

It is [FileHandle] (without backticks) in the signatures, so this needs to be synced.

Or this type can be added here to spare some doc bytes)

Member

vsemozhetbyt commented Jan 23, 2018

  1. This is going to be a talkative PR, so please, feel free to delete any of my comments if they are not appropriate or will not be hidden after addressing (or even if they will be hidden).

  2. Is the omission of the fs.write(fd, string[, position[, encoding]], callback) variant intended?

  3. If I get it right, the promises API doc part is not as full as the callback API part (some examples and notes are omitted). Is it worth to mention this in the intro part? Like "for more info see callback API counterparts"?

Owner

jasnell commented Jan 23, 2018

@vsemozhetbyt ... as always I love your reviews of my doc commits. Thank you for being so thorough. Yes, the omission of the fs.write(fd, string, position, encoding, callback) variant is intentional.

I plan to go back and add better examples either in an iteration of this PR or a future one.

lib/fs.js
+}
+
+fs.promises = {
+ async access(path, mode = fs.F_OK) {
@ofrobots

ofrobots Jan 23, 2018

Contributor

Given that you never await anything, does this need to be async?

@jasnell

jasnell Jan 23, 2018

Owner

To catch errors thrown, yes.

@ofrobots

ofrobots Jan 23, 2018

Contributor

@jasnell, can you elaborate? For my education, how does making the function async help with catching thrown errors? My understanding is that async merely enables the use of the await keyword.

@targos

targos Jan 23, 2018

Owner

Any error thrown within an async function is automatically caught and causes rejection of the promise returned by the function. With a regular function you could have two cases: synchronous throw or rejection depending on where the error happens.

@jasnell

jasnell Jan 23, 2018

Owner

Yeah, what @targos said... for instance, try the following cases:

function foo() {
  throw new Error('foo');  // error is thrown synchronously
  return new Promise(() => {});
}

async function bar() {
  throw new Error('bar'); // error causes promise rejection
  return new Promise(() => {});
}
Contributor

jinwoo commented Jan 23, 2018

Thanks for this. This is great.

Can we have a shorter namespace though? Something like fs.p (or fs.prom if fs.p is too terse)? It'd be painful if I have to type fs.promises all the time.

Member

vsemozhetbyt commented Jan 23, 2018

@jasnell You may also need to update the appropriate counterparts according to that PR (I've found those missing types reviewing this PR :) ).

Contributor

bmeurer commented Jan 23, 2018

@jasnell Your benchmark run seems to suggest that the promise performance is already quite good compared to the callback version. Although that is of course a micro-benchmark. Do you have a representative (macro-)benchmark?

Contributor

bmeurer commented Jan 23, 2018

Owner

jasnell commented Jan 23, 2018

Yep and rightfully so given that they mostly use exactly the same internal mechanisms. The differences are quite minimal thankfully. Where they are different is in the creation of the additional promise and the additional overhead of the promise hook, both of which are fairly minimal.

I do not yet have a macro benchmark in place. It'll likely take bit to get that.

One thing I did notice is that there are about twice as many GC runs in the promises microbenchmark, which is not at all surprising but is certainly worth noting.

Contributor

bmeurer commented Jan 23, 2018

@jasnell Did you see #17739 (comment) from @gsathya? That might be worth investigating.

src/env.cc
cb.cb_(this, cb.data_);
+ if (try_catch.HasCaught()) {
@joyeecheung

joyeecheung Jan 23, 2018

Contributor

It is worth updating the comment of SetImmediate that throwing errors in a native immediate callback would crash the process.

+ int ret = uv_fs_close(env()->event_loop(), &req, fd_, nullptr);
+ uv_fs_req_cleanup(&req);
+
+ struct err_detail { int ret; int fd; };
@joyeecheung

joyeecheung Jan 23, 2018

Contributor

Why not move this outside?

@jasnell

jasnell Jan 23, 2018

Owner

Because it's only used here. If it turns out to be useful elsewhere later, moving it is easy :-)

src/node_file.cc
+ AsyncCall(env, args, "open", UTF8, AfterOpenFileHandle,
+ uv_fs_open, *path, flags, mode);
+ } else {
+ SYNC_CALL(open, *path, *path, flags, mode)
@joyeecheung

joyeecheung Jan 23, 2018

Contributor

Can you use SyncCall here?

src/node_file.h
+
+ private:
+ bool finished_ = false;
+ double statFields_[14] {};
@joyeecheung

joyeecheung Jan 23, 2018

Contributor

Why making it an array here though? Maybe use a unique_ptr and only allocates memory and reset it when FillStatsArray is called?

@jasnell

jasnell Jan 23, 2018

Owner

to avoid an extra allocation. should be pretty cheap doing it this way overall.

test/parallel/test-fs-promises.js
+ }
+
+ const base = path.resolve(common.tmpDir, 'FOO');
+ assert(
@joyeecheung

joyeecheung Jan 23, 2018

Contributor

Does this work on windows? common.tmpDir would contain \ so it should not be able to be passed into new RegExp like that.

@jasnell

jasnell Jan 23, 2018

Owner

heh, good point. will have to double check

Owner

targos commented Jan 23, 2018

I don't know if we should discuss this here but I have a concern with the interaction of fs.promises and ESM (related to #18131).
We are committed to making ESM a first class feature in Node, but with the current way things are going, it will be difficult to provide the Promise APIs in an ESM-friendly way (no way to directly import a specific promisified method).
It seems the only way would be to have the Promise API in another module. Examples: require('fs.promise'), require('fs/promise')

Owner

jasnell commented Jan 23, 2018

@bmeurer ... Yeah, I'm working that way but wanted to get all of the js functions impl'd to make sure we would be able to completely eliminate creating the FSReqPromise object in js. I've got a way forward on that now that should work nicely, tho I'll have to test that it does not negatively impact the perf of the callback approach first.

Owner

jasnell commented Jan 23, 2018

@targos ... definitely a valid concern, will have to think about that a bit more. FWIW, we would have precisely the same issue with path.win32and path.posix, and likely several other bits of core API, so I'm not sure if solving it just for fs.promises is the right thing or if we should come with a more general solution.

Owner

jasnell commented Jan 23, 2018

@bmeurer ... updated to eliminate the creation of FSReqPromise in JS land, Definitely makes a difference.

james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 21,609.73968730913
fs/bench-stat-promise.js statType="lstat" n=200000: 13,606.886508071122
fs/bench-stat-promise.js statType="stat" n=200000: 20,361.793179169545
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 19,382.167235983314
fs/bench-stat-promise.js statType="lstat" n=200000: 14,694.606891503132
fs/bench-stat-promise.js statType="stat" n=200000: 13,698.616828686227
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 28,453.65135200578
fs/bench-stat-promise.js statType="lstat" n=200000: 13,697.273150949744
fs/bench-stat-promise.js statType="stat" n=200000: 18,633.918977446763
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 16,870.88056950238
fs/bench-stat-promise.js statType="lstat" n=200000: 13,472.76114383914
fs/bench-stat-promise.js statType="stat" n=200000: 26,460.504276421147
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat.js
fs/bench-stat.js statType="fstat" n=200000: 36,322.48835345608
fs/bench-stat.js statType="lstat" n=200000: 15,002.883243224593
fs/bench-stat.js statType="stat" n=200000: 22,319.499506552078
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat.js
fs/bench-stat.js statType="fstat" n=200000: 32,136.6980781016
fs/bench-stat.js statType="lstat" n=200000: 14,883.592505458588
fs/bench-stat.js statType="stat" n=200000: 25,631.456606611602
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat.js
fs/bench-stat.js statType="fstat" n=200000: 31,111.359958434878
fs/bench-stat.js statType="lstat" n=200000: 22,671.7262379716
fs/bench-stat.js statType="stat" n=200000: 17,104.025292210692
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 30,095.77601510028
fs/bench-stat-promise.js statType="lstat" n=200000: 24,965.011103171837
fs/bench-stat-promise.js statType="stat" n=200000: 27,054.78894406488
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 30,281.66760510009
fs/bench-stat-promise.js statType="lstat" n=200000: 23,380.61184604834
fs/bench-stat-promise.js statType="stat" n=200000: 27,438.146664427066
james@ubuntu:~/node/node$ ./node benchmark/fs/bench-stat-promise.js
fs/bench-stat-promise.js statType="fstat" n=200000: 26,104.278785435945
fs/bench-stat-promise.js statType="lstat" n=200000: 20,969.553527682114
fs/bench-stat-promise.js statType="stat" n=200000: 13,553.763324970801
Contributor

bmeurer commented Jan 23, 2018

@jasnell Cool! 👍 Kudos to @gsathya for the finding!

Contributor

guybedford commented Jan 23, 2018

@targos makes a great point in #18297 (comment).

I'd be a definite +1 on require('fs/promise') / require('fs-promise') / require('fs.promise') require('fsp') or similar to ensure ES module compatibility. ES module parity should be a primary consideration for all API changes at this point.

Contributor

guybedford commented Jan 23, 2018

@jasnell if we were implementing path.posix today the same suggestion would apply, but given it's a stable API it probably doesn't make sense to change now - import { posix } from 'path' seems less costly as it's a much rarer API use case anyway than import { readFile } from 'fs-promise'.

src/env.cc
cb.cb_(this, cb.data_);
+ if (try_catch.HasCaught()) {
+ FatalException(isolate(), try_catch);
+ }
@addaleax

addaleax Jan 23, 2018

Owner

Alternative suggestion to avoid setting up a TryCatchs for every individual callback:

diff in the fold
diff --git a/src/env.cc b/src/env.cc
index b05f0bec81df..f70945ca4c3a 100644
--- a/src/env.cc
+++ b/src/env.cc
@@ -290,13 +290,26 @@ void Environment::RunAndClearNativeImmediates() {
     size_t ref_count = 0;
     std::vector<NativeImmediateCallback> list;
     native_immediate_callbacks_.swap(list);
-    for (const auto& cb : list) {
-      cb.cb_(this, cb.data_);
-      if (cb.keep_alive_)
-        cb.keep_alive_->Reset();
-      if (cb.refed_)
-        ref_count++;
-    }
+    auto drain_list = [&]() {
+      v8::TryCatch try_catch(isolate());
+      for (auto it = list.begin(); it != list.end(); ++it) {
+        it->cb_(this, it->data_);
+        if (it->keep_alive_)
+          it->keep_alive_->Reset();
+        if (it->refed_)
+          ref_count++;
+        if (UNLIKELY(try_catch.HasCaught())) {
+          FatalException(isolate(), try_catch);
+          // Bail out, remove the already executed callbacks from list
+          // and set up a new TryCatch for the other pending callbacks.
+          std::move_backward(it, list.end(), list.begin() + (list.end() - it));
+          list.resize(list.end() - it);
+          return true;
+        }
+      }
+      return false;
+    };
+    while (drain_list()) {}
 
 #ifdef DEBUG
     CHECK_GE(immediate_info()->count(), count);

I’m happy to open a with PR that as a separate patch if you prefer.

In any case, we should probably run the timers benchmarks for setImmediate.

@jasnell

jasnell Jan 23, 2018

Owner

Nice. feel free to push a separate commit to this PR. Just isolate the change so I can squash it down with the existing one that edits this bit :-)... I'm good with this in whatever way you think is best, really

src/node_file.cc
+
+ // If the close was successful, we still want to emit a process warning
+ // to notify that the file descriptor was gc'd. We want to be noisy about
+ // this because not explicitly closing the garbage collector is a bug.
@addaleax

addaleax Jan 23, 2018

Owner

I think you mean something like “not explicitly closing the file descriptor is a bug”, right?

@jasnell

jasnell Jan 23, 2018

Owner

heh, yes

src/node_file.cc
+ resolver->Reject(env()->context(), reason);
+}
+
+FileHandle* FileHandle::CloseReq::fd() {
@addaleax

addaleax Jan 23, 2018

Owner

Maybe name this method file_handle()?

src/node_file.cc
-FSReqAfterScope::FSReqAfterScope(FSReqWrap* wrap, uv_fs_t* req)
+void FSReqPromise::Resolve(Local<Value> value) {
+ finished_ = true;
+ InternalCallbackScope callback_scope(this);
@addaleax

addaleax Jan 23, 2018

Owner

I’d always open the HandleScope first because InternalCallbackScope uses handles (in a way that would be kind of hard to rewrite)

src/node_file.cc
+ env()->promise_string()).ToLocalChecked();
+ CHECK(val->IsPromise());
+ Local<Promise> promise = val.As<Promise>();
+ Local<Promise::Resolver> resolver = promise.As<Promise::Resolver>();
@addaleax

addaleax Jan 23, 2018

Owner

Why not cast directly to Promise::Resolver?

@jasnell

jasnell Jan 23, 2018

Owner

heh, I suppose I could :-)

@@ -190,15 +380,15 @@ bool FSReqAfterScope::Proceed() {
}
void AfterNoArgs(uv_fs_t* req) {
- FSReqWrap* req_wrap = static_cast<FSReqWrap*>(req->data);
+ FSReqBase* req_wrap = static_cast<FSReqBase*>(req->data);
@addaleax

addaleax Jan 23, 2018

Owner

Side note (maybe try to remind me once this lands): There should be a ReqWrap<T>::From(T* uv_request) method that a) uses ContainerOf instead of requiring the lookup and b) doesn’t rely on blindlly casting void pointers to C++ non-base classes, because the latter is bound to go wrong at some point

src/node_file.cc
#define SYNC_RESULT err
+inline FSReqBase* GetReqWrap(Environment* env, const Local<Value>& value) {
@addaleax

addaleax Jan 23, 2018

Owner

nit: we usually pass Locals by value instead of references because they’re effectively just pointers

+
+ private:
+ bool finished_ = false;
+ AliasedBuffer<double, v8::Float64Array> stats_field_array_;
@addaleax

addaleax Jan 23, 2018

Owner

Can we avoid this in some way, i.e. create the array buffer + the typed array only once we’re actually at the resolve point and know that this is actually a stat call?

@jasnell

jasnell Jan 23, 2018

Owner

That's essentially what it was, then the change landed today that altered the signature for FillStatsArray to require an AliasedBuffer. I updated today just to get it working again but plan on revisiting before this lands.

Owner

jasnell commented Jan 26, 2018

@addaleax .. updated with your suggested change on the FatalException... will do a benchmark run now

Owner

jasnell commented Jan 26, 2018

@addaleax ... the benchmarks look good with the change you suggested. Will move forward with that.

Owner

jasnell commented Jan 27, 2018

@targos @guybedford ... went with a slightly different approach for now where fs.promises is a function that needs to be called to get the namespace object... e.g. const fsp = require('fs').promises() ... then awai fsp.open(...) and so forth. This is a bit of a stopgap until we figure out what the longer term strategy is.

Owner

jasnell commented Jan 27, 2018

fyi @bmeurer ....

Quick profile of a simple test app that performs 2e5 sequential stat and read operations, one after the other within an async function. Aside from the frequent gc, everything is looking healthy:

image

In contrast, here's the equivalent example using the callbacks model:

image

Note the patterns are nearly identical with one notable exception: the timescale for the promises version is much longer... Essentially, by enabling trace events and async_hooks instrumentation in order to capture the analysis, we kill the performance of the promises version. Without those hooks enable, performance is much closer to the callbacks version.

+added: REPLACEME
+-->
+
+* `file` {string|Buffer|[FileHandle][]} filename or `FileHandle`
+deprecated: REPLACEME
+-->
+
+* `path` {string|Buffer}
+deprecated: REPLACEME
+-->
+
+* `path` {string|Buffer}
+added: REPLACEME
+-->
+
+* `path` {string|Buffer}
+added: REPLACEME
+-->
+
+* `file` {string|Buffer|[FileHandle][]} filename or `FileHandle`
Contributor

bmeurer commented Jan 27, 2018

@jasnell Wow, that's pretty cool. I'm currently working on the GC issues. I think I've found a way to significantly reduce allocations for normal promise usage, so that should help with that.

Owner

jasnell commented Jan 27, 2018

Woo! Nice. The allocations are the only real issue I'm seeing here so if you're able to optimize that I think we're golden.

Of course, there's still the separate issue of the async hooks performance hit but we'll have to deal with that separately

Contributor

bmeurer commented Jan 27, 2018

@jasnell I wouldn't say golden, but at least the overhead should be less relevant.

My current thinking is also along the lines of first addressing the Promise problems, and deal with the async_hooks issue later, ideally redesigning the existing PromiseHooks API to reduce the overhead then. So for Node 10 we'll have to live with the performance drop I guess.

Owner

jasnell commented Jan 27, 2018

That's what I suspect as well. I think what we've got in this pr is an excellent starting point that we can iterate from.

Really appreciate the time you and your team have put in on optimizing this stuff

The code basically LGTM, I just have one question + something that I'd really like to see changed about the way the file handle API is exposed (and I know I've mentioned that before so there's a decent change you disagree? ;))

doc/api/fs.md
+The `fs.promises` API provides an alternative set of asynchronous file system
+methods that return `Promise` objects rather than using callbacks. The
+API is accessible via `fs.promises()` (for example:
+`const fsp = require('fs').promises()`).
@addaleax

addaleax Jan 28, 2018

Owner

Is the reason that this is not a getter ESM?

@TimothyGu

TimothyGu Jan 28, 2018

Member

Yes: #18297 (comment)

This is indeed pretty unidiomatic, for both CJS and ESM. I'd prefer the original fs.promises.* for CJS and import 'fs/promises'; for ESM.

If we can agree to a solution right now, then I don't think we necessarily have to implement it before this PR gets merged.

@pitaj

pitaj Jan 28, 2018

I don't see why "fs/promises" can't work for both CJS and ESM. It's probably better to be consistent.

@jasnell

jasnell Jan 29, 2018

Owner

Well, the most immediate reason is that our current CJS loader implementation would not support fs/promises as an option without modifications and I'm quite reluctant to do the necessary modifications within this PR.

@jasnell

jasnell Jan 31, 2018

Owner

I think, for now, I'm going to change this back to a getter. Hopefully we'll get a better solution figured out for ESM longer term but I'd rather not hold this PR up for this.

doc/api/fs.md
+ .catch(() => console.log('The file could not be copied'));
+```
+
+### fs.promises().fchmod(filehandle, mode)
@addaleax

addaleax Jan 28, 2018

Owner

Can't these be methods on FileHandle? That seems a lot cleaner to me, especially since FDs are intended to be object-oriented...

@jasnell

jasnell Jan 29, 2018

Owner

Eventually, yes, that's where I'd like to go with it and we likely should wrap the C++ provided FileHandle object with a pure JS object now in preparation for hanging additional things off of it. For now, as I've said, one of the goals is to have close parity between the new API and the existing API in order to make eventual migration easier. However, if there's enough support for it, I can get behind making this change now.

@jasnell

jasnell Jan 31, 2018

Owner

I'll go ahead and add this in now... however, one clarification: if I add a FileHandle.prototype.stat(), for example, would you prefer not having fs.promises.fstat()?

@addaleax

addaleax Feb 2, 2018

Owner

Sorry, missed the ping, especially in the light of what I’m about to say :)

however, one clarification: if I add a FileHandle.prototype.stat(), for example, would you prefer not having fs.promises.fstat()?

I really don’t care. 😄

+ stats_field_array_(env->isolate(), 14) {
+ auto resolver = Promise::Resolver::New(env->context()).ToLocalChecked();
+ object()->Set(env->context(), env->promise_string(),
+ resolver.As<Promise>()).FromJust();
@jasnell

jasnell Jan 31, 2018

Owner

@bmeurer ... just noting.... this is the bit we need to see if we can optimize (e.g. creating the promise in C++ using Promise::Resolver::New

+const fs = require('fs');
+
+const bench = common.createBenchmark(main, {
+ n: [20e4],
@psmarshall

psmarshall Feb 1, 2018

Contributor

This could just be 2e5?

@jasnell

jasnell Feb 2, 2018

Owner

Yeah. There are a couple like this in the benchmark suite. I'll update those all separately.

Owner

targos commented Feb 2, 2018

Does it make sense to put the promises API in a separate file?
I think about a file in the internal folder. Just required lazily when fs.promises is accessed.

Owner

targos commented Feb 2, 2018

fs.js is already quite big. That's why I ask.

Owner

mcollina commented Feb 2, 2018

It seems the best way to load this from esm would be as 'fs/promises' and not as fs.promises.
It would make sense to be future proof with this, and go for 'fs/promises' in both cases.

Owner

jasnell commented Feb 2, 2018

The key challenge with that approach right now is that the current cjs loader does not support it and would require changes that would block this pr from landing until it does

LGTM

@lacolaco lacolaco referenced this pull request in asciidwango/js-primer Feb 3, 2018

Open

node-cli: util.promisifyについて触れる #311

Owner

jasnell commented Feb 5, 2018

Marking this don't land on <= 9.x because it depends on semver-major's

Owner

jasnell commented Feb 5, 2018

@targos ... splitting it off into a separate file would require splitting some other bits out that I'd rather not do in this PR. A separate PR can move things around.

Owner

jasnell commented Feb 6, 2018

Ok, think we've finally got a good one here: https://ci.nodejs.org/job/node-test-pull-request/12963/

Owner

jasnell commented Feb 6, 2018

remaining failures in CI are unrelated.

jasnell added some commits Jan 19, 2018

fs: add FileHandle object fd wrapper
The `node::fs::FileHandle` object wraps a file descriptor
and will close it on garbage collection along with a
process warning. The intent is to prevent (as much as
possible) file descriptors from being leaked if the user
does not close them explicitly.
fs: add initial set of fs.promises APIs
Initial set of fs.promises APIs with documentation and one
benchmark.

jasnell added a commit that referenced this pull request Feb 6, 2018

src: handle exceptions in env->SetImmediates
PR-URL: #18297
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

jasnell added a commit that referenced this pull request Feb 6, 2018

fs: add FSReqPromise
PR-URL: #18297
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

jasnell added a commit that referenced this pull request Feb 6, 2018

fs: add FileHandle object fd wrapper
The `node::fs::FileHandle` object wraps a file descriptor
and will close it on garbage collection along with a
process warning. The intent is to prevent (as much as
possible) file descriptors from being leaked if the user
does not close them explicitly.

PR-URL: #18297
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

jasnell added a commit that referenced this pull request Feb 6, 2018

fs: add initial set of fs.promises APIs
Initial set of fs.promises APIs with documentation and one
benchmark.

PR-URL: #18297
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Owner

jasnell commented Feb 6, 2018

Landed in df34029, 7154bc0, 85b37db and 329fc78. woot!

@jasnell jasnell closed this Feb 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment