Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
fs: add promises API #18297
Conversation
nodejs-github-bot
added
C++
lib / src
labels
Jan 22, 2018
jasnell
changed the title from
Fs promises
to
fs: add promises API
Jan 22, 2018
|
@addaleax ... because I know you'll ask.. this intentionally does not do the |
This was referenced Jan 22, 2018
jasnell
added
semver-minor
experimental
labels
Jan 22, 2018
|
My intent would be to land this as an experimental feature |
|
Just fyi.. benchmark comparison for bench-stat vs bench-stat-promises (running in a vm on a laptop...)
|
vsemozhetbyt
added
promises
fs
labels
Jan 22, 2018
| +`fs.promises.open()` method. | ||
| + | ||
| +Unlike the callback-based `fs.fstat()`, `fs.fchown()`, `fs.fchmod()`, | ||
| +`fs.ftruncate()`, `fs.read()`, and `fs.write()`, operations -- all of which |
vsemozhetbyt
Jan 23, 2018
Member
Is it supposed to be an exhaustive list? If so, maybe fs.appendFile(), fs.fdatasync(), fs.fsync(), fs.readFile(), fs.futimes(), and fs.writeFile() should be added.
| +--> | ||
| + | ||
| +* Returns: {Promise} A `Promise` that will be resolved once the underlying | ||
| + file descriptor is closed, or will reject if an error occurs while closing. |
| +* `fs.constants.X_OK` - `path` can be executed by the calling process. This has | ||
| +no effect on Windows (will behave like `fs.constants.F_OK`). | ||
| + | ||
| +If the accessible check is successful, the `Promise` is resolved with no |
| + * `encoding` {string|null} **Default:** `'utf8'` | ||
| + * `mode` {integer} **Default:** `0o666` | ||
| + * `flag` {string} **Default:** `'a'` | ||
| + |
| + | ||
| +The `file` may be specified as a `FileHandle` that has been opened | ||
| +for appending (using `fs.promises.open()`). The `FileHandle` will | ||
| +not be closed automatically. |
vsemozhetbyt
Jan 23, 2018
Member
Does this mean that any FileHandle opened for appending will not be closed automatically? If so, should this be added to FileHandle description as an exception?
| +* Returns: {Promise} | ||
| + | ||
| +Changes the ownership of a file then resolves the `Promise` with no arguments | ||
| +upon success |
| + | ||
| +Asynchronously copies `src` to `dest`. By default, `dest` is overwritten if it | ||
| +already exists. No arguments other than a possible exception are given to the | ||
| +callback function. Node.js makes no guarantees about the atomicity of the copy |
| +Truncates the file represented by `filehandle` then resolves the `Promise` | ||
| +with no arguments upon success. | ||
| + | ||
| +If the file referred to by the file descriptor was larger than `len` bytes, only |
| +If the file referred to by the file descriptor was larger than `len` bytes, only | ||
| +the first `len` bytes will be retained in the file. | ||
| + | ||
| +For example, the following program retains only the first four bytes of the file |
| +* `mtime` {number|string|Date} | ||
| +* Returns: {Promise} | ||
| + | ||
| +Change the file system timestamps of the object referenced by the supplied file |
| +* `path` {string|Buffer|URL} | ||
| +* Returns: {Promise} | ||
| + | ||
| +Asynchronous lstat(2). The `Promise` is resolved with the [`fs.Stats'][] object |
| + | ||
| +*Note*: The behavior of `fs.promises.open()` is platform-specific for some | ||
| +flags. As such, opening a directory on macOS and Linux with the `'a+'` flag - | ||
| +see example below - will return an error. In contrast, on Windows and FreeBSD, |
| + | ||
| +Following successful read, the `Promise` is resolved with an object with a | ||
| +`bytesRead` property specifying the number of bytes read, and a `buffer` property | ||
| +that is a referece to the passed in `buffer` argument. |
| + * `encoding` {string} **Default:** `'utf8'` | ||
| +* Returns: {Promise} | ||
| + | ||
| +Reads the contents of a directory then resolves with `Promise` with an array |
vsemozhetbyt
Jan 23, 2018
Member
resolves with Promise with an array -> resolves the Promise with an array?
| + | ||
| +The optional `options` argument can be a string specifying an encoding, or an | ||
| +object with an `encoding` property specifying the character encoding to use for | ||
| +the filenames passed to the callback. If the `encoding` is set to `'buffer'`, |
| +* `options` {Object|string} | ||
| + * `encoding` {string|null} **Default:** `null` | ||
| + * `flag` {string} **Default:** `'r'` | ||
| + |
| +returned. | ||
| + | ||
| +Any specified `FileHandle` has to support reading. If a `FileHandle` is | ||
| +specified as the `path`, it will not be closed automatically. |
vsemozhetbyt
Jan 23, 2018
Member
"it will not be closed automatically" — if this is true, is it worth to be mentioned as an exception in the FileHandle description?
| + | ||
| +The optional `options` argument can be a string specifying an encoding, or an | ||
| +object with an `encoding` property specifying the character encoding to use for | ||
| +the link path passed to the callback. If the `encoding` is set to `'buffer'`, |
| +* Returns: {Promise} | ||
| + | ||
| +Determines the actual location of `path` using the same semantics as the | ||
| +`fs.realpath.native()` function then resolves the `Promise` with the resolved |
jasnell
Jan 23, 2018
Owner
No, I don't think so. The current fs.realpath() function is a legacy artifact that we're pretty much stuck with thanks to existing userland code. Since this is a new API, we have the opportunity not to continue that forward.
| + | ||
| +The optional `options` argument can be a string specifying an encoding, or an | ||
| +object with an `encoding` property specifying the character encoding to use for | ||
| +the path passed to the callback. If the `encoding` is set to `'buffer'`, |
| +no arguments upon success. | ||
| + | ||
| +Using `fs.promises.rmdir()` on a file (not a directory) results in the | ||
| +`Promise` being reject with an `ENOENT` error on Windows and an `ENOTDIR` error |
| +- Values can be either numbers representing Unix epoch time, `Date`s, or a | ||
| + numeric string like `'123456789.0'`. | ||
| +- If the value can not be converted to a number, or is `NaN`, `Infinity` or | ||
| + `-Infinity`, a `Error` will be thrown. |
| + | ||
| +The `Promise` is resolved with an object containing a `bytesWritten` property | ||
| +identifying the number of bytes written, and a `buffer` property containing | ||
| +the a reference to the `buffer` written. |
| +should be written. If `typeof position !== 'number'`, the data will be written | ||
| +at the current position. See pwrite(2). | ||
| + | ||
| +The callback will be given three arguments `(err, bytesWritten, buffer)` where |
| +The callback will be given three arguments `(err, bytesWritten, buffer)` where | ||
| +`bytesWritten` specifies how many _bytes_ were written from `buffer`. | ||
| + | ||
| +Note that it is unsafe to use `fs.write` multiple times on the same file |
| +`bytesWritten` specifies how many _bytes_ were written from `buffer`. | ||
| + | ||
| +Note that it is unsafe to use `fs.write` multiple times on the same file | ||
| +without waiting for the `Promise` to resolve. For this scenario, |
| + * `encoding` {string|null} **Default:** `'utf8'` | ||
| + * `mode` {integer} **Default:** `0o666` | ||
| + * `flag` {string} **Default:** `'w'` | ||
| + |
| +The `encoding` option is ignored if `data` is a buffer. It defaults | ||
| +to `'utf8'`. | ||
| + | ||
| +If `options` is a string, then it specifies the encoding. Example: |
| + | ||
| +Any specified `FileHandle` has to support writing. | ||
| + | ||
| +It is unsafe to use `fs.writeFile` multiple times on the same file without |
| +waiting for the `Promise` to be resolved (or rejected). | ||
| + | ||
| +If a `FileHandle` is specified as the `file`, it will not be closed | ||
| +automatically. |
vsemozhetbyt
Jan 23, 2018
Member
"it will not be closed automatically" — if this is true, is it worth to be mentioned as an exception in the FileHandle description?
jasnell
Jan 23, 2018
Owner
good point, will need to figure out the different wording here. The FileHandle will be closed if it is allowed to gc .... but if a FileHandle is provided to this function, the function itself will not call filehandle.close(), it's still up to the user to do that.
| @@ -3446,6 +4203,7 @@ The following constants are meant for use with the [`fs.Stats`][] object's | ||
| [`AHAFS`]: https://www.ibm.com/developerworks/aix/library/au-aix_event_infrastructure/ | ||
| [`Buffer.byteLength`]: buffer.html#buffer_class_method_buffer_bytelength_string_encoding | ||
| [`Buffer`]: buffer.html#buffer_buffer | ||
| +[`FileHandle`]: #fs_class_filehandle |
vsemozhetbyt
Jan 23, 2018
Member
It is [FileHandle] (without backticks) in the signatures, so this needs to be synced.
Or this type can be added here to spare some doc bytes)
|
|
@vsemozhetbyt ... as always I love your reviews of my doc commits. Thank you for being so thorough. Yes, the omission of the I plan to go back and add better examples either in an iteration of this PR or a future one. |
| +} | ||
| + | ||
| +fs.promises = { | ||
| + async access(path, mode = fs.F_OK) { |
ofrobots
Jan 23, 2018
Contributor
@jasnell, can you elaborate? For my education, how does making the function async help with catching thrown errors? My understanding is that async merely enables the use of the await keyword.
targos
Jan 23, 2018
Owner
Any error thrown within an async function is automatically caught and causes rejection of the promise returned by the function. With a regular function you could have two cases: synchronous throw or rejection depending on where the error happens.
jasnell
Jan 23, 2018
Owner
Yeah, what @targos said... for instance, try the following cases:
function foo() {
throw new Error('foo'); // error is thrown synchronously
return new Promise(() => {});
}
async function bar() {
throw new Error('bar'); // error causes promise rejection
return new Promise(() => {});
}|
Thanks for this. This is great. Can we have a shorter namespace though? Something like |
|
@jasnell Your benchmark run seems to suggest that the promise performance is already quite good compared to the callback version. Although that is of course a micro-benchmark. Do you have a representative (macro-)benchmark? |
|
cc @psmarshall |
|
Yep and rightfully so given that they mostly use exactly the same internal mechanisms. The differences are quite minimal thankfully. Where they are different is in the creation of the additional promise and the additional overhead of the promise hook, both of which are fairly minimal. I do not yet have a macro benchmark in place. It'll likely take bit to get that. One thing I did notice is that there are about twice as many GC runs in the promises microbenchmark, which is not at all surprising but is certainly worth noting. |
|
@jasnell Did you see #17739 (comment) from @gsathya? That might be worth investigating. |
| cb.cb_(this, cb.data_); | ||
| + if (try_catch.HasCaught()) { |
joyeecheung
Jan 23, 2018
Contributor
It is worth updating the comment of SetImmediate that throwing errors in a native immediate callback would crash the process.
| + int ret = uv_fs_close(env()->event_loop(), &req, fd_, nullptr); | ||
| + uv_fs_req_cleanup(&req); | ||
| + | ||
| + struct err_detail { int ret; int fd; }; |
jasnell
Jan 23, 2018
Owner
Because it's only used here. If it turns out to be useful elsewhere later, moving it is easy :-)
| + AsyncCall(env, args, "open", UTF8, AfterOpenFileHandle, | ||
| + uv_fs_open, *path, flags, mode); | ||
| + } else { | ||
| + SYNC_CALL(open, *path, *path, flags, mode) |
| + | ||
| + private: | ||
| + bool finished_ = false; | ||
| + double statFields_[14] {}; |
joyeecheung
Jan 23, 2018
Contributor
Why making it an array here though? Maybe use a unique_ptr and only allocates memory and reset it when FillStatsArray is called?
jasnell
Jan 23, 2018
Owner
to avoid an extra allocation. should be pretty cheap doing it this way overall.
| + } | ||
| + | ||
| + const base = path.resolve(common.tmpDir, 'FOO'); | ||
| + assert( |
joyeecheung
Jan 23, 2018
Contributor
Does this work on windows? common.tmpDir would contain \ so it should not be able to be passed into new RegExp like that.
|
I don't know if we should discuss this here but I have a concern with the interaction of fs.promises and ESM (related to #18131). |
|
@bmeurer ... Yeah, I'm working that way but wanted to get all of the js functions impl'd to make sure we would be able to completely eliminate creating the FSReqPromise object in js. I've got a way forward on that now that should work nicely, tho I'll have to test that it does not negatively impact the perf of the callback approach first. |
|
@targos ... definitely a valid concern, will have to think about that a bit more. FWIW, we would have precisely the same issue with |
|
@bmeurer ... updated to eliminate the creation of
|
|
@targos makes a great point in #18297 (comment). I'd be a definite +1 on |
|
@jasnell if we were implementing |
| cb.cb_(this, cb.data_); | ||
| + if (try_catch.HasCaught()) { | ||
| + FatalException(isolate(), try_catch); | ||
| + } |
addaleax
Jan 23, 2018
Owner
Alternative suggestion to avoid setting up a TryCatchs for every individual callback:
diff --git a/src/env.cc b/src/env.cc
index b05f0bec81df..f70945ca4c3a 100644
--- a/src/env.cc
+++ b/src/env.cc
@@ -290,13 +290,26 @@ void Environment::RunAndClearNativeImmediates() {
size_t ref_count = 0;
std::vector<NativeImmediateCallback> list;
native_immediate_callbacks_.swap(list);
- for (const auto& cb : list) {
- cb.cb_(this, cb.data_);
- if (cb.keep_alive_)
- cb.keep_alive_->Reset();
- if (cb.refed_)
- ref_count++;
- }
+ auto drain_list = [&]() {
+ v8::TryCatch try_catch(isolate());
+ for (auto it = list.begin(); it != list.end(); ++it) {
+ it->cb_(this, it->data_);
+ if (it->keep_alive_)
+ it->keep_alive_->Reset();
+ if (it->refed_)
+ ref_count++;
+ if (UNLIKELY(try_catch.HasCaught())) {
+ FatalException(isolate(), try_catch);
+ // Bail out, remove the already executed callbacks from list
+ // and set up a new TryCatch for the other pending callbacks.
+ std::move_backward(it, list.end(), list.begin() + (list.end() - it));
+ list.resize(list.end() - it);
+ return true;
+ }
+ }
+ return false;
+ };
+ while (drain_list()) {}
#ifdef DEBUG
CHECK_GE(immediate_info()->count(), count);I’m happy to open a with PR that as a separate patch if you prefer.
In any case, we should probably run the timers benchmarks for setImmediate.
jasnell
Jan 23, 2018
Owner
Nice. feel free to push a separate commit to this PR. Just isolate the change so I can squash it down with the existing one that edits this bit :-)... I'm good with this in whatever way you think is best, really
| + | ||
| + // If the close was successful, we still want to emit a process warning | ||
| + // to notify that the file descriptor was gc'd. We want to be noisy about | ||
| + // this because not explicitly closing the garbage collector is a bug. |
addaleax
Jan 23, 2018
Owner
I think you mean something like “not explicitly closing the file descriptor is a bug”, right?
| + resolver->Reject(env()->context(), reason); | ||
| +} | ||
| + | ||
| +FileHandle* FileHandle::CloseReq::fd() { |
| -FSReqAfterScope::FSReqAfterScope(FSReqWrap* wrap, uv_fs_t* req) | ||
| +void FSReqPromise::Resolve(Local<Value> value) { | ||
| + finished_ = true; | ||
| + InternalCallbackScope callback_scope(this); |
addaleax
Jan 23, 2018
Owner
I’d always open the HandleScope first because InternalCallbackScope uses handles (in a way that would be kind of hard to rewrite)
| + env()->promise_string()).ToLocalChecked(); | ||
| + CHECK(val->IsPromise()); | ||
| + Local<Promise> promise = val.As<Promise>(); | ||
| + Local<Promise::Resolver> resolver = promise.As<Promise::Resolver>(); |
| @@ -190,15 +380,15 @@ bool FSReqAfterScope::Proceed() { | ||
| } | ||
| void AfterNoArgs(uv_fs_t* req) { | ||
| - FSReqWrap* req_wrap = static_cast<FSReqWrap*>(req->data); | ||
| + FSReqBase* req_wrap = static_cast<FSReqBase*>(req->data); |
addaleax
Jan 23, 2018
Owner
Side note (maybe try to remind me once this lands): There should be a ReqWrap<T>::From(T* uv_request) method that a) uses ContainerOf instead of requiring the lookup and b) doesn’t rely on blindlly casting void pointers to C++ non-base classes, because the latter is bound to go wrong at some point
| #define SYNC_RESULT err | ||
| +inline FSReqBase* GetReqWrap(Environment* env, const Local<Value>& value) { |
addaleax
Jan 23, 2018
Owner
nit: we usually pass Locals by value instead of references because they’re effectively just pointers
| + | ||
| + private: | ||
| + bool finished_ = false; | ||
| + AliasedBuffer<double, v8::Float64Array> stats_field_array_; |
addaleax
Jan 23, 2018
Owner
Can we avoid this in some way, i.e. create the array buffer + the typed array only once we’re actually at the resolve point and know that this is actually a stat call?
jasnell
Jan 23, 2018
Owner
That's essentially what it was, then the change landed today that altered the signature for FillStatsArray to require an AliasedBuffer. I updated today just to get it working again but plan on revisiting before this lands.
|
@addaleax .. updated with your suggested change on the |
|
Benchmark for timers/immediates: https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/106/ |
|
@addaleax ... the benchmarks look good with the change you suggested. Will move forward with that. |
|
@targos @guybedford ... went with a slightly different approach for now where |
|
fyi @bmeurer .... Quick profile of a simple test app that performs 2e5 sequential stat and read operations, one after the other within an async function. Aside from the frequent gc, everything is looking healthy: In contrast, here's the equivalent example using the callbacks model: Note the patterns are nearly identical with one notable exception: the timescale for the promises version is much longer... Essentially, by enabling trace events and async_hooks instrumentation in order to capture the analysis, we kill the performance of the promises version. Without those hooks enable, performance is much closer to the callbacks version. |
| +added: REPLACEME | ||
| +--> | ||
| + | ||
| +* `file` {string|Buffer|[FileHandle][]} filename or `FileHandle` |
| +deprecated: REPLACEME | ||
| +--> | ||
| + | ||
| +* `path` {string|Buffer} |
| +deprecated: REPLACEME | ||
| +--> | ||
| + | ||
| +* `path` {string|Buffer} |
| +added: REPLACEME | ||
| +--> | ||
| + | ||
| +* `path` {string|Buffer} |
| +added: REPLACEME | ||
| +--> | ||
| + | ||
| +* `file` {string|Buffer|[FileHandle][]} filename or `FileHandle` |
|
@jasnell Wow, that's pretty cool. I'm currently working on the GC issues. I think I've found a way to significantly reduce allocations for normal promise usage, so that should help with that. |
|
Woo! Nice. The allocations are the only real issue I'm seeing here so if you're able to optimize that I think we're golden. Of course, there's still the separate issue of the async hooks performance hit but we'll have to deal with that separately |
|
@jasnell I wouldn't say golden, but at least the overhead should be less relevant. My current thinking is also along the lines of first addressing the |
|
That's what I suspect as well. I think what we've got in this pr is an excellent starting point that we can iterate from. Really appreciate the time you and your team have put in on optimizing this stuff |
addaleax
reviewed
Jan 28, 2018
The code basically LGTM, I just have one question + something that I'd really like to see changed about the way the file handle API is exposed (and I know I've mentioned that before so there's a decent change you disagree? ;))
| +The `fs.promises` API provides an alternative set of asynchronous file system | ||
| +methods that return `Promise` objects rather than using callbacks. The | ||
| +API is accessible via `fs.promises()` (for example: | ||
| +`const fsp = require('fs').promises()`). |
TimothyGu
Jan 28, 2018
Member
Yes: #18297 (comment)
This is indeed pretty unidiomatic, for both CJS and ESM. I'd prefer the original fs.promises.* for CJS and import 'fs/promises'; for ESM.
If we can agree to a solution right now, then I don't think we necessarily have to implement it before this PR gets merged.
pitaj
Jan 28, 2018
I don't see why "fs/promises" can't work for both CJS and ESM. It's probably better to be consistent.
jasnell
Jan 29, 2018
Owner
Well, the most immediate reason is that our current CJS loader implementation would not support fs/promises as an option without modifications and I'm quite reluctant to do the necessary modifications within this PR.
jasnell
Jan 31, 2018
Owner
I think, for now, I'm going to change this back to a getter. Hopefully we'll get a better solution figured out for ESM longer term but I'd rather not hold this PR up for this.
| + .catch(() => console.log('The file could not be copied')); | ||
| +``` | ||
| + | ||
| +### fs.promises().fchmod(filehandle, mode) |
addaleax
Jan 28, 2018
Owner
Can't these be methods on FileHandle? That seems a lot cleaner to me, especially since FDs are intended to be object-oriented...
jasnell
Jan 29, 2018
Owner
Eventually, yes, that's where I'd like to go with it and we likely should wrap the C++ provided FileHandle object with a pure JS object now in preparation for hanging additional things off of it. For now, as I've said, one of the goals is to have close parity between the new API and the existing API in order to make eventual migration easier. However, if there's enough support for it, I can get behind making this change now.
jasnell
Jan 31, 2018
Owner
I'll go ahead and add this in now... however, one clarification: if I add a FileHandle.prototype.stat(), for example, would you prefer not having fs.promises.fstat()?
addaleax
Feb 2, 2018
Owner
Sorry, missed the ping, especially in the light of what I’m about to say :)
however, one clarification: if I add a
FileHandle.prototype.stat(), for example, would you prefer not havingfs.promises.fstat()?
I really don’t care.
| + stats_field_array_(env->isolate(), 14) { | ||
| + auto resolver = Promise::Resolver::New(env->context()).ToLocalChecked(); | ||
| + object()->Set(env->context(), env->promise_string(), | ||
| + resolver.As<Promise>()).FromJust(); |
jasnell
Jan 31, 2018
Owner
@bmeurer ... just noting.... this is the bit we need to see if we can optimize (e.g. creating the promise in C++ using Promise::Resolver::New
| +const fs = require('fs'); | ||
| + | ||
| +const bench = common.createBenchmark(main, { | ||
| + n: [20e4], |
jasnell
Feb 2, 2018
Owner
Yeah. There are a couple like this in the benchmark suite. I'll update those all separately.
|
Does it make sense to put the promises API in a separate file? |
|
|
|
It seems the best way to load this from esm would be as |
|
The key challenge with that approach right now is that the current cjs loader does not support it and would require changes that would block this pr from landing until it does |
lacolaco
referenced this pull request
in asciidwango/js-primer
Feb 3, 2018
Open
node-cli: util.promisifyについて触れる #311
jasnell
added
dont-land-on-v4.x
dont-land-on-v6.x
dont-land-on-v8.x
dont-land-on-v9.x
labels
Feb 5, 2018
|
Marking this don't land on <= 9.x because it depends on semver-major's |
|
New New New CI: https://ci.nodejs.org/job/node-test-pull-request/12961/ |
|
@targos ... splitting it off into a separate file would require splitting some other bits out that I'd rather not do in this PR. A separate PR can move things around. |
|
Another CI Run: https://ci.nodejs.org/job/node-test-pull-request/12962/ |
|
Ok, think we've finally got a good one here: https://ci.nodejs.org/job/node-test-pull-request/12963/ |
|
remaining failures in CI are unrelated. |


jasnell commentedJan 22, 2018
fs.Promises, the actual PR (replaces #17739)
/cc @mcollina @addaleax
/cc @bmeurer and @nodejs/v8 @nodejs/chakracore ... it would be excellent if we could have y'all take a look at this and do some performance testing.
Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passesAffected core subsystem(s)
fs