Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.
Sign upmodules: Package exports validations and fallbacks #28949
Conversation
nodejs-github-bot
added
the
C++
label
Aug 3, 2019
guybedford
requested a review
from jkrems
Aug 3, 2019
guybedford
force-pushed the
guybedford:exports-validations
branch
from
b158078
to
01f017b
Aug 3, 2019
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Should we be preventing nested node_modules paths?
|
bmeck
requested changes
Aug 3, 2019
|
nits / question, seems fine if we just add tests to answer the question of inner packages and avoid breakage in future. i have no strong opinions on throwing vs allowing. |
ljharb
reviewed
Aug 3, 2019
| This is just a convention that works because `false`, just like `{}`, has no | ||
| iterable own properties. | ||
|
|
||
| Any invalid exports entries will be ignored. This includes exports not |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
guybedford
Aug 4, 2019
Author
Contributor
The problem with warnings in Node.js is that when you warn for behaviour of packages in node_modules, there isn't much the user can do. This is in contrast to the browser, where warnings are for developers not users (console = developers), wheras in Node.js the users get the console.
So if we just spit out these warnings and we have a future-compatibility path, then it's just going to be spam, so I'd be against that.
Perhaps we can explicitly make these debug messages though under NODE_DEBUG=exports or similar in a follow-on PR.
This comment has been minimized.
This comment has been minimized.
ljharb
Aug 4, 2019
Member
That’s a fair point, but it’s the same with that event emitter memory leak issue.
This comment has been minimized.
This comment has been minimized.
jkrems
Aug 5, 2019
Contributor
It's not quite the same as the memory leak warning because we expect that there are invalid entries and ignoring them on older versions is exactly what should be happening on those. I would compare it to warning on unknown package.json properties. It would catch mian: lib.js but it would be much more likely to warn on all kinds of stuff that is perfectly correct.
This comment has been minimized.
This comment has been minimized.
ljharb
Aug 5, 2019
Member
It’s the same in that it’s not actionable to the top level app, because it originates in an installed package.
The same is true for npm audit warnings.
I think this kind of warning is important for encouraging proper ecosystem behavior.
This comment has been minimized.
This comment has been minimized.
guybedford
Aug 6, 2019
Author
Contributor
The latest commit I just added gives the following for invalid export resolutions -
eg -
{
"exports": {
"./x": "in:valid"
}
}import 'pkg/x' will then provide the error:
Error: Cannot resolve package exports target 'in:valid' matched for './x' in C:\Users\Guy\Projects\node\node_modules\pkg\package.json, imported from C:\Users\Guy\Projects\node\x.mjs
at Loader.resolve [as _resolve] (internal/modules/esm/default_resolve.js:59:13)
at Loader.resolve (internal/modules/esm/loader.js:73:33)
at Loader.getModuleJob (internal/modules/esm/loader.js:149:40)
at ModuleWrap.<anonymous> (internal/modules/esm/module_job.js:43:40)
at link (internal/modules/esm/module_job.js:42:36) {
code: 'ERR_MODULE_NOT_FOUND'
This comment has been minimized.
This comment has been minimized.
ljharb
Aug 6, 2019
Member
what about for any processing of that package.json? I’d expect a warning when the package is read, not just when that path is imported.
This comment has been minimized.
This comment has been minimized.
guybedford
Aug 6, 2019
Author
Contributor
You could imagine a package that provides one export pkg/newfeature for modern Node.js, and another pkg/legacyfeature for legacy Node.js. If we validate all exports on load we get the same issue discussed above with warnings for things that aren't used.
As mentioned, I'd be open to a top-level flag for this kind of global validation, but I'd be against making it the default behaviour for the reason that it is simply entirely unactionable to the user - you can't even post a PR to the dependency package itself like other Node.js warnings for node_modules.
This comment has been minimized.
This comment has been minimized.
ljharb
Aug 6, 2019
Member
again, I’m talking about invalid ones, not nonexistent ones.
I think it’s appropriate to always warn when using a package whose package.json has any exports that can’t be used in my node version. The actionable item is “stop using that package” or “upgrade node”.
This comment has been minimized.
This comment has been minimized.
guybedford
Aug 7, 2019
•
Author
Contributor
Along the lines of Jan's comment above, this is like saying that an API exporting:
exports.legacy = function () {
// uses fs.oldAPI();
}
exports.modern = function () {
// uses fs.newAPI();
}should warn in legacy environments that the modern function exists just because it might call an undefined core library if you run pkg.modern().
There is nothing wrong with pkg.modern function existing if it isn't called in legacy environments. Similarly there should be nothing wrong with defining exports that only apply to modern use cases.
As I've mentioned, I'm happy to have a global validation mode as opt-in, but it shouldn't be the default for these reasons.
lib/internal/modules/cjs/loader.js Outdated
lib/internal/modules/cjs/loader.js Outdated
src/module_wrap.cc Outdated
This comment has been minimized.
This comment has been minimized.
|
Re importing from node_modules, I could get behind explicitly restricting that case as well, if others agree? |
This comment has been minimized.
This comment has been minimized.
|
@guybedford easier to restrict for now and ease it later. |
This comment has been minimized.
This comment has been minimized.
|
Note that if we want to do this restriction, then we should also restrict the package boundary detection to not pass through node_modules as well. I can implement both together here. |
ljharb
reviewed
Aug 4, 2019
| This is just a convention that works because `false`, just like `{}`, has no | ||
| iterable own properties. | ||
|
|
||
| Any invalid exports entries will be ignored. This includes exports not |
This comment has been minimized.
This comment has been minimized.
ljharb
Aug 4, 2019
Member
That’s a fair point, but it’s the same with that event emitter memory leak issue.
This comment has been minimized.
This comment has been minimized.
|
Another interesting thing about blocking |
This comment has been minimized.
This comment has been minimized.
|
Blocking node_modules inside the package also breaks reasonable bundles
dependencies or other packages with a genuine node_modules directory inside
of the package. I’m -1 on special-casing it but not strongly enough to
block on it.
…On Sat, Aug 3, 2019 at 10:00 PM Guy Bedford ***@***.***> wrote:
Another interesting thing about blocking node_modules is that "exports":
{ "./": "./" } is no longer a noop as it opts-in to not being able to
load from node_modules.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#28949?email_source=notifications&email_token=AAEKR5FOI2I7NWNWLQHRUF3QCYZ5VA5CNFSM4IJCID4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3PY4ZA#issuecomment-517967460>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAEKR5E6F2HFVZQWMJKMD4DQCYZ5VANCNFSM4IJCID4A>
.
|
This comment has been minimized.
This comment has been minimized.
|
Ok I've implemented the node_modules restrictions, we can always relax this later too if need be, but it is difficult to add later so this seems the cautious route. |
guybedford
requested a review
from bmeck
Aug 4, 2019
vsemozhetbyt
reviewed
Aug 4, 2019
doc/api/esm.md Outdated
vsemozhetbyt
reviewed
Aug 4, 2019
doc/api/esm.md Outdated
bmeck
reviewed
Aug 5, 2019
lib/internal/modules/cjs/loader.js Outdated
bmeck
reviewed
Aug 5, 2019
lib/internal/modules/cjs/loader.js Outdated
bmeck
requested changes
Aug 5, 2019
|
minor request to ensure query and hash of URL don't get included in the path check. |
This comment has been minimized.
This comment has been minimized.
|
@jkrems I think the problem here is we don't have an overall goal for what this feature is trying to aid. Is it trying to prevent package boundary crossing at all, is it trying to deal with package installation locations being unreliable, etc. I think the most conservative approach is the right one until we agree upon a goal so that we can relax as needed. For example, validating bundled dependencies could be checked via |
| f. If either the key or exports[key] do not end with a slash (`/`), | ||
| throw "not found". | ||
| g. Return DIR/name/${exports[key]}${subpath.slice(key.length)}. | ||
| e. let RESOLVED_URL = |
This comment has been minimized.
This comment has been minimized.
guybedford
and others
added some commits
Aug 2, 2019
guybedford
force-pushed the
guybedford:exports-validations
branch
from
677ee4e
to
5e58810
Aug 6, 2019
guybedford
force-pushed the
guybedford:exports-validations
branch
from
e83c354
to
5a09519
Aug 6, 2019
guybedford
added some commits
Aug 6, 2019
guybedford
force-pushed the
guybedford:exports-validations
branch
from
5a09519
to
a7e3e76
Aug 6, 2019
guybedford
requested a review
from bmeck
Aug 6, 2019
This comment has been minimized.
This comment has been minimized.
|
I've added a new commit here that refines the validation failure errors to include some contextual information so users can see how to correct it. We can hopefully continue to refine this over time. |
This comment has been minimized.
This comment has been minimized.
|
(Travis failure seems unrelated) |
jkrems
approved these changes
Aug 6, 2019
jkrems
added this to In progress
in Support "exports" in package.json
via automation
Aug 6, 2019
This comment has been minimized.
This comment has been minimized.
jkrems
moved this from In progress
to Review in progress
in Support "exports" in package.json
Aug 6, 2019
This comment has been minimized.
This comment has been minimized.
bmeck
approved these changes
Aug 6, 2019
Support "exports" in package.json
automation
moved this from Review in progress
to Reviewer approved
Aug 6, 2019
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
@ljharb LGTY? |
ljharb
approved these changes
Aug 8, 2019
|
I still think we should be more aggressive on warning for any invalid export in any processed package.json - we can remove warnings later if needed, more easily than adding them. |
guybedford
added
the
author ready
label
Aug 9, 2019
This comment has been minimized.
This comment has been minimized.
|
I'll aim to land this in the next day or so if no one beats me to it. Happy to continue the discussion on validations for potential follow-ups. |
jkrems
reviewed
Aug 10, 2019
lib/internal/modules/cjs/loader.js Outdated
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Landed in 2103ae4. |
guybedford commentedAug 3, 2019
•
edited
This clarifies and specifies the package "exports" proposal validations roughly in a way similar to as specified for import maps (except slightly more restrictive - no URLs or backtracking).
The validations implemented are:
/pkg/map/../../below).../or/are not permittednode_modulespath segmentnode_modulessegments are not permittedInstead of throwing on array targets, a very simple version of fallback arrays is added to allow for forwards compatibility in future, behaving just like import maps where the first valid match of the array is selected, based on the validation rules above.
The implementation is provided for both CJS and ESM, with associated specification changes for both as well.
Along with banning
node_modulessegments for exports, for consistency an associated addition to the package scope lookup algorithm is provided to avoid checking package scope package.json lookups through any node_modules path segment.In addition a test has been included for the use of
%2Fin exports targets and user paths.Checklist
make -j4 test(UNIX), orvcbuild test(Windows) passes