I like the idea and find it useful, but think it would be far more useful (I'm thinking specifically in the case of a DSL) if I can get a single string expansion. So if I've built a cmd.exe compatible string in my PS, I have a way to string verbatim to cmd.exe. Perhaps

&n -command $string

That would potentially also let me specify my interpreter:

&n -shell /bin/sh -command $string
or
&n -shell cmd.exe -command $string

in terms of the name/operator, is "&!" in use? That has similarity to shbang( "#!" ).

$! is a good suggestion!

The shell can be specified by simply specifying it as the command:

&! /bin/sh -c blah

The $var expansion creates a problem in that $ may need to be literal to be passed to the native command/shell. One of the problems this tries to avoid is when people need to figure out how to properly escape everything. If you require variable expansion, you could always do:

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

However, in cases where you want to mix PowerShell with the native command, it's probably best to use the current syntax and just be aware of what needs to be escaped properly as I would consider that advanced usage.

$mycommand = "/bin/sh ls"
Invoke-Expression "&! $mycommand"

I can't believe you just suggested iex. :-)

@essentialexch there are a few times where it's appropriate :)

Note that in the example, expectation is that the user properly validated the contents of $mycommand before executing!

Great writeup and a solution for problems like the one you described are definitely needed.
I myself didn't know about --% and don't remember ever seeing it unfortunately. Therefore the proposed &n operator might suffer from similar discoverability problems and it doesn't look natural to me to combine a symbol with a character (and it reminds me of %f in C printf statements). Since it's a breaking change anyway Is there a reason why it couldn't be for example &&?
Maybe it would me more intuitive to use something like brackets (or double braces?) to mark the area that you want to execute. Example: & [ wsl ls ] | less

I agree &! is the more intuitive way to go for those coming from other shells, however in super-duper-linter I am usually running native commands by building a parameter array and then splatting it to the command.

$command = 'linter'
$myargs = @(
    '-config'
    'linterconfig.path'
)
if ($Test) {$myargs += 'test'}
& $command @myargs

So this is the best way in case of conditional and whatnot for me

@bergmeister I originally thought of && as visually it's similar to & which many people know today. However, && being a pipeline chain operator may cause confusion about what it is supposed to do. I like the &! suggestion currently.

@JustinGrote, there's no reason to not continue to use that. the &! syntax is really for cases where you just want to cut and paste or have some args that conflict with PowerShell syntax and don't want to or don't know how to escape properly

Oh man, I remember having discussions with Bruce and Jason about this a decade ago. My gut feeling is that inventing another operator seems unnecessary here. I know that some people on this thread have not heard about --% but I'll wager than there are more than you think. What's wrong with:

# run ls in wsl, return results to powershell, then pipe to wsl again, to grep.
& wsl ls | wsl grep -i "foo"  

#  the entire pipeline right of wsl will run in wsl. 
& --% wsl ls | grep -i "foo"

Why hasn't anyone suggested this? It's logically consistent with usage of --% elsewhere to say "everything after this is to be passed without special powershell treatment."

@oising well for one, that command doesn't work because you have to preface it with the command you want to run, are you suggesting adding the functionality?

This kinda does what is expected:
function Invoke-LiteralCommand ($command) {Invoke-Expression "& $command --% $args"}

Invoke-LiteralCommand ping -W 200 www.google.com | grep icmp

@JustinGrote Yes, I know it doesn't work now :) I am suggesting that instead of &n or &! or whatever else is being talked about. It just makes more sense to me: & is call and --% is suppress powershell parsing; together they are coherent and more discoverable than adding something new. I don't like the idea of having a "call" and a "call native" operator.

I've expanded my example to show the differences.

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

Not at all. It's for the PowerShell user that doesn't understand the complex quoting rules between PowerShell/cmd.exe/bash. As the issue title says "call native", for calling native executables.

@oising I'd be cool with that over a new operator, though that is a lot of obtuse typing for "new powershell user who knows bash" which is what I assume this would be meant for.

Not at all. It's for the PowerShell user that doesn't understand the complex quoting rules between PowerShell/cmd.exe/bash. As the issue title says "call native", for calling native executables.

True, or for those of us who'd rather not have to think about them at all.

Yeah I'm with @oising. This concept exists, it's just woefully insufficient. If we're going to implement something completely new we're better off deprecating/removing the old syntax.

I feel oftentimes these ideas are voiced, not enough weight is given to their actual target demographic. If users are already struggling to find the existing methods, and finding the existing methods lacking when they are found, it's a call to a) improve documentation, and b) improve the actual functionality of the existing operators.

Instead we get a weird option c) which is supposed to somehow address the past issues whilst introducing yet another operator which is relying on documentation which is already not accessible enough for users to find it naturally.

I agree that error messages and/or the suggestions system should be used to help introduce these concepts. I don't agree that we need a third? fourth? fifth? (I've literally lost count, someone help me out here) way to invoke commands. The existing methods are insufficient -- that means we should improve them, not leave them behind like cobwebs to confuse users further when they start digging into them.

The first question I tend to get when people realise there's 5 ways of doing something is "why are there 2-3 ways that literally do the same thing but don't work as well", and my answer is as it always has been -- the PS team is way too focused on making sure everything from the past decade++ still works, when trying to add/improve functionality. Here we see it again, we have existing but insufficient implementations that need revision and improvement, and the solution proposed is to further muddy the pool by adding another potentially incomplete implementation.

We should be finishing what we start, not starting another implementation all over again, IMO.

I believe if we are going to be using & for more than a few Unique use cases we should start thinking about making this considered a verb or look at standardization across the language.

The last thing I would want is confusion about what & is meant to do. In its entirety.

I'd consider 3 scenarios:

1. Full expansion - current PowerShell behavior.
2. Full literal - that is proposed in the OP.
3. Partial expansion - wsl ls $path still works

I wonder where @mklement0 comments? :-)

At start of reading this thread I thought; great idea! But after reading comments it started me thinking. I have always found the & command 'not PS standards worthy" and something reminding me of 'PERL days' (ugh). Introducing another non-standard PS command (not being noun-verb) will not help. Certainly not those less savvy at PS.

I did not know about the --% parameter either. I use the same principle as @JustinGrote as a solution.

Cutting/pasting commands into PS shell has never been my main 'thing'.
Would we not be better of replacing those native commands with PowerShell one's?
Make path for replacing cmd.exe entirely...

I vote for exploring improving existing commands. And - indeed - improving some documentation on --% and & usage

@iSazonov the use case @SteveL-MSFT is proposing is the 'cut paste' scenario. Partial expansion would make things much more difficult I think (on the AST side of things)

Immediately though of “#!” and then “!#”. Liking the “&!”.

We can probably look at adding token accelerators to function calls in the tokenizer. This would allow for the use of the following.

Iex -l
Or Invoke-Expression -literal that stops parsing before passing up.

I feel adding more unique tokens at the parsing level makes the barrier to entry go up and discoverability for this feature goes down.

Get-Help & for example doesn't show up with anything. And we have to look for it in about_Operators.

However there are unique cases for the call_operator that doesn't get documented.
& modulename { command }
allows you to scope into a module. And I am certain there are more. But the discoverability on it is so low that it becomes difficult to find within the native documentation. And the community knows about it only through JSnover and a talk he gave showing us cool things.

I believe whatever is decided on it needs to fully take into account how discoverable this "feature" is, from a new user perspective and keep in mind that newer users will try to use this on Powershell 5 not without knowing its new to pwsh core if the discoverability is too low.

I'd love us to change the existing behaviour, but changing such a fundamental API in PowerShell would just break so many things. We could possibly consider migrating over with a configuration.

Breaking things more directly might once have been possible before PS 6 went GA, but honestly, even then it would have been a serious threat to backward compatibility (not unlike Python's print() function).

In terms of passing arguments to subprocesses, I think both synchronous invocation and Start-Process invocation already have storied issues discussing an overhaul, and my feeling is that both need investing in at once for consistency. Start-Process in particular needs to update its support of passing an array of arguments.

For synchronous invocation with cmdline arg passing, I see four kinds of argument passing possible:

• Current behaviour, which would become legacy, which is subject to CommandLineToArgvW on Windows and has unexpected results
• Default behaviour, that should pass arguments by their expression value but apply the usual bareword token rules, so that things like > and | separate commands. In this mode, the value a subprocess takes from the expression should be what is displayed by Write-Host $val
• Verbatim behaviour, where all tokens are interpreted as bareword strings until the escape token is seen. This is what --% is supposed to do today, but ideally would have only one end token that can be embedded on the same line like --% ... %--
• Bash readline behaviour, where an alternate, sh-oriented escaping logic is applied, allowing simpler compatibility

I think the first behaviour should be slowly phased out by introducing the second as an experimental feature and then swapping them.

The second and third behaviours could have their own sigils like &! and &# or perhaps +% ... -% or something. But for the verbatim mode, I think an important facet would be simplifying the escape token so that more tokens are taken verbatim. Similar to a heredoc.

If the request is only to address copy-paste scenario like "Any text content after this operator will call into the "default shell" of the OS to execute." why do not say this explicitly by shell?

PS> shell wsl ls | less
PS> (shell wsl ls *.txt) | Select-String Hello

It is more discoverable and readable than cryptic operators in Forth/APL style.

I absolutely don't see how this would resolve #1995: See #1995 (comment)

Also, the problem with WSL is a problem of the behavior of wsl.exe (see #12975 (comment)) not of powershell.
(Ok, there is a problem with powershell, but that's #1995)

Oh, and isn't this almost a duplicate of #12975 ? Because I can essentially mostly repeat, what I mentioned there:

@bitcrazed

The problem is that the lack of an ability to delimit a portion of a command-line to be passed varbatim to the receiving command/script is something that trips users up all the time.

What do you mean by "portion of a command-line to be passed varbatim"?

Do you mean

1. pass some sequence of characters verbatim to the called executable, such that the called executable has this sequence of characters in one element of its argument array (e.g. those characters are then available in argv[1] in main)

Or do you mean

2. insert some sequence of characters verbatim into the lpCommandLine parameter of CreateProcess

---

If you mean (1.), then it essentially already works:

PS /home/User> /bin/echo 'cd / && ls . | cowsay'
cd / && ls . | cowsay
PS /home/User>

(Except for the problem of embedded quotes as discussed in #1995)

One could argue, that adding a one-line-here-string would improve some usecases, but I think, that's not really the point of this issue.

As this does already work more or less, I assume, you meant (2.)

---

If you mean (2.), then let me state my opinion on that in a somewhat dramatic way:

Please please please don't add special syntax for this. This is basically what --% tried to do, which also should have never ever been implemented.

Why am I so strongly against this?

1. It is a Windows only problem, so adding syntax would mean that powershell on Windows has different syntax than on Linux. (Or the syntax would be supported but is totally meaningless, as it is currently the case for --%)

2. If the main commandline shell on Windows published by Microsoft adds a first-class feature (via special syntax opposed to via a cmdlet) to call executables that don't follow the typical commandline parsing rules (if the tool follows the typical rules, you don't need (2.), you can usually better use (1.)), then that encourages authors of command line tool, to not follow these rules, which only worsens the "Windows command line anarchy". The less people follow the typical rules, the harder it is to programmaticly call external executables or generally write cross platform code, so I definitely think, that program authors should be encouraged to follow those typical rules.

3. I strongly doubt, that this is a common use-case

   And this isn't just an issue that affects WSL: It also affects Windows Terminal's wt command-line invocations, and many other tools.

   Could you add some examples, where such problems occur? Because in case of WSL, I'd say that WSL's parsing of the commandline is simply broken (issue was about bash.exe but situation is by default not better with wsl.exe) in the default case - I'd consider every tool, that doesn't follow the typical commandline parsing rules broken, but WSL's default behavior is IMHO not even properly documented...

   I said the "default" behavior of wsl.exe is broken - while writing this response, I noticed, that wsl.exe actually seems to behave as expected, when using -e:

PS C:\> wsl -e bash -c 'cd / && ls . | cowsay'
 _______________________________________
/ acct bin boot cache cygdrive data dev \
| etc home init lib lib64 lost+found    |
| media mnt opt proc root run sbin snap |
\ srv sys tmp usr var                   /
 ---------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
PS C:\>

So the only thing missing for this explicit usecase is IMO a parameter to wsl.exe to call the default shell with the commandline arguments parsed as in any other normal exe.

The only thing regarding "portion of a command-line to be passed varbatim to the receiving command/script", that would be improved with this operator would be meaning (2.) above, and as mentioned, I think developing in that direction is bad.

If all you want to do is a shortcut do copy&paste existing command lines, why don't you add a cmdlet (e.g. Invoke-Shell, that calls bash -c on linux and cmd /c on windows?
Then you could just do

Invoke-Shell @'
whatever existing comandline containg '"quotes"' or whatnot
'@

If it must be one line, then add one-line-here-string syntax, but please don't implement a special operator, that can't be properly found and just complicates everything.

And please don't abandon #1995 for this!

@SP3269, we can cross that bridge when we get to it 😁.

But seriously:

If you think of "native" as "written specifically for the host platform [family]", the name would still fit, even if PowerShell should ever become the default system shell on a given platform (here's hoping, but I don't think that's realistic, at least not in the foreseeable future).

As for alternatives:

• Invoke-[System]DefaultShell would be the most accurate name (though it would unambiguously no longer apply should PowerShell ever become a system's default shell), but "native" seems to be the more commonly used term in the PowerShell world.

• Invoke-LegacyShell has some justification on Windows, but I don't think Unix users would take too kindly to that term.

I think I am beginning to unravel the utter confusion I'm experiencing trying to keep track of everyone's motives, ideas and issues here. It seems that we're (@mklement0 and I, at least) are looking at this as two different solutions to two different views of the same conceptual problem. What I was suggesting was using --% as a hint to the parser to change the way things are parsed in conjunction with the call & operator (not necessarily using --% standalone.) On the other hand, Michael seems to be looking at ins as a drop-in cmdlet to replace powershell's own internal native command broker and argument parser, and is not looking to change powershell's parser, instead, using strings/here-strings to capture the intended parameters (which I am saying could be used with --% also.)

--% is lacking a closing delimiter, which prevents it use in Powershell pipelines

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Regardless of whatever way is chosen, it seems that there will be a need to choose a native shell to dispatch the commandline. I suggest we use the environment variables COMSPEC on Windows (defaults to %systemroot%\system32\cmd.exe) and SHELL on Linux (defaults to /bin/bash) -- if SHELL doesn't exist, we should target /bin/sh (which in most cases is a symlink to bash anyway.)

--% is lacking a closing delimiter, which prevents it use in Powershell pipelines

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Basically you can't call things like cmd --% /c someapp | someOtherApp and have cmd handle the pipeline; PowerShell still interprets a pipeline (and a few other things like && and || which would otherwise be useful for Unix folks) as being a PowerShell token instead of passing it to the native command as part of the argument string.

and SHELL on Linux (defaults to /bin/bash)

No: the system shell on Unix-like platforms is invariably /bin/sh.
This is distinct from a given user's interactive shell (reflected in $env:SHELL - but, unfortunately, currently not if PowerShell is the user's shell, see #12150), which is (a) configurable and (b) often defaults to /bin/bash, but even that is not a given, as evidenced by macOS recently transitioning to /bin/zsh as the default interactive shell for new users.

I suggest we use the environment variables COMSPEC on Windows

Good point, that's the better way to refer to the system shell (command interpreter) on Windows - I've updated the sample command above accordingly.

One of the other features I really want in PS v7.x is the ability to have a native command exiting with an error code stop my script, like set -e as discussed in this RFC. If the solution to that is something like an Invoke-NativeCommand, would that get confusing with Invoke-NativeShell?

Just trying to play the movie forward a bit because as much as I want the call native feature, I really want my build & test scripts to error out when calling msbuild, cl, cmake, Conan, npm, etc and those native commands exit with a non-zero exit code without having to remember to check $LASTEXITCODE all the time. If the implementation is via just another preference variable e.g. $PSNativeCommandInErrorActionPreference then I suppose that would impact the invocation of the native shell - it being a native "command" and all?

We have Start-NativeExecution in our build module for that purpose.

I've seen that. How exactly would you combine Start-NativeExecution with Invoke-NativeShell if you'd like the later to effectively throw on a non-zero exit code? I really hope that something like PR #3523 makes it in along with this call native feature because I want the two to work together. I suppose if call native winds up being implemented as a cmdlet (Invoke-NativeShell vs --%) then -ErrorAction Stop could be implemented to have it throw (terminating error) on a non-zero exit code.

--% is lacking a closing delimiter, which prevents it use in Powershell pipelines

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Basically you can't call things like cmd --% /c someapp | someOtherApp and have cmd handle the pipeline; PowerShell still interprets a pipeline (and a few other things like && and || which would otherwise be useful for Unix folks) as being a PowerShell token instead of passing it to the native command as part of the argument string.

Yes, I understand the current behaviour @vexx32 . But we're not (at least I'm not) talking about the current behaviour of --% -- we're talking about enhancing it, no? Why do I get the feeling that we're all talking around each other here? :)

As I said way, way up above, we could enhance & to work with --% so this is possible. I also think we need to be clear about implicit native exec versus explicit invocation of a shell with arguments. This is a constant source of confusion - even for seasoned windows users - that somehow cmd.exe (a shell) is needed to "run" other executables; the same applies to linux/osx.

Yes, I understand the current behaviour @vexx32 . But we're not (at least I'm not) talking about the current behaviour of --% -- we're talking about enhancing it, no? Why do I get the feeling that we're all talking around each other here? :)

So the current pitch for adjusting --% is that if it's where the command name would typically be, everything to the right of it is parsed as is (up to a new line) and sent directly to bash/cmd. There's no room any PowerShell syntax like here-strings and what not because then it has all the same problems that --% and the native command processor currently have.

I've seen that. How exactly would you combine Start-NativeExecution with Invoke-NativeShell if you'd like the later to effectively throw on a non-zero exit code? I really hope that something like PR #3523 makes it in along with this call native feature because I want the two to work together. I suppose if call native winds up being implemented as a cmdlet (Invoke-NativeShell vs --%) then -ErrorAction Stop could be implemented to have it throw (terminating error) on a non-zero exit code.

Currently it will be Start-NativeExecution { Invoke-NativeShell @whatever }.

This point I don't get at all. It's no more lacking a closing delimiter than ins/invoke-nativecommand is. If you need to delimit or feed from the LHS, use here-strings, else it is considered as a single statement.

Invoke-NativeShell does not need any closing delimiter because it is a regular command, whereas the horror of --% is not.

and SHELL on Linux (defaults to /bin/bash)

No: the system shell on Unix-like platforms is invariably /bin/sh.

I think the thing should be equivalent to making an executable script file and running it, which delegates the task of choosing the right shell to the operating system and we should not be bothered. Including, if it starts with #!, so be it.

The idea behind Invoke-NativeShell is to provide a convenience wrapper around the native shell's CLI.

• On Unix, that is fully sufficient, and creating an auxiliary script file not only introduces extra overhead, but then reports a random value in $0 (the script file path), whereas invocation via the CLI it predictably contains /bin/sh and can even be set explicitly by following the code-argument with another argument (e.g., sh -c 'echo $0' foo)

  • (As an aside: creating a shell script with shebang line #!/bin/sh means targeting the system shell by full path just as explicitly as invoking /bin/sh directly).

• On Windows, the execution via a batch file would bring advantages (but not for delegating the task of locating the system shell, which $env:ComSpec predictably does), as it would avoid the problems with % escaping and for behavior mentioned above.

To address the latter as a courtesy (so that users don't have to write their own aux. batch files), for conceptual clarity, we could offer an
-AsBatchFile switch as an opt-in.

That said, if there is consensus that the majority of public cmd command lines used for cut-and-paste are written with batch-file semantics in mind (%%i rather than %i as a loop variable, ability to escape a verbatim % as %%), perhaps invariably using an aux. batch file on Windows (while sticking with the CLI on Unix) is the better solution - I do not feel strongly about this, but it would have to be clearly documented, especially given that it means exhibiting behavior that is different from other scripting languages.

@rkeithhill

To me, the integration-of-native-errors RFC you link to is worth addressing just as urgently as #1995:

Both represent necessary steps toward making external programs (native utilities) first-class citizens in PowerShell (as much as conceptually possible), which they should always have been.

Being a first-class citizen means: direct invocation (with & needed only for syntactic reasons, situationally), not via a cmdlet.
In other words: there should never be an Invoke-NativeCommand cmdlet or a Start-NativeExecution cmdlet.

(As an aside: Start-NativeExecution is misnamed, as many functions in the build module are; Start signals asynchronous operation, whereas most of these function are synchronous - see the discussion about Start-Sleep at MicrosoftDocs/PowerShell-Docs#4474).

Therefore, Invoke-NativeShell requires no special consideration: PowerShell, as it already does, will set $LASTEXITCODE based on the exit code reported by the native-shell executable process (cmd / sh) invoked.

Then the mechanism proposed in the RFC will act on it, as it would act on directly invoked executables (e.g., if $PSNativeCommandErrorAction = '/old?u=https%3A%2F%2Fgithub.com%2FPowerShell%2FPowerShell%2Fissues%2FStop&y=1999' is set, a script-terminating error would occur if $LASTEXITCODE is nonzero.)

(A quick aside - this discussion doesn't belong here: As for a per-call mechanism: something like
/bin/ls nosuch || $(throw 'ls failed) works, as does /bin/ls nosuch || $(exit $LASTEXITCODE) (the equivalent to a POSIX shell's /bin/ls nosuch || exit); #10967 discusses why we unfortunately cannot avoid $(...) (without major changes to the grammar); similarly, needing to refer to $LASTEXITCODE explicitly probably cannot be avoided.)

(As an aside: creating a shell script with shebang line #!/bin/sh means targeting the system shell by full path just as explicitly as invoking /bin/sh directly).

That is not what I wanted to say. I wanted to say that PowerShell does not need to know what the default shell is because operating systems have this feature built in. Linux knows how to execute an executable script in case it does not start with #! or it starts with something else, like #!/usr/bin/env python and Windows knows what to do to call a .CMD script, so we should not peek into%COMSPEC%’n’stuff either IMHO. Windows can also execute other scripts but it is based on the script file’s extension, so it obviously does not apply to this case.

Again, Invoke-NativeShell's mandate should be calling the native shell's CLI, not creating auxiliary script files (with side effects) behind the scenes (except for different reasons, as discussed above).

With respect to robustly determining the system shell, there is no problem to solve here: hard-coding /bin/sh on Unix is perfectly appropriate, as is consulting $env:ComSpec on Windows.

No, Unix platforms at the system-call level do not know how to execute an executable plain-text file without a shebang line; that you still can invoke such files is a convenience feature of POSIX-like shells: they try exec, then fall back to interpreting such files as written for them; that is, it is whatever POSIX-like shell that is being run that ends up executing the file - whereas PowerShell simply fails, because it doesn't implement this courtesy fallback (you'll get Program 'foo' failed to run: Exec format error).

Needless to say, it is therefore ill-advised to create such scripts.

No, Unix platforms at the system-call level do not know how to execute an executable plain-text file without a shebang line; that you still can invoke such files is a convenience feature of POSIX-like shells: they try exec, then fall back to interpreting such files as written for them; that is, it is whatever POSIX-like shell that is being run that ends up executing the file - whereas PowerShell simply fails, because it doesn't implement this courtesy fallback (you'll get Program 'foo' failed to run: Exec format error).

csh is not POSIX-like and it does not fail on exec a bare script. It invokes sh. So does perl.

I hope it is obvious that this makes my point: it's up to each shell to decide what to do, and different shells / scripting languages do different things; PowerShell currently just fails - if it had to make a choice, and you wanted that choice to be /bin/sh, we're back at square one: /bin/sh must be assumed.

I hope it is obvious that this makes my point: it's up to each shell to decide what to do, and different shells / scripting languages do different things; PowerShell currently just fails - if it had to make a choice, and you wanted that choice to be /bin/sh, we're back at square one: /bin/sh must be assumed.

exec in perl is a direct system call, perl does not modify it in any way. In particular, it does not choose the interpreter.

Of course, that assumes that users know the difference between single-quoted (verbatim) and double-quoted (interpolating) strings and about selective `-escaping of $, but I think it's an important (advanced) option to have.

Maybe it is just me, but I would trade a thousand "’s for one -f.

@yecril71pl

exec in perl is a direct system call, perl does not modify it in any way. In particular, it does not choose the interpreter.

That can not be true if it works with scripts without #! line. The exec syscall on linux does not work without #!, as one can easily test:

user@Programming-PC:/tmp/exec_test$ ls
test.c  testscript
user@Programming-PC:/tmp/exec_test$ cat test.c
#include <unistd.h>
#include <stdio.h>
 
int main(int argc, char **argv) {
  char *binaryPath = "./testscript";
 
  execl(binaryPath, binaryPath, NULL);

  perror("Error");
 
  return 0;
}
user@Programming-PC:/tmp/exec_test$ gcc test.c -o testexecutable
user@Programming-PC:/tmp/exec_test$ chmod +x testscript 
user@Programming-PC:/tmp/exec_test$ cat testscript 
echo test from script
user@Programming-PC:/tmp/exec_test$ #works from shell:
user@Programming-PC:/tmp/exec_test$ ./testscript 
test from script
user@Programming-PC:/tmp/exec_test$ #doesn't work via exec syscall:
user@Programming-PC:/tmp/exec_test$ ./testexecutable 
Error: Exec format error
user@Programming-PC:/tmp/exec_test$ vim testscript 
user@Programming-PC:/tmp/exec_test$ cat testscript 
#!/bin/sh
echo test from script
user@Programming-PC:/tmp/exec_test$ #exec syscall works with #! line:
user@Programming-PC:/tmp/exec_test$ ./testexecutable 
test from script
user@Programming-PC:/tmp/exec_test$ uname -a
Linux Programming-PC 4.4.0-176-generic #206-Ubuntu SMP Fri Feb 28 05:02:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
user@Programming-PC:/tmp/exec_test$ 

Edit: I just noticed, that if you use the library function execlp then according to the man page:

If the header of a file isn't recognized (the attempted execve(2) failed with the error ENOEXEC), these functions will execute the shell (/bin/sh) with the path of the file as its first argument. (If this attempt fails, no further searching is done.)

However this is not a feature of "linux" but of the gnu c library - and again according to the man page it uses explicitly /bin/sh and not some userdefinable shell. (Hardcoded in the source of posix/execvpe.c (new_argv[0] = (char *) _PATH_BSHELL;) and sysdeps/unix/sysv/linux/paths.h (or sysdeps/generic/paths.h - I don't know which header is used...) (#define _PATH_BSHELL "/bin/sh"))

However this is not a feature of "linux" but of the gnu c library - and again according to the man page it uses explicitly /bin/sh and not some userdefinable shell. (Hardcoded in the source of posix/execvpe.c (new_argv[0] = (char *) _PATH_BSHELL;) and sysdeps/unix/sysv/linux/paths.h (or sysdeps/generic/paths.h - I don't know which header is used...) (#define _PATH_BSHELL "/bin/sh"))

Since it comes from sysdeps/**/linux, it is a Linux thing. It may also be the generic thing but it is overridable — obviously not locally but per OS.

Yes, I understand the current behaviour @vexx32 . But we're not (at least I'm not) talking about the current behaviour of --% -- we're talking about enhancing it, no? Why do I get the feeling that we're all talking around each other here? :)

So the current pitch for adjusting --% is that if it's where the command name would typically be, everything to the right of it is parsed as is (up to a new line) and sent directly to bash/cmd. There's no room any PowerShell syntax like here-strings and what not because then it has all the same problems that --% and the native command processor currently have.

Oh lordy... Is everyone just trying to troll me? 😄 No, Patrick -- if you want to use powershell expressions, then use here-strings with --%. If you want multiline, use here-strings. If you want multline without variable substitution, you could use single-quote here-strings.

Anyway, I don't have any skin in this game and I think I've said enough. I still think there's something seriously wrong about using a cmdlet to defer to another shell from within a shell. It's just clumsy. It's especially clumsy when you're using a cmdlet to execute another shell to execute a native command that has no need for that secondary shell in the first place. The whole thing just smells bad.

edit: last word

The title of this item is "call native operator." I see a wasted opportunity when we already have a call operator & and we already have a token that was designed to deal with passing arguments to native commands, --%. Right now, using & with --% is NOT recognized and as such will NOT introduce a breaking change should we decide to enable this scenario and give it specific behavior to fix the problems being discussed here.

if you want to use powershell expressions, then use here-strings with --%. If you want multiline, use here-strings.

The point of the operator is that it doesn't parse literally any PowerShell syntax. There's no such thing as here-strings. There's no such thing as single quoted string constants. It's "send the text to the right as is" not "evaluate the expression to the right, convert to string and send it's value"

So for example, if you had this:

--% @'
echo my native command
'@

it would translate to:

cmd.exe /c "@'"
echo my native command
'@

Except you'd get a parser error because '@ would just be that start of a single quoted string constant containing @.

I think maybe you're picturing the implementation differently so that it's more or less the same thing as the pitch for ins, but the above is the current pitch being discussed for --%.

Also worth noting that this is how the existing functionality works as well. This example works mostly the same way:

cmd /c --% @'
echo my native command
'@

I repeat:

Right now, using & with --% is NOT recognized/undefined and as such will NOT introduce a breaking change should we decide to enable this scenario and give it specific behavior to fix the problems being discussed here.

Let me spell it out even clearer: Allow here-strings to follow --% when used with &

Hah that's a big oof - yeah I missed that, sorry @oising.

About that idea I think it would be a little confusing to have --% mean stop parsing in some contexts and "do a better job at native argument binding" in others (likewise with making it work like Invoke-NativeShell if that's what you mean instead).

Thanks for digging deeper re exec, @TSlivede. Let me try to summarize and therefore hopefully close this tangent; applies to both Linux and macOS:

There is no system (or library) function called exec on Unix, there is on a family of related functions with exec in their name (most of them as a prefix).

• The system functions (man section 2), execve at its heart, fail when you try to execute a shebang-less script.

• Among the library functions that build on the system functions (man section 3 - man 3 exec), only those that have p (for "path", I presume) have the pass-to-the-system-shell fallback (e.g., execlp).

  • The GNU library implementations of these functions hard-code /bin/sh, as @TSlivede has shown, whereas the BSD-based ones use just sh, and rely on sh to be in $env:PATH (though, curiously, the man page states /bin/sh).

As stated, for direct execution the major POSIX-like shells (bash, dash, ksh, and zsh) fall back to executing shebang-less scripts themselves, which implies that they do not use the fallback variants among the exec library functions; by contrast, perl's choice for its own exec function was to rely on the fallback; similarly, the exec builtins of the major POSIX-like shells rely on the fallback, except in bash.

Since on Unix the use of a script file behind the scenes is unnecessary, we can make the same assumptions about the system shell path as the fallback library functions, and I recommend /bin/sh over sh, for security and predictability:

Despite the POSIX spec not mandating the location of sh (that is, strictly speaking, the BSD library functions are the compliant ones):

• /bin/sh is safe to assume, because it is the de facto standard location, not least because writing portable Unix shell necessitates referring to sh by its full path, given that shebang lines support only full, literal paths (#!/bin/sh).

• Conversely, relying on locating sh via $env:PATH can be a security risk, given that $env:PATH could be manipulated to invoke a different sh.

The same risk applies to locating cmd.exe via $env:ComSpec, by the way, so perhaps the better way is to call the GetSystemDirectory WinAPI function and append \cmd.exe to its result (we need a fallback anyway, if $env:ComSpec happens not to be defined).

/bin/sh is safe to assume, because it is the de facto standard location, not least because writing portable Unix shell necessitates referring to sh by its full path, given that shebang lines support only full, literal paths (#!/bin/sh).

Normally you say #!/usr/bin/env sh (except in system scripts).

• Normally, you don't: googling "#!/bin/sh" yields about 3,900,000 matches, "#!/usr/bin/env sh" yields about 34,100 matches.

• Normally, you shouldn't: you want to predictably target the system shell, /bin/sh, not whatever sh utility happens to come first in $env:PATH.

The only reason to target an executable named sh is to portably target the lowest-common-denominator-assume-POSIX-features-only system shell, i.e. /bin/sh.

Normally, you don't: googling "#!/bin/sh" yields about 3,900,000 matches, "#!/usr/bin/env sh" yields about 34,100 matches.

There is no way to limit a Web search to user scripts, especially since many user scripts are not marked executable at all, and even if they are, they may rely on execlp; but even if most user scripts said that, we should not take customary for normal. The scripts to be run by PowerShell are user scripts; when users want a shell, they call sh, not /bin/sh, unless they are paranoid.