The Magic of [email protected]

Hey 👋

Recently while lurking some geeky subreddits, I stumbled upon an interesting question about bash. I have a bit of a soft spot for shell programming, so I decided to share some insight in a blog-worthy response. If you too are interested in the magic variable that is [email protected], read on.

I promise you'll learn something 🎓

So, here's the question:

Can anyone explain this... I have a script which runs commands in docker containers with docker run and sh -c. I pass the arguments of the script to the command with [email protected] Now if I do this directly it only passes the first argument, but if I assign it to a different variable first and then pass that variable to docker run I get all the arguments. Let's say I have a file bin/test and the contents is this:

#!/usr/bin/env bash

# Outputs all arguments
echo "[email protected]"

# Only outputs fist argument
docker run --rm -ti node:10.9 sh -c "echo [email protected]"

# Outputs all arguments
[email protected]
docker run --rm -ti node:10.9 sh -c "echo $args"
And I run it with bin/test foo --bar the output would be:

foo --bar
foo
foo --bar

Why does the second command only pass the first argument?

A few words about words

As some background, whenever a command is parsed and run, a bunch of things happen, including two important steps:

• Parameter substitution: for example, replacing variable references like $var and ${bar} with the contents of variables var and bar respectively.
• Word splitting: splitting a command line into a list of words suitable for a call to execve(2). In English, this means starting with a command line like grep pattern file and splitting it into a command name, grep and a list of arguments, pattern and file.

Although other interesting things happen to command lines as the shell reads and executes them, those two steps happen in that order. This is why seemingly correct scripts like:

#!/bin/bash

file="abc def ghi.txt"
cat $file

...don't quite work out as expected.

[email protected] is special 🌟

[email protected] is a special variable when it comes to parameter substitution and word splitting.

If [email protected] is evaluated while it's not surrounded by double quotes, it simply expands to a space delimited list of the positional parameters (the arguments to the enclosing script or function).

However, if [email protected] is evaluated while it's directly surrounded by double quotes, it temporarily overrides word splitting such that each of the positional parameters are treated as separate words—one word per positional parameter. This even includes the positional parameters that have spaces in them.

This is what makes this script crash and burn:

$ cat foo.sh
#!/bin/bash

grep PASSWORD [email protected]
$ ./foo.sh 'a file with spaces.txt'
grep: a: No such file or directory
grep: file: No such file or directory
grep: with: No such file or directory
grep: spaces.txt: No such file or directory

...while this script works out:

$ cat foo.sh
#!/bin/bash

grep PASSWORD "[email protected]"
$ ./foo.sh 'a file with spaces.txt'
MY PASSWORD IS passw0rd

This property also makes the output of even trivial scripts deviate ever so slightly from expectations:

$ cat foo.sh
#!/bin/bash

echo You provided: [email protected]
$ ./foo.sh "a   parameter   with   tripled   spaces"
a parameter with tripled spaces 🤔

Comparing the argument to foo.sh and its output, notice that sequences of multiple spaces are all folded into one. This is because the shell first expands [email protected] into a space delimited list of the command line arguments, and then proceeds to split the resulting command line into words, starting with ./foo.sh and ending in spaces.

Maybe a little too special

There's even more to [email protected] 😱. The moment a double-quoted [email protected] is encountered, the shell temporarily goes into a word emitting mode

What this means is: if a double quoted [email protected] appears directly after or right before what would otherwise be a word (with no separating whitespace), the first word emitted by [email protected] will be joined to the word directly before it and the last word emitted by [email protected] will be joined to word that directly follows it.

Feels like an example is in order:

#!/bin/bash

echo a b c"[email protected]"d e 

Here, echo will be called with the arguments: a, b, c$1, $2, ... , ${n}d, e, and f. The same would happen if [email protected] appeared in double quotes with other text. Like, echo a b "[email protected]"d e f: The first and last word from [email protected] would be fused with c and d.

$* is just plain ordinary

If you ever want to avoid [email protected]'s special behaviour, use $*. It's just like [email protected], but it doesn't cause the script to go into funky modes. It just expands to the positional parameters, separated by single spaces. No muss...no fuss.

So why's my script broken?

So, back to the problem script:

echo "[email protected]"

This works as expected because it's being run, un-adulterated by the shell that's interpreting the script. The double quotes directly surrounding the [email protected] have the shell translate its command line parameters directly into words.

docker run --rm -ti node:10.9 sh -c "echo [email protected]"

This works strangely because the shell that's interpreting your shell script sees the [email protected] and flips into word emitting mode: Although word splitting will occur on the rest of this command line later, when the shell expands [email protected] it immediately starts turning it into words that will bypass the real word-splitting phase: echo [email protected] becomes the following words, echo (1st arg) and (2nd arg), (3rd arg), (...the rest of the script's arguments as individual words).

By the time the entire command line is split into words, it becomes: docker run --rm -ti node:10.9 sh -c echo (1st arg) (2nd arg) ... (3rd arg). Docker then turns around and runs sh -c 'echo (1st arg)' '(2nd arg)' '(3rd arg)' ..., which means "run the script echo (1st arg) with the command line arguments, (2nd arg), (3rd arg), ... ."

To get around this strange behavior, you could try replacing [email protected] with a variable that doesn't have special word-splitting-bypassing properties, like $*.

[email protected]; docker run --rm -ti node:10.9 sh -c "echo $args"

This works as expected because the value of [email protected] is stored in the args variable. Even if it were double quoted, word splitting doesn't occur when you assign a value to a regular variable, so the variable just receives the list of command line args, separated by a space character as a single string. Since $args is a plain ol' variable, it's expanded without any funky word splitting nonsense in the -c scriptlet and runs as expected.

Hope this post is as helpful to you as it was to the reddit user who posted it.

Love shell? Hate it? Not sure why it does the things it does?

Leave a comment and let's bash it out.

~ chris