Welcome back 👋🏽

As usual, if you find my ramblings interesting and want to read more, or think they're rubbish and want to tell me about it, remember to scroll down to leave a comment before, during, or after your read.

tl;dr

If you're here to see the smallest Bash script, look no further.

Here it is:

#!/bin/bash

Read on if you're wondering why it's not just an empty file like 10-years-ago-me.

The Smallest Kinda Sorta Bash Script in the Universe

I'll let you in on a little secret.

Back in the day I had this misconception that a Bash script was any text file with execute permission. In other words, I thought that shell_script in the following example would always be executed using bash as the interpreter:

# Create an empty file
$ : >shell_script

# Mark that file as executable
$ chmod u+x shell_script

# Run that file
$ ./shell_script
$

To explain why shell_script is only kinda sorta the smallest Bash script, we gotta go a little deeper and learn how programs are executed on Linux.

fork and execve

Most programs running on Linux run at least some code from the C programming language.

If they're not directly written in C, they're very likely either running code from the C runtime or are running in an interpreter that uses the C runtime. And BTW, for the purposes of this post you can think of the C runtime as a bunch of functions that have already been written for C programmers. This code sits in a shared library, ready to run.

The two functions we'll be focusing on are ones you wouldn't ever want to implement yourself as they interact with the kernel to do fun and exciting things that you don't want to get wrong. Whoever had to review my implementations of these functions in that university operating systems course can attest to that. I'm sorry for subjecting you to that, professor I Forgot What your Name Is.

Let's follow a C program named shelly as it runs on a Linux box and executes another program named script. Whenever shelly wants to run another program, it needs to run two functions from the C runtime: fork() and execve().

Process Splitting with fork

fork() tells the kernel to clone the currently running process into two processes. It's sort of like how cells split in a petri dish. In Linux, however, the resulting pair of processes forms a hierarchy with the original process becoming the parent of the new process. Once it's done, the first instruction of either process is a return from fork().

So after fork() completes, there will be two shelly processes.

Process Replacement with execve

execve() is a little more interesting.

It asks the kernel to replace the program running in the current process with some other program. execve() quite literally obliterates the memory image of the running program and replaces it with a new memory image for the desired program. When its done, execution continues at the first instruction of that new program. So in our example above, after execve() completes, there will be one process running the code for the shelly program and one process running the code for the shell_script program.

Since no remnants of shelly 's code will survive execve(), the call accepts data that the kernel will pass to the new program when it starts. This data is passed in two forms which you may have heard of:

• environment variables (a list of key-value pairs)
• arguments (an array of strings).

Arguments usually describe what a process should call itself and what precisely it should do. Environment variables usually describe the system that a process is running in or other software systems that a process may need to work with.

Since the environment of a process will very likely be the same as its children, that set of key-value pairs is usually just copied from the already running program to execve() with maybe a few additions.

Arguments are usually invocation-specific, and so are instead passed to execve() explicitly every time it's run.

What does execve actually do?

execve()'s only purpose in life is to ask the kernel to run programs from executable files.

It does this by asking the kernel to identify the type of program in an executable file and to run the appropriate kernel code to load it and set it executing. The kernel has handlers for programs in a handful of different binary formats, but the handler we're interested in is right here.

Scripts Must Begin with #!

Here's the beginning of the real magic of executing a script file:

The most important code (highlighted) checks the first two characters of the first line of the program file. If these characters are # followed by !, the program is considered a script. It then goes on to parse out the text following the #! into two words: an interpreter and an optional argument to that interpreter.

With these values in hand, the kernel repeats what it was asked to do with a few changes:

• Instead of loading and executing the script file, load and execute the interpreter.
• Use the optional argument from the #! line as the first argument to the interpreter.
• Use the script's name as the second argument.
• For the remaining arguments, simply copy them from the arguments execve() was originally given.

Running the Actual Smallest Bash Script in the Universe

So let's say shelly tried to run execve() on a program with the following contents:

$ cat shell_script
#!/bin/bash
$

Even if shelly forks and executes shell_script, the kernel will execute /bin/bash with a first argument of shell_script.

The Smallest Bash Script

And that's why the smallest Bash script is:

#!/bin/bash

...and why it's important to always begin shell scripts with #!.

The end.

Um...really?

You may notice that if you list a bunch of commands in a text file and run it in Bash, those commands still execute as if the file were a Bash script:

# Create a file containing two lines of shell
$ cat > shell_script <<COMMANDS
echo Hello
echo world
COMMANDS

# Mark the file as executable
$ chmod u+x shell_script

# Execute the file
$ ./shell_script
Hello
world
$

I haven't been lying to you. Attempts to ask the kernel to execute a text file that doesn't start in #! will always fail. If Bash is doing the executing, though, the shell will do you a solid.

Don't believe me? Take a peek at the source:

Your script may not always be executed by Bash (e.g. from a cron job, by the backend of a web service, or by a continuous integration system) so it's important to always start scripts with #! followed by the interpreter and an optional argument.

Further Reading

Hope you found that interesting and helpful. If it was, be sure to click Recommend below and may be even leave a comment.

For further reading on how Bash scripts are executed, see:

• How programs get run, Linux Weekly News
• binfmt_script.c, The Linux Kernel
• execve(2), The Linux Manual
• execute_command.c, GNU Bash

Hey 👋

Recently while lurking some geeky subreddits, I stumbled upon an interesting question about bash. I have a bit of a soft spot for shell programming, so I decided to share some insight in a blog-worthy response. If you too are interested in the magic variable that is $@, read on.

I promise you'll learn something 🎓

So, here's the question:

Can anyone explain this... I have a script which runs commands in docker containers with docker run and sh -c. I pass the arguments of the script to the command with $@. Now if I do this directly it only passes the first argument, but if I assign it to a different variable first and then pass that variable to docker run I get all the arguments. Let's say I have a file bin/test and the contents is this:

#!/usr/bin/env bash

# Outputs all arguments
echo "$@"

# Only outputs fist argument
docker run --rm -ti node:10.9 sh -c "echo $@"

# Outputs all arguments
args=$@
docker run --rm -ti node:10.9 sh -c "echo $args"
And I run it with bin/test foo --bar the output would be:

foo --bar
foo
foo --bar

Why does the second command only pass the first argument?

A few words about words

As some background, whenever a command is parsed and run, a bunch of things happen, including two important steps:

• Parameter substitution: for example, replacing variable references like $var and ${bar} with the contents of variables var and bar respectively.
• Word splitting: splitting a command line into a list of words suitable for a call to execve(2). In English, this means starting with a command line like grep pattern file and splitting it into a command name, grep and a list of arguments, pattern and file.

Although other interesting things happen to command lines as the shell reads and executes them, those two steps happen in that order. This is why seemingly correct scripts like:

#!/bin/bash

file="abc def ghi.txt"
cat $file

...don't quite work out as expected.

$@ is special 🌟

$@ is a special variable when it comes to parameter substitution and word splitting.

If $@ is evaluated while it's not surrounded by double quotes, it simply expands to a space delimited list of the positional parameters (the arguments to the enclosing script or function).

However, if $@ is evaluated while it's directly surrounded by double quotes, it temporarily overrides word splitting such that each of the positional parameters are treated as separate words—one word per positional parameter. This even includes the positional parameters that have spaces in them.

This is what makes this script crash and burn:

$ cat foo.sh
#!/bin/bash

grep PASSWORD $@
$ ./foo.sh 'a file with spaces.txt'
grep: a: No such file or directory
grep: file: No such file or directory
grep: with: No such file or directory
grep: spaces.txt: No such file or directory

...while this script works out:

$ cat foo.sh
#!/bin/bash

grep PASSWORD "$@"
$ ./foo.sh 'a file with spaces.txt'
MY PASSWORD IS passw0rd

This property also makes the output of even trivial scripts deviate ever so slightly from expectations:

$ cat foo.sh
#!/bin/bash

echo You provided: $@
$ ./foo.sh "a   parameter   with   tripled   spaces"
a parameter with tripled spaces 🤔

Comparing the argument to foo.sh and its output, notice that sequences of multiple spaces are all folded into one. This is because the shell first expands $@ into a space delimited list of the command line arguments, and then proceeds to split the resulting command line into words, starting with ./foo.sh and ending in spaces.

Maybe a little too special

There's even more to $@ 😱. The moment a double-quoted $@ is encountered, the shell temporarily goes into a word emitting mode

What this means is: if a double quoted $@ appears directly after or right before what would otherwise be a word (with no separating whitespace), the first word emitted by $@ will be joined to the word directly before it and the last word emitted by $@ will be joined to word that directly follows it.

Feels like an example is in order:

#!/bin/bash

echo a b c"$@"d e 

Here, echo will be called with the arguments: a, b, c$1, $2, ... , ${n}d, e, and f. The same would happen if $@ appeared in double quotes with other text. Like, echo a b "c$@"d e f: The first and last word from $@ would be fused with c and d.

$* is just plain ordinary

If you ever want to avoid $@'s special behaviour, use $*. It's just like $@, but it doesn't cause the script to go into funky modes. It just expands to the positional parameters, separated by single spaces. No muss...no fuss.

So why's my script broken?

So, back to the problem script:

echo "$@"

This works as expected because it's being run, un-adulterated by the shell that's interpreting the script. The double quotes directly surrounding the $@ have the shell translate its command line parameters directly into words.

docker run --rm -ti node:10.9 sh -c "echo $@"

This works strangely because the shell that's interpreting your shell script sees the $@ and flips into word emitting mode: Although word splitting will occur on the rest of this command line later, when the shell expands $@ it immediately starts turning it into words that will bypass the real word-splitting phase: echo $@ becomes the following words, echo (1st arg) and (2nd arg), (3rd arg), (...the rest of the script's arguments as individual words).

By the time the entire command line is split into words, it becomes: docker run --rm -ti node:10.9 sh -c echo (1st arg) (2nd arg) ... (3rd arg). Docker then turns around and runs sh -c 'echo (1st arg)' '(2nd arg)' '(3rd arg)' ..., which means "run the script echo (1st arg) with the command line arguments, (2nd arg), (3rd arg), ... ."

To get around this strange behavior, you could try replacing $@ with a variable that doesn't have special word-splitting-bypassing properties, like $*.

args=$@; docker run --rm -ti node:10.9 sh -c "echo $args"

This works as expected because the value of $@ is stored in the args variable. Even if it were double quoted, word splitting doesn't occur when you assign a value to a regular variable, so the variable just receives the list of command line args, separated by a space character as a single string. Since $args is a plain ol' variable, it's expanded without any funky word splitting nonsense in the -c scriptlet and runs as expected.

Hope this post is as helpful to you as it was to the reddit user who posted it.

Love shell? Hate it? Not sure why it does the things it does?

Leave a comment and let's bash it out.

~ chris

Recently, I started ramping up on a little something called Docker. In its simplest form, Docker lets you treat a runtime environment like a codebase under source control.

Want to record how an environment looks?

docker commit <container> <tag>

Made a few unfortunate changes, and want to roll it back?

docker rm <container>
docker run <tag> <command>

It's more than a little intruiging and I'll definitely get back to how to make the most of it, but first an admission. Early on, I was routinely confused by all of the terms. "Images, containers and processes, oh my," I recall saying to myself on more than one occasion while staring aimlessly at an xterm. To help me keep all this straight in my head, I whipped up a simple analogy.

In docker...

An image is like a blueprint for a jail cell.

A docker image contains a starting point for a runtime environment. This includes all of the files on disk at the time that environment starts. In there as well are tips on what network ports the process in the environment may need to listen on, the current working directory of this process, as well as a default command to run when the environment is started.

A container is like a completed jail cell...with internet access.

A dockerized process is like a prisoner.

A docker container is the runtime environment mentioned above. Processes bound to a container may do whatever they want—create or delete files on disk, produce diagnostics, create users, bind network ports, or even create new processes. The useful thing is that the effects of these actions will extend no further than the container walls. This means that processes in two different containers can simultaneously, e.g. write different content to /etc/foo.conf or listen on port 8080 without conflict.

The docker daemon is part prison guard; part construction crew.

The docker daemon does a lot of work. But the important bits involve creating containers from images, running dockerized processes, and recording their output. The daemon also manages network traffic destined to containers (connections from other containers as well as ones routed in through the docker host), and helps manage requests to turn containers into new images.

A linux machine with lxc enabled and docker installed is like a prison.

Docker is effectively a thin veneer over a still maturing set of kernel features collectively called Linux Containers. Containers are a not so easy way to setup isolated zones of execution on a Linux machine. Docker makes these features extremely easy to use.

And that's it—Docker in a nutshell. This will definitely save the day the next time I'm staring at my xterm. Hope it saves yours, too :-)

The other day I was reading up on a new Node.js backend framework and was intrigued as it was based on a new Javascript language feature: Generators. Knowing nothing about this bleeding edge Javascript tech, I set out on a bit of an adventure. It turns out that generators are part of a yellow brick road that the TC39 hopes will lead developers away from zigzagging pyramid-like programs to code that more closely resembles the plain old synchronous variety almost all of us are used to.

But where does this journey begin? Consider the following slow function...and just a note that all of these code examples should run in NodeJS v5.4.1 with no special flags or modules.

/*
 * A slow routine.
 */
function slowlyIncrement(i, cb) {
        setTimeout(function() {
                cb(null, i + 1);
        }, 1000);
}

This function follows a pretty well defined pattern where the last argument is a piece of code pushed in by the caller. Internally, the function does some work and calls the provided callback, cb, sending it an optional error and a result.

Pretty simple.

But what if you need to run several slow routines in sequence?

slowlyIncrement(0, function(err, i) {
        // TODO: handle errors
        slowlyIncrement(i, function (err, j) {
                // TODO: handle errors
                slowlyIncrement(j, function (err, k) {
                        // TODO: handle errors
                        slowlyIncrement(k, function (err, x) {
                                // TODO: handle errors
                                slowlyIncrement(x, function (err, y) {
                                        // TODO: handle errors
                                        slowlyIncrement(y, function (err, z) {
                                                // TODO: handle errors
                                                console.log(z); // prints 6
                                        });
                                });
                        });
                });
        });
});

The main issue—the one that pops right out like one of those 3D stereograms from a few decades ago—is what I like to call the bad kind of horizontal scaling. Another more subtle issue is how errors are handled. Because potential errors are passed as arguments to each callback, there's no way to defer error handling: Every callback must have error handling code. Both of these issues may be jarring to someone more used to synchronous code and exceptions.

Arrow functions

One neat feature introduced in ES6/ES2015 is a shorthand for anonymous functions. It's pretty simple.

In compatible runtimes, this:

function (i) {
        console.log(i);
}

...is equivalent to this:

(i) => {
        console.log(i);
}

So, our crazy zigzagging code can be reduced to this:

slowlyIncrement(0, (err, i) => {
        slowlyIncrement(i, (err, j) => {
                slowlyIncrement(j, (err, k) => {
                        slowlyIncrement(k, (err, x) => {
                                slowlyIncrement(x, (err, y) => {
                                        slowlyIncrement(y, (err, z) => {
                                                console.log(z); // prints 6
                                        });
                                });
                        });
                });
        });
});

Promises

Although arrow functions are a great way to reduce a little repetition, they don't really address the structural issues with our code. Enter promises.

As I mentioned above, when a slow function is provided a callback, the programmer is pushing an entity (a piece of code) into that function. At least to me, this concept was pretty confusing when I started working with Javascript. I was used to functions—slow or fast—return-ing values which got pulled out via a function call.

Promises are an attempt at turning this unintuitive convention back on its head. Don't spend too much time studying it too closely, but consider the following new and improved slow function:

function slowlyIncrement (i) {
        return new Promise((resolve, reject) => {
                setTimeout(() {
                        resolve(i + 1);
                }, 1000);
        });
}

One immediate improvement here is that we're no longer expecting callers to push code to our function. Instead, callers pass in arguments and pull out an object that represents the eventual result.

For example:

var result = slowlyIncrement(0);

Now, the trick is coaxing the actual value out of this object. This is accomplished via it's only documented method, then:

var result = slowlyIncrement(0);
result.then((i) => {
        console.log(i); // prints 1
});

The idea is that the callback provided to then will be called when the value underlying the promise is known or resolved.

A neat bit of trivia is that because the only standard method of a promise is then, a promise is usually described as a thenable object. But more on that later.

More then meets the eye

then has a few more tricks up it's sleeve; actually, two pretty simple properties:

1. then always returns a promise. Let's call this promise thenResult.
2. thenResult resolves to whatever the callback provided to then returns. Specifically:
3. If the callback provided to then returns a run of the mill value, thenResult will immediately be resolved with that value.
4. If the callback provided to then returns a thenable, thenResult will be resolved whenever (and with whatever) that thenable is resolved with.

Here's an example of a run of the mill value being propagated:

var result = slowlyIncrement(0);
result
.then((i) => {
        return i + "'s the result";
})
.then((i) => {
        console.log(i);  // prints "1's the result"
});

Here's an example of a promise being chained:

var result = slowlyIncrement(0);
result
.then((i) => {
        return slowlyIncrement(i); // a promise is returned here
}) // this then() returns a promise which resolves to whatever value the promise above does
.then((i) => {
        return i + "'s the result";
})
.then((i) => {
        console.log(i); // prints "2's the result"
});

Scaling vertically

Now we have all the knowledge required to tackle that nesting in our original code snippet. You remember...the code that performed five increments in sequence and spanned three screen widths.

Here's the Cole's Notes version:

slowlyIncrement(0)
.then((i) => {
  slowlyIncrement(i);
})
.then((j) => {
  slowlyIncrement(j);
})
.then((k) => {
  slowlyIncrement(k);
})
.then((x) => {
  slowlyIncrement(x);
})
.then((y) => {
  console.log(y);
})

...which can be reduced even further as:

// slowlyIncrement just so happens to accept one param
// ditto for console.log.
slowlyIncrement(0)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(console.log)

Oh yeah. Error handling.

Although I didn't get into it earlier, there was something I didn't mention about the then method of a promise. It actually takes two callbacks.

The first one is called whenever the value underlying the promise is known (or resolves). The second one is called whenever an error occurs while arriving at a value. When this happens, it's up to the promise returning function to raise an error by rejecting the promise.

Here's an example of a more complete slowlyIncrement which rejects where the provided value is not a number:

function slowlyIncrement(i) {
        return new Promise(function (resolve, reject) {
                if (Number.isNaN(Number(i))) {
                        reject(new Error(i + ": Not a number"));
                } else {
                        setTimeout(() => {
                                resolve(i+1);
                        }, 1000);
                }
        });
}

Here's an example usage with proper error handling:

// Here, no error occurs.
slowlyIncrement(0)
.then(
  console.log, // prints 1
  (e) => {
    console.log(e.stack);
  }
);

Here's an example usage where an error actually occurs:

// Here, an error occurs because "abcde" isn't a number
slowlyIncrement("abcde")
.then(
  console.log,
  (e) => {
    console.log(e.stack);  // displays a stack trace
  }
);

The Weakest Link

Just like resolved values propagate down the promise chain, so too do errors. Here, errors will still be caught, even if we decide not to handle them for each and every call to then:

slowlyIncrement("abcde")
.then(slowlyIncrement)
.then(
  console.log,
  (e) => {
    console.log(e.stack);
  }
);

This is looking much more familiar, eh? Let's close the loop. With promises, that grimy, nested, error prone mess all the way at the top becomes this:

function slowlyIncrement(i) {
        return new Promise(function (resolve, reject) {
                if (Number.isNaN(Number(i))) {
                        reject(new Error(i + ": Not a number"));
                } else {
                        setTimeout(() => {
                                resolve(i+1);
                        }, 1000);
                }
        });
}

slowlyIncrement(0)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(slowlyIncrement)
.then(
  console.log,
  (e) => {
    console.log(e.stack);
  }
)

Now I know what you're thinking:

The Good

• We can cut our use of the keyword function down considerably with arrow functions.
• With promises, we no longer need to push code to slow functions. We simply call those functions and fetch a deferred result in the form of a promise.
• Because promises may be chained, we no longer have to contend with very wide source files or hard to match parentheses.
• Promises allow error handling code to be consolidated, instead of needing to always be spread out throughout an asynchronous call chain.

The Bad

• We're still pushing code to fetch a value, but to the then method of a promise. This still feels a bit unnatural.
• A promise is a type that controls the execution path of a program. Shouldn't this be part of the language's syntax? This way, the compiler will be privy to the desired control flow and may be able to optimize or at least confirm syntax.
• The implementation of the slow running function is still kind of unnatural. This is probably related to the point above about the lack of language support for any of this.

In my next post, I'll get into the part of the road closer to the Emerald City—that place where async code can look as clean and familiar as the synchronous code of old.

This leg of the journey is definitely under construction, with some pretty crazy twists and turns. The exciting part, however, is how much of this plumbing is being brought behind the curtain of the Javascript language where it belongs.

Until next time,

— chris