Kindness City Blog
23 Aug 2021

Exploring Bash Process Substitution

Exploring Bash Process Substitution

Recently, while debugging some old bash scripts, my pair and I came across a command that looked like yq eval "$KEY = \"$VAL\"" -- <(echo "$CURRENT_NOTES")

My pair hadn't previously had the chance to fully grok the <( things ) construct, so I attempted to explain. This post is a more considered version of the conversation that followed.

If you run a command that contains the term <( things ), then bash will do the following:

  • create an imaginary file for us
  • pretend that that file contains the output of the command things
  • pass that file as an argument to the command we're running (in our case, the yq command above)

This is called Process Substitution

You can actually see some of the nuts and bolts of process substitution in action if you use another cool bash feature: set -x.

A quick detour – set -x

If you're not familiar with set -x, this is a command which makes bash print out everything it does until you tell it to stop. So, you could have a script that looks like:

cd /tmp
mkdir my-new-directory
cd my-new-directory
ls -a

If you run this script, all you will see output is this:

.  ..

This is the output of your ls -a command, in the empty directory you just created.

However, if you add set -x at the beginning of the script, the output changes to:

+ cd /tmp
+ mkdir my-new-directory
+ cd my-new-directory
+ ls -a
.  ..

Each line beginning with a + is bash printing the command it's about to run, before it runs it. You can tell bash to stop doing this by using set +x (note the + instead of the -).

To read more about this (and all the other cool things you can do with the set command) you can do info "(bash) The Set Builtin" on any mac or linux box

Using set -x to watch Process Substitution happening

Ok, so we want to understand process substitution, which looks like <( things ) , using the tool set -x. Let's write a tiny script:

set -x
cat <(echo "Badgers!")

If you run that script, you'll see:

+ cat /dev/fd/63
++ echo 'Badgers!'

Let's break that down

The first thing that bash does is run cat /dev/fd/63

But what is this /dev/fd/63 ?!?

Well, I think it's like a temporary file (but see ⚠ below). So our cat command is going to read whatever is in that file, and print it to the terminal.

The next thing that bash does is echo 'Badgers!'

Notice that my double-quoted string has become a single-quoted one. That's because bash has already done all the variable substitution it's going to do, so this is exactly the command that's actually going to run. Also notice that this line begins with ++ not just +. This is because bash needs to finish running this line before it can complete the last one that begin with a single +.

The result of that command goes "into" /dev/fd/63 ready for cat to pick it up.

Finally, cat does its job, and the string Badgers! appears on the terminal. And the script exits.

⚠ I've skirted close to implying that /dev/fd/63 is a regular file, like a temporary file. It's not – or at least not always. Often it will be a thing called a "named pipe" or a "FIFO". But that's a topic for another post.

You can read more about process substitution by typing info "(bash) Process Substitution" on any mac or linux box. There's also lots more cool bash documentation in info bash. If using the info system feels weird, you can do info info to learn about it.

Tags: bash programming scripting unix linux texinfo

There's no comments mechanism in this blog (yet?), but I welcome emails and tweets. If you choose to email me, you'll have to remove the .com from the end of my email address by hand.

You can also follow this blog with RSS.

Other posts