The Unix Shell

At a high level, computers do four things:

• run programs;
• store data;
• communicate with each other; and
• interact with us.

They can do the last of these in many different ways, including direct brain-computer links and speech interfaces. Since these are still in their infancy, most of us use windows, icons, mice, and pointers. These technologies didn't become widespread until the 1980s, but their roots go back to Doug Engelbart's work in the 1960s, which you can see in what has been called "The Mother of All Demos".

Going back even further, the only way to interact with early computers was to rewire them. But in between, from the 1950s to the 1980s, most people used a technology based on the old-fashioned typewriter. That technology is still widely used today, and is the subject of this chapter.

When I say "typewriter", I actually mean a line printer connected to a keyboard (Figure 1). These devices only allowed input and output of the letters, numbers, and punctuation found on a standard keyboard, so programming languages and interfaces had to be designed around that constraint—though if you had enough time on your hands, you could draw simple pictures using just those characters (Figure 2).

                    ,-.             __
                  ,'   `---.___.---'  `.
                ,'   ,-                 `-._
              ,'    /                       \
           ,\/     /                        \\
       )`._)>)     |                         \\
       `>,'    _   \                  /       ||
         )      \   |   |            |        |\\
.   ,   /        \  |    `.          |        | ))
\`. \`-'          )-|      `.        |        /((
 \ `-`   .`     _/  \ _     )`-.___.--\      /  `'
  `._         ,'     `j`.__/           `.    \
    / ,    ,'         \   /`             \   /
    \__   /           _) (               _) (
      `--'           /____\             /____\

Today, this kind of interface is called a command-line user interface, or CLUI, to distinguish it from the graphical user interface, or GUI, that most of us now use. The heart of a CLUI is a read-evaluate-print loop, or REPL: when the user types a command, the computer reads it, executes it, and prints its output. (In the case of old teletype terminals, it literally printed the output onto paper, a line at a time.) The user then types another command, and so on until the user logs off.

This description makes it sound as though the user sends commands directly to the computer, and the computer sends output directly to the user. In fact, there is usually a program in between called a command shell (Figure 3). What the user types goes into the shell; it figures out what commands to run and orders the computer to execute them.

A shell is a program like any other. What's special about it is that its job is to run other programs, rather than to do calculations itself. The most popular Unix shell is Bash, the Bourne Again SHell (so-called because it's derived from a shell written by Stephen Bourne—this is what passes for wit among programmers). Bash is the default shell on most modern implementations of Unix, and in most packages that provide Unix-like tools for Windows, such as Cygwin.

Using Bash, or any other shell, sometimes feels more like programming that like using a mouse. Commands are terse (often only a couple of characters long), their names are frequently cryptic, and their output is lines of text rather than something visual like a graph. On the other hand, the shell allows us to combine existing tools in powerful ways with only a few keystrokes, and to set up pipelines to handle large volumes of data automatically. In addition, the command line is often the easiest way to interact with remote machines. As clusters and cloud computing become more popular for scientific data crunching, being able to drive them is becoming a necessary skill.

The part of the operating system responsible for managing files and directories is called the file system. It organizes our data into files, which hold information, and directories (also called "folders"), which hold files or other directories.

Several commands are frequently used to create, inspect, rename, and delete files and directories. To start exploring them, let's log in to the computer by typing our user ID and password. Most systems will print stars to obscure the password, or nothing at all, in case some evildoer is shoulder surfing behind us.

login: vlad
password: ********
$

Once we have logged in we'll see a prompt, which is how the shell tells us that it's waiting for input. This is usually just a dollar sign, but which may show extra information such as our user ID or the current time. Type the command whoami, then press the Enter key (sometimes marked Return) to send the command to the shell. The command's output is the ID of the current user, i.e., it shows us who the shell thinks we are:

$ whoami
vlad
$

More specifically, when we type whoami the shell:

1. finds a program called whoami,
2. runs it,
3. waits for it to display its output, and
4. displays a new prompt to tell us that it's ready for more commands.

Getting Around

Next, let's find out where we are by running a command called pwd (which stands for "print working directory"). At any moment, our current working directory is our current default directory, i.e., the directory that the computer assumes we want to run commands in unless we explicitly specify something else. Here, the computer's response is /users/vlad, which is Vlad's home directory:

$ pwd
/users/vlad
$

To understand what a "home directory" is, let's have a look at how the file system as a whole is organized (Figure 4). At the top is the root directory that holds everything else the computer is storing. We refer to it using a slash character / on its own; this is the leading slash in /users/vlad.

Inside that directory (or underneath it, if you're drawing a tree) are several other directories: bin (which is where some built-in programs are stored), data (for miscellaneous data files), users (where users' personal directories are located), tmp (for temporary files that don't need to be stored long-term), and so on. We know that our current working directory /users/vlad is stored inside /users because /users is the first part of its name. Similarly, we know that /users is stored inside the root directory / because its name begins with /.

Underneath /users, we find one directory for each user with an account on this machine (Figure 5). The Mummy's files are stored in /users/imhotep, Wolfman's in /users/larry, and ours in /users/vlad, which is why vlad is the last part of the directory's name. Notice, by the way, that there are two meanings for the / character. When it appears at the front of a file or directory name, it refers to the root directory. When it appears inside a name, it's just a separator.

Let's see what's in Vlad's home directory by running ls, which stands for "listing":

$ ls
bin          data      mail       music
notes.txt    papers    pizza.cfg  solar
solar.pdf    swc
$

ls prints the names of the files and directories in the current directory in alphabetical order, arranged neatly into columns. To make its output more comprehensible, we can give it the flag -F by typing ls -F. This tells ls to add a trailing / to the names of directories:

$ ls -F
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/
$

As you can see, /users/vlad contains seven sub-directories (Figure 6). The names that don't have trailing slashes—notes.txt, pizza.cfg, and solar.pdf—are plain old files.

Now let's take a look at what's in Vlad's data directory by running the command ls -F data. The second parameter—the one without a leading dash—tells ls that we want a listing of something other than our current working directory:

$ ls -F data
amino_acids.txt   elements/     morse.txt
pdb/              planets.txt   sunspot.txt
$

The output shows us that there are four text files and two sub-sub-directories. Organizing things hierarchically in this way is a good way to keep track of our work: it's possible to put hundreds of files in our home directory, just as it's possible to pile hundreds of printed papers on our desk, but in the long run it's a self-defeating strategy.

Notice, by the way that we spelled the directory name data. It doesn't have a trailing slash: that's added to directory names by ls when we use the -F flag to help us tell things apart. And it doesn't begin with a slash because it's a relative path, i.e., it tells ls how to find something from where we are, rather than from the root of the file system (Figure 7).

If we run ls -F /data (with a leading slash) we get a different answer, because /data is an absolute path:

$ ls -F /data
access.log    backup/    hardware.cfg
network.cfg
$

The leading / tells the computer to follow the path from the root of the filesystem, so it always refers to exactly one directory, no matter where we are when we run the command.

What if we want to change our current working directory? Before we do this, pwd shows us that we're in /users/vlad, and ls without any parameters shows us that directory's contents:

$ pwd
/users/vlad
$ ls
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/
$

We can use cd followed by a directory name to change our working directory. cd stands for "change directory", which is a bit misleading: the command doesn't change the directory, it changes the shell's idea of what directory we are in.

$ cd data
$

cd doesn't print anything, but if we run pwd after it, we can see that we are now in /users/vlad/data. If we run ls without parameters now, it lists the contents of /users/vlad/data, because that's where we now are:

$ pwd
/users/vlad/data
$ ls
amino_acids.txt   elements/     morse.txt
pdb/              planets.txt   sunspot.txt
$

OK, we can go down the directory tree: how do we go up? We could use an absolute path:

$ cd /users/vlad
$

but it's almost always simpler to use cd .. to go up one level:

$ pwd
/users/vlad/data
$ cd ..

.. is a special directory name meaning "the directory containing this one", or, more succinctly, the parent of the current directory.

Sure enough, if we run pwd after running cd .., we're back in /users/vlad:

$ pwd
/users/vlad
$

The special directory .. doesn't usually show up when we run ls. If we want to display it, we can give ls the -a flag:

$ ls -F -a
./           ../       bin/       data/
mail/        music/    notes.txt  papers/
pizza.cfg    solar/    solar.pdf  swc/

-a stands for "show all"; it forces ls to show us file and directory names that begin with ., such as .. (which, if we're in /users/vlad, refers to the /users directory). As you can see, it also displays another special directory that's just called ., which means "the current working directory". It may seem redundant to have a name for it, but we'll see some uses for it soon.

Everything we have seen so far works on Unix and its descendents, such as Linux and Mac OS X. Things are a bit different on Windows. A typical directory path on a Windows 7 machine might be C:\Users\vlad. The first part, C:, is a drive letter that identifies which disk we're talking about. This notation dates back to the days of floppy drives; today, different "drives" are usually different filesystems on the network.

Instead of a forward slash, Windows uses a backslash to separate the names in a path. This causes headaches because Unix uses backslash for input of special characters. For example, if we want to put a space in a filename on Unix, we would write the filename as my\ results.txt. Please don't ever do this, though: if you put spaces, question marks, and other special characters in filenames on Unix, you can confuse the shell for reasons that we'll see shortly.

Finally, Windows filenames and directory names are case insensitive: upper and lower case letters mean the same thing. This means that the path name C:\Users\vlad could be spelled c:\users\VLAD, C:\Users\Vlad, and so on. Some people argue that this is more natural: after all, "VLAD" in all upper case and "Vlad" spelled normally refer to the same person. However, it causes headaches for programmers, and can be difficult for people to understand if their first language doesn't use a cased alphabet.

$ ls north-pacific-gyre/2012-07-03/

$ ls no

$ ls north-pacific-gyre/

We now know how to explore files and directories, but how do we create them in the first place? Let's go back to Vlad's home directory, /users/vlad, and use ls -F to see what it contains:

$ pwd
/users/vlad
$ ls -F
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/
$

Let's create a new directory called thesis using the command mkdir thesis (which has no output):

$ mkdir thesis
$

As you might (or might not) guess from its name, mkdir means "make directory". Since thesis is a relative path (i.e., doesn't have a leading slash), the new directory is made below the current one:

$ ls -F
bin/         data/     mail/      music/
notes.txt    papers/   pizza.cfg  solar/
solar.pdf    swc/      thesis/
$

However, there's nothing in it yet:

$ ls -F thesis
$

Let's change our working directory to thesis using cd, then run an editor called Nano to create a file called nano draft.txt:

$ cd thesis
$ nano draft.txt

Nano is a very simple text editor that only a programmer could really love. Figure 9 shows what it looks like when it runs: the cursor is the blinking square in the upper left, and the two lines across the bottom show us the available commands. (By convention, Unix documentation uses the caret ^ followed by a letter to mean "control plus that letter", so ^O means Control+O.)

Let's type in a short quotation (Figure 10) then use Control-O to write our data to disk. Once our quotation is saved, we can use Control-X to quit the editor and return to the shell.

nano doesn't leave any output on the screen after it exits. But ls now shows that we have created a file called draft.txt:

$ ls
draft.txt
$

If we just want to quickly create a file with no content, we can use the command touch.

$ touch draft_empty.txt
$ ls
draft.txt   draft_empty.txt
$

The touch command has allowed us to create a file without leaving the command line. As with text editors like nano, touch will create the file in your current working directory.

If we touch a file that already exists, the time of last modification for the file is updated to the present time, but the file is otherwise unaltered. We can demonstrate this by running ls with the -l flag to give a long listing of files in the current directory which includes information about the date and time of last modification.

$ ls -l
-rw-r--r-- 1 username username 512 Oct  9 15:46 draft.txt
-rw-r--r-- 1 username username   0 Oct  9 15:47 draft_empty.txt
$ touch draft.txt
$ ls -l
-rw-r--r-- 1 username username 512 Oct  9 15:48 draft.txt
-rw-r--r-- 1 username username   0 Oct  9 15:47 draft_empty.txt

Note how the modification time of draft.txt changes after we touch it.

We can run ls with the -s flag (for "size") to show us how large draft.txt is:

$ ls -s
   1  draft.txt        0  draft_empty.txt
$

Unfortunately, Unix reports sizes in disk blocks by default, which might be the least helpful default imaginable. If we add the -h flag, ls switches to more human-friendly units:

$ ls -s -h
 512  draft.txt        0  draft_empty.txt
$

Here, 512 is the number of bytes in the draft.txt file. This is more than we actually typed in because the smallest unit of storage on the disk is typically a block of 512 bytes.

Let's start tidying up by running rm draft.txt:

$ rm draft.txt
$

This command removes files ("rm" is short for "remove"). If we run ls, only the draft_empty.txt file remains:

$ ls
draft_empty.txt
$

Now we'll finish cleaning up by running rm draft_empty.txt:

$ rm draft_empty.txt
$

If we run ls again, its output is empty, which tells us that both of our files are gone:

$ ls
$

Let's re-create draft.txt as an empty file, and then move up one directory to /users/vlad using cd ..:

$ pwd
/users/vlad/thesis
$ touch draft.txt
$ ls
draft.txt
$ cd ..
$

If we try to remove the entire thesis directory using rm thesis, we get an error message:

$ rm thesis
rm: cannot remove `thesis': Is a directory
$

This happens because rm only works on files, not directories. The right command is rmdir, which is short for "remove directory": It doesn't work yet either, though, because the directory we're trying to remove isn't empty:

$ rmdir thesis
rmdir: failed to remove `thesis': Directory not empty
$

This little safety feature can save you a lot of grief, particularly if you are a bad typist. If we really want to get rid of thesis we should first delete the file draft.txt:

$ rm thesis/draft.txt
$

The directory is now empty, so rmdir can delete it:

$ rmdir thesis
$

$ rm -r thesis
$

Let's create that directory and file one more time. (Note that this time we're running touch with the path thesis/draft.txt, rather than going into the thesis directory and running touch on draft.txt there.)

$ pwd
/users/vlad/thesis
$ mkdir thesis
$ touch thesis/draft.txt
$ ls thesis
draft.txt
$

draft.txt isn't a particularly informative name, so let's change the file's name using mv, which is short for "move":

$ mv thesis/draft.txt thesis/quotes.txt
$

The first parameter tells mv what we're "moving", while the second is where it's to go. In this case, we're moving thesis/draft.txt to thesis/quotes.txt, which has the same effect as renaming the file. Sure enough, ls shows us that thesis now contains one file called quotes.txt:

$ ls thesis
quotes.txt
$

Just for the sake of inconsistency, mv also works on directories—there is no separate mvdir command.

Let's move quotes.txt into the current working directory. We use mv once again, but this time we'll just use the name of a directory as the second parameter to tell mv that we want to keep the filename, but put the file somewhere new. (This is why the command is called "move".) In this case, the directory name we use is the special directory name . that we mentioned earlier):

$ mv thesis/quotes.txt .
$

The effect is to move the file from the directory it was in to the current directory. ls now shows us that thesis is empty, and that quotes.txt is in our current directory:

$ ls thesis
$ ls quotes.txt
quotes.txt
$

(Notice that ls with a filename or directory name as an parameter only lists that file or directory.)

The cp command works very much like mv, except it copies a file instead of moving it. We can check that it did the right thing using ls with two paths as parameters—like most Unix commands, ls can be given thousands of paths at once:

$ cp quotes.txt thesis/quotations.txt
$ ls quotes.txt thesis/quotations.txt
quotes.txt   thesis/quotations.txt
$

To prove that we made a copy, let's delete the quotes.txt file in the current directory, and then run that same ls again. This time, it tells us that it can't find quotes.txt in the current directory, but it does find the copy in thesis that we didn't delete:

$ ls quotes.txt thesis/quotations.txt
ls: cannot access quotes.txt: No such file or directory
thesis/quotations.txt
$

At this point we have a basic toolkit which allows us to do things in the shell that you can normally do in Windows, Mac OS, or GNU/Linux with your mouse. For example, we can create, delete and move individual files. We will now learn about wildcards, which will allow us to expand our toolkit so that we can perform operations on multiple files with a single command. We'll start in an empty directory and create four files:

$ touch file1.txt file2.txt otherfile1.txt otherfile2.txt
$ ls
file1.txt   file2.txt   otherfile1.txt   otherfile2.txt
$

If we want to see all filenames that start with "file", we can run the command ls file*:

$ ls file*
file1.txt   file2.txt
$

There are four other basic wildcard characters we will explore:

• The question mark, ?, matches any single character, so text?.txt would match text1.txt or textz.txt, but not text_1.txt.
• The square brackets, [ and ], which can be used to specify a set of possible characters. For example, text[135].txt would match text1.txt, text3.txt, and text5.txt. You can also specify a natural sequence of characters using a dash. For example, text[1-6].txt would match text1.txt, text2.txt, ..., text6.txt.
• The exclamation mark, !, which can be used with square brackets to negate characters. For example, text[!135].txt would match anything of the form text?.txt except for text1.txt, text3.txt, and text5.txt. (Note that the exclamation mark has to be used right after the first square bracket as in the example. An exclamation mark outside of square brackets is not a wildcard, but serves another function of accessing previous commands from your history, as we will later see.)
• The curly braces, {}, are similar to the square brackets, except that they take a list of items separated by commas, and that they always expand all elements of the list, whether or not the corresponding filenames exist. For example, text{1,5}.txt would expand to text1.txt text5.txt whether or not text1.txt and text5.txt actually exist. The elements of the list need not be single characters, so text{12,5}.txt expands to text12.txt text5.txt. Natural sequences can be specified using .., i.e. text{1..5}.txt would expand to text1.txt text2.txt text3.txt text4.txt text5.txt.

We can use any number of wildcards at a time: for example, p*.p?* matches anything that starts with a 'p' and ends with '.', 'p', and at least one more character (since the '?' has to match one character, and the final '*' can match any number of characters). Thus, p*.p?* would match preferred.practice, and even p.pi (since the first '*' can match no characters at all), but not quality.practice (doesn't start with 'p') or preferred.p (there isn't at least one character after the '.p').

Let's see some more examples of wildcards in action. First we will remove everything in the current directory:

$ rm *
$ ls
$

Now that our working directory is empty, let's create some empty files using the touch command and the curly braces:

$ touch file{4,6,7}.txt file_{a..e}.txt
$ 

The curly braces expand the second token file{4,6,7}.txt into three separate file names, and the third token file_{a..e}.txt expands into five separate file names. So, now if we list files using the asterisk wildcard, we get:

$ ls file*
file4.txt   file7.txt    file_b.txt   file_d.txt
file6.txt   file_a.txt   file_c.txt   file_e.txt
$

We have matched all files in our current directory that begin with file. Now, let us list a subset of those files using the question mark:

$ ls file?.txt
file4.txt   file6.txt    file7.txt
$

Note how this glob pattern matched only the numbered files. That's because only the numbered files have a single character between file and .txt. The lettered files have two characters between file and .txt:

$ ls file??.txt
file_a.txt   file_b.txt    file_c.txt   file_d.txt   file_e.txt
$

Let's try out using the ! wildcard with square brackets. The pattern file[!46]* would match everything beginning with "file", unless the next character is a 4 or a 6. Thus, continuing with our example, if we do rm file[!46]*, we will delete everything except file4.txt and file6.txt:

$ ls
file4.txt   file7.txt    file_b.txt   file_d.txt
file6.txt   file_a.txt   file_c.txt   file_e.txt
$ rm file[!46]*
$ ls
file4.txt   file6.txt
$

It's also possible to combine and nest wildcards. For example, file{4,_?}.txt would match file4.txt and file_?.txt, where ? matches any single character. To demonstrate this, let's re-create all of our files:

$ touch file{4,6,7}.txt file_{a..e}.txt
$ ls
file4.txt   file7.txt    file_b.txt   file_d.txt
file6.txt   file_a.txt   file_c.txt   file_e.txt
$ 

Now let's run ls file{4,_?}.txt:

$ ls file{4,_?}.txt
file4.txt   file_b.txt   file_d.txt
file_a.txt   file_c.txt   file_e.txt
$

Finally, it is important to understand what happens if a wildcard expression doesn't match any files. In this case, the wildcard characters are interpreted literally. For example, if we were to enter ls file[AB].txt, the system will first try to find fileA.txt or fileB.txt. If it found either of those, then the wildcard expression would be expanded appropriately, but when it fails to find either of them, the expression remains unevaluated as file[AB].txt, and is then passed to ls. At this point, ls will try to find a single file that is actually named file[AB].txt. We can see that this is the case if we try ls file[AB].txt:

$ ls file[AB].txt
ls: file[AB].txt: No such file or directory
$

The system tells us that file[AB].txt doesn't exist since this was the file it was looking for after the wildcard expression found no matches. Still not convinced? Let's actually create a file named file[AB].txt. To do this, we need some way of telling the computer that we actually want the [ and ] characters to be in the filename, and that we don't mean them as wildcards.

Now we can create the file file[AB].txt:

$ touch "file[AB].txt"
$ ls
file4.txt   file7.txt      file_a.txt   file_c.txt   file_e.txt
file6.txt   file[AB].txt   file_b.txt   file_d.txt
$

If we do ls file[AB].txt, it will find the file we've just created:

$ ls file[AB].txt
file[AB].txt
$

But if we create a file named fileA.txt, this will be found first by the wildcard expansion, since it first looks for fileA.txt and fileB.txt:

$ touch fileA.txt
$ ls file[AB].txt
fileA.txt
$

If we really do want to look for something called file[AB].txt, and not fileA.txt or fileB.txt, then we have to escape the special characters "[" and "]". We can do this:

$ ls file\[AB\].txt
file[AB].txt
$

or this:

$ ls "file[AB].txt"
file[AB].txt
$

Single quotes would work as well in this case.

Now that we know a few basic commands and have learned how to use wildcards, we can finally look at the shell's most powerful feature: the ease with which it lets you combine existing programs in new ways. We'll start with a directory called molecules that contains six files describing some simple organic molecules. The .pdb extension indicates that these files are in Protein Data Bank format, a simple text format that specifies the type and position of each atom in the molecule.

$ ls molecules
cubane.pdb    ethane.pdb    methane.pdb
octane.pdb    pentane.pdb   propane.pdb
$

Let's go into that directory with cd and run the command wc *.pdb. wc is the "word count" command; it counts the number of lines, words, and characters in files:

$ cd molecules
$ wc *.pdb
  20  156 1158 cubane.pdb
  12   84  622 ethane.pdb
   9   57  422 methane.pdb
  30  246 1828 octane.pdb
  21  165 1226 pentane.pdb
  15  111  825 propane.pdb
 107  819 6081 total
$

If we run wc -l instead of just wc, the output shows only the number of lines per file:

$ wc -l *.pdb
  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total
$

We can also use -w to get only the number of words, or -c to get only the number of characters.

Now, which of these files is shortest? It's an easy question to answer when there are only six files, but what if there were 6000? That's the kind of job we want a computer to do.

Our first step toward a solution is to run the command:

$ wc -l *.pdb > lengths

The > tells the shell to redirect the command's output to a file instead of printing it to the screen. The shell will create the file if it doesn't exist, or overwrite the contents of that file if it does. (This is why there is no screen output: everything that wc would have printed has gone into the file lengths instead.)

ls lengths confirms that the file exists:

$ ls lengths
lengths
$

We can now send the content of lengths to the screen using cat lengths. cat stands for "concatenate": it prints the contents of files one after another. In this case, there's only one file, so cat just shows us what it contains:

$ cat lengths
  20  cubane.pdb
  12  ethane.pdb
   9  methane.pdb
  30  octane.pdb
  21  pentane.pdb
  15  propane.pdb
 107  total
$

Now let's use the sort command to sort its contents. This does not change the file. Instead, it sends the sorted result to the screen:

$ sort lengths
  9  methane.pdb
 12  ethane.pdb
 15  propane.pdb
 20  cubane.pdb
 21  pentane.pdb
 30  octane.pdb
107  total
$

We can put the sorted list of lines in another temporary file called sorted-lengths by putting > sorted-lengths after the command, just as we used > lengths to put the output of wc into lengths. Once we've done that, we can run another command called head to get the first few lines in sorted-lengths:

$ sort lengths > sorted-lengths
$ head -1 sorted-lengths
  9  methane.pdb
$

Giving head the parameter -1 tells us we only want the first line of the file; -20 would get the first 20, and so on. Since sorted-lengths the lengths of our files ordered from least to greatest, the output of head must be the file with the fewest lines.

If you think this is confusing, you're in good company: even once you understand what wc, sort, and head do, all those intermediate files make it hard to follow what's going on. How can we make it easier to understand?

Let's start by getting rid of the sorted-lengths file by running sort and head together:

$ sort lengths | head -1
  9  methane.pdb
$

The vertical bar between the two commands is called a pipe. It tells the shell that we want to use the output of the command on the left as the input to the command on the right. The computer might create a temporary file if it needs to, or copy data from one program to the other in memory, or something else entirely: we don't have to know or care.

Well, if we don't need to create the temporary file sorted-lengths, can we get rid of the lengths file too? The answer is "yes": we can use another pipe to send the output of wc directly to sort, which then sends its output to head:

$ wc -l *.pdb | sort | head -1
  9  methane.pdb
$

This is exactly like a mathematician nesting functions like sin(πx)² and saying "the square of the sine of x times π": in our case, the calculation is "head of sort of word count of *.pdb".

Inside Pipes

Here's what actually happens behind the scenes when we create a pipe. When a computer runs a program—any program—it creates a process in memory to hold the program's software and its current state. Every process has an input channel called standard input. (By this point, you may be surprised that the name is so memorable, but don't worry: most Unix programmers call it stdin.) Every process also has a default output channel called standard output, or stdout (Figure 11).

The shell is just another program, and runs in a process like any other. Under normal circumstances, whatever we type on the keyboard is sent to the shell on its standard input, and whatever it produces on standard output is displayed on our screen (Figure 12):

When we run a program, the shell creates a new process. It then temporarily sends whatever we type on our keyboard to that process's standard input, and whatever the process sends to standard output to the screen (Figure 13):

Here's what happens when we run wc -l *.pdb > lengths. The shell starts by telling the computer to create a new process to run the wc program. Since we've provided some filenames as parameters, wc reads from them instead of from standard input. And since we've used > to redirect output to a file, the shell connects the process's standard output to that file (Figure 14).

If we run wc -l *.pdb | sort instead, the shell creates two processes, one for each process in the pipe, so that wc and sort run simultaneously. The standard output of wc is fed directly to the standard input of sort; since there's no redirection with >, sort's output goes to the screen (Figure 15):

And if we run wc -l *.pdb | sort | head -1, we get the three processes shown in Figure 16 with data flowing from the files, through wc to sort, and from sort through head to the screen.

This simple idea is why Unix has been so successful. Instead of creating enormous programs that try to do many different things, Unix programmers focus on creating lots of simple tools that each do one job well, and work well with each other. Ten such tools can be combined in 100 ways, and that's only looking at pairings: when we start to look at pipes with multiple stages, the possibilities are almost uncountable.

This programming model is called pipes and filters. We've already seen pipes; a filter is a program that transforms a stream of input into a stream of output. Almost all of the standard Unix tools can work this way: unless told to do otherwise, they read from standard input, do something with what they've read, and write to standard output.

The key is that any program that reads lines of text from standard input, and writes lines of text to standard output, can be combined with every other program that behaves this way as well. You can and should write your programs this way, so that you and other people can put those programs into pipes to multiply their power.

$ cd north-pacific-gyre/2012-07-03
$ wc -l *.txt

 300 NENE01729A.txt
 300 NENE01729B.txt
 300 NENE01736A.txt
 300 NENE01751A.txt
 300 NENE01751B.txt
 300 NENE01812A.txt
 ... ...

$ wc -l *.txt | sort | head -5
 240 NENE02018B.txt
 300 NENE01729A.txt
 300 NENE01729B.txt
 300 NENE01736A.txt
 300 NENE01751A.txt

$ wc -l *.txt | sort | tail -5
 300 NENE02040A.txt
 300 NENE02040B.txt
 300 NENE02040Z.txt
 300 NENE02043A.txt
 300 NENE02043B.txt

$ ls *Z.txt
NENE01971Z.txt    NENE02040Z.txt