When you click on the Terminal icon on your Desktop you are loading a program
which in turn loads a shell. The commands you write are not executed directly
by the kernel but are first interpreted by the shell.

When you execute a program from Python you can choose to execute it directly
with the kernel or via a shell. If you are executing it directly you won't be
able to write your commands in quite the same way as you would when using a
shell like bash.

Let's look at the different functionality you will be used to using on
the shell before looking at how to achive similar results with subprocess.

In UNIX and Linux, there are three I/O channels known as streams which
connect a computer program with its environment such as a text terminal (eg
gnome-terminal' running Bash) or another computer program (eg a Python program
using the ``subprocess` module). These I/O channels are called the standard
input, standard output and standard error respectively and can also be
refered to by their corresponding file descriptors which are the numbers 0, 1
and 2 respectively.

Handle

Name

Description

0

stdin

Standard input

1

stdout

Standard output

2

stderr

Standard error

Here you can see that standarf input is often called stdin, standard output
called stdout and standard error called stderr.

The streams work as follows: input from the terminal is sent via the standard
input stream to the program, normal output is returned from the program via the
standard output and error messages are returned to the environment standard
error. The diagram from wikipedia below illustrates this:

Occasionally you might want to redirect a stream from a program to another
location. Let's look at that next.

You can redirect the standard output from a program to a file using the shell
operator > in Bash (other shells may have slightly different syntax).
Here's an example:

$ program1 > file1

This results in program1 being executed and its standard output stream
being written to file1, replacing any existing data in file1. If you'd
wanted to append data to the end of the file instead you could use the shell
operator >> in Bash:

$ program1 >> file1

The shell operator < can be used to read data from a file and send it to
the standard input stream of a program:

$ program1 < file1

Again, program1 is executed but this time file1 is the source of the
data for standard input instead of the keybaord.

You can actually combine shell operators to achieve more sophisticated results.
In the example below program1 is executed and data from file1 is sent
to its standard input. The standard output from running program1 with the
input from file1 is then written to file2:

$ program1 < file1 > file2

There may be times when you want the output from one program to be read as the
input to another program. You can achieve this using temporary files like this:

$ program1 > tempfile1
$ program2 < tempfile1
$ rm tempfile1

This is a bit cumbersome though so shells provide a facility called piping.

As well as redirecting the stadard output, you can also redirect other
streams, for example to send the standard error data to standard output. In
Bash the >, < and >> operators we've already discussed can also be
prefixed by the file descriptor (remeber the numbers 0, 1 and 2 in the table earlier) to redirect that stream. If the number
is omitted it is assumed to be 1 for the standard output which is why the
commands we've used so far work.

This command executes program1 and send's any data it writes to standard error to file1.

program1 2> file1

executes program1, directing the standard error stream to file1.

Here's an example program you can use to test it. Save it as redirect1.py:

This program constantly polls stdin and echos any message it recieves to both stdout and stdout.

In shells derived from csh (the C shell), the syntax instead appends the &
character to the redirect characters, thus achieving a similar result.

Another useful capability is to redirect one standard file handle to another.
The most popular variation is to merge standard error into standard output so
error messages can be processed together with (or alternately to) the usual
output. Example:

find / -name .profile > results 2>&1

will try to find all files named .profile. You need the & character
even in Bash this time. Executed without redirection, it will output hits to
stdout and errors (e.g. for lack of privilege to traverse protected
directories) to stderr. If standard output is directed to file results, error
messages appear on the console. To see both hits and error messages in file
results, merge stderr (handle 2) into stdout (handle 1) using 2>&1 .

It's possible use 2>&1 before ">" but it doesn't work. In fact, when the
interpreter reads 2>&1, it doesn't know yet where standard output is redirected
and then standard error isn't merged.

If the merged output is to be piped into another program, the file merge
sequence 2>&1 must precede the pipe symbol, thus:

find / -name .profile 2>&1 | less

In Bash a simplified form of the command:

command > file 2>&1

is:

command &>file

or:

command >&file

but don't use these shortcuts or you might get confused. Better to be more
verbose and explicit.

The &> operator redirects standard output (stdout) and standard error (stderr)
at the same time. This is simpler to type than the Bourne shell equivalent
command > file 2>&1.

The redirection and piping tokens can be chained together to create complex
commands. For example:

ls | grep '\.sh' | sort > shlist

lists the contents of the current directory, where this output is filtered to
only contain lines which contain .sh, sort this resultant output
lexicographically, and place the final output in shlist. This type of
construction is used very commonly in shell scripts and batch files.

Now that we've discussed the sort of functionality offered on the command line,
let's experiment with the subprocess module. Here's a simple command you
can run on the command line:

$ echo "Hello world!"
Hello world!

Let's try to run this from Python.

In the past, process management in Python was dealt with by a large range of
different Python functions from all over the standard library. Since Python 2.4
all this functionality has been carefully and neatly packaged up into the
subprocess module which provides one class called Popen which is all
you need to use.

Note

If you are interested in how the new Popen class replaces the old
functionality the subprocess documentation
has a section explaining how things used to be done and how they are done now.

As you can see, this prints Hello world! to the standard output as before
but the interactive console also displays that we have created an instance of
the subprocess.Popen class.

If you save this as process_test.py and execute it on the command line you
get the same result:

$ python process_test.py
Hello world!

So far so good.

You might we wondering which shell is being used. On Unix, the default shell is
/bin/sh. On Windows, the default shell is specified by the COMSPEC
environment variable. When you specify shell=True you can customise the
shell to use with the executable argument.

The errors.txt file will be empty because there weren't any errors.
Interstingly on my computer, the Popen instance is displayed before the
HELLO WORLD! message is printed to the standard output this time. Pipes and
redirects clearly work anyway.

While it is handy to be able to execute commands pretty much as you would on
the command line, you often need to pass variables from Python to the commands
you are using. Let's say we wanted to re-write this function to use echo:

You can actually specify a single string as the argument when
shell=False but it must be the program itself and is no different from just
specifying a list with one element for args. If you try to execute the same
sort of command you would when shell=False you get an error:

Since we are still passing it as a string, Python assumes
the entire string is the name of the program to execute and there isn't a
program called echo "Hello world!" so it fails. Instead you have to
pass each argument separately.

This function is also very useful for checking whether a user has a program
your Python program requires installed on their PATH.

Of course you can also find out the location of programs with the whereis
command on the command line:

$ whereis echo
echo: /bin/echo /usr/share/man/man1/echo.1.gz

Note

Notice that whether shell is True or False we haven't had to
specify the full path to an executable. As long as the executable is on the
PATH environmant variable, you can execute it from Python. Of course, there
is no harm in specifying the full path if you prefer.

If you want to be slightly perverse you can specify the executable argument
rather than having the executable as the first argument in the args list.
This doesn't seem well documented but this is what it does on my computer:

The drawback of executing programs without the shell is that you can't so
redirection, piping, use here documents, shell expansion or many other
techniques you are used to in quite the same way as you would on the command
line. Instead you have to handle these features programatically we'll look at
this later.

When you run programs with Popen, output is just sent to stdout as usual
which is why you've been able to see the output from all the examples to far.

If you want to be able to read the standard output from a program you have to
set the stdout argument in the initial call to Popen to specify that a
pipe should be opened to the process. The special value you set is
subprocess.PIPE:

subprocess.PIPE

Special value that can be used as the stdin, stdout or stderr argument to
Popen and indicates that a pipe to the standard stream should be opened.

The Popen instance also has attributes stdout and stderr which you
can write to as if they were file-like objects and a stdin attribute which
you can read from like a file. You can use these instead of getting and sending
data via communicate() if you prefer. We'll see them next.

When a program exits it can return an integer value to indicate the exit
status. Zero is considered “successful termination” and any nonzero value is
considered “abnormal termination” by shells and the like. Most systems require
it to be in the range 0-127, and produce undefined results otherwise. Some
systems have a convention for assigning specific meanings to specific exit
codes, but these are generally underdeveloped; Unix programs generally use 2
for command line syntax errors and 1 for all other kind of errors.

You can access the return code from an exited child process via the
.returncode attribute of a Popen instance. Here's an example:

The returncode value is not ever set by the child process, it starts off
with a default value of None, and remains None until you call a method
in the subprocess such as poll() or wait(). Those methods set and
then return returncode. As a result, if you want to know what the status of
the child process is, you have to call either poll() or wait().

The poll() and wait() methods are subtley different:

Popen.poll()

Check if child process has terminated. Set and return .returncode attribute.

Popen.wait()

Wait for child process to terminate. Set and return .returncode attribute.

Notice that sys.argv[0] always contains the name of the script which was
executed, regardless of how Python was invoked. sys.argv[1] and onward are
the command line arguments. You can also invoke the program using Python's
-m switch to run it as a module import: