Systems, Tools, and Terminal Science

Main menu

Tag Archives: command

You can tell a lot about a shell user by looking at their prompt. Most shell
users will use whatever the system’s default prompt is for their entire career.
Under many Linux distributions, this prompt includes the username, the
hostname, and the current working directory, along with a $ sigil for regular
users, and a # for root.

tom@sanctum:~$

This format works well for most users, and it’s so common that it’s often used
in tutorials to represent the user’s prompt in lieu of the single-character
standard $ and #. It shows up as the basis for many of the custom prompt
setups you’ll find online.

Some users may have made the modest steps of perhaps adding some brackets
around the prompt, or adding the time or date:

[10:11][tom@sanctum:~]$

Still others make the prompt into a grand multi-line report of their shell’s
state, or indeed their entire system, full of sound and fury and signifying way
too much:

Then there are the BSD users and other minimalist contrarians who can’t abide
anything more than a single character cluttering their screens:

$

Getting your prompt to work right isn’t simply a cosmetic thing, however. If
done right and in a style consistent with how you work, it can provide valuable
feedback on how the system is reacting to what you’re doing with it, and also
provide you with visual cues that could save you a lot of confusion — or even
prevent your making costly mistakes.

I’ll be working in Bash. Many of these principles work well for Zsh as well,
but Zsh users might want to read Steve Losh’s article on his prompt setup
first to get an idea of the syntactical differences.

Why is this important?

One of the primary differences between terminals and windowed applications is
that there tends to be much less emphasis on showing metadata about the
running program and its environment. For the most part, to get information
about what’s going on, you need to request it with calls like pwd,
whoami, readlink, env, jobs, and echo $?.

The habit of keeping metadata out of the terminal like this unless requested
may appeal to a sense of minimalism, but like a lot of things in the command
line world, it’s partly the olden days holding us back. When mainframes and
other dinosaurs roamed the earth, terminal screens were small, and a long or
complex prompt amounted to a waste of space and cycles (or even paper). In our
futuristic 64-bit world, where tiny CRTs are a thing of the past and we think
nothing of throwing gigabytes of RAM at a single process, this is much less of
a problem.

Customising the prompt doesn’t have many “best practices” to it like a lot of
things in the shell; there isn’t really a right way to do it. Some users will
insist it’s valuable to have the path to the terminal device before every
prompt, which I think is crazy as I almost never need that information, and
when I do, I can get it with a call to tty. Others will suggest that
including the working directory makes the length of the prompt too variable and
distracting, whereas I would struggle to work without it. It’s worth the effort
to design a prompt that reflects your working style, and includes the
information that’s consistently important to you on the system.

So, don’t feel like you’re wasting time setting up your prompt to get it just
right, and don’t be so readily impressed by people with gargantuan prompts in
illegible rainbow colors that they copied off some no-name wiki. If you use
Bash a lot, you’ll be staring at your prompt for several hours a day. You may
as well spend an hour or two to make it useful, or at least easier on the eyes.

The basics

The primary Bash prompt is stored in the variable PS1. Its main function is
to signal to the user that the shell is done with its latest foreground task,
and is ready to accept input. PS1 takes the form of a string that can
contain escape sequences for certain common prompt elements like usernames and
hostnames, and which is evaluated at printing time for escape sequences,
variables, and command or function calls within it.

You can inspect your current prompt definition with echo. If you’re using
Linux, it will probably look similar to this:

tom@sanctum:~$ echo $PS1
\u@\h:\w\$

PS1 works like any other Bash variable; you can assign it a new value,
append to it, and apply substitutions to it. To add the text bash to the
start of my prompt, I could do this:

tom@sanctum:~$ PS1=bash-"$PS1"
bash-tom@sanctum:~$

There are other prompts — PS2 is used for “continued line” prompts, such as
when you start writing a for loop and continue another part of it on a new
line, or terminate your previous line with a backslash to denote the next line
continues on from it. By default this prompt is often a right angle bracket
>:

$ for f in a b c; do
> do echo "$f"
> done

There are a couple more prompt strings which are much more rarely seen. PS3
is used by the select structure in shell scripts; PS4 is used while
tracing output with set -x. Ramesh Natarajan breaks this down very well in
his article on the Bash prompt. Personally, I only ever modify PS1,
because I see the other prompts so rarely that the defaults work fine for me.

As PS1 is a property of interactive shells, it makes the most sense to put
its definitions in $HOME/.bashrc on Linux or BSD systems. It’s likely there’s
already code in there doing just that. For Mac OS X, you’ll likely need to put
it into $HOME/.bash_profile instead.

Escapes

Bash will perform some substitutions for you in the first pass of its
evaluation of your prompt string, replacing escape sequences with appropriate
elements of information that are often useful in prompts. Below are the escape
sequences as taken from the Bash man page:

\a an ASCII bell character (07)
\d the date in "Weekday Month Date" format (e.g., "Tue May 26")
\D{format}
the format is passed to strftime(3) and the result is inserted into
the prompt string; an empty format results in a locale-specific time
representation. The braces are required
\e an ASCII escape character (033)
\h the hostname up to the first `.'
\H the hostname
\j the number of jobs currently managed by the shell
\l the basename of the shell's terminal device name
\n newline
\r carriage return
\s the name of the shell, the basename of $0 (the portion following
the final slash)
\t the current time in 24-hour HH:MM:SS format
\T the current time in 12-hour HH:MM:SS format
\@ the current time in 12-hour am/pm format
\A the current time in 24-hour HH:MM format
\u the username of the current user
\v the version of bash (e.g., 2.00)
\V the release of bash, version + patch level (e.g., 2.00.0)
\w the current working directory, with $HOME abbreviated with a tilde
(uses the value of the PROMPT_DIRTRIM variable)
\W the basename of the current working directory, with $HOME
abbreviated with a tilde
\! the history number of this command
\# the command number of this command
\$ if the effective UID is 0, a #, otherwise a $
\nnn the character corresponding to the octal number nnn
\\ a backslash
\[ begin a sequence of non-printing characters, which could be used to
embed a terminal control sequence into the prompt
\] end a sequence of non-printing characters

As an example, the default PS1 definition on many Linux systems uses four of
these codes:

\u@\h:\w\$

The \! escape sequence, which expands to the history item number of the
current command, is worth a look in particular. Including this in your prompt
allows you to quickly refer to other commands you may have entered in your
current session by history number with a ! prefix, without having to actually
consult history:

Keep in mind that you can refer to history commands relatively rather than
by absolute number. The previous command can be referenced by using !! or
!-1, and the one before that by !-2, and so on, provided that you have left
the histexpand option on.

Also of particular interest are the delimiters \[ and \]. You can use these
to delimit sections of “non-printing characters”, typically things like
terminal control characters to do things like moving the cursor around,
starting to print in another color, or changing properties of the terminal
emulator such as the title of the window. Placing these non-printing characters
within these delimiters prevents the terminal from assuming an actual character
has been printed, and hence correctly manages things like the user deleting
characters on the command line.

Colors

There are two schools of thought on the use of color in prompts, very much
a matter of opinion:

Focus should be on the output of commands, not on the prompt, and using
color in the prompt makes it distracting, particularly when output from
some programs may also be in color.

The prompt standing out from actual program output is a good thing, and
judicious use of color conveys useful information rather than being purely
decorative.

If you agree more with the first opinion than the second, you can probably just
skip to the next section of this article.

Printing text in color in terminals is done with escape sequences, prefixed
with the \e character as above. Within one escape sequence, the foreground,
background, and any styles for the characters that follow can be set.

For example, to include only your username in red text as your prompt, you
might use the following PS1 definition:

PS1='\[\e[31m\]\u\[\e[0m\]'

Broken down, the opening sequence works as follows:

\[ — Open a series of non-printing characters, in this case, an escape
sequence.

\e — An ASCII escape character, used to confer special meaning to the
characters that follow it, normally to control the terminal rather than to
enter any characters.

[31m — Defines a display attribute, corresponding to foreground red,
for the text to follow; opened with a [ character and terminated with an
m.

\] — Close the series of non-printing characters.

The terminating sequence is very similar, except it uses the 0 display
attribute to reset the text that follows back to the terminal defaults.

The valid display attributes for 16-color terminals are:

Style

0 — Reset all attributes

1 — Bright

2 — Dim

4 — Underscore

5 — Blink

7 — Reverse

8 — Hidden

Foreground

30 — Black

31 — Red

32 — Green

33 — Yellow

34 — Blue

35 — Magenta

36 — Cyan

37 — White

Background

40 — Black

41 — Red

42 — Green

43 — Yellow

44 — Blue

45 — Magenta

46 — Cyan

47 — White

Note that blink may not work on a lot of terminals, often by design, as it
tends to be unduly distracting. It’ll work on a linux console, though. Also
keep in mind that for a lot of terminal emulators, by default “bright” colors
are printed in bold.

More than one of these display attributes can be applied in one hit by
separating them with semicolons. The below will give you underlined white text
on a blue background, after first resetting any existing attributes:

\[\e[0;4;37;44m\]

It’s helpful to think of these awkward syntaxes in terms of opening and
closing, rather like HTML tags, so that you open before and then reset after
all the attributes of a section of styled text. This usually amounts to
printing \[\e[0m\] after each such passage.

These sequences are annoying to read and to type, so it helps to put commonly
used styles into variables:

A more robust method is to use the tput program to print the codes for
you, and put the results into variables:

color_red=$(tput setaf 1)
color_blue=$(tput setaf 4)

This may be preferable if you use some esoteric terminal types, as it queries
the terminfo or termcap databases to generate the appropriate escapes for
your terminal.

The set of eight colors — or sixteen, depending on how you look at it — can
be extended to 256 colors in most modern terminal emulators with a little
extra setup. The escape sequences are not very different, but their range is
vastly increased to accommodate the larger palette.

A useful application for prompt color can be a quick visual indication of the
privileges available to the current user. The following code changes the prompt
to green text for a normal user, red for root, and yellow for any other user
attained via sudo:

Otherwise, you could simply use color to make different regions of your prompt
more or less visually prominent.

Variables

Provided the promptvars option is enabled, you can reference Bash variables
like $? to get the return value of the previous command in your prompt
string. Some people find this valuable to show error codes in the prompt when
the previous command given fails:

The usual environment variables like USER and HOME will work too, though
it’s preferable to use the escape sequences above for those particular
variables. Note that it’s important to quote the variable references as well,
so that they’re evaluated at runtime, and not during the assignment itself.

Commands

If the information you want in your prompt isn’t covered by any of the escape
codes, you can include the output of commands in your prompt as well, using
command substitution with the syntax $(command):

This requires the promptvars option to be set, the same way variables in the
prompt do. Again, note that you need to use single quotes so that the command
is run when the prompt is being formed, and not immediately as part of the
assignment.

This can include pipes to format the data with tools like awk or cut, if
you only need a certain part of the output:

$ PS1='[$(uptime | cut -d: -f5) ]\$ '
[ 0.36, 0.34, 0.34 ]$

For clarity, during prompt setup in .bashrc, it makes sense to use
functions instead for reasonably complex commands:

prompt_load() {
uptime | cut -d: -f5
}
PS1='[$(prompt_load) ]'$PS1

Note that as a normal part of command substitution, trailing newlines are
stripped from the output of the command, so here the output of uptime appears
on a single line.

A common usage of this pattern is showing metadata about the current directory,
particularly if it happens to be a version control repository or working copy;
you can use this syntax with functions to show the type of version control
system, the current branch, and whether any changes require committing. Working
with Git, Mercurial, and Subversion most often, I include the relevant logic
as part of my prompt function.

When appended to my PS1 string, $(prompt vcs) gives me prompts that look
like the following when I’m in directories running under the appropriate VCS.
The exclamation marks denote that there are uncommitted changes in the
repositories.

In general, where this really shines is adding pieces to your prompt
conditionally, to make them collapsible. Certain sections of your prompt
therefore only show up if they’re relevant. This snippet, for example, prints
the number of jobs running in the background of the current interactive shell
in curly brackets, but only if the count is non-zero:

It’s important to make sure that none of what your prompt does takes too long
to run; an unresponsive prompt can make your terminal sessions feel very
clunky.

Note that you can also arrange to run a set of commands before the prompt is
evaluated and printed, using the PROMPT_COMMAND. This tends to be a good
place to put commands that don’t actually print anything, but that do need to
be run before or after each command, such as operations working with history:

PROMPT_COMMAND='history -a'

Switching

If you have a very elaborate or perhaps even computationally expensive prompt,
it may occasionally be necessary to turn it off to revert to a simpler one.
I like to handle this by using functions, one of which sets up my usual prompt
and is run by default, and another which changes the prompt to the minimal $
or # character so often used in terminal demonstrations. Something like the
below works well:

You can then switch your prompt whenever you need to by typing prompt_on and
prompt_off.

This can also be very useful if you want to copy text from your terminal into
documentation, or into an IM or email message; it removes your distracting
prompt from the text, where it would otherwise almost certainly differ from
that of the user following your instructions. This is also occasionally helpful
if your prompt does not work on a particular machine, or the machine is
suffering a very high load average that means your prompt is too slow to load.

Further reading

Predefined prompt strings are all over the web, but the above will hopefully
enable you to dissect what they’re actually doing more easily and design your
own. To take a look at some examples, the relevant page on the Arch Wiki
is a great start. Ramesh Natarajan over at The Geek Stuff has a great article
with some examples as well, with the curious theme of making your prompt as
well-equipped as Angelina Jolie.

Finally, please feel free to share your prompt setup in the comments (whether
you’re a Bash user or not). It would be particularly welcome if you explain why
certain sections of your prompt are so useful to you for your particular work.

If you set environment variables like your EDITOR in your .bashrc file
that refer to commands that you expect to be available on the system, it’s
prudent to check that an appropriate command actually exists before making the
setting.

A common way of approaching this is using which, in a syntax similar to the
below, which sets EDITOR to the first executable instance of vi found in
PATH, or blank if not found:

EDITOR=$(which vi)

Because the behaviour of whichcan be unexpected, it’s better practice
to use one of Bash’s builtins to do this, either command, type, or hash.
I prefer using hash, which searches PATH for a command of the given name,
and loads it into Bash’s command hash table if found. An implementation like
the below works well:

if hash vi 2>/dev/null; then
export EDITOR=vi
fi

This ignores any error output from the hash call by redirecting it to the
null device, and only defines the value for EDITOR if a matching command is
found.

You can compact this syntax into one line:

hash vi 2>/dev/null && export EDITOR=vi

Thanks to commenter Yu-Jie Lin for pointing out the above abbreviation.

For tools like diff that work with multiple files as parameters, it can be
useful to work with not just files on the filesystem, but also potentially with
the output of arbitrary commands. Say, for example, you wanted to compare the
output of ps and ps -e with diff -u. An obvious way to do this is to
write files to compare the output:

$ ps > ps.out
$ ps -e > pse.out
$ diff -u ps.out pse.out

This works just fine, but Bash provides a shortcut in the form of process
substitution, allowing you to treat the standard output of commands as files.
This is done with the <() and >() operators. In our case, we want to direct
the standard output of two commands into place as files:

$ diff -u <(ps) <(ps -e)

This is functionally equivalent, except it’s a little tidier because it doesn’t
leave files lying around. This is also very handy for elegantly comparing files
across servers, using ssh:

$ diff -u .bashrc <(ssh remote cat .bashrc)

Conversely, you can also use the >() operator to direct from a filename
context to the standard input of a command. This is handy for setting up
in-place filters for things like logs. In the following example, I’m making a
call to rsync, specifying that it should make a log of its actions in
log.txt, but filter it through grep -vF .tmp first to remove anything
matching the fixed string .tmp:

$ rsync -arv --log-file=>(grep -vF .tmp >log.txt) src/ host::dst/

Combined with tee this syntax is a way of simulating multiple filters for a
stdout stream, transforming output from a command in as many ways as you see
fit:

By default, the Bash shell keeps the history of your most recent session in the
.bash_history file, and the commands you’ve issued in your current session
are also available with a history call. These defaults are useful for keeping
track of what you’ve been up to in the shell on any given machine, but with
disks much larger and faster than they were when Bash was designed, a little
tweaking in your .bashrc file can record history more permanently,
consistently, and usefully.

Append history instead of rewriting it

You should start by setting the histappend option, which will mean that when
you close a session, your history will be appended to the .bash_history
file rather than overwriting what’s in there.

shopt -s histappend

Allow a larger history file

The default maximum number of commands saved into the .bash_history file is a
rather meager 500. If you want to keep history further back than a few weeks or
so, you may as well bump this up by explicitly setting $HISTSIZE to a much
larger number in your .bashrc. We can do the same thing with the
$HISTFILESIZE variable.

HISTFILESIZE=1000000
HISTSIZE=1000000

The man page for Bash says that HISTFILESIZE can be unset to stop
truncation entirely, but unfortunately this doesn’t work in .bashrc
files due to the order in which variables are set; it’s therefore more
straightforward to simply set it to a very large number.

If you’re on a machine with resource constraints, it might be a good idea to
occasionally archive old .bash_history files to speed up login and reduce
memory footprint.

Don’t store specific lines

You can prevent commands that start with a space from going into history by
setting $HISTCONTROL to ignorespace. You can also ignore duplicate
commands, for example repeated du calls to watch a file grow, by adding
ignoredups. There’s a shorthand to set both in ignoreboth.

HISTCONTROL=ignoreboth

You might also want to remove the use of certain commands from your history,
whether for privacy or readability reasons. This can be done with the
$HISTIGNORE variable. It’s common to use this to exclude ls calls, job
control builtins like bg and fg, and calls to history itself:

HISTIGNORE='ls:bg:fg:history'

Record timestamps

If you set $HISTTIMEFORMAT to something useful, Bash will record the
timestamp of each command in its history. In this variable you can specify the
format in which you want this timestamp displayed when viewed with history. I
find the full date and time to be useful, because it can be sorted easily and
works well with tools like cut and awk.

HISTTIMEFORMAT='%F %T '

Use one command per line

To make your .bash_history file a little easier to parse, you can force
commands that you entered on more than one line to be adjusted to fit on only
one with the cmdhist option:

shopt -s cmdhist

Store history immediately

By default, Bash only records a session to the .bash_history file on disk
when the session terminates. This means that if you crash or your session
terminates improperly, you lose the history up to that point. You can fix this
by recording each line of history as you issue it, through the
$PROMPT_COMMAND variable:

The command line in Vim for ex commands can be edited with a few of the GNU
Readline key combinations that may be familiar to Bash or Emacs users, so it’s
reasonably easy to edit it, for example to type in complex search patterns and
replacements.

However, if you want the full facility of Vim editing for your commands, it can
be helpful to use Vim’s command line window, which will allow you to enter
commands and edit previous ones with the usual normal and insert mode.

You can open the command line window from normal mode in one of four ways:

q: — Open with a command history from normal mode

q/ — Open with a search history from normal mode (to search forward)

q? — Open with a search history from normal mode (to search backward)

Ctrl+F — Open with a command history from command mode

Note that this doesn’t work while you’re recording macros, since pressing q
stops the recording.

The window is always immediately above the status bar, and its height can be
set via the cmdwinheight option.

Command line window

Once the command line window is opened with q:, you can browse through it and
press Enter on any line to issue the same command again. You can also edit it
beforehand, perhaps to fix a mistyped command. The window will close when this
is done, but you can close it with Ctrl+W, C the same as any other window if
you change your mind.

Vim
command line window

Note that you can’t move to another window while this one is open, nor can you
load another buffer into it.

Search window

Similar to the above, if you open the command window with q/ or q?, it
shows a history of your searches, and pressing enter on any line will issue the
same again, in the appropriate direction.

Vim
search window

For more information on how the command window works, check out :help
command-line-window.

By default, the Bash shell uses GNU Readline, which provides a set of
Emacs-friendly key bindings that are a pretty workable way to edit long and
complicated commands. If you learn a little about the chords available to you
in editing Bash commands through this library, and combine that with a little
old-school command-line magic, you can work at high speed with decent
accuracy quite easily. Alternatively, if you don’t like the Emacs-style
keybindings and would prefer to stick to a vi-friendly model, you can use set
-o vi to edit your command line directly with vi keybindings.

However, if you’re building a particularly complex string of commands involving
a lot of pipes, escapes, and redirections, it often turns out to be handy to
actually load them into your favourite editor, to give you full facility to
edit them in any way you wish. Bash provides a method for this in the form of
its Ctrl+X, Ctrl+E binding.

To use it, you can type anything at the command prompt (including nothing at
all) and press Ctrl+X, Ctrl+E to bring up your EDITOR, be it Vim, Emacs, or
Nano, with the contents of the command line there to edit. As soon as you save
and quit, the command will be run as stated, and will be entered into the
command history as if you typed it out on the shell directly.

There’s also a handy built-in Bash shortcut, fc (short for “fix command”) to
open the previous command in your editor, allow you to edit it, and then run
it automatically when you quit. This is particularly useful if you’ve made a
small mistake in a complex line of shell code.

If this happens to bring up the wrong editor, perhaps because your choice
doesn’t match that of the system administrator, you can set your personal
preference of editor like so:

$ export EDITOR=/usr/bin/vim

You can confirm this is working by checking your environment variables:

$ env

One downside of this method is that without special setup within your editor,
you lose some of the benefits of things like tab completion. Fortunately it
only takes a little creative mapping to make this work in Vim, taking advantage
of the Ctrl+X, Ctrl+F file completion that’s already built in. You could even
bind that straight to the Tab key if you don’t otherwise use it.

I very often “fat-finger” some of the most common commands in Vim by holding
down Shift too long after typing a colon. This means that instead of typing
:w I end up typing :W, or :Qa instead of :qa, and because I use these
commands so often, I type them so rapidly and reflexively that mistakes become
quite common.

Since the uppercase versions of these oft-mistyped commands don’t actually
correspond to any other valid command, I considered it safe to map them so that
it would quietly accept this common mistake from me and understand what I
actually meant to do. I did this with the following lines in my .vimrc:

Note the -bang and <bang> parts of each line; this allows me to include an
exclamation mark in my mistyped command to force the command if a buffer is
unsaved, which otherwise wouldn’t work. Additionally, I use command! with an
exclamation mark to prevent errors if I reload my .vimrc file having already
loaded it once. Finally, the first four commands, which can optionally take an
argument, are set up to do that in their capitalised form as well.

You can list these commands and any others you or your plugins have defined by
typing just :command by itself. Check out :help command for a bit more
information on mapping commands in this manner.

The wrong way

This is quite different from the approach I often see recommended to work
around this problem, which is using cnoreabbrev like so:

cnoreabbrev W w
cnoreabbrev Q q

I think this is a pretty bad idea because if I wanted to search for a single
uppercase Q or W, it gets automatically mapped back to its lowercase equivalent
in the search window. When I tried these mappings out I noticed this very
quickly, and because I use case-sensitive search it rapidly got very
frustrating. I am much happier with the solution I describe above, and
particularly recommend it if like myself you prefer to keep ignorecase off.

Remapping for a different approach

If you don’t mind a slightly more drastic remapping, you could use another
character that doesn’t require holding Shift to initiate commands, such as a
semicolon, which means you won’t accidentally capitalise these common commands
anymore:

nnoremap ; :

I don’t like this solution either, because the semicolon already has a function
in repeating the linewise character searches you can do with f, t, F, and
T, but I have seen it in several other people’s .vimrc files.