The Linux Operating System

This booklet is designed to help with common tasks on a Linux system. It
is designed to be presentable as a series of "recipes" for accomplishing
common tasks. These recipes consist of a plain English one-line
description, followed by the Linux command which carries out the task. The
document is focused on performing tasks in Linux using the 'command line'
or 'console'. The format of the booklet was largely inspired by the "Linux
Cookbook" www.dsl.org/cookbook

GETTING

HELP

The traditional Unix help system is called 'man' or 'manual' pages. And
they can be good. It is one of the ironies and frustrations of Unix that a
man page only really becomes helpful and interesting once one already
knows what a program does and how to basically use it.

UNIX

CULTURE SHOCK ....

Those users who have no experience with a Unix-style operating
system, but are familiar with the Microsoft Windows operating system
will experience a kind of 'culture shock' when they begin to
use Linux. This is true even if the user is familiar with using
the 'command-line' on a Microsoft computer. This section attempts to
point out some of the things which may seem baffling and illogical
to the new Linux user.

@@ You cant just download an 'exe' and click on it to run it
However you can easily install new software with
'sudo apt-get install <program-name>'
@@ file names dont have to have extensions (such as .txt), but can
To an experienced Microsoft command line user, the idea of
just calling a text file 'report' rather than 'report.txt' is
rather disorientating. "How will the computer know what type
of file it is?" The microsoft user thinks.
File names can and do begin with a dot '.'
File names can begin with a tilde '~'
@@ Linux commands are very short and cryptic
Why is the 'list files' command, called 'ls' and not 'list'?
It would be so much more memorable as 'list' than 'ls'. This
is the "Unix way". If you like typing you can do
alias list='ls'

@@ The folder hierarchy in Linux seems very cryptic and you cant
just put files anywhere

BASIC

USER COMMANDS ....

* log in to the system with a username of 'bob'

ubuntu login: bob

* log out of the system

logout

* switch to the fourth virtual console

press [ALT]-[F4]

* switch from the fourth to the third virtual console, press:

[ALT]-[<-]

* switch from X to the first virtual console, press:

[CTRL]-[ALT]-[F1]

* run the hostname tool to find the name of the computer

hostname

* output the version of the hostname tool

hostname --version

* run hostname and specifying that the file 'host.info' be read from

hostname -F host.info

* change your password

passwd

* output your username

whoami

* see who is currently logged in

who

* see who is currently logged in and what they are doing

w

* display information about recent use of the system

last | less

* find out when the user 'mjb' last logged in

last mjb

NOTE

: The last tool gets its data from the system file '/var/log/wtmp';
the last line of output tells how far this file goes back.

BASIC

PROCESS COMMANDS ....

* list the processes in your current shell session

ps

* list all the processes that user 'hst' has running on the system

ps -u hst

This command is useful for listing all of your own processes

* list all of the processes and give their user-names

ps aux ##(there could be a lot of output, even on single user systems)

* display a continually updated display of the current system processes

top

* list all the processes containing a reference to an 'sbin' directory

ps aux | grep sbin

* list any processes whose process IDs contain a 13 in them

ps aux | grep 13

* list the process whose PID is 344

ps -p 344

BASIC

SOFTWARE COMMANDS ....

* Run skype using your GTK theme

skype --disable-cleanlooks -style GTK

* output a list of programs that pertain to consoles

apropos consoles

* output a list of all tools whose pages in the system manual contain a reference to consoles

man -k consoles

* list all of the software packages installed on the system

dpkg -l

* list all of the packages whose name or description contains the text "edit," regardless of case

dpkg -l | grep -i edit

* peruse descriptions of the packages that are available

less /var/lib/dpkg/available

* get a description of the who tool

whatis who

* view the manual page for w

man w

* view all of the Info manuals on the system

info

* read the Info documentation for the tar tool

info tar

This command opens a copy of The GNU tar Manual in info.
To read the contents of a file written in Info format, give the name of

* read 'faq.info', an Info file in the current directory

info -f faq.info

* read 'faq.info', an Info file in the current directory, beginning with the node Text

info -n 'Text' -f faq.info

* view the HTML version of the Debian FAQ in the lynx Web browser

lynx /usr/share/doc/debian/FAQ/debian-faq.html

The above only works on a Debian flavour of Linux.

* view the compressed text version of the Debian FAQ in zless,

zless /usr/doc/debian/FAQ/debian-faq.txt.gz

BASIC

COMMAND LINE USAGE ....

* repeat the last command entered

[^|]

The [^|] key moves the last command you typed back to the input line,
and executes it.

* retrieve the last command you entered with the string 'grep' in it

<control-r> ##(press the <control> key and the 'r' key)

(reverse-i-search)'': grep

* put the 3rd-last command you entered with 'grep' in it on the input line

* output a tree graph of your home directory and all its subdirectories

tree ~ ##(this shows files as well as folders)

* show a just the start of a folder tree for the home folder

tree -d /usr/local | head -20

[>>0.3 image/eg-tree.png]

* peruse a tree graph of the '/usr/local' directory tree

tree -d /usr/local | less

COPYING

FILES ....

* copy the file 'old' to the file `new'

cp old new

* copy files preserving the file attributes

cp -p file.txt new-copy.txt

* copy the folder 'reports' verbosely (showing what is being done)

cp -vr reports ~/new/

* copy a folder tree verbosely, but only copy files which are newer

cp -vur docs/office ~/docs

* copy the folder 'public_html', and subfolders, to 'private_html'

cp -R public_html private_html

The cp '-R' option doesnt copy symbolic links. But the "man" page for cp
states that -r and -R are equivalent

* make an archive copy of the directory tree 'public' to 'private'

cp -a public_html private_html

MOVING

FILES ....

* move the file 'notes' in the current working directory to '../play'

mv notes ../play

* move the file '/usr/tmp/notes' to the current working directory,

mv /usr/tmp/notes .

This command moves the file '/usr/tmp/notes' to the current working

* move the directory 'work' in the current working directory to 'play'

mv work play

* rename the file 'notes' to `notes.old'

mv notes notes.old

NOTE

: to rename multiple files use 'rename'

DELETING

FILES ....

* remove the file 'notes' in the current working directory

rm notes

* remove the directory 'waste' and all of its contents

rm -R waste

* remove the directory 'empty'

rmdir empty

* use tab completion to remove the file 'No Way' in the current directory

rm No[TAB] Way

* delete the file '^Acat' in a directory that also contains the files 'cat' and `dog'

rm -i ?cat ##(rm: remove '^Acat'? y )

* remove the file '-cat' from the current directory

rm -- -cat

SYMBOLIC

LINKS ....

* create a hard link from 'seattle' to `emerald-city'

ln seattle emerald-city

* create a symbolic link from 'seattle' to `emerald-city'

ln -s seattle emerald-city

* list all files in the '/usr/bin' directory that have the text 'tex' anywhere in their name

ls /usr/bin/*tex*

* copy all files whose names end with '.txt' to the `doc' subdirectory

cp *.txt doc

* output a verbose listing of all files whose names end with either a '.txt' or '.text' extension, sorting the list so that newer files are listed first

ls -lt *.txt *.text

* remove all files in the current working directory that begin with a hyphen and have the text 'out' somewhere else in their file name

rm -- -*out*

* join text files whose names have an 'a' followed by 2 or more characters

cat a??*

The original files are unchanged, but the joined together text files
are displayed on the screen.

THE

BASH PROMPT ....

* change your shell prompt to 'Your wish is my command: '

PS1='Your wish is my command: '

* change your prompt to the default bash prompt (the current folder)

PS1='\w $ '

* change the prompt to the current date, space, the hostname, and a '>'

PS1='\d (\h)>'

* clear the screen every time you log out,

clear ##(put this in the file '.bash_logout')

INSTALLING

SOFTWARE

* search for games software which are available to install

apt-cache search game

* update the repository cache

sudo apt-get update

Add new repositories to the file '/etc/apt/sources.list'

SOUND

A linux system is capable of playing, recording and editing
sound (audio) files with a variety of open-source software.
This section only provides what should be a very succint overview
of the most important sound tasks with linux. For more detailed
information, please consult the booklet listed below.

LESS

The humble 'less' program is worthy of a second look. Less allows
one to peruse and search a text file, but not alter it. I am
documenting version 429 (year 2008). Less uses vi-like keys
to move around and search.

* view the text file 'doc.txt' one screen page at a time

less doc.txt

* view the text file 'days.txt' starting at the end

less +G days.txt

some common 'less' commands
.. [space-bar] - forward one window
.. [esc] + [space-bar] - forward one window (with multiple files)
.. b - back one window
.. j - down one line (the same as 'vim')
.. k - up one line (the same as 'vim')
.. F - go to end of file and 'follow' new data (like tail -f)
.. G - go to the last line of the file
.. g - go to the first line of the file
.. /pattern - Search forward for (N-th) matching line.
.. ?pattern - Search backward for (N-th) matching line.
.. n - Repeat previous search (for N-th occurrence).
.. N - Repeat previous search in reverse direction.
.. ESC-n - Repeat previous search, spanning files.
.. v - edit the current file with $VISUAL or $EDITOR
..

* convert from the local character set to the latin1 encoding saving to "new.txt"

recode ..lat1 < file.txt > new.txt ##(the original file is unchanged)

* convert to html

recode ..HTML < file.txt > file.html

* convert from utf8 to html with verbose output

recode -v u8..h < file.txt

* convert from MS Windows utf8 to the local character set

recode utf-8/CRLF.. file-to-change.txt

COMPARING

AND PATCHING TEXT FILES ....

The process of 'patching' a text or code file is very important in the
world of open-source development (and therefore in the development of Linux
itself). Patching allows non-linear changes to be made to a file and is
usually used in conjuction with 'diff'

data jargon
@@ delimiter
This is a something which separate 2 field or marks the beginning or
end of some data. If we have 2 numbers separated by a space
character as in '23.2 200' then we say that the data is
'delimited' by spaces.
@@ field
A field is a unit of data in a record. If we have data about
people, then 3 fields may be a persons first name, second name and
age. However the way that data is broken up into fields is
subjective and option culturally dependent. Therefore there is
no universally accepted data format
@@ data record
a record is a set of data fields which relate to a particular
entity. For example if we store data about people, then a
record will contain data for one person. The way the record is
stored is dependant upon the data format.
@@ export data
The process of exporting data involves converting data to a particular
format and saving it in a file or files.
@@ import data

* wrap long lines in the file 'doc.txt' without disturbing special patterns

par doc.txt

STREAM

EDITING TEXT FILES ....

'stream editing' or 'batch editing' text files refers to the process
of modifying a text file without opening a text editor. In other words
the text file is modified via a series of commands which are applied
to a file or many files with a program such as sed, awk, perl, etc.
This is very useful when a large number of modifications need to be made
in many files and where those modifications are similar; For example
where a persons name needs to be changed in a large number of files.

* Randomize lines (opposite of | sort)

random -f <file>

* uniq for unsorted data

awk '!_[$0]++{print}'

* Add a line to a file using sudo

echo "foo bar" | sudo tee -a /path/to/some/file

* Remove duplicate entries in the file 'list.txt' without sorting.

awk '!x[$0]++' list.txt

SORTING

TEXT DATA

The unix 'sort' utility is a powerful and flexible way to reorder lines of a
text file based on the data in one or more of the 'fields' of each line. A
field is a chunk of text separated from another chunk by a 'delimiter'
character such as a space or a colon. For example if the line is 'italy 3
4/oct/1999' and the delimiter is a space then the fields are 'italy', '3' and
'4/oct/1999'

By default, sort divides a line into fields based on spaces or tabs but
can use another character (see the -t option)

VIEWING

HELP FOR SORT ....

* view some examples of using the 'sort' program

info coreutils 'sort invocation'

* view a confusing description about how 'sort' works

man sort ##(this is a classic example of an accurate but baffling man page)

BASIC

USAGE ....

* sort in alphabetical order (abc...) the lines of the text file 'data.txt'

* sort in descending numeric order using the first 3 digits of the 2nd field

sort -rnk2.1,2.3 data.txt

SORTING

BY DATE VALUES ....

The problem of sorting by a date value is somewhat tricky, mainly
because of the daunting array of different formats which a date can
appear in, without even thinking about intercultural differences.
However 'sort' is capable of sorting by date value, where the dates
are in a consitent format.

Where the date string is of a variable length the problem gets harder.
It may be possible to use the -t delimiter to divide up each field of
the date (eg use '-t/' for '1/january/08' and '02/oct/2001')

* Use the 'M' modifier to recognise month names and abbreviated month names

* sort lines by month name, assuming that field 3 is an english month name

sort -k3M data ##(field 3 can be something like 'feb', 'oct' or 'january')

* sort by month names using the 3rd character of the 2nd field as the start point

sort -k2.3M data ##(field 2 may be '--jan' or '::august' etc)

* sort lines by the 1st field, a date in the format dd/mm/yyyy (eg '09/11/1922')

sort -k 1.7n -k 1.4n -k 1.1n -k 4.14,4.21 data.txt

##(note that '4/11/1920' will not sort well but '04/11/1920' will)

* reverse sort lines by a date field in the format dd/mm/yyyy

sort -k 1.7nr -k 1.4nr -k 1.1nr data.txt

* sort lines with the second field a date in format dd/MON/yyyy '02/Apr/2010'

sort -k 1.8n -k 1.4M -k 1.1n data.txt

##(the month abbreviations must be 3 letters, and in english ...)

* sort lines by a date/time value such as '01/Jan/2004:06:31:51'

sort -k 1.8n -k 1.4M -k 1.1n -k 1.13,1.21 data.txt

##(this assumes the date is the first field of each line)

FIELDS

AND PARTIAL FIELDS ....

* sort lines starting the key at the second character of the 2nd field

sort -k2.2 data

* sort using characters 2 to 8 as the alphabetic sort key

sort -k2.2,2.8 data

OTHER

TOOLS ....

* tsort is a mysterious tool

tsort

* sort text files by the number of lines which they contain, reverse order

TYPESETTING

TEXT DOCUMENTS

This section explains how to prepare documents to be printed
on paper. This is actually a really large topic. See also the
latex-book.txt file for detailed information about the Latex
typesetting system.

OVERVIEW

....

This section tries to give an overview of the numerous tools available
to produce 'typeset' documents in a format ready to be sent to a
printer (such as pdf or postscript).

@@ markdown
The user writes a plain text document adhering to certain
formatting rules ... and markdown produces a formatted html
document
@@ halibut
The user embeds simple 'markup' codes into the text document
and the tool can convert the document into a variety of
output formats including pdf
@@ reStructured text
another philosophy similar to markdown

GREP

Grep is the classic and venerable unix tool for searching for text
within files.

* count how many times a string appears in a tree of files

grep -or string path/ | wc -l

* permanently let 'grep' colorize its output

echo alias grep=\'grep --color=auto\' >> ~/.bashrc ; . ~/.bashrc

* Search through files, ignoring .svn

grep <pattern> -R . --exclude-dir='.svn'

* Output files without comments or empty lines

grep -v "^\($\|#\)" <filenames>

* Colorize matching string without skipping others

egrep --color=auto 'usb|' /var/log/messages

* find string

grep -iR find_me ./

SED

STREAM EDITOR ....

This section provides only a very brief overview to the sed stream editor
attempting to cover only the most common tasks which may be performed
with sed. For more information see the booklet 'sed-book.txt'

PERL

Perl is a language which was originally inspired by the Bash shell
syntax, as well as by the idea of writing terse but powerful programs.
Perl initially rose to fame through its suitability for writing web-server
cgi scripts, since perl, like the unix shells, uses plain text as a kind
of "data interchange format".

This section only provides the briefest of introduction to perl, with
an emphasis on using short, one-line programs for carrying out
a simple task from the bash-shell.

* indent Test.java with an indent of 2 spaces, writing to standard output

indent -gnu --indent-level2 -st Test.java | less

##(the gnu style puts the curly braces underneath the previous line)

* bli is the brace indent

* indent the java code file 'original', with an indent of 2 spaces

astyle -js2 < original > new

astyle --mode=java --indent=2 < original > new ##(the same)

CONVERTING

TO SYNTAX HIGHLIGHTED DOCUMENTS ....

The following tools can convert source code in a text file to another
format, such as HTML, rtf or LaTeX, with the syntax of the source code
highlighted for easier reading. For HTML, the tool 'webcpp' seems to
produce the nicest looking output (at least with the default settings).
All the tools support a number of programming languages for the input
code file.

RECIPES

* browse the '.gif' images in the '/usr/doc/imagemagick/examples' directory

display 'vid:/usr/doc/imagemagick/examples/*.gif'

* browse through all image files in the current directory

display 'vid:*'

In the preceding example, only those files with image formats supported
by display are read and displayed.

* put the image 'tetra.jpeg' in the root window

display -window root tetra.jpeg

Use zgv to view images in a virtual console (not in X). You can use zgv

* browse the images in the current directory

zgv

* browse the images in the '/usr/share/gimp/scripts' directory,
type:

zgv /usr/share/gimp/scripts

Use the arrow keys to navigate through the file display; the red border

* view the file '/usr/share/images/mondrian-15.jpeg'
file:/usr/share/images/mondrian-15.jpeg
Notice that the given [36]file: URL only has one preceding slash,
pointing to the root directory, and not two, as in [37 http://.

* browse the images on the PhotoCD disc mounted on '/cdrom'

xpcd /cdrom

The preceding example will open two new windows -- a small xpcd command
bar window, and a larger window containing thumbnails of all PhotoCD

* view the PhotoCD file 'driveby-001.pcd'

xpcd driveby-001.pcd

NOTE

: You can also use display to view a '.pcd' PhotoCD image file (see

* resize 'phoenix.jpeg' to 480x320 pixels

mogrify -geometry 480x320 phoenix.jpeg

This transforms the original 'phoenix.jpeg' file to:
editing-images-scaling-01

* create a montage from the files 'owl.jpeg', `thrush.jpeg', and
'warbler.jpeg' and write it to `endangered-birds.png'

montage owl.jpeg thrush.jpeg warbler.jpeg endangered-birds.png

NOTE

: In this example, three JPEGs were read and output to a PNG file;

* combine two images, 'ashes.jpeg' and `phoenix.jpeg', into a new
file 'picture.jpeg'

combine ashes.jpeg phoenix.jpeg picture.jpeg

You can specify the percentage to blend two images together with the

* combine the image files 'phoenix.jpeg' and `ashes.jpeg' so that
the blended image contains 70 percent of the second image

combine -blend 70 ashes.jpeg phoenix.jpeg picture.jpeg

This command combines the two images and writes a new image file,

* make a morphed image of the files 'ashes.jpeg' and
'phoenix.jpeg', and write it to `picture.jpeg'

combine -compose difference ashes.jpeg phoenix.jpeg picture.jpeg

The result in file 'picture.jpeg' is:

* convert the JPEG file 'phoenix.jpeg' to a PNG image

convert phoenix.jpeg phoenix.png

This command converts the JPEG image 'phoenix.jpeg' to PNG format and
writes it to a new file, 'phoenix.png'.

* convert the PNM file 'pike.pnm' to non-interlaced JPEG while
sharpening the image by 50 percent and adding both a 2x2 border and
a copyright comment

convert -interlace NONE -sharpen 50 -border 2x2

* list available scanner devices

scanimage --list-devices

device 'umax:/dev/sgb' is a UMAX Astra 1220S flatbed scanner

* list available options supported by the device listed in the previous example

scanimage --help -d 'umax:/dev/sgb'

NOTE

: For all scanimage commands, specify the scanner device you want

* test the UMAX scanner listed previously

scanimage --test -d 'umax:/dev/sgb'

Debian: 'netpbm'

* make a 72 dpi scan of a color image 200 pixels wide and 100
pixels tall, using the UNIX scanner from previous examples, and
writing to a file called 'scan.ppm'

scanimage -d umax:/dev/sgb --resolution 72 -x 200 -y 100 >

* make a 300 dpi scan of a black and white image 180 pixels wide
and 225 pixels tall, using the UMAX scanner from previous examples,
and writing to a file called 'scan.pbm'

scanimage -d umax:/dev/sgb --resolution 300 --mode lineart

* extract the highest resolution from the file 'slack.pcd' and
save it to a PPM file named 'slack.ppm'

pcdtoppm -r5 slack.pcd slack.ppm

* [37]Converting PhotoCD: Converting PhotoCD images to other formats.

* convert the file 'slack.ppm' to non-interlaced JPEG, sharpen the
image, add a two-pixel by two-pixel border, and annotate the image,
type (all on one line):

convert -interlace NONE -sharpen 50 -border 2x2 -comment

* remove the "green haze" from a PhotoCD image, do the following:
+ First, open the extracted image in the GIMP (see [44]Editing
Images with the GIMP).
+ Then, click through the Image menu to the Colors submenu and

VIDEO

LOG

FILES

In Unix and Linux 'log' files are used by software to record things
that it has done or happened to it. In particular, server software
makes great use of log files. These log files are always stored as
plain text.

* convert all of the '.DOC' Word files in the current directory to LaTeX files with maximum line widths of 40 characters

word2x -f latex -w 40 *.DOC

* convert the Word file 'resume.doc' to a plain text file called 'resume'

word2x -f text resume.doc resume

* search the text of the Word file 'resume.doc' for the string 'linux' regardless of case

word2x resume.doc - | grep -i linux

POSTSCRIPT

DOCUMENTS

Postscript is a document format very well suited to being printed.
However it is more difficult to view than the adobe pdf format
(since the acrobat viewer is more widely installed on Microsoft
Windows computers)

Note: if the pdf file contains page numbers, then the -trim
function will probably not work as well as you were hoping.

* Convert PDF to JPEG using Ghostscript

gs -dNOPAUSE -sDEVICE=jpeg -r144 -sOutputFile=p%03d.jpg file.pdf

PDF

TO POSTSCRIPT ....

* make a copy of 'file.pdf' in postscript format called 'file.ps'

pdftops file.pdf

PDF

TO TEXT ....

* convert a pdf document to plain text (without any formatting)

pdftotext file.pdf ##(a file called "file.txt" is created)

* convert the pdf file "file.pdf" to a plain text file "ouput.txt"

pdftotext file.pdf output.txt

ps2ascii file.pdf output.txt ##(more or less the same)

* pstotext

FROM

OTHER FORMATS ....

* convert a web page into a pdf

touch $2;firefox -print $1 -printmode PDF -printfile $2

THE

INFO HELP READER

* Nice info browser

pinfo

The info reader.

For some incomprehensible reason, the www.gun.org people decided that they
didnt like "man" pages and so they invented a whole new help document reader
called "info". Its a nasty little second rate hypertext viewer which forces
you to learn a whole new set of arbitrary keystrokes in order to move around
the document (maybe they're "emacs" keystrokes, who knows). Whats more,
instead of just calling "pages" pages, info calls pages "nodes" to try and be
as pretentious as it possibly can. But anyway, you may have to use it.

* view the "info" help document for Latex

info latex

Some "info" reader commands
.. [space], move forward a page
.. [del], move back a page, or to the previous node if at the top of the page
.. q, exit the Info document viewer
.. ?, show some keystrokes which are available
.. h, show an "info" tutorial, if available
.. [tab], go to the next link on the page
.. [enter], jump to the link under the cursor
.. ???, how to go to previous link is anyones guess [M-TAB] ???
.. l, go to the last page that was viewed (like a browser "back button")
.. n, open the next page in the document
.. p, open the previous page in the document
.. u, go "up" one level in the document (usually to a table of contents)
.. t, go to the main table of contents for the document
.. d, view a list of all Info documents
.. b, go to the top of the current page
.. e, go to the end of the current page
.. m, type the name of a link to jump to (can type partial names)
.. i, type the name of any node to jump to
.. /, search for a string in the current page
..

XML

EDITING

TEXT

In the traditional unix world, the choice of text editor
is often seen as a choice between vim and emacs. But many other
good choices exist.

@@ tea - ?
@@ sam
An editor written by Rob Pike which is in turn used
by various veteran unix forefathers. Sam makes more use of
the mouse.
@@ acme
Another editor by Rob Pike.
@@ joe 'joes own editor'
possibly the simplest text editor to use ...
@@ nano, pico
small reasonably easy to use text editors originating from
the pine email program.
@@ vim 'visual editor improved'
this is a programmers' editor for people who write a
great deal of text and preferibly can type without looking
at the keyboard. It is strange and annoying for new users.
@@ emacs 'editing macros'
capable of just about anything but dont use it, because I think
its bad.

* graphical text editors
.. gedit
.. kate
..

BATCH

EDITING TEXT FILES ....

Batch editing of text files involves editing several or many text files
at once, usually in a non-interactive way.

use vim with the "argdo" function

use perl with the "-i" switch

some versions of "sed" do not have the "-i" switch, so perl must be used

## Before doing a "batch" edit of text files, it is probably a good
## idea to check what changes will take place without doing anything

a few common file-system types
.. fat32 - created 1996, by microsoft, appeared in windows 95b
.. vfat - ?
.. ntfs v5.1 - created 2001 by microsoft and used in windows xp
.. ntfs v6 - 2006 by microsoft for windows vista
.. ext2 - created in 1993 and used by linux and hurd
.. ext3 - created 1999 used in linux
.. reiserfs - created 2001 by namesys, used in linux
.. google file system - created 2003 by google for linux
.. gfs2 - 2006 by red hat for linux
.. ext4 - 2006 for linux
..

* view all mounted file system types

df -T

MOUNTING

AND UNMOUNTING FILESYSTEMS ....

* Mount proc

mount -t proc{,,}

* currently mounted filesystems in nice layout

mount | column -t

* Mount a temporary ram partition

mount -t tmpfs tmpfs /mnt -o size=1024m

* Mount folder/filesystem through SSH

sshfs name@server:/path/to/folder /path/to/mount/point

* Mount a partition from within a complete disk dump

lomount -diskimage /path/to/your/backup.img -partition 1 /mnt/foo

* umount all nfs mounts on machine

umount -a -t nfs

* mount directories in different locations

mount --bind /old/directory/path /new/directory/path

SYMBOLIC

LINKS ....

* Find broken symlinks and delete them

find -L /path/to/check -type l -delete

* Find broken symlinks

find . -type l ! -exec test -e {} \; -print

* Propagate a directory to another and create symlink to content

lndir sourcedir destdir

* show the status of all symlinks in the current folder

symlinks -r $(pwd)

* List all symbolic links in current directory

find /path -type l

* Find dead symbolic links

find . -type l | perl -lne 'print if ! -e'

* Show the disk usage for files pointed to by a symbolic link

find /usr/lib -maxdepth 1 -type l -print0 | xargs -r0 du -Lh

* Eliminate dead symlinks interactively in /usr/ recursevely

find /usr/ -type l ! -xtype f ! -xtype d -ok rm -f {} \;

* Get the canonical, absolute path given a relative and/or

readlink -f ../super/symlink_bon/ahoy

DIRECTORIES

Directories, which are also known as 'folders' are a way of
organising files in a heirarchical kind of way.

SEARCHING

FOR FILES AND DIRECTORIES

This section covers techniques to search for and find files and
folders on a Unix system. This mainly involves using the "find"
program from the command line. This program is flexible and useful,
but its syntax can initially seem to be unnecessarily complicated.
For example "find filename" will not work.

The 'find' tool has an enormous number of options and operators to
construct truly abstruse find commands.

* create a compressed archive of the folder 'dir' in file 'dir.tar.gz'

tar -cvz --exclude *.exe -f dir.tar.gz dir

##(excludes all filenames ending with '.exe')

* Compress files found with find

find ~/bin/ -name "*sh" -print0 | xargs -0t tar -zcvf foofile.tar.gz

* Create a zip archive excluding all SVN folders

zip -r myfile.zip * -x \*.svn\*

ARCHIVES

OF FILES

A file archive is a way to combine many different files within
one file, optionally compressing (reducing the size) of each file
included in the "archive" file. The purpose of creating file archives
is to facilitate making backups of files or for transfering files from
one computer to another.

SCP

....

The secure copy program is allegedly faster than 'sftp' mode, and uses
ssh encryption. For transfering entire folders the 'rsync' program should
probably used instead. Scp appears incapable of transfering 2 different
files to 2 different folders. For that, see sftp. Without the '-r'
switch Scp wont transfer folders.

* transfer 'book.txt' to the books folder of a sourceforge site server

scp books.txt user,project@web.sf.net:htdocs/books/

* upload files 'f' and 'g' into the webfolder on a sourceforge server

scp f g user,project@web.sourceforge.net:htdocs/

* download files 'f' and 'g' from a sourceforge server to this folder

scp user,project@web.sourceforge.net:"htdocs/f htdocs/g" .

* upload (overwriting) all files in the current folder to the folder
* 'doc/books/' to the 'eva' account on the server 'far.net'

scp * eva@far.net:doc/books/

* upload and overwrite all '.txt' files to the 'doc/' folder on far.net

SFTP

....

sftp transfers files using the encrypted ssh (secure shell) protocol
and is thus 'safer' and more modern than 'ftp'. sftp is designed to
be used interactively by the user, but can also run in batch mode
with the -b switch

* using sftp in batch mode to upload to the sourceforge web server

sftp -b a.txt user,project@web.sf.net > /tmp/sftp.log

the file 'a.txt' contains 'cd books; put chap1.txt; bye;'

RSYNC

Rsync is a very powerful network file copying program. It is particularly
nice because it only transfers the parts of the files which have changed
(the 'difference' to the old files). This saves a great deal of bandwidth
and is probably the most efficient and pleasant way to develop a website.
It is a program with a very large number of options to fine tune the way
the file transfers take place.

Note: If the source path ends with "/" then the contents of the folder are
copied but the folder itself is not

rsync tools
.. grsync - a very basic graphical version of rsync with gtk
..

GRAPHICAL

TEXT

MODE WEB SURFING ....

Text-mode web surfing means using a program running from a 'console'
'terminal' or 'command-line' to view web-pages. This generally means
that it is not possible to view the images contained in the web-pages,
only the text. While this is aesthetically inferior, it is faster and
sometimes more useful and less distracting.

for i in $(cat listofservers.txt); do konsole --new-tab -e ssh $i; done

* browse files on an ssh server

mc

EXPECT

Expect is a useful tool to allow the scripting of tasks
which would normally require some user interaction. Expect
includes its own simple scripting language. Expect is available
on the majority of unix type operating systems, including
Mac OSX.

Entire books exist about expect, Expect is an extension to
the 'tcl' language.

PROXY

MAC

ADDRESSES ....

A 'mac' address is a unique number associated with a given
network interface on a computer. A network interface is a wireless card,
ethernet card, modem, etc. No 2 network devices in the world have
the same mac address, or at least shouldnt.

the 2 keys
.. your public - other people use it to send you stuff
.. your private - decrypt things sent to you
.. other persons public - you use it to send things to them
.. other person private - they use it to decrypt things sent to them

* see if gpg is installed

which gpg

which gpg2 ##(the newer version)

* make a public key for somebody to send you stuff

gpg --gen-key

input name and email address

The new key gets put on a key-ring

* see what keys are on your keyring (like the one you just made)

gpg --list-keys

* show the fingerprint for a key

gpg --fingerprint

'fingerprints' are used to make sure that the public key for
somebody really is from them.

* export the key with the given id to a file 'key.asc' and send to people

gpg --armor --export 86d68542 --output key.asc

* send the public key with given id to the mit 'public key server'

gpg --keyserver=x-hkp://pgp.mit.edu --send-keys 86d68542

or use the web interface at http://pgp.mit.edu to submit a public key
paste the '.asc' file created before with --armor

* import a key file containing somebodies public key

gpg --import key.asc

Then you use this key to encrypt things which you are going to
send to them.

* search the 'mit' public key server for a public key belonging to Ben stewart

gpg --keyserver=x-hkp://pgp.mit.edu --search-keys Ben Stewart

Then enter a number corresponding to the desired user to enter that
key into your keyring

KEYRING

....

* private keys normally stored in

/home/user/.gnupg/pubring.pgp

* public keys stored in

/home/user/.gnupg/secring.pgp

SIGNING

KEYS ....

In order to verify that a public key really is from somebody, signing
is used. The more genuine people sign a key as authentic the more likely
it is that it will be authentic

* sign the public key for 'bob'

pgp --edit-key bob@stew.net

Then send this signed key to the public key server with the same command
as before.

* list who has signed the public key for 'bob stewart'

gpg --list-sigs bob@stew.net

ENCRYPTING

....

* encrypt 'file.pdf' for sending to user bob@stew.net (using his public key)

gpg --encrypt --recipient 'bob@stew.net' file.pdf

creates a file 'file.pdf.gpg' which is encrypted

* encrypt a file for yourself (just use your own public key to encrypt it)

gpg --encrypt --recipient 'me@here.net' file.pdf

DECRYPTING

....

* decrypt a file sent to you (uses your private key)

gpg --output f.txt --decrypt f.txt.asc

this prompts for your keyring passphrase

* create a file 'leaks.txt.asc' which is signed as being from you but not encrypted

gpg --clearsign leaks.txt

* verify that a 'signed' file is from the person who it says it is

gpg --verify leaks.txt

* create a revocation certificate which is useful if you forget a password

gpg --gen-revoke 8657698

Send the revocation certificate to the public key server if you forget
a password for a key

CHANGING

SERVICES

'services' are a special type of process which are generally
always running and are often started when the computer starts up.
Examples of services are a web-server, an ftp server ...

@@ debian: sysvconfig
allows a user to use the redhat-style 'service' command

* show what services are available on the computer

ls /etc/init.d

* start or stop a service on a debian-style linux

sudo /etc/init.d/servicename start|stop

* restart the 'sshd' (secure shell) service

sudo /etc/init.d/sshd restart

* another way

update-rc.d ... invoke-rc.d

* stop the 'apache' service (with the sysvconfig package, or on redhat style)

sudo service apache stop

* other tools: rcconf, update-rc.d

STARTING

SERVICES AT COMPUTER START UP ....

Linux has a concept of 'run-levels' which are stages the
computer reaches as it boots up, and as it shuts down. The higher the
level the more 'booted-up' is. One can configure a server to start
at any particular 'run-level'.

* configure apache to start up when the computer starts

update-rc.d apache2 defaults ##(this is debian-specific)

update-rc.d apache2 start 20 2 3 4 5 . stop 80 0 1 6 . ##(the same)

* disable the apache webserver from starting when the computer starts

update-rc.d -f apache2 remove

* create automatic startup links manually (this is the older way)

cd /etc/rc5.d/

ln -s /etc/init.d/apache2 S20apache2

# when the computer enters run-level 5 (rc5.d) is will start (S) the
# 'apache2' service with a priority of '20', that is, before all other
# services which have a priority number greater than 20

PIPES

The unix pipeline is possibly the most important concept
in the unix world

* cat large file to clipboard with speed-o-meter

pv large.xml | xclip

PROCESSES

At the heart of a unix-type operating system is the concept of the
'process'. Processes allow the computer to do more than one thing
at a time.

Processes are essentially running programs (applications, software).
Some applications, when they run, are visible because they use
a 'window' or they display information or data on the command-line.
However other processes are 'invisible'; that is, they are running, but
you as the user doesnt see any visible activity.

KILLING

PROCESSES ....

In order to stop a running program (which doesnt have a window) it
is necessary to 'kill' (or stop) the associated process. In order
to do this first it may be necessary to find out the process
identification number ('pid') of the running application.

* start and then kill the 'leave' process using its exact name

leave +20; killall -9 leave; ps

The command above will permanently stop ('kill') all processes which
are named exactly 'leave', that is, all processes which were started
by the 'leave' reminder program.

* make a bash function 'psk' to kill processes with 'ave' in their name

* add a folder to the executable path (put in ~/.bash_profile file to save it)

export PATH=/path/to/folder:"${PATH}"

##(programs in this folder can then be executed with 'programname')

* make all terminals write to the same history file

shopt -s histappend ##(put in .bashrc, on single user systems)

* enable changing to a folder by typing only the folder name (not path)

export CDPATH='.:~:/some/folder:/path/to/folder'

##(place this in .bashrc with commonly used folders)

TIMING

PERFORMANCE

* time how long a "grep" command takes to execute

time grep -rl big *

* time the execution time of 2 commands at the same time

time { find / -name '*what*'; locate '*.cc' ; }

SCHEDULING

tools
.. batch -
.. cron - a scheduling tools
.. crontab - a file of scheduled tasks
.. at - run a command at a certain time
.. nice - run with a certain priority
..

AT ....

* retrieve the page 'elpais.com' at 12:00 oclock

echo 'wget http://elpais.com' | at 12:00

* Schedule a script or command to run in 2 hours

( ( sleep 2h; your-command your-args ) & )

* Execute a command at a given time

echo "ls -l" | at midnight

* Run a command only when load average is below a certain threshold

echo "rm -rf /unwanted-but-large/folder" | batch

CRON

....

Cron is the traditional unix tool for scheduling tasks; that is for
instructing the computer to automatically carry out a task at a given time,
or at given intervals, without the user explicitly starting the task.
Each task is placed in the 'crontab' file.

lshw -html > hardware.html ##('hardware.html' can be viewed in a browser)

* show the linux kernel version

uname -r

* show kernel startup messages

dmesg

* show usb devices

lsusb

MODULES

AND DEVICE DRIVERS

In Linux, 'device drivers' (which make bits of hardware work) are also
known as 'kernel modules'. If a piece of hardware is not working, then
usually a kernel module must be loaded. This may involve; Find the
product code for the device (looks like 0341:4561). Then find the module
for that product. Then download the module source, compiling the module
and installing it.

VERSION

CONTROL SYSTEMS

A version or revision control system is designed to keep track of
different versions of a text document, or other file or files.
These systems are often employed when writers or programmers want
to be able to revert to a previous version of a document or code file.

CVS

....

* Override and update your locally modified files through cvs..

cvs update -C

* List only locally modified files with CVS

cvs -Q status | grep -i locally

SUBVERSION

....

Subversion is a reasonably modern versioning system which was
supposed to replace 'cvs'.

This will back up the file 'eg.txt' to 'eg.txt.13may2010.10-45am' for
example where the filename used represents the current date and time.

* a function to back up a file with a date-time stamp name

bak(){ [ -z "$1" ] || cp $1{,.$(date "+%d%b%Y.%I-%M%p")}; }

SETTING

THE SYSTEM DATE AND TIME ....

There are a number of ways to set the date and time on the computer.
One way is to set the time just locally. Another way is to use a
network time server to set the time, which should be more accurate in
the long run

* advance the clock by 3 minutes

date -s '+3 mins'

* set the hardware clock

hwclock

* set the system date

rdate -s time-A.timefreq.bldrdoc.gov

* use a network time server to set the date

man ntp

CONVERTING

TIME AND DATE FORMATS ....

Date and times come in many different text formats, depending
on the country and whim of the date-time writer. This is very
thorny issue, but often must be dealt with. The problem is confounded
by cultural differences and ways of measuring time.

Unix-type operating systems such as Linux and Mac OSX have traditionally
used the number of milliseconds since 1970 as a universal date-time
format, but this has obvious limitations.

* Unix time to local time

date -R -d @1234567890

CONVERTING

DATE AND TIME TIMEZONES ....

If you need to know the equivalent date and time in a different
timezone, then you will have to do a conversion.

UNIX

Unix time is a number which represent the number of seconds since 1970-01-01
00:00:00 UTC

* Easily decode unix-time (function)

utime(){ perl -e "print localtime($1).\"\n\"";}

* show the number of seconds since 1970

date +%s

* Unix alias for date command that lets you create timestamps

alias timestamp='date "+%Y%m%dT%H%M%S"'

CONVERT

TO UNIX TIME ....

It can be useful to convert to unix-time when arithmetic needs to be performed
on dates or times, or when dates need to be sorted. The 'date' command does a
very good job of parsing diverse date strings (such as feb 7 2009 or
01/11/2001) with its '-d' option, but it has its limitations

* convert a date string to the number of seconds since 1970

date -d "Sat Feb 7 00:37:06 EST 2009" +%s

* convert tomorrows date (midnight, I suppose) into unix-time

date -d'tomorrow' +%s ##(prints something like '1264168894')

* print the final unix date

date -ud @$[2**31-1]

* convert the date 'february 7, 2009' into unix-time

date -d "Sat Feb 7 00:00:00 EST 2009" +%s

date -d "feb 7 2009" +%s

date -d "7 feb 2009" +%s ##(the same)

date -d "february 7 2009" +%s ##(the same)

date -d "7/2/2009" +%s ##(the same, expects mm/dd/yyyy)

date -d "7/2/09" +%s ##(the same)

* these formats do NOT work (gnu date version 7.4)

date -d "feb/7/2009" +%s ##(NO!! doesnt work, at least for me)

date -d "7/feb/2009" +%s ##(Nor does this work!)

date -d "feb 2009" +%s ##(Nor does this work!)

* convert the date 'Dec 11 01:25:00 2008' to a unix time value

perl -e 'use Time::Local; print timelocal(0,25,1,11,11,2008), "\n";'

##(should print '1228919100')

CONVERT

FROM UNIX TIME ....

* convert unix time (seconds since 1970) to something human readable

date -d @1234567890

* convert unix-time (seconds since 1970) to something more human-readable

UNIX

PROGRAM NAMING ....

The names for Unix programs appear cryptic and seem to have no relation to
their function. However there is usually an explanation for the name. Unix
commands are traditionally very short to save typing.

GENEOLOGY

BLOG

JARGON

@@ easter egg
Is an unexpected function hidden in a piece of software
@@ Unix-style operating system,
This is an operating system which is somewhat 'unix-like'
an included Apple OSX, Linux, FreeBSD, etc
@@ Linux flavour, Linux Distribution
Linux comes in a number of different distributions, such as
Debian, Fedora, and Ubuntu to name only some of many.
@@ regular expression
A search pattern
@@ recursive
Recursive means to do an action for some object, and then
repeat the same action for all objects which that object
contains.
@@ concatenate
join together two or more things. For example join together
2 files with 'cat a.txt b.txt > c.txt'
@@ command line
@@ shell
A shell is a piece of software which allows you to communicate
with the computer by typing (rather cryptic) commands, and
watching the output of those commands. This book focusses on
the 'Bash' shell, which is probably the most popular shell
on Unix, Linux and Apple OsX.
@@ terminal
The terminal is a piece of software which allows you to use
the 'shell'. Examples are 'Konsole' for KDE
@@ program
A program is really just a piece of software which does
something on the computer.
@@ parse, parsing
parsing is the process of analysing and splitting into its
structure a text pattern. This is a very important task on all
computer systems, since a great deal of data processing involves
the parsing of one text pattern, and then the transformation of
that pattern into another format.
@@ thread
@@ daemon
@@ process
These are similar but important concepts

printing jargon
@@ pretty-print
This refers to the process of preparing some piece of text
automatically in some way which is pleasing to read once printed
on a piece of paper or on a computer screen. The term is
ambiguous since it can refer to the application of colour to
a text file or to the changing and formatting of indentation and
paragraph width

NOTES

AND IDEAS

This section should contain references to things which I
think would be worthwhile investigating but which I havent
and may well never get round to doing.

ORGANISATION

.. zim - note taker
.. tomboy - note taker
.. fzapper
..

DOCUMENT

-NOTES:

# this section contains information about the document and
# will not normally be printed.

# a small icon image, filename relative to the "htdocs" folder
document-icon: /img/icon/linux-penguin.png
#
document-image:
# what sort of document is this
document-type: book
# in what kind of state (good or bad) is this document
document-quality: good
# when was this document last updated
last-revision: jan 2010

# a history of work on the document
document-history:
@@ 2009
document begun
@@ jan 2010 revision
@@ 5 jan 2010
incorporated rewritten recipes from dsl.org
@@ 8 jan 2010
got 4032 recipes from commandlinefu.com
@@ early jan 2010
more detailed enscript description. This enscript chapter
has now (april 2010) been moved to the 'linux-documentation-book.txt'
although a few enscript recipes remain.
@@ 24 jan 2010
added information about the 'halibut' and 'markdown' text
transformation systems.
@@ 25 jan 2010
removed a large quantity of unclassified recipes which had
been incorporated from 'commandlinefu.com'
@@ 26 march 2010
removed linux sound chapters into a separate book since it
was getting to big
@@ 27 march 2010
some general tidying up, classifying recipes. The task of
cleaning and verifying all the commandfu recipes is a
big one
@@ 30 march 2010
added some zenity info
@@ 2 april 2010
continuing with the rather overwhelming task of attempting
to analyse all the recipes from commandlinefu.com and
classify and weed them out.
@@ 21 april 2010
some tidying up. wrote a vim command (:Bsi) to create screenshots of
oneline commands and save them in the 'image' subfolder. Also the
':Bci' command just creates and displays a test image without saving it
into the image folder. When building with the 'booktolatex.sed',
'booktolatex.cgi' scripts, we still have the problem that right aligned
images 'push' the recipes out of alignment.
@@ 6 may 2010
a little clean up of the 'introduction to the command line'. I
had the idea of writing a 'culture shock' section for Microsoft
users who are coming to Linux.

# work that needs to be done on this document
to-do:
A great deal of work needs to be carried out
@@ enscript
reorganise the enscript recipes which came from the
linux cook-book.