Powerhouse Programs of Linux

Wed, Apr 29, 2009

rsync

There are hundreds of different backup applications out there. Rsync is one of those programs what once you learn how to harness its power you will never even think of backing up using anything else. Rsync is a staple utility that is used in almost all professional backup solutions. Not only is it the preferred method of backing up large amounts of data over a slow network, but also so efficient that it is the preferred way for transferring files to Amazon’s S3 service. When you pay for bandwidth by the bit you cannot afford anything but rsync.

Common Uses

dd

Not only is dd a court recognized bit for bit hard drive forensic imaging utility, it is also a great tool for making your own backups. Dd has proven itself time and time again. You can use it to copy CDs and other media. You can also use it to move data to another hard drive or cat out strings stored in RAM.

Common Uses

grep

Grep, Global Regular Expression Print, is an incredibly useful utility for finding strings of text in a folder full of files. Getting its roots directly from Unix this command has some tricks up its sleeves. Grep is often times paired with other commands and connected using the pipe, ‘|’.

Common Uses

Search history for the ‘mount’ command

history | grep mount

Find the process named, ‘firefox‘

ps -A | grep firefox

This searches all files and folders in the current directory recursively looking for, ‘some string’.

grep -r “some string” *

Find out what service uses port 25

cat /etc/services | grep 25

find

Find is an awesome utility for finding files. By itself it isn’t really that great but when you pipe the file output to other commands you can do amazing things. This is one of those commands that can increase your productivity 10 fold.

Common Uses

xargs

Build and execute command lines from standard input. Xargs is a tool in which you a lot of complex commands are based on. You can use xargs to execute a specific command using multiple file names. Xargs is often used with the find command.

Common Uses

Delete more than one month old thumbnails from home directory

find ~/.thumbnails/ -type f -atime +30 -print0 | xargs -0 rm

history

History of commands

Common Uses

Replace a string in the previous command

^typo^correction^

Search previous commands for strings that match ‘mount‘

history | grep mount

alias

Make your own commands. Alias is a great way to take an otherwise long command and map it down to something of your own. Oftentimes SSH connections are mapped to smaller commands as well as others.

Common Uses

gzip

Gzip is a very fast compression that is often used to compress HTML and CSS data on the web. It is also used to compress files on a Linux machine most notably log files. Take a peak in ‘/var/log’ to see what i mean.

Common Uses

Taking the dd command that we talked about earlier, we can backup an entire hard drive into a gzip file.

dd if=/dev/hdz1 | gzip -9 > /mnt/backup.gz

ssh

SSH is a secure remote login program and is absolutely awesome. I use ssh to login to remote servers every day. You can also use it to create encrypted tunnels for transferring data and also NAT traversal using reverse tunnels.

Common Uses

Login to a remote host

ssh user@host.com

Create a reverse tunnel

ssh -R 10002:localhost:22 middleuser@middle

scp

Secure file copy. I am usually using ssh to do most of my work and I use scp every time I need to move files remotely. No longer do I use the insecure protocol, FTP.

Common Uses

scp user@remote:/path/to/remote /path/to/local

tail

Output the last parts (tail) of a file(s)

Common Uses

View the last kernel messages (helpful to detect boot problems)

dmesg | tail

head

Like tail this ouputs part of a file. Head outputs the beginning of a file.

Common Uses

Print the first few lines of ‘filename.txt’

head filename.txt

Print the last modified file

ls -t1 | head -n1

ln

Make Links between files

Common Uses

Create a symbolic link

ln -s file/name linkName

curl

cURL seems to be the defacto standard application for integrating online web service APIs. I have personally used cURL to do some pretty cool stuff with my own PHP code as well as local bash/perl scripts. It is often used to emulate a web browser in automation scripts.

wget

wget is like the command line version of cURL. There is no library for wget and is usually not integrated into programming languages directly like cURL but is definitely a powerful tool to have in your arsenal.

Common Uses

wc

wc is a simple app that packs a handy feature. It has the ability to output counts for: bytes, chars, lines, longest line, words. You can even give it a list of files to check by using the, ‘–files0-from’ flag.