dd Cheat Sheet

dd is one of the most versatile IO tools available for Linux. It’s used in a variety of ways ranging from Disk Benchmarking through to creating SWAP files and copying downloaded disk images to physical disks.

dd takes the following common switches:

if is the input file name and location.

of is the name and location of the output file.

bs is the block size that will be used to read and/ or write the file. Increasing this can help with performance or dictate how much data will be read or written.

count is the number of blocks that will be used.

seek is the number of blocks on the output file that will be skipped before writing any data.

skip is the number of blocks that will be skipped on the input file before starting to read data.

conv is a comma separated list of additional parameters that can be used. See the man dd for more information.

The below headings will list a few example uses of dd in a typical Linux environment.

Backup disk partition with dd

You can use dd to copy an entire disk partition to a virtual disk file. This can be useful for creating a backup or to clone the disk to another machine.

1

ddif=/dev/sda1 of=~/localdisk_sda1.img

You can use this method to read a CD-ROM, USB drive or Flash disk to a file in the same way – just make sure the device is inserted and point the if= part of the dd command to the relevant /dev/ device.

You could also compress the image as part of the process with gzip.

1

ddif=/dev/sda1|gzip-c>~/localdisk_sda1.img.gz

Restore disk partition with dd

Similar to the above command, you can use dd to replace a disk’s partition with a virtual disk file.

1

ddif=~/localdisk_sda1.imgof=/dev/sda1

If you compressed the image then you can decompress it first all in one go:

1

gunzip-c~/localdisk_sda1.img.gz|ddof=/dev/sda1

Create a fixed size file with dd

You can create a fixed size file with DD that will be created in the location you specify.

1

ddif=/dev/zero of=/root/testbs=1024count=1

This will create a file in /root/test of 1024 bytes in size. Increase either bs or count to change the size of the file. The resulting size will be bs * count. You can also use shorhand sizes such as K, M and G with bs, for example bs=1G,

1

dd if=/dev/zero of=upload_test bs=file_size count=1

Create a SWAP file with dd

dd can be used to create a SWAP file that can be used as a SWAP device by your computer. This is often needed with smaller instances on Cloud providers such as AWS.

The starting point is the same as the above command to create a file with the size that you’d like to use for swap. See my other blog post for more info.

Split a file with dd

dd can be used to read just part of a file, given offset and length coordinates. The below example will skip the first 100 bytes of the file and output the proceeding 10 bytes (byte 101 – 111).

1

ddif=filetosplit of=partfile bs=1count=10skip=100

You could repeat this process to split a large file into multiple smaller files, to be able to email it for example.

1

2

3

dd if=filetosplit of=partfile1 bs=1count=100

dd if=filetosplit of=partfile2 bs=1count=100skip=100

dd if=filetosplit of=partfile3 bs=1count=100skip=200

Merge multiple files with dd

You can merge multiple files into a single file with dd. Following on from the above split example, the below will rejoin the 3 file parts into a single file.

1

2

3

ddif=partfile1 of=joinedfile bs=1count=100

ddif=partfile2 of=joinedfile bs=1count=100seek=100

ddif=partfile3 of=joinedfile bs=1count=100seek=200

Convert text to lower case with dd

You can use the conv switch with dd to transform ascii text from upper case to lower case and visa-versa. Using lcase and ucase in the conv switch will instruct dd to convert the text as it’s written.

The below example will convert all characters in the filetoconvert.txt. file to lower case.

Install OpenSSL

Create Private Key

The last argument in the below line is the key length. This can be changed to 2048 or 4096 if required for better encryption.

1

openssl genrsa-des3-out server.key1024

Generate a CSR (Certificate Signing Request)

You will be asked for the details of the certificate such as domain name and address when running this command.

1

openssl req-new-key server.key-out server.csr

Remove Passphrase from Key

Some applications do not allow for the private key to have a passphrase. The below commands will remove the passphrase – be careful as it will mean the key is no longer protected and can be viewed by anyone with read access to the file.

1

openssl rsa-inserver-with-passphrase.key-out server.key

Generating a Self-Signed Certificate

Once you have generated a key and CSR you will need to sign the request and generate the public certificate. If you do not have a certificate authority you can sign the certificate yourself. The below will generate a certificate which is valid for one year.

Debian and Ubuntu use the service command to control services and update-rc.d for adding and removing services from start up. Using the service command we can start, stop, restart and display all available services. With update-rc.d we can add and remove services and add them to the Ubuntu/ Debian start up scripts. As Linux operating systems have multiple states, or runlevels, you need to make sure you add any new services to the correct runlevels. For example, you would not want to start a web service application before starting networking.

What services are available for startup?

Use the status-all switch to list all services which are registered with the OS and issues them a status command. You will then get one of the following displayed next to each service:

[ + ] – Services with this sign are currently running.

[ – ] – Services with this sign are not currently running..

[ ? ] – Services that do not have a status switch.

1

service--status-all

Sample output:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

service--status-all

[?]acpid

[-]apparmor

[?]apport

[?]atd

[-]bootlogd

[?]console-setup

[?]cron

[?]dbus

[?]dmesg

[?]dns-clean

[?]friendly-recovery

[?]glusterfs-server

[-]grub-common

[?]gssd

[?]hostname

[?]hwclock

[?]hwclock-save

[?]idmapd

[?]irqbalance

[?]killprocs

[?]module-init-tools

[?]network-interface

[?]network-interface-container

[?]network-interface-security

[?]networking

[?]ondemand

[?]passwd

[?]plymouth

[?]plymouth-log

[?]plymouth-ready

[?]plymouth-splash

[?]plymouth-stop

[?]plymouth-upstart-bridge

[?]portmap

[?]portmap-wait

[?]pppd-dns

[?]procps

[?]rc.local

[?]resolvconf

[?]rpcbind-boot

[-]rsync

[?]rsyslog

[?]screen-cleanup

[?]sendsigs

[?]setvtrgb

[+]ssh

[?]statd

[?]statd-mounting

[-]stop-bootlogd

[-]stop-bootlogd-single

[?]sudo

[?]udev

[?]udev-fallback-graphics

[?]udev-finish

[?]udevmonitor

[?]udevtrigger

[?]ufw

[?]umountfs

[?]umountnfs.sh

[?]umountroot

[-]unattended-upgrades

[-]urandom

[?]whoopsie

Start a service

Starting a service is done using the command service followed by the service name and the command to start the service.

1

service apache2 start

Stop a service

Use the stop keyword with service to stop a service.

1

service apache2 stop

Check the status of a service

Each service has a status, usually running or not running. Some services, such as network, may have a different output and output more information on the service.

1

service apache2 status

Remove a service

Use the remove keyword with update-rc.d to remove the service start up command for an application. You will need to use the -f switch if the applications /etc/init.d start up file exists.

1

update-rc.d-fapache2 remove

Add a service

Adding a service to Ubuntu or Debian is done with the update-rc.d command. You can specify which runlevels to start and stop the new service or accept the defaults. The init.d file will be added to the relevent rc.d startup folders.

1

update-rc.dapache2 defaults

Setting Start and Kill priority

If you need more control over when your service is asked to start and stop, you may need to set the start and kill (S and K) values.

For a given runlevel, you may have several services starting. For example, you may have apache2 and mysql both starting on runlevel 3. Ideally, you’d want mysql to start before apache2 and shutdown after apache2. In this case we need to give mysql the priority in startup, but apache2 the priority in shutdown.

When starting, the lower the number, the earlier it will start. A start priority of 10 will start before a priority of 20. When killing, it’s the opposite. A higher number will be killed before a lower number.

To set the start and kill priority we simply add them to the above update-rc.d command with the start priority first, followed by the kill priority.

1

update-rc.dapache2 defaults[START][KILL]

The below command will start mysql first, then apache2. On shutdown, the kill will be the reverse of the start with apache2 being killed first and mysql second.

1

2

update-rc.dapache2 defaults9090

update-rc.dmysql defaults1010

Because, in our example, both start and kill priorities are the same we can shorted the command to just

1

2

update-rc.dapache2 defaults90

update-rc.dmysql defaults10

Manually set the RunLevel to Start and Kill a service

You can manually specify the Linux RunLevel that the system must be in to Start and Kill your service. See my other blog post for more information on RunLevels.

To extend the above example, we can specify exactly which RunLevel apache2 will be started and stopped.

1

update-rc.dapache2 start102345.stop90016.

apache2 will be started (as long as it isn’t already) when the system enters RunLevel 2, 3, 4 or 5 with a priority of 10. It will then be asked to stop when the system enters RunLevel 0, 1 or 6 with a priority of 90.

Most Linux applications create log files that need to be managed by either archiving or deleting old log files. This process is called log file rotation. The most common log rotation utility for Linux is conveniently called logrotate. logrotate is configured using the main configuration file, or the logrotate configuration folder. The configuration file usually holds the global configuration and the pointer to the configuration folder. In common Linux distributions such as Ubuntu and Debian, the logrotate configuration file can be found:

1

/etc/logrotate.conf

And the configuration folder can be found:

1

/etc/logrotate.d/

The configuration folder is usually where you would add new log file configurations which are unique for a particular log file, or set of log files. For example, if you were to add a new log rotation action for the Ubuntu package manager apt, you may use something like below:

1

vi/etc/logrotate.d/apt

1

2

3

4

5

6

7

8

9

10

11

12

13

14

/var/log/apt/term.log{

rotate12

monthly

compress

missingok

notifempty

}

/var/log/apt/history.log{

rotate12

monthly

compress

missingok

notifempty

}

logrotate runs each day by default, and is invoked using the daily cron job. The below subject detail the common components of creating a logrotate configuration file.

Create a new empty template

To create a new logrotate configuration file, you need to create a new file in /etc/logrotate.d/. You will then need to add a reference to the log files you wish to rotate. This can be directly to a single file or use pattern matching to match a group of log files. The below example matches all log files in /var/log/myapp/ which have a .log extension.

1

2

3

/var/log/myapp/*.log{

}

You will need to add further commands to this template before it becomes useful. Further commands will be added inside the { and } tags.

Rotate based on log file size

Use the size keyword to rotate the log file when it exceeded a given file size. The below example rotates a file when it reaches 10 KB.

1

2

3

/var/log/myapp/*.log{

size10k

}

Rotate based on time (Month, Week or Day)

You can rotate logs using the monthly, weekly or daily keyword to create a new log based on duration. The keywords explain them selves, and they can be used in conjunction with the size keyword to rotate on which ever criteria is met first.

1

2

3

4

/var/log/myapp/*.log{

size10k

weekly

}

Limit how many log files are kept after rotation by number

The rotate keyword allows you to specify how many old, rotated, log files are kept before logrotate deletes them. The rotate keyword requires an integer to specify how many old log files are kept.

1

2

3

4

5

/var/log/myapp/*.log{

size10k

weekly

rotate8

}

The above example will keep 8 old log files.

Limit how many files are kept after rotation by date

You can specify how long to keep rotated files using the maxage keyword. Any rotated log files which are older than maxage will be deleted. The below example will keep rotated log files for 56 days.

1

2

3

4

5

/var/log/myapp/*.log{

size10k

weekly

maxage56

}

Compress rotated log files

Log files which have been rotated can be compressed to save disk space. Gzip is used by default.

1

2

3

4

5

6

/var/log/myapp/*.log{

size10k

weekly

rotate8

compress

}

You can change the default gzip compression to another format by specifying the compresscmd command and a different executable to use. The below example changes the compression format to bzip2 for better file compression.

1

2

3

4

5

6

7

/var/log/myapp/*.log{

size10k

weekly

rotate8

compress

compresscmd/bin/bzip2

}

Ignore missing log files

If a log file does not exist when logrotate is running then an error will be thrown. You can use the keyword missingok to avoid this scenario and instruct logrotate to ignore the log file if it does not exist.

1

2

3

4

5

6

/var/log/myapp/*.log{

size10k

weekly

rotate8

missingok

}

Continue writing to the same log file after rotation

Usually when a log file is rotated the log file is moved to a new location. Some applications may throw an error, and others may continue to write to the relocated file. The copytruncate keyword copies all the log in the file to a new file and then truncates the original file. This keeps the original log file in place and also allows rotation to continue.

Controlling startup services in Linux distributions such as Red Hat Enterprise Linux (RHEL), CentOS and Oracle Enterprise Linux (OEL) can be done using a Gnome GUI or a command line utility. The command line utility is called chkconfig and can list existing, add new or remove services from the operating systems startup list.

As Linux operating systems have multiple states, or runlevels, you need to make sure you add any new services to the correct runlevels. For example, you would not want to start a web service application before starting networking. See my post on runlevels.

What services are available for startup?

Use the –list switch to see your existing services and when they should be running.

1

chkconfig--list

An example output is below. This shows all of the machine runlevels and what the state of the service will be.

In this example, there are three services available. netconsole is not available at any runlevels and is therefore never started automatically, network is available only in runlevel 3 and modules_dep is available in runlevel 2 and runlevel 3.

You could also use the example below to detail the runlevels of a single service only.

1

chkconfig network--list

Add a new service with chkconfig

Adding a new service is easily done with the below command. The below example shows the service network being enabled to start at the next machine boot.

1

chkconfig network on

Use the –level switch to enable the service at specific runlevels. Use the below example to enable the service at runlevel 3.

1

chkconfig network on--level3

Remove a service with chkconfig

Removing an existing service is done with the below command. The below example shows the service network being disabled from automatic start.

1

chkconfig network off

Use the –level switch to remove the service from specific runlevels. Use the below example to disable the service at runlevel 3.

1

chkconfig network off--level3

Start a service

Starting a service is done using the command service followed by the service name and the command to start the service.

1

service network start

Stop a service

Use the stop keyword with service to stop a service.

1

service network stop

Check the status of a service

Each service has a status, usually running or not running. Some services, such as network, may have a different output and output more information on the service.