Sed - 23 Jan 2006

When you're wearing your security administrator hat, you know you should regularly scan your Security and other logs for threats, but who has time? Automated text file manipulation dramatically simplifies log file analysis and lets you easily analyze huge amounts of data from network devices, firewalls, and servers. You can use the sed tool to help parse and process output files from any of these devices.

Sed comes from the UNIX world and stands for Stream EDitor. You can think of sed as a middleman—a tool for manipulating the output of one program or command on the fly and displaying the output to the console or even feeding the output to another program or command.

Sed gives you extremely granular control over parsing and reformatting of text files and—even more usefully—output from other commands. For example, you can use sed to reformat or strip alerts from a firewall log to show only the attacker's IP address. Or you can filter your VPN server log files to obtain a list of users who are connected to the service and when they connected, stripping out extraneous data that the server might have added.

Although sed can read a file from disk, it really shines when accepting data from an I/O stream. Sed inspects the stream line by line looking for a specified pattern. When it finds a match, it edits the data according to rules and filters that you define. You can download a Win32 port of sed as part of the Win32 UNIX Tools package at http://unxutils.sourceforge.net/UnxUtils.zip. To find Help options for sed, use Google to search the Web for "man sed" or "sed how-to." You'll find many online guides describing its features.

Let's look at two sed examples: one that reads a file from disk and another that manipulates data from an I/O stream. We'll focus on basics such as using sed's substitute parameter.

Sed in Action
Seeing sed in action is the best way to see what it can do. Let's look at a simple example that shows how sed can manipulate data from a file.

Figure 1 shows the contents of a file named data.txt. Let's use sed to substitute the word "one" for all occurrences of the number 1. This is easy to do using sed's substitute parameter (s). The substitute syntax is

sed s/SearchString/ReplaceString/
Filename

where SearchString is the pattern you're looking for, ReplaceString is the string you want to put in its place, and Filename is the name of the file you want to search. So, to replace the numeral 1 in data.txt, you'd type

sed s/1/one/ data.txt

Figure 2 shows the output from this command. As you can see, the command worked partially. Notice that only the first numeral 1 on each line was changed. By default, sed stops analyzing a line after it detects the first occurrence of a pattern. To instruct sed to look for every match in a line, use the global (g) parameter at the end of the regular expression:

sed s/1/one/g data.txt

This command results in the output in Figure 3, which is just what we want.

A More Practical Example
For an example that's geared toward systems administration, let's use sed with other tools to display only the IP address of a system. We'll do this by streaming the output data from one command as input into another. Most Win32 ports of UNIX tools such as sed support I/O streams, and this example will show you how valuable this support can be.

We use the pipe (|) character to stream the output of one program into another. To obtain a system's IP address, we pipe the output of Ipconfig into grep, then pipe grep's output into sed. Streaming the data from one tool to another creates a compact, easy-to-follow set of commands. In fact, the solution to showing just the system's IP address is

c:\ipconfig | grep "IP Address"
| sed -r
s/^.\{0,\}:./My\x20IP:\x20/

which outputs exactly what we want and nothing more:

My IP: 192.168.0.100

We can look at the output of each tool independently to isolate its purpose.

The c:\ipconfig command returns a lot of IP address-related information and puts each bit of information on its own line, as Figure 4 shows. The first step in extracting the data we want is to identify it. We want only the IP address (192.168.0.100), so we need to trim the extraneous data. Our target data appears at the end of the second line of Ipconfig's output, so we'll use grep to fetch just that line. (If you're new to grep, see Toolbox, "Grep," August 2005, InstantDoc ID 46869, for an introduction-to this indispensable tool.) Running the command

ipconfig | grep "IP Address"

returns only the line

IP Address. . . : 192.168.0.100

Now, let's pipe this output through sed and tell sed to replace the text that precedes the IP address with the string "My IP:". Like grep, sed uses regular expressions for pattern matching. Unlike grep, sed lets you edit the data that you match.

In "Grep," I introduced regular expressions. The one I use here might look a bit intimidating, though, so let's walk through it. The sed command that substitutes the text preceding the IP address is

sed -r s/^.\{0,\}:./My\x20IP:\x20/

The -r parameter instructs sed to use extended regular expressions. Extended regular expressions are necessary for some versions of sed (including the Win32 port we're using) that require a more complex syntax, such as the braces (\{\}) we use here.

In this command, the expression

^.\{0,\}:./My\x20IP:\x20/

instructs sed to start at the beginning of the string (^), look for any character (.) that occurs 0 or any number of times (\{0,\}) until it finds a colon (:), then look for one more character (.) and replace that character with the text "My IP: ". We represent the replacement text as the regular expression

My\x20IP:\x20

The \x20 expression occurs twice and is a regular expression for a space (" ") in hexadecimal. We must represent the space in hex because the cmd.exe parser would interpret an actual space as the end of the command.

Just the Beginning
I've given you just the basics of sed and how to use it with other programs to strip or reformat data. Sed is powerful, but don't be intimidated—you can do a lot with just these basics and discover its more powerful features as you need them.