Editor: Text-file and command-line purist Tom Baker contacted us after seeing Lifehacker's todo.txt script to say he'd developed a text list manager of his own called shawkle. Today Baker is kind enough to explain how he manages his personal information in TXT using shawkle. Shawkle is a clever bit of command line nerdery that measures high on the geek Richter scale, but it's an incredibly useful and flexible tool for plain text file enthusiasts.

Back in 1993 I was maintaining personal information in a handful of plain text files using the Unix editor vi. I often had to open several files in order to keyboard overlapping bits of information — e.g., a reference to a project (todo.txt), the phone number of someone related to that project (phone.txt), and a scheduled appointment with that person (calendar.txt). After the appointment was over, I had to open yet another file to move the reference from calendar.txt to log.txt.

Flipping between the files was tedious, even with vi, so I wrote a little script to automatically move any lines containing the keyword PHONE to phone.txt, any lines with NOW to todo.txt, and so on. When I saw that keywords could be associated with files using rules, and those rules could be split out into a separate file, I wrote a script called shuffle (now shawkle).

shuffle took a directory full of TXT-format lists, dumped their contents into one big file, deleted the originals (after backing them up!), and generated a new set of lists according to a rule file. Shortly after shuffle was published in a UnixWorld column, I replaced its central function, based on grep, with a slightly more sophisticated function based on awk, and I renamed the script shawkle. shawkle has now reliably served as my list organizer for over twelve years.

Rule-based list processing

The idea behind "rule-based list processing" is that lists are managed with regular expression-based rules that control the contents and sorting order of plain text files. It's a form of "lazy information management" inasmuch it takes loosely structured input and provides ways to let the information "fall into place." Lines of information can be typed into any (arbitrarily named) text file, in any order, tagged with keywords, and the rules will ensure that the lines end up in specified files, sorted in specified orders. Lists are edited, rules are tweaked, shawkle is run, and fresh lists are generated, made to order.

Requirements and Download

shawkle is a command-line tool that can run at a Unix command line or emulator like Cygwin on Windows. Download shawkle from here, chmod 755 to make it executable, and follow along with the example below to get started.

shawkle in action

Let's start with two files:

foobar.txt, an unsorted list of appointments, tasks, and reference information:

When applied to all text files in a directory (shawkle *.txt), the script verifies that each source file consists of plain text with no blank lines, moves a copy of each source file to the "hidden" directory .backup, and aggregates the contents of the source files into one big temporary target file (combined.data). At this point the rules are processed. Taken rule-by-rule:

Lines containing at least one character (i.e., all of the lines!) are moved from the source file combined.data into a target file, default.txt, which is then sorted alphabetically.

Lines starting with "B " (a B followed by a blank space) are moved from default.txt into done.txt.

Lines starting with "NOW" are moved from default.txt into todo.txt.

Lines starting with "LATER" are also moved from default.txt into todo.txt.

Note that in this example, the last three rules are "commented out" with hash signs. If "uncommented," they would mean:

Lines starting with "Jan" are moved from default.txt into calendar.txt.

Lines starting with "Feb" are also moved from default.txt into calendar.txt, which is then sorted by month and day.

Lines starting with "PHONE" are moved from default.txt into phone.txt.

After shawkle is run there are two new text files (in place of foobar.txt):

Tweaking the example

Let's now edit the file .rules, "uncommenting" the last three rules by deleting the hash signs. Next, let's edit default.txt to make one of its lines match a different rule: change the date string "Jan 06" to read "B 2006-01-06". Run the script again (shawkle *.txt). Now there are four files:

How I use shawkle

I always carry a folded printout of my current to-do list in my pocket. The sheets serve as a collection bucket for phone numbers, URLs, and random thoughts. Away from my computer — on the train, standing in line, over coffee — the printouts provide a space to brainstorm, set priorities, turn vague ideas into Next Actions, and best of all, to cross things out. The sheets fill with arrows and scribbles. Back at the computer, I enter the additions and corrections at the keyboard, tweak .rules to change the order of keywords or their grouping into files, run shawkle again, print a fresh edition, and the cycle begins anew.

Lists and rules co-evolve over time. They reflect my changing projects and priorities. My preparing-for-a-conference list morphs into a writing-an-article list, then into a going-on-vacation list. The file .rules is an evolving vocabulary of keywords; no other documentation is needed because it documents itself.

Having used this script continually for twelve years, I have amassed a large corpus of completed actions, email addresses, phone numbers, reading excerpts, and time-stamped meeting notes, which I query using simple shell scripts based on grep. Each time shawkle is run, another shell script mirrors the latest lists to HTML files, making any URIs and local filenames (such as mail folders in mbox format) clickable in a browser. The file todo.html serves as my default homepage.

David Allen suggests choosing a list organizer with which one is inspired to play. The editability of .rules makes it easy to tag data on multiple axes — priorities (TODAY, SOON, SOMEDAY), big projects (DCMI, KIM, SWD), small projects (DINNER), incubators (NEWIDEA), contexts (CALL), reminders (EXPECT), references (PHONE, LINKS), or events (YYYY-MM-DD, Mmm DD). There is no "one best way" to do this. Iteratively editing the data, tweaking the rules, and generating different views is a form of thinking.

Rule-based processing of plain-text lists with a plain-vanilla editor bypasses the fancy interfaces to put a user directly in touch with data. Plain-text editing has long superseded the limitation of 80-column-wide terminal consoles. Today, lines 120 or even 130 columns wide are beautifully readable in a medium-font text window or in a landscape printout. The constraint on width inspires a certain pithiness, even poetry.

About shawkle

shawkle (as its predecessor shuffle) was developed using MKS Toolkit, a pre-Cygwin Unix shell emulator, running on DOS 3.3. Some commands in the script, such as print -u2, are specific to the Korn shell. The script works fine with the public-domain Korn shell (pdksh) now widely available. Some advice:

Save your text files and exit the editor before running shawkle.

If a sanity check causes shawkle to exit before it has finished aggregating the data into one file, the file combined.data will hold the contents of any files that have been processed to that point. (To reassure yourself, compare the size of combined.data with the combined size of the files in .backup.) To get back on track, fix the problem described in the error message, rename combined.data to somethingelse.txt, and re-run shawkle.

Do not run shawkle twice, simultaneously, on the same data.

Do not use non-existent sort options in the rule file (e.g., sort -e); data may be lost. If data has been lost, shawkle will let you know. Just fix the offending rule, recover the original files from .backup, and re-run shawkle.

Note that the Unix program /bin/file may sometimes interpret a text file as being something other than "text", causing shawkle to exit with an error message. If this happens, test the offending file with /bin/file, possibly using an alternative magicfile, and consider changing the keywords or punctuation in the text file that are causing the problem.

The 1994 UnixWorld column provides additional detail about the script. One reviewer found that, using agrep, shuffle could be modified to handle multi-line text such as mbox mail folders and Usenet news articles.

For clarity of exposition, the script presented in this article is slightly simpler than the script I actually use, which automatically moves specified files between directories. Lines with completed actions, such as 2006-11-06.1230.done.txt, are pushed to an archive directory, with its own local .rules file, for merging into a log.txt. Note that the awk-based regular expressions can match any numbered whitespace-delimited field within a line, and a rule tree can have more than one branch.