Q & A with the Authors of "UNIX and Linux System Administration Handbook"

Evi Nemeth, Ben Whaley, Garth Snyder, and Trent Hein are all experienced sysadmins who bring a deep knowledge of UNIX and Linux to the table. Matthew Helmke recently had the opportunity to ask them some questions about their new book, interacting with readers, and whether after 4 editions they consider themselves sysadmins or authors.

From the author of

From the author of

UNIX and Linux System Administration Handbook is now in its fourth edition. This classic text is consistently one of the highest rated and most respected books on the topic. In the 20 years since the first edition, the book has continually upped the ante. Already the world's best-selling book on UNIX system administration, it has been improved for this edition with material on several of the leading Linux distributions, such as Ubuntu, RHEL, and OpenSUSE. It is also being offered as an eBook for the first time ever.

I had the chance to ask the authors a few questions. Ben Whaley, Evi Nemeth, Garth Snyder, and Trent Hein are all experienced sysadmins who bring a deep knowledge of UNIX and Linux to the table, combine it with a desire to share their knowledge with others who have an interest in the topic, and add to it a bit of wit. The end product is a book that is filled with clearly presented, quality information that is enjoyable to read. It turns out they are also enjoyable to interview.

Matthew Helmke: What is your experience with UNIX and Linux system administration?

Ben Whaley: I started with Red Hat Linux in 1999 while working for very small, family-owned computer repair shop in Wyoming. It was a real trial by fire. I remember that an FTP server I ran was hacked and served pirated software for some unknown length of time. I exchanged email with the author of the exploit (the software was left in plain text on the hacked system) and he said, "Hmm, that works only on poorly configured servers." I learned that lesson the first time, and that's really where my interest in security began.

At the University of Colorado I had more experience administering Solaris servers in the central UNIX services department. After I left school and started at AppliedTrust, I quickly expanded to larger UNIX and Linux environments. I focused on Red Hat technologies since they remain the leader in enterprise Linux deployments.

Garth Snyder: My first exposure to system administration came while working for Evi at the University of Colorado during the summers I was in college in the late 1980s. Three of the authors (me, Trent, and Scott Seebass, a coauthor of the first edition) emerged from that milieu. The systems—mostly Sun workstations, Pyramids, and DEC Vaxes running BSD—were incredibly primitive by today's standards, but in many ways it's surprising how little the major issues of system administration have changed.

Evi Nemeth: I started with Bell Labs version 6 on a PDP11/45 that I used for teaching computer science at SUNY-Utica. We had no sysadmins, just people like me who liked to tinker and explore. Over the years I migrated from the trenches to supervising student sysadmins and teaching UNIX system administration and networking in the computer science program at the University of Colorado.

Trent Hein: I started using UNIX in 1984 on Honeywell minicomputer, but really became a system administrator at the University of Colorado in 1987 under Evi's guidance. I've administered Linux systems since the early 1990's.

Matthew: How did you decide to write this book (or join the project)?

Ben: I was fortunate enough to work alongside Trent at Applied Trust. He asked if I'd be interested in working on a few chapters for the 2nd edition of the Linux Administration Handbook, which I gladly accepted. Apparently my contributions weren't too harmful to the book, and the primary author team offered to add me as an author to ULSAH/4E, which I also happily agreed to.

Garth: The original book was all Evi's idea. There was a huge need for a book to cover this territory, but nothing was available, so we knew the opportunity was there. We set aside a summer to work full time on the first edition, and as ludicrous as that seems now, that the book has swelled to almost 1,350 pages, we pretty much completed the initial draft in that time frame. These days, it takes several years to prepare a new edition.

Evi: In the late 1980s, as I was figuring it out and teaching others, there were no reference materials other than the man pages and UNIX documents. They told you what to do, but not why or when. Visiting publisher representatives, who would often ask if I didn't want to write a book, planted the seed, and I recruited two undergraduates who worked with me summers (Garth Snyder from Swarthmore College and Scott Seebass from UC Berkeley) to help. We cranked out a couple of chapters and sent them and the table of contents to three publishers; Prentice Hall accepted us.

Trent: I appeared on the scene at the University of Colorado around the time that Evi, Garth, and Scott were working on USAH/1E. I was a reviewer for that edition and joined as an author for USAH/2E in 1995.

Matthew: There are many books on this topic; what distinguishes UNIX and Linux System Administration Handbook from all the rest?

Ben: I'm a latecomer to the series; my first exposure was using the USAH 3rd edition as the text in a course in school. The series is very well written and highly opinionated. Unlike most dry material on the subject, ULSAH makes light of poor design decisions or confusing aspects of the operating system. No vendor is spared, but Microsoft is ridiculed more than most.

Garth: Our multiplatform approach is also quite unusual, I think—not because we do it in a particularly strange way, but because relatively few technical books address such a wide variety of systems. Comparisons among systems have been a feature of our book from the very start. ULSAH4E is one of the few resources you can consult to get a feel for how different it is to manage, say, Ubuntu vs. OpenSolaris systems.

Evi: In the late 80s, technical books were dry and formal and no fun to read. With 2/3 of our team smart undergraduates, we aimed for a tone that would be interesting to read, disrespectful when things were done wrong by our vendors, and fun. This was furthered when Prentice Hall, a bit worried about new authors, asked for a sample chapter for their copy editors to review and see how badly we wrote. We sent the finished adduser chapter. The copy editor informed us that company policy was to never use the word "etc.", but to use "and so on" instead. Needless to say, the /"and so on"/passwd file would not fly, and gave us license to ignore copy editors and incorporate an undergraduate tone, making the book both technically correct and fun to read. This tone was enriched by the cartoons at the beginning of each chapter, done by a University of Colorado CS undergraduate, Tyler Stevens. We have all new cartoons for this edition, thanks to Lisa Haney, a professional artist.

Trent: We're professional system administrators, not professional writers. We really do this stuff every day.

Matthew: What has changed in this edition? What new things have you added?

Ben: Lots of changes in this version! We've added completely new chapters on server virtualization, scripting technologies, and green IT. AIX is now a covered UNIX variant, as well as three Linux distributions. The cover art and chapter cartoons are all new thanks to Lisa Haney's excellent work.

Garth: More time and effort have been dedicated to this edition than to any prior version of the book. In addition to all the new chapters and the new extras (such as the UNIX history section by Peter Salus and a short dialogue with one of our technical commenters, Dan Foster, about AIX), we've expanded the scope of many existing chapters. For example, the discussion of email systems now includes all three of the major platforms: sendmail, Exim, and Postfix. The storage chapter covers technologies such as solid state hard drives, Solaris's ZFS filesystem, and the explosive growth of SAN networks.

Evi: We started this edition as a 20th anniversary edition, expecting it to be published in 2009, and trying to incorporate at least 20 new topics. But alas, we missed our deadline by almost a year; however, 21 is an important birthday, too, at least in the USA. We've tried to embrace the latest technology and also to lose some of our prior Berkeley UNIX biases (e.g., bind and sendmail) with coverage of other competing systems. We've also had to admit that Microsoft Windows is here to stay, and we had better embrace it rather than gripe about it; there is a concerted effort to show how to use the best of both Windows and UNIX in a mixed environment.

Trent: My favorite new addition to this edition is the Green IT chapter. System administrators can save the planet!

Matthew: What was one of the most surprising things you learned while preparing, writing, and revising this book?

Ben: I'm not sure that this should be a surprise, but it's clear that modern IT moves more quickly than the writing process. Keeping material current and relevant was my biggest challenge. Also, really rooting out the facts of some complex and unclear protocols and applications (for instance, identity mapping in NFS version 4) is a tough task with all the misinformation on blogs, forums, and the like.

Garth: Prior to the debut of OpenSolaris, we were seeing the major UNIX vendors gradually cede a lot of ground to Linux distributions in the areas of innovation, configuration management, and general fit and finish. But OpenSolaris and Solaris 11 have surprised and impressed me with their ability to shake off some of the shackles of the past and to incorporate new and better approaches from the Linux world. It's going to be very interesting to see what becomes of these systems now that Oracle is in charge.

Evi: My biggest surprise, as always, was the huge amount of time it takes to revise a chapter that is only a couple of years out of date. I was happily exploring the San Blas islands of Panama on my sailboat when the call went out for a 20th anniversary edition. I came home in February, 2009 to work on the book and only flew back a year later on the last day my ticket was valid. The other authors picked up the slack in integrating external reviewers' comments and indexing for my chapters.

Trent: As a system administrator, it's hard to keep track of changes in OS versions (and in the industry), such as Oracle's surprise renaming of Sun Solaris to Oracle America Solaris. It's even harder as a writer.

Matthew: Do readers ever contact you about the book? Why?

Garth: They do, they do! Usually it's pretty friendly—despite all the inflammatory statements in the book, we don't receive a lot of hate mail. The most common request is for answers to the exercise questions at the end of each chapter. Unfortunately, many of those questions (e.g., "Is UNIX doomed?") are designed more to provoke thought and provide opportunities for structured analysis than to check for specific factual knowledge. So evaluating someone's answers demands some subjective judgment and familiarity with the subject matter.

Evi: Yes, in fact a few years ago we had to dump all mail to admin.com (our domain name for the book) because the spammers instructed recipients to email postmaster@admin.com if they wanted to be removed from the spam mailing list. We typically answer email from users, although as Garth says, it’s often [from readers] requesting answers to the exercises.

Trent: We hear from readers a lot, actually. Usually they have a cool story about how the book helped them, or they have a question about some specific situation. I hope that more than anything we give folks a foundation of knowledge that they can use to solve any problem, not just the problems specifically covered in the book.

Matthew: When you read a technical book, what do you look for?

Ben: I buy books for accuracy, relevance, and principles. I can find command-line flags and syntax answers via Google. For me, the purpose of a book is to understand higher-level concepts and implementation design. I hope that ULSAH offers a little of both.

Garth: A book's intended audience makes a huge difference. I get the most out of books that assume a technical background similar to mine and that try to take me to the specific places I want to go. If those factors are a match, I find that many of the other details don't matter so much. Unfortunately, that's a hard lesson for us to internalize as authors since we know we have readers at many levels of technical skill. We try to level the playing field by eliminating material that isn't of direct practical benefit to the majority of readers.

Evi: I don't read many technical books any more, but in sailing books, the thing I value most is accuracy. Incorrect coordinates on a reef are a total drag! Our review process, first by our coauthors and then by external experts, tends to keep USLAH pretty accurate. We try to correct any typos or errors that remain in subsequent printings and, of course, list them on our website.

Trent: Technical accuracy!! It drives me nuts when I find errors.

Matthew: There are four people listed as coauthors of this edition, but you and I know many more are involved in the process of publishing a book. Tell us a little bit about the collaborative process; both working together as authors as well as working with the people whose names are not on the cover.

Ben: The author team primarily worked together through Skype conference calls, video conferences, and email. Evi spent much of the writing effort in Boulder with Trent and me. Garth lives and works in Seattle. During the heavy writing phase, we would spend Tuesday evenings with Garth's face projected on our conference room wall via Skype while working on our individual chapters.

We had several contributing authors to manage as well—quality and timeliness were the biggest challenges. Garth is really the project manager/editor extraordinaire behind the writing effort. We each took a few different subtasks to help push the book forward. For example, I worked with IBM to obtain an AIX example system. Trent helped us with an HP-UX and Solaris system. The network of industry contacts was also immensely useful. I'm very proud and humbled that Tim O'Reilly wrote the foreword.

Garth: Every chapter has a primary owner who prepares an initial draft and who shepherds the chapter through a gauntlet of reviews and development. However, each of the primary authors reads and annotates all the material several times, so it's very much a collaborative process, even at the finest level of granularity.

Evi: On the collaborative front, our biggest challenge was managing the "ghosties," as we call them—the people who are better experts in one particular area than we are and who allow themselves to be arm-twisted into writing a chapter or part of a chapter. They all have full-time jobs and families; half are better at deadlines than we are, but the other half needed an extra push now and then to get the chores done. We also learned that a good sysadmin in a particular area doesn't always mean a good writer of the facts for that area.

Trent: At this point, there's such a community behind this book that it's amazing how much help and support we get. We acknowledge dozens of people in the front of the book, and that only scratches the surface. As far as primary authors go, I absolutely loved working with our team on this edition—it's the best group we've ever had. And, for the most part, we're still speaking to each other!

Matthew: When you think about yourself, do you consider yourself primarily a writer or a sysadmin? Why?

Ben: Interesting question. I think of myself as a technical sysadmin with a penchant for writing. A solid background in writing is terribly useful as a technical person, of course. Being able to communicate technical concepts clearly is a rare skill in IT, and one that I continue to hone in my career.

Garth: I love software and software systems generally. At this point, I've spent more time as a software developer than as a system administrator, but to tell the truth, I don't see much difference between the two. System administration has many of the same rewards and challenges as, say, web development.

Evi: Since retiring from the Computer Science faculty at the University of Colorado, I've been more of a sailor than either a sysadmin or a writer. But I've probably spent more time writing than fighting with the administration of my laptop.

Trent: I'm a system administrator. That's my day job. Writing is just a way of sharing that.

Matthew: Do you have any advice for people who want to become sysadmins or writers?

Ben: For sysadmins, I'd say the most important thing is to get your hands dirty. Practice, practice, practice. I was lucky enough to start at a very young age (14) and quickly logged a large number of hours doing work on the computer. Knowledge is easily transferable, so lessons learned on one system/application/environment are very portable to other areas.

Garth: If you can think logically, you can learn to write. For someone that wants to write about technical subjects, I'd suggest putting the emphasis on the subject matter rather than on writing per se. An entire editorial industry is devoted to fixing up people's imperfect English. However, relatively few people can catch technical errors in your flawlessly written prose.

Evi: Do it. You don't really understand something unless you have to do it or teach it. Sysadmins can grab any number of open-source operating systems, install them on commodity hardware, and play away. Volunteering to help out often opens doors in student labs. Try your hand at writing things like a lab manual for your school computing lab.

If software is your strength, volunteer for one of the open source projects. You can't just say I'm Joe/Jane Undergrad and I want to work on the Linux kernel, too; but you can start contributing patches to software bugs you find or tools that you have written. Reading the mailing lists of open-source projects can acquaint you with the community.

Trent: The world needs more sysadmins—I think everyone who's detail-oriented, a good problem solver, and likes multitasking should consider it as a career. As I tell all the folks I work with, this is one job where you are NEVER bored.