Search

Sysadmin 101: Ticketing

This is the third in a series of articles on system administrator
fundamentals where I focus
on some lessons I've learned
through the years that might be obvious to longtime sysadmins, but news
to someone just coming into this position.

In the first
article, I discussed how to approach alerting
and on-call rotations as a sysadmin. The second
article covered how
to automate yourself out of a job. In this article, I explore something
that on the surface may seem boring or mundane but is absolutely critical to
get right if you want to be an effective sysadmin: ticketing.

By ticketing, I'm referring to systems that allow sysadmins to keep track of
tasks both internally and those requested by their coworkers or customers. There
are many ways to get ticketing wrong so that it becomes a drain on an
organization, so many sysadmins avoid or it use it begrudgingly. Also,
ticketing approaches that work well for developers may be horrible for
sysadmins, and vice versa. If you don't currently use a ticketing system, I
hope by the end of this article, I've changed your mind. If you do use
tickets, but you wish you didn't, I hope I can share how to structure a
ticketing system that makes everything easier, not more difficult.

Why Tickets Are Important

Like documentation, tickets are one of those important things in a mature
organization that some administrators think are unnecessary or even a waste
of time. A ticketing system is important no matter the size of your
organization. In a large organization, you have a large volume of tasks you
need to keep track of distributed among a group of people. In a small
organization, you often have one person taking on many roles. This leads me
to the first reason why tickets are important.

Tickets Ensure That Tasks Aren't Forgotten

Sysadmins are asked to do new tasks constantly. These days, there are any
number of ways a coworker might ask for your help, from an email, to a phone
call, to a message in a chat program, to a tap on the shoulder. If you
weren't doing anything else, you immediately could start working on that
task,
and everything would be fine. Of course, usually sysadmins have to balance needs
from many different people at the same time. Even requests that come in
through email have a tendency to fall through the cracks and be forgotten.
By storing every request in a ticket, no matter how you got the request, it
is captured, so that even if you do forget about it, you'll remember it the
next time you look at your ticketing system.

Tickets Make Sure the Task Is Done Right

Even if you can remember what someone wants you to do, you may not remember
on Monday all of the details that someone told you in person on Friday. A ticket
lets you capture exactly what people want done in their own words and
provides a way for them to confirm that you completed the task the way they
wanted before you close the ticket.

Tickets Help You Prioritize Tasks

Every request is important to the person who makes it. Every request may not
be as urgent to you or your team compared to your other tasks,
however. When
all of your tasks are captured in tickets, the team lead or manager can go
through and re-prioritize tasks so they are worked on in the right order.
This ends up being more fair for everyone; otherwise, new tasks have a way of
cutting in line, especially when the person asking for something is standing
over your shoulder.

With a ticketing system, team leads or managers have a full list of
important tasks they can point to when they need to explain why they aren't
dropping everything for a new request. At the very least, it will help direct the
conversation about why a particular task should be put at the head of the
line.

Tickets Distribute the Work

If you have only one sysadmin, distributing tickets and projects is
easy. Once your team grows though, it's important to distribute the work so
no member of the team gets burned out. Coworkers have a tendency of finding
that senior member of your team who is most productive and going to them
directly when they have any issue. Of course, that team member is probably
already working on plenty of other tasks or may be trying to focus on an
important
project.

When a task is captured in a ticket, the team lead or manager can assign and
reassign tickets to different members of the team to make sure no one gets
burned out, and also to ensure that everyone learns how to do things.
Otherwise, you end up
cultivating specialists within the team that always take tickets related to
certain systems, which leads to problems later when that team member goes on
vacation.

Tickets Provide an Audit Trail for Changes

Every time you change a system, you create an opportunity for something to
break. If you are lucky, things break immediately after you make the change.
More often, you'll find that it takes some time for a change to cause a
problem. You'll discover two weeks later that something stopped working, and
with a ticketing system, you can pull up all of the tasks that were worked on
around that time. This makes it much easier to pinpoint potential causes of
a problem.

Tickets also provide an audit trail for tasks that require approval or proof
of completion, like creating or revoking accounts, granting new privileges
or patching software. When someone asks who said it was okay for Bob to get
access to production and when it happened, you can answer the question. If
you need to prove that you applied a security patch, you can point to
command output that you capture and then store in the corresponding ticket.

Qualities of an Effective Ticketing System

Many different ticketing systems exist, and sometimes when you
hear people complain about tickets, what they are really complaining about
is a bad ticketing system. When choosing between ticketing systems,
you should look for a few things.

Some systems that developers use to track code through the development
process result in very complicated workflows. For a sysadmin though, the
simpler the ticketing system the better. Because you already are asking a
sysadmin to take time out of solving a problem to document it in a ticket,
it helps if the ticketing process is fast and simple. I prefer very simple
ticket workflows for sysadmins where there may be only a few states: open,
assigned, in progress, resolved and closed. (I'll talk more about how I
treat each of those states in the next section.)

The fewer required fields in a ticket, the better. If you want to add extra
fields for tags or other information, that's fine, just don't make those
fields mandatory. The goal here is to allow sysadmins to create tickets
based on someone walking up and tapping them on the shoulder in less than a
minute.

Ideally, the ticketing system would allow you some other way to generate tickets
from a script, either from sending an email to a special address or via an
exposed API. If it has an API that lets you change ticket state or add
comments, all the better, as you potentially can integrate those into your
other automation scripts. For instance, I've created a production deployment
script that integrates with my ticketing system, so that it reads the
manifest of packages it should install from the ticket itself and then
outputs all of the results from the deployment as comments in the
ticket.
It's a great way to enforce a best practice of documenting each of your
software releases, but it does it in a way that makes it the path of least resistance.

Favor ticketing systems that allow you to create dependencies or other links
between tickets. It's useful to know that task A depends on task B, and so
you must complete task B first. These kinds of ticketing systems also make it
easier to build a master ticket to track a project and then break that
large project down into individual tickets that describe manageable tasks.
These kinds of systems often show all of the subordinate tickets in the
master ticket, so a quick glance at the master ticket can give you a clue
about where you are in a project.

How to Manage and Organize Tickets

Each ticketing system has its own notion of ticket states, but in my opinion,
you should, at a minimum, have the following:

Open: a task that needs to be completed, but hasn't been assigned to
anyone.

Assigned: a task that's in a particular person's queue, but they
haven't started work on it yet. Tasks in this state should be safe to
reassign to someone else.

In progress: a task that has been assigned to someone who is
currently working on the task. You definitely should communicate with
the assignee before you reassign tickets in this state.

Resolved: the sysadmin believes the task has been completed and is
waiting for confirmation from the person who filed the ticket before
closing it.

Closed: the task has been completed to everyone's satisfaction.

A well run ticketing system should provide the team with the answers to a
few important questions. The first question is "What should I work on
now?"
To answer that question, each member of the team should be able to claim
tickets, and team leads or managers should be able to assign tickets to
individual members of the team. It's important for people to claim tickets
and start work only after they are claimed; otherwise, it's easy (and common)
for two members of the team to start working on the same task without realizing
it. Then everyone on the team can start working on tickets in their personal
queue, starting with the highest-priority tasks.

The next question a good ticketing system should answer is "What should I
work on next?" Once sysadmins' personal queues are empty, they should be able
to go to the collective queue and see a list of tasks ordered by priority.
It should be clear what tasks they should put on their queue, and if there's
any question about it, they can go to the team lead or manager for some
clarity. Again, ticket priority helps inform everyone on the team about
what's next—higher-priority tasks trump lower-priority ones, not
necessarily because they are less important (a ticket is always important to
the person who filed it), but because they are less urgent.

I approach ticket priority as a way for users to help inform the team about
how important the ticket is to them, but not how urgent it is for the
team.
The fact is, there's no way every employee in the company can know all of
the other important tasks the sysadmin has to perform for other people nor
can they be expected to weigh the importance of their need against everyone
else's needs.

A good manager should reserve the right to weigh the priority assigned to a
ticket against the other tickets in the queue and change the priority up or
down based on its urgency relative to the other tasks. It also may be the
case where a task that was low urgency two weeks ago has become urgent now
because of how long it was in the queue, so a good manager would be aware of
this and bump the priority. If you are going to start the practice of
changing ticket priorities though, be sure to inform everyone of your
intentions and how you will determine the urgency of a ticket.

Another key to managing tickets is to make sure all of your requests are
captured in the ticketing system. Sometimes a coworker can be guilty of
trying to skip ahead in line by messaging you with a request or
walking directly to your desk to ask you to do something. Even in those
cases where you really are going to drop everything to work on their
request, you should insist on capturing the request in a ticket so you can
track the work. This isn't just so you can prioritize it based on other
tasks or so you don't forget it, it's so in a week when some problem crops
up based on this urgent change, you'll see this ticket along with other
tasks completed that day and it will help you track down the cause.

Finally, as a manager, be careful to distribute work fairly among your team.
Even if one member of the team happens to be an expert on a particular
service, don't assign that person every task related to that service; it's
important for everyone on the team to cross-train. Pay attention if
employees try to get tickets assigned to their favorite member of the
team,
and don't be afraid to reassign tasks to spread the work around evenly.
Finally, every ticket
queue has routine, mundane grunt work that must be done. Be sure to
distribute those tasks throughout the team so no one gets burnt out.

Kyle Rankin is a Tech Editor and columnist at Linux Journal and the Chief Security Officer at Purism. He is the author of Linux Hardening in Hostile Networks, DevOps Troubleshooting, The Official Ubuntu Server Book, Knoppix Hacks, Knoppix Pocket Reference, Linux Multimedia
Hacks and Ubuntu Hacks, and also a contributor to a number of other O'Reilly books. Rankin speaks frequently on security and open-source software including at
BsidesLV, O'Reilly Security Conference, OSCON, SCALE, CactusCon, Linux World Expo and Penguicon. You can follow him at @kylerankin.