Zeno's Interview Question

I hope to teach a "how to interview" class at an upcoming conference. Here's one of the points I'll be making.

How can one interview question help me understand what the candiate does and doesn't know Unix?

Here's the question:

What happens with I type this at a shell prompt:

telnet www.wikipedia.org 80 RETURN

Usually the candiate will explain just the command: "It opens a connection to port 80 on wikipedia". That's a good answer.

The follow-up question is, "Please give me more detail".

At this point they might explain how DNS works or how TCP network connections work. Whichever they explain, you can ask about the other one.

That is, of course, skipping the fact that the command was started: You can ask them to explain how a process is created in Unix. You can ask them to explain how command lines are parsed.

Basically you can drill down and down and down and down. Any time the candiate says, "I don't know" ask them to guess. After the guess, move to some other aspect of the system.

You can do this in 15 minutes or 15 hours depending on how detailed you ask the candidate to get.

The point is that you will learn a lot about what the candidate does and doesn't know. Don't get hung up if they don't know half the issues you bring up, just make sure the parts that are known are the parts the job opening requires knowledge. Watch the "guesses" to see if they are logical or do they require what I will politely call "magical thinking". Not knowing how DNS works is forgivable for some positions. Thinking that TCP works with hostnames instead of IP addresses is not forgivable. Thinking that "the computer just knows" is a magical thinking and is not forgivable.

I call this "Zeno's Interview Question" because you never get to the end. There is always more to discuss if you break the situation down into smaller and smaller parts to be described. Generally the conversation doesn't go linearly. You often ask "what happens before that?" When all is said and done you probably have traversed the timeline back and forth and back and forth; moving up and down layers of abstractions again and again.

You can use different command lines depending on if the position is for a sysadmin or software developer. The above example is good for a sysadmin because it involves DNS and networking. A systems programmer might be asked "cat *" instead, especially if you want to introduce that there is a file called "-f" in the current directory. You'll learn a lot about whether or not the candidate understands how command line parsing works.

Here is a list of topics this can generate:

How USB keyboards transmit keystrokes.

How keystrokes get processed through the kernel.

How the unix concept of pty "raw vs. cooked" mode.

How the shell parses lines (quotes then variables then {} then globs).

How the shell executes commands: aliases then relative path or search $PATH.

How commands are executed: all about fork() and exec()

How a process starts: the Unix disk I/O system, virtual memory, paging executables into memory, bss vs data. Jumping to the first instruction.

How argv is passed to the process.

How a process parses flags.

How DNS lookups work (in the simple case)

How DNS lookups work (in the recursive case)

To send the DNS packet we need to discuss:

How L3 routing works

How L2 packet passing works (Ethernet Switching, "cam" tables)

VLANs

Finally we have an IP address. Now we discuss...

How a TCP connection is created.

How sequence numbers work.

How a TCP connection sends data.

Nagle's Algorithm

My fingers are getting tired.

(Note: the above list is incomplete, by definition. It probably also includes a few errors.)

We discussed this at the monthly meeting of the New Jersey Chapter of LOPSA a few years ago. It was 60 minutes before we got to the point that a packet hit the network. At the 2-hour mark we stopped the meeting because, to be honest, we were all pretty exhausted.

If you use this as an interview question I highly recommend you first tell the candidate how much time you've allocated for this question. If they know you want this to last 20 minutes, for example, it helps them give an indication of how much detail is being expected.

The best part of this question is that there's no way to "cheat". If the candidate has heard this question before it doesn't help them. The only way to "cheat" is to read all the books and other resources required to learn all of this. That's called education.

No TrackBacks

SO much "this", as the cool kids say! I had a similar interview question sprung on me many years ago involving ssh to a host, and the interviewer kept saying "...what happens before that?" We spent a good three hours on it, and I've co-opted it for my own interviews ever since. It's a WONDERFUL exercise, and while it's not perfect, I've found it quickly cuts through the BS on what a candidate does or doesn't know. I use 'ssh' when trying out linux-based candidates, 'http' when (most recently) I interviewed a mid-level/senior Windows guy for a junior linux person.

I forgot to add, I think this is a much better interview question than the ones asked me by a Google head-hunter not too long ago. I felt that I didn't get to flex just how deep my *nix-knowledge went when asked trivia like "how many bits does the permissions-mask on an ext3-filesystem take up?"

Interesting - we ask almost the exact same question (only it's "what happens when you type http://www.somewebsite.com/ into a browser" - the telnet example is a little better as you don't have to get bogged down with how the browser works) with similar variation in response detail.

The level of detail given can also give hints to how a candidate approaches explaining technical issues to people. In cases where a lot of customer interaction is required, giving less detail initially is a better response than someone who immediately dives deep into the innards of the network stack, which would be a hint that they may give overly technical descriptions of problems to non-technical people where it wasn't called for.

Another good "getting started" question for someone in a job interview that claims to be a Unix expert is to ask them what their favorite shell is and why. It will give you good insight into what they have done on Unix and how long they've been using it.

For example, someone who is a very competant scripter may prefer BASH because it is so easy to program with. From there, you can quiz them on the features of BASH that make is so much better than bsh or ksh for scripting.

Also, it's a good "bozo bit" question. I'm consistently amazed by interviewees who claim to be an 8 or 9 out of 10 in Unix proficiency who don't even know what shells they have used in the past, or that the shell even exists.

I'm glad I stumbled on this article. I enjoy reading this blog specially if you're a sysadmin. I had a similar experience when I got interviewed by Google 2 years ago Unix Application Admin position based in Sydney. Knowing how TCP 3-way handshake work is helpful since it is a basic networking concept. But, I got bogged down about how to handle a zombie process. Well, I know what a zombie is by its definition. To get rid of it is somehow I've never had even encountered or a kind of thing I wasn't aware of because of my lack of experience managing large scale Linux installations. Three days later, Google sent me a very polite Thank you letter.

To my knowledge, a warm reboot is the only option left to clean up zombies. But then I quickly realized that a risk factor is very high if you do a reboot on a production systems and hey!, its about dealing with hundreds of thousands of these machines. The interview was a good one-- learned a lot from it. So just to give you related conversations on this topic:

Lazlo Bock on how Google's hiring process works.

"We hire for capability and learning ability before we hire for expertise."

In the meantime, it's pretty hard in my current location getting a job on sysadmin role and hope to find one :). Anyways, I've taken steps to learn Puppet and reading books like Unix and Linux System Administration ,4e and The Practice of System and Network Administration, 2e. These two books complement each other. I've seen a university professor who teaches a course on system administration using the combination of these two books.