Cracking the DevOps Interview

In January, I thought that maybe it was time to start looking for a new job. My new title at work was DevOps Engineer and before that, I was a Software Engineer. I gave myself the title of Business Development Consultant at my friend’s startup, though the most crucial part of developing the business–sales growth–was firmly out of my hands. Supported by a degree that was both an accredited engineering degree and business degree, I figured that I’d be able to compete for opportunities in an assortment of industries.

Initial conversations with recruiters went well. I was received warmly. As I spoke to more people in my industry (trading) and others (general finance, consulting, big tech, government), I quickly realized that DevOps was a point of fixation for nearly everybody. With it came certain benefits (e.g. demand severely outpaced supply) but also an overarching sense of ambiguity. What is DevOps? I can tell you, and I feel that I can phrase it quite well, but this blog post aims to deal with a more important point–how do you prepare for a DevOps interview?

I didn’t perform well in my first few phone screens. I can’t even use “DevOps is ambiguous” as an excuse, because my CS fundamentals were lacking. I would get questions wrong and know that the interview was over because they were things that any new grad would know cold. Further, and perhaps most interestingly, none of the positions I interviewed for were termed “DevOps Engineer.” None of them were “Software Engineer,” either. Some were more SE, some were what you’d generically term Reliability Engineering, some were more systems-focused, and there’s one that I’m still not sure what it was.

Not knowing exactly what I wanted to do but with the confidence-boost of knowing that some of the coolest companies in the world were willing to interview me, I formulated a study-plan that became the highest priority in my life for roughly three months. During this time, I estimate that I spent thirty hours per week preparing. I also saved a lot of money by effectively canceling my social life.

The gist of my preparation was that I wasn’t willing to dedicate time to “gaming” the interview. I had a demanding full-time job and was involved with a successful startup. A promise I made to myself early on was that I would only spend 10% of my time doing prep that wouldn’t be directly useful for my current or any future job (mainly, esoteric coding interview questions with solutions more fit for careful research than committal to memory).

The result of my work was that I received multiple job offers, including one that I ended up accepting. The time I spent preparing remains a point of pride because it was through several months of self-imposed exile that I finally proved to myself that I was worthy to compete for jobs at elite companies and also, more abstractly, that I could indeed accomplish whatever I put my mind to.

I’ve written this post as both a way of helping those who are also preparing for interviews and also a way to thank those who have developed some of the most useful prep material.

1. The Coding Interview

You still must know how to code and quite well at that. Sure, we operate in a world where many coding questions you’ll encounter on the job can be solved by a quick Google search. Unfortunately, the expectation for your problem-solving intuition is that you have committed to memory the fundamentals of algorithms & data structures. Some people disagree with this style of interview. I see merit to both arguments, but the point is that, especially as someone interviewing for a junior position, you’ll be better off if you take the time to master these building blocks. Society has somehow evolved to encourage and reward people to play the victim, but channeling your frustration about coding challenges in this way will not lead you to any type of fulfillment–or that job you want.

If you’re an undergraduate at a competitive school, you have the luxury of being able to cast a wide net. This means you’ll be inundated with the classic “Cracking the Coding Interview” (CtCI) questions in pre-screens (usually fully-automated online testing problems perhaps not requiring any hiring manager involvement). If you find yourself in a position where you can easily get first-round interviews with a dozen of the hottest tech companies, it’s worth focusing on Leetcode questions. Just as it’s economical for large companies to construct walls of coding challenges, it’s an economical use of your time to learn how to scale these walls with speed, accuracy, and poise.

For those who are in a similar position to me–two years after graduating summa cum ordinary at a non-target school–it’s more important to focus on the fundamentals. So, rather than diving right into CtCI or Leetcode, pick up CLRS and treat it as you would as an undergraduate taking a real class. Spend hours each night reading and practicing problems. Fall asleep with the book next to you. Not a requirement, but these things happen.

I recommend a CLRS-centric prep plan because eventually, you will realize that there are only so many types of problems and so many applicable heuristics. Initially, I found that it was easier to associate solutions with fundamental concepts (and corresponding proofs and things like loop invariants) than with other problems. If you don’t know the fundamentals and you’ve focused on memorizing “most popular” question solutions, when you get stuck in a real interview, you’re 1. Pretty much screwed and 2. Bound to sound like a blabbering idiot.

Once you feel you have a solid grasp on CLRS, I do recommend CtCI. Time yourself, push yourself, and, most importantly, don’t lie to yourself. Enforced by CLRS, after just two weeks I was flying through my older version of CtCI. The benefits of CtCI over the online practice sites are that it provides 1. Fewer distractions and less temptation to cheat since it’s offline 2. More realistic practice for whiteboarding (e.g. no compiler or code analysis) and 3. Most patterns used in common interview questions, including some light system design.

As far as online courses, I did several run-throughs of Interview Cake. I liked the format and felt it was worth the money, but the way the site is constructed makes it seem that there are many, many more interactive questions when, in actuality, there are about 50. I felt that the best value was Coderust 2.0. Though the questions aren’t interactive, I found it to be more effective and more representative of what I encountered in interviews than Cracking the Coding Interview.

Aside from this, I used Hackerrank to focus on my language implementation skills (speed and correctness). Then, I would typically complete one or two Leetcode problems a day. I will admit that I never reached my ideal level of fluidity on Leetcode. I practiced talking through problems regularly, and this helped me a lot, but even in the end I found myself regularly stumped by the hard problems.

2. The Ops Part

There are two common interpretations here. In one, Ops means incident triage and debugging. In the other, Ops is more related to automation and, broadly, operational process. There’s no way to know what you’ll be asked beyond studying the job description or the background of your interviewer. In my experience, in-house recruiters won’t be particularly helpful in spelling it out for you and it isn’t defined anywhere on the internet. This category of questions usually won’t be covered in Software Engineering interviews.

Part 1

The DevOps engineer supporting production systems and helping SEs use their time more efficiently will wear many hats. I found that Linux and what may have been previously categorized as SysAdmin questions were prevalent. Unfortunately, it’s nearly impossible to study for this category. There is just so much material and if you don’t have hands-on experience, it will be blatantly obvious to your interviewer.

For the positions I interviewed for, it seemed that a pragmatic rather than obsessive knowledge of Linux was being tested. Though I saw on Quora that Facebook’s Production Engineer position will test you up and down the operating system stack (all the way down to what system calls are getting made), nothing I encountered was that hardcore. The book Understanding the Linux Kernel was recommended. Though I see its merit and read a couple hundred pages, it was too dense and academic to be immediately useful to my daily work or the positions I was interviewing for.

As a general guide, I can’t recommend the UNIX and Linux System Administration Handbook enough. I read it front to back–over one thousand pages. It immediately had an impact on my day job, my side projects, and my fledgling interview processes. Judging it by the cover, it might seem like it’s outdated or for dummies, but it’s a fun read and there’s nothing out there like it.

A useful go-between was Advanced Programming in the UNIX Environment. It’s in some ways a blend between the SysAdmin Handbook and the Linux Kernel deep dive. It includes code samples of common Linux commands which helped me to internalize different concepts. Most people probably had a few C assignments in college where they had to implement some Linux functions–I see this as the extended workbook that you’d make your way through if you really needed extra credit.

You have to be putting knowledge into practice. If you don’t have the luxury (or curse) of already having a job while you are searching for a new one, I recommend fooling around with projects in the cloud. DigitalOcean is the best, in my opinion, because it’s the simplest way for you to practice being a SysAdmin, or Systems Engineer, or whatever you want to call yourself. If your learning has been hampered by not having root access at work, working in the cloud forces you to take responsibility for your own systems. Now, if you’re interviewing for a position that requires cloud expertise (many Fortune 500 companies require it, or at least think that they require it), you have to follow a more accelerated learning path. However, be aware that going from having worked on one AWS project while an undergraduate to a mid-level position requiring cloud expertise might present a significant even once you get on the job. So please be honest with yourself and don’t misrepresent your experience.

When it came to questions in this realm, I found that I improved significantly from interview to interview. I knew what to brush up on and what peripheral topics to explore. Though it was a deep time sink, I developed a study guide/flashcard system with topics I was asked about as well as questions I came up with on my own (some nonsensical, some scenario based). Again, I can’t begin to speculate on how much time I spent on this document, which today spans more than twenty pages. However, by the end of my job search, I was systematically (ha-ha) nailing interview questions, was performing better at my job, and had dozens of ideas for intriguing blog posts.

For example, do you need to read an entire book about networking? Did I even learn about networking in college? It can seem overwhelming, but spending a couple hours here and a couple hours there diving into ping, ssh, and multicast allowed me to answer pretty much every networking question I was asked. Granted, I dove quite deep, but it was worth it and I never felt like a high level of proficiency was unattainable.

Part 2

The other part of Ops, as I see it, is the release, configuration, and monitoring of systems. Technically a part of the development workflow, testing is also a major component. Talking about testing is a great way for a hiring manager to screen an experienced candidate as well as a great way for a job candidate to screen a potential employer.

Because much of my work at my previous job revolved around our build, release, testing, and monitoring systems, there wasn’t much prep to do here aside from thoroughly documenting and reviewing what I’d already done. I had a unique advantage in that I had somehow come to be the lead (sole) maintainer of the build system and monitoring system. My day-to-day was quite stressful, but this increase in responsibility served me well when it came time to interview.

I don’t want to forget the topic of Docker and containerization. There is a neat chapter on containerization in the SysAdmin book. My favorite books for brushing up on Docker are Docker in Practice and Docker Deep Dive. I found that both books helped me reconcile my understanding and usage of Docker with how it was being embraced by the industry as a whole.

Both part 1 and part 2 are difficult lines of questioning to answer without at least a year of professional experience. Obsessive prep work can only be helpful up to a point. Where many people swear by grinding through Leetcode questions for coding interviews, you really need to get your hands dirty to gain the same level of proficiency with these topics. I highly recommend embarking on complex self-guided projects if you feel your work experience is not sufficient.

3. System Design & Problem Breakdown

It seems that now engineers are widely expected to have full-stack capabilities, sharp capacity planning skills, and the ability to architect a complex system in a narrow span of time. When I had my first onsite interview during this cycle, I was totally unprepared. I didn’t even remember this being mentioned when I was prepping for my first internship interview. My mental math was embarrassingly bad while under pressure. I guess it’s worth mentioning that my last onsite interview before Winter 2018 was in December of 2014.

Anyway, CtCI does have a useful section on capacity planning. The book Programming Pearls, which I have trouble recommending as a front-to-back read, is an outdated book that has some aspects of capacity planning as well as some of the original editions of interview questions commonly seen today.

What was most helpful for me was the course called Grokking the System Design Interview. These problems are a lot of fun to work through. Using real-world system examples, it’s easy to internalize why different design decisions were made. I ended up spending a lot of time on this course because it was totally new to me. I hadn’t had any experience with this type of estimation at my previous job.

Once I felt comfortable with System Design, I began mentally deconstructing and weighing supposed design decisions for every website I visited. This was a useful, though ultimately imprecise, exercise. I also started a project to build a better version of Goodreads so I could put such design into practice, but I never ended up taking it beyond a single blog post and a few sheets of looseleaf paper.

4. Your Story

I subscribe to the belief commonly spouted by those seeking high-finance jobs that, especially when you are coming from a non-target school, you must have a story. Not a dramatic fairytale of a story like “My house burned down when I was six so then I had the drive to become an NCAA Division I race-walker and then taught myself all the necessary skills to rebuild my childhood home without the use of power tools.” But you need a cohesive narrative that relates to why you’re seeking one job in particular and, if applicable, why you’re leaving your current job.

To be honest, I was unhappy in my previous role and I spent many months anxiously turning over different grievances in my head. The result was that, when I finally stepped into an interview room, the way I told “my story” was so lucid, succinct, and, sometimes, relatable that interviewers would be taken aback. Also, internally, I knew exactly what I wanted from a new job and what red flags and landmines to watch out for as the conversation progressed.

This wasn’t something I consciously practiced. I think that with the influx of students studying Computer Science just because it leads to stable, high-paying jobs, a lot of them pursue jobs for the wrong reason with little underlying passion. Who am I to judge–I guess–but there is something fundamentally wrong and disingenuous about there being “soft skills” guides with bulleted questions and canned answers on Github.

In this section, your “story” should really just be a list of talking points perhaps starting with why you chose a certain school and major. It should also revolve around your resume and you knowing each line-item in detail without ever having to glance at a copy of your resume.

If you’re asked a particularly bizarre question, there’s nothing wrong with stopping and carefully thinking of an answer. As a whole, though, any conversation should flow freely and shouldn’t seem rehearsed. As a final test, it’s useful to have a friend call you at a random time and drill you on high-level resume and behavioral interview questions.

5. Intangibles and Extras

There are certain parts of your candidacy that go beyond your work experience and education. Sometimes, this might be your intangibles, your human qualities that can’t be faked or improved upon. When people refer to intangibles, it’s usually a good thing, but oftentimes might be used as an argument in favor of a candidate who might not necessarily have nailed every technical question in an interview. Other times, there are extra things, such as your hobbies, that can drastically swing an interviewer’s opinion of you or, in many cases, might be the reason you get asked to interview in the first place. I think it would be disingenuous for you to go and “get cool hobbies” after reading a rambling interview guide on a no-name blog, but there’s a simple question that all candidates should be aware of.

Are you someone that I want to be stuck in an airport with?

I’ve seen hiring managers, specifically at LinkedIn, come out and say that they don’t care if you pass the airport test. However, at small and mid-sized companies (and, often, in small and mid-sized teams in large companies) the airport test is often the final step in determining who to extend a job offer to. In my opinion, my high airport testability has served me well in the tech world. I possess the self-awareness to say that in some cases it has allowed me to secure opportunities ahead of people who are objectively smarter than me.

Sure, I’ve read the book Soft Skills. In fact, I lay claim to having the highest-rated critical review of the book on Amazon. Like I said above, I don’t think that it’s appropriate to manipulate your interactions with people in the interview when those are the people you’ll be spending hours a day with for the next several years. Be yourself. If being yourself means working hard to develop a sense of confidence and introspection that allows you to navigate interpersonal relationships with ease, then do that. If it means ensuring that you have proper table manners and can get through an interview dinner without making an ass of yourself, then do that.

This being said, it’s meaningful to a hiring manager if you have hobbies that embody your interests, personality, and passion. I’ve actually seen resume guides that recommend not listing your hobbies or unrelated side projects. This is nonsensical. Some of the best exchanges that I’ve had in interviews have occurred when I’ve strayed off the beaten path and talked about things like my blog, my book reviews, my side projects, my fraternity, and my dreams. Sure, talking about these things presents a certain risk, but it helps separate you from the dozens of other qualified candidates interviewing for the position.

With this being said, I’m out of things to say. I hope that at least one person can use my experience to better prepare for interviews.