A few months ago, I was asked to speak to graduates aspiring to be PMs at a startup school about my product management interview process. Now, to be clear, I haven’t worked for any of the valley unicorns and I’ve surprisingly never had formal interview training. Most of my process is “startup organic” by making mistakes and absorbing feedback from others, as well as books likeCracking the PM Interview.

So, with that in mind, here’s my interview process and some questions.

Background Notes

I like to ask the same questions. I find asking from the same pool of questions, enables me (or a team) to better pattern match the candidate’s response

I lean towards open-ended case style questions where the question is really a story and we build along the way with follow-up questions

I prefer questions that don’t have a right or wrong answer. This is much more about understanding how someone thinks and less so about remembering exact frameworks

I generally avoid looking at resumes and/or LinkedIn profiles. I sometimes can’t avoid this if I have to solicit and/or screen the candidate but when I can, I find this helps me eliminate any potential bias

Each of my questions have a specific goal, whether it’s strategy, design, analytics or other. Ultimately, I believe product management is a mindset

Questions

Q1: Tell me about a (software or hardware) product experience that you disliked? (something you haven’t worked on)

How do you describe the product and background? From a UX perspective or more high-level? From a customer perspective?

Are you specific?

Let’s talk through some edge cases

Are you inductive and/or deductive?

How would you fix it? Let’s design, whiteboard, and iterate?

What if the fix didn’t work?

If not obvious, my goal with this question is to see if you have product empathy, if you can focus on what you think the problem is, and how you would design to fix and test it. I generally steer this question towards design (personal bias) but that’s not required.

The “not so great” answers are often very high-level discussions about why iTunes sucks — but without any product detail. Use the whiteboard and start at the top-left so you have enough room to answer.

Q2: You are the PM of Google Music and you need to present to the CEO a strategy to compete with Apple Music. Discuss

Do you ask clarifying questions? How opinionated are you? Do you create a hypothesis? Are you open to alternative ideas?

How do you organize your thoughts?

And I like to interject with alternate potential angles-of-attack (e.g. geo, biz model, go-to-market etc.)

Honestly, this question could literally be about any major product between two companies. I intentionally choose something crazy big to see how the candidate narrows, organizes, and focuses.

Using the whiteboard is a must here. The worst answers are very disorganized in their approach and often only consider one insertion point and/or may not have a strong hypothesis. Sometimes, I may even ask the candidate to try to outline some slides to see if they can organize their ideas.

Q3: You are the CEO of an airline and you have unlimited budget (full reign to change anything you want with the plane or the airport). Design a faster way to board a plane.

How out-of-box and/or creative are you with your thinking?

Are you analytical? Did you think about the edge cases? I usually hint along with scenarios you may have missed (e.g. handicap, families, seating preferences)

Failing to ask any clarifying questions is not good (and this is thematically true amongst most of my questions). As mentioned before, there are generally no right or wrong answers. Some answer with very creative ideas: “park the plane into the airport and let people board directly as they arrive to the gate.” Others have very in-the-box answers but maybe very customer-centric: “mandatory use of a mobile boarding pass and potentially giving people a real-time seat assignment based on preferences and order of arrival.”

As I’ve written, the goal from these questions is not necessarily to rule you out as a candidate but rather to identify your strengths. Are you creative, analytical, or organized? Then I determine if that best matches what we need at the company at that time.

Have you really managed and iterated on a product with an engineering team?

How do you deal with different scenarios (e.g. CEO asks for a feature last minute; or your team keeps missing dates)?

What ratios and graphs would you like to have or review in this process? Sometimes, I’ll draw some odd plots of odd ratios or parameters and ask the candidate to interpret (for example, a burn-down graph that doesn’t burn down until the day before the sprint ends).

Usually, I can pretty quickly ascertain whether the candidate has actually iterated a product with an engineering team from v0 to v1 and to v1.XX. This is not to say that all product managers need to be hands-on project or program managers, but this question provides more data points to match against the role we need in the company.

Q5: Psuedo-code a load balancer

Can you write an algorithm?

Can you draw a simple architecture diagram?

Do you know basic data structures?

Can you optimize and determine edge cases?

With this question, I’d always start with an architecture diagram while asking clarifying questions. The best outcomes usually start simple and then iterate on edge cases and performance. I don’t really care about programming language or syntax but mostly about whether the candidate has some analytical chops to earn the respect of the engineering team and whether he/she can think technically particularly when writing requirements for something like backend APIs. I like this example of a Load Balancer because there are so many opportunities to increase complexity while continuing to optimize, so you can easily determine the technical ability of the candidate.

Q6: You are the PM of the Gmail app for the iPhone (generally indifferent about which mail app and which platform, but it should be something the candidate uses). You’ve been asked to present an analytics report to the management team. Outline your slides

Can you define assumptions and goals to measure?

Can you identify the key metrics? What ratios are relevant and how would you represent this data (e.g. over time, per user, per session?)

How would you get and normalize this data?

I love this question, and the product can really be anything, but I like to choose products that the candidate has used. The “not so great” answers tend to be very disorganized and/or are lacking very obvious actionable metrics. I can usually tell pretty quickly how much time this potential PM has spent in growth or engagement — the vocabulary they use to describe an action or metric can be revealing. That said, just because you may not know the right terms, doesn’t mean you can’t deduce it and I’ve definitely had experiences interviewing candidates that had obviously never performed analytics but were smart enough to deduce what they would want to do.

The second part of this question requires organizing these metrics into presentable slides (at an outline level). I come back to slides often because I feel a large part of product management success is championing a strategy and plan and winning over your various constituents (employees, peers, management) and so becoming a pseudo-salesperson is a must.

Q7: What are you most proud of from your career?

This question is very telling. I find those that have managed teams often discuss moments where they had to carry a team through a tough period. Those that have been more IC (individual contributor) will often talk about how they designed a specific feature which was received well. Those that lean more project management will often highlight how they shipped their product on time and so forth. The answers are varied but they can signal strengths and humility.

The “not so great” answers are always too broad and do not highlight a specific accomplishment. This is certainly in the bucket of questions I’d suggest prospective interviewees to think about before starting an interview process.

Summary

In short, these are a subset of ~15 types of question templates I use. I’ll decide on the pool of questions to ask with my interview team based on what we need for the role (e.g. hiring a VP of Product would put greater emphasis on being able to mentor product managers, resolve resources, recruit, break strategy down into tactical roadmaps).

Lastly, I immediately (almost always the same day) summarize my feedback on the candidate via email to the interview team (after everyone has had a chance to complete their interview). I will often also score the candidate’s performance on the questions I was responsible for asking. In the end, we, the interview team, have a quick meeting, discuss the candidate, and make a decision.

I would love to hear reader responses on how I can improve my process, what I might be missing in terms of questions, and what tweaks I could make.

It should also be noted that I’m not sure I could reasonably pass my own interview process! But, hey, it’s all about surrounding yourself with smarter people.

This past weekend, I had the pleasure of moderating the “Testing Your (Aritificial) Intelligence” panel at SXSW. On the panel, we had Dror Oren from Kasisto (vertical messaging assistant for banking), Alex Lebrun from Facebook M (horizontal messaging assistant) and Dimitra Vergryi from SRI (runs the speech research lab). Much of our early discussion was regarding the future of assistants in-part drawing on some of my experiences from SRI and Tempo AI but we quickly moved to the hot topic of messaging bots and the role of AI.

If you’re not familiar with messaging bots, I encourage you to read “The Bot Paradigm” from The Information as well as Jonathan Libov’s super-aggregation of messaging UX design. In short, messaging (communication) represents our primary workflow (day-to-day) and as exemplified by applications like WeChat, messaging can be used to facilitate other experiences (eg shopping, sending money, customer support).

This new modality has the disruptive potential of replacing all app experiences and if so, would represent an opportunity as significant as the iPhone App Store was to publishers.

Some panel takeaways:

Horizontal (general purpose) assistants are hard! Users do not know what to ask nor remember what primary use case to use the assistant for. In addition, the technical complexity of the system is exponentially greater in that you have to deal with out of context and extreme queries. For example, Alex mentioned they no longer intend to support “pet delivery” in FB M since that’s a use case out of their machine automation wheelhouse.

Standalone messaging apps don’t stand a chance. I found this feedback interesting in that they are advocating that messaging bot experiences need to be built upon the existing large players (eg WhatsApp, Slack, Facebook Messenger, Google Hangouts etc). This intuitively makes sense since the App Store is no longer a great growth channel.

App bot discovery and the ultimate app store for messaging bots is still unclear. Having played with the app store within FB Messenger, I found the workflow to be sub-par. Alex and Dror suggested displaying related app bots as you type within Messenger (in-context). This certainly would be an improvement at smaller scale but will remain a challenge if we have 100s of K of messaging bots. The ultimate design of the APIs that the messaging apps provide will be critical.

Personas are necessary for messaging bots to. I found this interesting in that personas can certainly lighten the mood and set expectations but can also be tiring when you want expedience. See slides I presented on mobile AI/UX design at SXSW a previous year (wish I had a video of the presentation with the voice-over).

Machine-powered conversational NLP is a very long-way out. Dror felt you must go vertical and thus their focus on the banking sector. Alex said that the FB research team described conversational NLP as level 10 technical difficulty and AlphaGo as level 1 (my FB post on the AlphaGo AI milestone). All of the panelists also acknowledged that building a data corpus (collection of queries/conversation) for training is the real challenge in any AI application.

Key initial industries to be disrupted by messaging bots were financial services, customer service, commerce and travel. Not surprisingly, some of these categories are identical to what we saw with early vertical Siri-clones.

In summary, messaging bots represent the next growth channel and will spawn new billion dollar opportunities; excited to see it happen!