Unpacking Interview Questions:

Types of Interview Questions

This is a follow-up to my series on unpacking interview questions where I dissected some of my favorite questions to ask in job interviews. In that series, I touched behavioral interviewing and wrote a little bit about why it’s valuable. But to go deeper, let’s dissect the different types of interview questions, and where they’re valuable (or not).

There are three types of interview questions¹: behavioral, hypothetical, and trivia. Behavioral questions are the gold standard; they’re the most effective at predicting job performance. Hypothetical questions can be useful in certain circumstances, if used correctly. Avoid trivia.

Behavioral Questions

Behavioral questions are ones where you ask a candidate to tell you about some situation in the past, and how they behaved (what they said, what they did, etc.) They usually start “tell me about a time when…” or “describe a situation where…”, and so forth. An example of a good behavioral question is this question about disagreeing with a coworker. I explained in that post why behavioral questions are so important when asking this type of question:

Behavioral questions are especially valuable when asking questions like this one. If you ask in a hypothetical or general way (“how do you handle conflict at work?”) you’re likely to get trite, best-case-scenario answers (“oh, it’s important to understand both sides…”). Most people know intellectually that there’s a professional way to resolve conflict, and if you ask them to speak in generalities they can usually talk a good game.

But, faced with real conflict, not everyone responds as well as they’d like. Someone might know that they should attempt to understand both sides of a conflict, but still get caught up in their own opinion and not actually do that in real life.

Behavioral questions are the gold standard since they are the most effective type of question. By “effective” I mean: of these three types of questions, performance on behavioral questions best correlate to actual job performance. There isn’t much research here, but the research that does exist seems to validate that behavioral questions are more effective. My experience backs this up: the limited dataset I have on hires I’ve made job performance shows that performance on behavioral questions has the strongest correlation to subsequent performance.

So: whenever possible, you should try to ask behavioral questions.

Hypothetical Questions

However, not everything you want to measure in an interview lends itself to a “tell me about a time…"-style question. For example, I was recently interviewing for an CEO role at a small startup. I wanted to understand how candidates would approach leading this organization. I could use a version of my tell me about a project you led question, and that would tell me some things about how this person led other organizations. But, small organizations can differ pretty widely, and this startup has some unique challenges. To make a good hiring recommendation, I needed to understand how candidates would lead this organization.

So, knowing they’d already been told a fair bit about the company, its staff, and the challenges, I asked a line of questions like,

• “Based on what you’ve seen so far, how would you approach leading this startup?”
• “How would you structure your executive team?”
• “How would you work with the small product/engineering staff?”

These are hypothetical questions. They’re not about behavior in the past; they’re hypothetical questions about potential future behavior.

Hypothetical questions aren’t as valuable as behavioral ones, as outlined above: people are often terrible at predicting their own future behavior. But, there are times when they can be useful, as in this example where the specific approach to this company was important to understand.

Hypothetical questions can also be useful in asking certain types of engineering questions. For example, a question I often ask of more senior engineers is, “if you were starting a new project today, what technology stack would you choose?” If I only asked about previous experience, I might miss better tools that someone might choose if given the opportunity.

The best way to use hypothetical questions is by pairing them with behavioral ones. In the case of the tech stack question, I usually also ask about the tech stack of a current project. In the case of the leadership question, I paired it with a question about previous leadership work. This can help you validate that a candidate’s prediction of their own future performance is at least plausible. It can also set up a nice contrast, where you get a candidate to talk about the differences between the past and hypothetical future, and explain the rationale behind differences.

Trivia

Finally, we come to what I call trivia. Trivia is any question with a single, specific, “right” answer. If Google will turn up the answer to a question, it’s trivia.

Trivia make poor interview questions; don’t use them. That is, there are a small handful of cases where trivia can be useful (I’ll discuss a couple below), but as a general rule they don’t work and are to be avoided.

Some trivia I’ve been asked in the past:

• What’s the current version of TLS?
• In SQL, what’s the difference between an outer and an inner join?
• What’s the standard library Python module for making HTTP requests? ²

The goal of trivia is to quickly assess a candidate’s level of knowledge in some specific domain. That’s a fine goal. But in practice, trivia rarely does this effectively. The problem is: in the real world, memorization of specific APIs isn’t necessary – we all code with our memory augmented by documentation, Google, StackOverflow, etc. There’s a vast amount of trivia available in any domain, and you’re only going to pick a few to ask about. This makes it quite likely even qualified candidates won’t know some of your questions off the top of their head.

To make this concrete: Django’s documentation runs over 2,000 printed pages. Every page has at least a few facts that could be used as a trivia question. So there’s something like 5,000 - 10,000 potential trivia questions about Django. I’m one of the three people who created Django, but there’s no way I have those thousands of facts memorize. Even I write Django code with the documentation open³ right next to my text editor.

If an interviewer asks me some random Django question, I’m quite likely to not know the answer. So, if your trivia question is likely to predict that I don’t know how to use Django – is it really a good question?

Of course not. If you want to measure that expertise, instead try:

1. Asking behavioral questions about a area you’re looking for, and ask the candidate to go deep on details with follow-ups For example, “tell me about a site you built with Django”, followed by asking them to get specific about the technical details.
2. Have some sort of practical exercise: ask them to write code (before or during the interview); look at code they’ve written in the past; ask them to review your code; etc. I’ll have more to say about practical exercises in the future.
3. Check references. If someone claims to have shipped a project using Django, call someone they worked with and ask about their work and their role. Checking references is also another post; I’ll have more to say about that in the future, too.

I hope this clarifies why I focus on behavioral questions, and why I think you should, too. It’s hard to measure interviewing techniques concretely, but there’s a bit of data (and ample anecdotes) that show that behavioral interview questions work better than the alternatives. If you’re not using them already, I hope you’ll try. I expect you’ll see better results.