Work Sample Tests:

Pair Programming

Welcome back to my series on work sample tests. This post describes one of the kinds of work sample test I use (and suggest others use as well). For some theory and background, see the first few entries in the series: introduction, the tradeoff between inclusivity and predictive value, and my “rules of the road” for good work sample tests.

Background

I tend to prefer asynchronous work sample tests, like coding homework. I think they more closely simulate real work where engineers write software with a great deal of flexibility. The flexible scheduling of asynchronous exercises (i.e. “work on this whenever you like”) works better for the majority of candidates.

But for some candidates, and some roles, synchronous exercises work better. By “synchronous” I mean: work sample tests that are explicitly scheduled, and that has both the interviewer and the candidate working directly together at the same time. In these cases, I often turn to pair programming.

Who this is for: like coding homework, this is suitable for any role where writing code is a major part of the job, but usually only as a second or third choice option. I like to offer it to give candidates as much choice as possible (rule #4), but I don’t recommend it be the default option. The exception is if pair programming is something your organization does so frequently that it’s essentially required. If the job requires pair programming, you probably want to see that a candidate can perform under that structure. Pair programming is a bit controversial; some developers can’t stand it. Requiring pair programming as a work sample test will turn off many candidates, which is only a good thing if they will be required to pair at work. If so, letting them self-select out of your process is good: they wouldn’t be happy in the job anyway.

What it measures: again, mostly the same as coding homework: the ability to write and reason about software. It also measures the ability to perform in a pair programming work style, as discussed above. Additionally, a pair programming work sample test gathers more signal about collaboration than asynchronous coding homework, which can be nice for roles that’ll have a strong mentorship/technical leadership component.

Interviewers, be warned: pair programming is NOT a good exercise if you don’t have skill at pair programming! Pair programming is more difficult than solo programming for many (most?) programmers. To be most effective, pair programming depends on a foundation of trust and psychological safety. This won’t exist during a job interview, where there’s a massive power imbalance. You need to be skilled enough at pair programming to get through some of the tough spots that might crop up and skilled enough at interviewing to filter out the noise caused by the awkward context (more on this in the discussion of behaviors to look for, below).

This is why, again, I tend to only offer pair programming as an option – not the default. Few candidates chose it; I’d estimate maybe 5-10%. But that’s enough that I like having it as an option.

The Exercise

For the most part, the exercises used for pair programming are the same as those used for coding homework. You can use most of the same exercises covered there. Most of the same structure surrounding the exercise applies, including choice of languages, minimizing dependencies, including tests and documentation, and sending a complete briefing ahead of time.

Turning to the structure of the exercise, the scheduling and logistics of the pairing itself, there are a few practices I suggest:

• Because pair programming shows how a candidate writes code (you get to watch them), you don’t need a separate follow-up interview. I use a short debrief after the coding parts, and between that and the pairing itself feel very confidant about the information I’ve gathered from the test. This means we’ve saved a bit of time from the overall selection process, so it’s OK to break the 3-hour limit slightly (see my suggested schedule, below).
• If you do intend on making the exercise longer than about 90 minutes, schedule some break. Interviews are exhausting and pair programming can be too; be nice to your candidates and give them time to get some food or water, use the bathroom, and decompress.
• Practice real pair programming – you need to write some code, too! If you (the interviewer) aren’t spending some of the session writing code, it’s not pair programming. It’s watching someone code, which is terrifically stressful for the candidate and thus a very bad interview practice.
• It’s better to choose a pairing structure beforehand (and tell the candidate which you’ll be using), rather than trying to negotiate that structure in the moment.

Given these points, here’s a typical schedule for a pairing work sample test:

Post-exercise interview

As noted above, I don’t suggest a separate post-exercise interview, just a short debrief. I suggest treating this like a classic retrospective, asking questions like:

• How’d that go for you overall?
• What went well?
• What could have gone better?
• With 20/20 hindsight, what would you have done differently?

I also recommend you review the post-exercise interview questions from the coding homework exercise. If they come up naturally during the pairing sessions, take a minute to discuss them in the moment. You’ll probably find most of them will get answered just by observing what happens, and you can use them to help guide the debrief if needed.

Behaviors to look for

For the most part, this is the same as for the coding homework exercise; see that exercise’s discussion of behaviors, positive signs, and red flags.

Beware of reading too much into behaviors that stem from discomfort with pair programming! Pair programming is difficult for many under ideal circumstances, and the interview context is pretty dang far from ideal. Julia Evans has a good summary of what people find difficult about pair programming. As she writes about, feelings of exhaustion or frustration are common, as in anxiety about being watched while coding, and so forth. Signs like these should be greatly discounted or ignored entirely: it’s much more likely to be caused by the context than by the candidate.

Discussion

Pair programming is more difficult to execute than coding homework, and many candidates will find it difficult or uncomfortable. I like to present it as an option since some candidates prefer it. It does have some advantages – particularly in how much information you can gather about a candidate’s technical collaboration and communication skills. Still, it’s best as a second (or third) option, not as a default, unless pair programming is an expected part of the day-to–day at your workplace.

The technical side

So far I’ve avoided discussing the technical details of how to conduct a pair programming session because that’s a sticky one. In person, pair programming is easy enough, but I mostly assume remote interviews. And the options for remote pair programming are… well, they can be pretty great… when they work. Unfortunately, nothing I’ve found is reliable enough that I don’t experience some failures from time to time.

My tool of choice these days is VS Code Live Share. VS Code is free, runs nearly anywhere, and the Live Share extension is easy enough to set up and works most of the time.

Whatever you choose, make sure you tell your candidates well ahead of time what you’ll be using, and give them detailed setup instructions. I also suggest offering to schedule some time a day or so ahead of the interview to test it out and debug any issues.

Your tooling still might fail (e.g., for me, Live Share simply refuses to connect about 5% of the time, seemingly randomly.) So whatever you pick, you need to be prepared to be flexible, to fall back on more simple tooling (like a simple video call with screenshare), and, most importantly, do not penalize candidates for technical failures.

Alternate versions of this exercise

Like coding homework, there’s a nearly endless range of exercises you could adapt to pair programming. However, the exercise itself generally needs to have an even smaller scope than a solo coding exercise. The ritual and ceremony around pairing will add some time. You may spend some time fighting with technology. And the coding itself will go slower when pairing than if the candidate was solo¹.

There are also a few different pairing structures that could work well in an interview context. I’ve only used the “driver/navigator” structure, which is what I’ve used in the example above. I imagine “ping-pong” might work very well, too².

Questions?

If you have questions about this example or anything in the series, send me an email or tweet at me. If I get some good questions, I’ll answer them in a series wrap-up.