Work Sample Tests:

Bring Your Own Code

Welcome back to my series on work sample tests. This post describes one of the kinds of work sample test I use (and suggest others use as well). For some theory and background, see the first few entries in the series: introduction, the tradeoff between inclusivity and predictive value, and my “rules of the road” for good work sample tests.

Background

It’s time for an embarrassing admission: I’m something of a hypocrite about coding work sample tests. I strongly advocate for their use in hiring engineers, and yet I get salty when asked to code myself! That’s because I’ve written a ton of open source code in my career. If you want to verify that I can write code, you can go look at my GitHub. Writing some new code feels redundant.

If you’re hiring engineers, some candidates will be in the same boat. They’ll already have code they can share: side projects, open source, and so on. It’s a bit silly to give these candidates coding homework if already have code they share. The work sample test isn’t about solving the arbitrary problem you’ve created; it’s about seeing that the candidate can code and talk about it.

So, if you’re going to use coding homework as a work sample test, you should also offer to let candidates submit something they’ve previously written. This should not be required: you can’t just ask for candidates’ GitHub accounts and dismiss those who don’t have code to share. If you do, you’ll be missing out on some terrific candidates. Some of the best engineers I’ve worked with don’t have much in the way of a public persona: they write amazing code at work, but because it’s all work product they don’t have anything they can legally share.

Who this is for: anyone interviewing for a technical role, as an option instead of coding homework or pair programming.

What it measures: the same things as the previous two exercises: the ability to write and reason about software.

The Exercise

This one’s fairly simple: candidates submit a small bit of code they’ve previously written, you review it, and then interview them about their code.

I do recommend putting a few guidelines around what you’ll accept:

• The code sample should be short: something around 500-750 lines of code is ideal. It needs to be short enough for you to understand in a reasonable amount of time. I don’t recommend setting a hard limit (either minimum or maximum), but instead simply tell folks to aim for something around this size, but that shorter or longer is OK, within reason.
• The code should mostly stand alone - again, it needs to be something you can understand without a huge amount of context. A small part of a larger codebase can be OK if it’s easy enough to understand a small part without much context. For example, I’ve had candidates send me a large Django app, but tell me to focus on a couple of models, or a few views, etc. This can work just fine if the codebase is cleanly factored.
• Open source code with multiple contributors is fine but the candidate must have written the vast majority of the code they submit. It’s OK if there have been a few patches from other contributors here and there¹, but the vast majority (like, 90%+) should have been written by the candidate.
• Ideally, the code should be something you can run (with any instructions you might need). This isn’t critical, but it’s nice to be able to execute the code and see what happens as part of your review.
• Because real-world programming includes writing tests and documentation, ask for tests and documentation as part of the submitted code sample.
• Finally, include any restrictions around language, dependencies, etc., if you have them. See the coding homework exercise for a discussion of those limits.

Interview

Like all the other exercises, the code itself is the start of a discussion, not a pass/fail.

After you receive a candidate’s code, you’ll want to interview them. This is a very similar interview to the coding homework interview, with just a couple differences relative to that interview:

• Schedule more time: at least a full hour, and up to 90 minutes. With coding homework, you’ll understand the problem space very well, and most candidates’ submissions will look similar. So you can get right into the more interview-y questions. But with code the candidate wrote in the past, you’ll find you need to spend a good deal of time just having the candidate walk you through the code. It’s also OK to take more time here because they candidate won’t have spent as much time writing the code as for the other options.
• So, you should focus on the “explain this to me”-style questions. You should ask more questions like: What were the problems you were trying to solve? What’s the background? How does the code work? How does it fit into other parts? And so forth.

Other than that: all the potential questions, behaviors, and positive/negative signs are the same here as they are for the coding homework interview.

Discussion

As I discussed earlier in the article about my framework for good work sample tests, you should provide as much choice as possible. This is why when hiring engineers, I offer three different options, all of which I’ve now covered:

1. Coding homework.
2. Pair programming.
3. Submit previously-written code (this article).

In practice, most candidates seem to choose the first option – I’d guess around 70% do the coding homework. A small but sizable minority, maybe 20-25%, submit code they’ve written in the past. Only a few (5-10%) choose pair programming, but it’s enough candidates that I like to include the option.

There’s no escaping the fact that this is a lot of work! Preparing for three different options, briefing candidates on all three, working through their submissions… it’s a lot. It can also make decisions difficult: if Jules did coding homework, but Vincent submitted previously-written code, and both were strong, how do you decide who gets an offer?

I think it’s reasonable to only offer a single option (if you do: make it coding homework). But, you need to be comfortable with the fact that you’ll lose some candidates who are unwilling to do your assignment but would take a different option. These are often the best candidates: those with the deepest resumes and most experience can usually get offers from companies that don’t require work sample tests at all. So, you want to make sure you have enough options that the best candidates will want to engage. Lack of choice will also harm any diversity efforts: you’ll tend to get a narrow range of candidates who are comfortable with your work sample test style.

I’m always going to choose more work on the side of the hiring manager to get a broader and more diverse candidate pool, but, yeah, it’s a lot. I won’t blame you for offering less choice… but if you can, you should.

Next up

Now that I’ve covered the most common exercises I use, the next parts of this series will cover a couple of exercises that I use less frequently, but that can be helpful for certain kinds of roles. We’ll talk about flipping the script and asking a candidate to review your code, and after that cover simulations and “lab” environments.

Finally, I’ll cover some of the kinds of exercises I don’t recommend – and explain why – before wrapping up the series by answering some questions. If you have a question that you’d like to see included in that wrap-up, send me an email or tweet at me.