The biggest interview fail I ever tried was the take-home problem

Hiring was always one of the biggest challenges in running DoubleMap. Our limited budget, small team, and geographic location put a lot of constraints on hiring, and I was always interested in learning about how other companies interviewed candidates.

Around 2013/2014-ish, I read about companies giving candidates take-home coding problems and decided to try it out. The basic idea was that a take-home problem offered candidates more time flexibility because it’s async and is a better simulation of the work compared to whiteboard questions.

I liked the idea, so I cooked up two take-home problems that candidates could choose from: a front-end problem where you were asked to write some HTML/JS to talk to a JSON API, and a back-end problem where you were asked to create some JSON API endpoints given a database.

In both scenarios, the scope was minimal (2-3 API endpoints, very simple data model) and a live API or database was provided to develop against. As the candidate, you could choose whatever libraries you liked (and even whatever programming language you liked for the API implementation). In addition, candidates were explicitly told that they could ask any question about the task and expect a timely response.

There was plenty of room for varying implementation details and also plenty of topics to discuss as a follow-up. It seemed great… in theory.

In practice, it was not a success.

Why? I made two big mistakes in giving out this problem.

Mistake 1 was including new grads. For people who have had experience with web apps, this task should have been a walk in the park. For new grads who didn’t have experience with web stuff from their coursework, it was jumping in the deep end.

Theoretically, we had lower expectations for what a new grad would come up with, but a hard fail not only doesn’t give much data, but it’s also demoralizing for everyone involved. Even worse, at the time we were interviewing a lot of new grads relative to more experienced candidates.

This wouldn’t have been so bad were it not for the second mistake.

Mistake 2 was not time-boxing it strictly enough. We explicitly told candidates that they should spend no more than a couple hours on it, but we gave them multiple days to actually do it. This was supposed to give them the flexibility to find time in their potentially unpredictable schedules.

The result was that some people failed at the problem, then continued to spend time beyond the expected time to do… something? I distinctly remember someone turning in a collection of web pages that did not function, but they had obviously spent a lot of time on graphics, flavor text, and other things not asked for by the task. (It wasn’t aesthetically great, either.)

I felt terrible. Sure, maybe this told us that we shouldn’t hire this person, but there must have been better ways to figure that out without taking up so much of their time.

What I should have done is ask them for a 2-3 hour window and given them the task at the start of the window, expecting an emailed archive at the end of the window. The candidate may fail, and fail hard, but at least there’s a tighter cap on how long they spend on it.

There were some other minor issues with this type of interview, like the relatively high creation and maintenance time (especially with providing live instances). And in the end, we abandoned the take-home problem not too long after we started trying it out.

It was certainly a lesson learned in unforeseen consequences and although we might have been able to fix these mistakes with more practice, we got a more traditional interview flow to a decent place and used it to grow steadily until we were acquired.

Side note: there was another interview strategy that I did quite like – the choose-your-own-adventure format – that I wanted to write about, but I was pleasantly surprised to see that Julia Evans created some very similar debugging puzzles that you can play in your browser.

Did you have more success with giving take-home problems? I’d love to hear other people’s experiences with this type of interview, both as the interviewer and as the candidate.