The technical interview practice gap, and how it keeps underrepresented groups out of software engineering

I’ve been hiring engineers in some capacity for the past decade, and five years ago I founded interviewing.io, a technical recruiting marketplace that provides engineers with anonymous mock interviews and then fast-tracks top performers—regardless of who they are or how they look on paper—at top companies.

We’ve hosted close to 100K technical interviews on our platform and have helped thousands of engineers find jobs. Since last year, we’ve also been running a Fellowship program specifically for engineers from underrepresented backgrounds.

All that is to say that even though I have strong opinions about “diversity hiring” initiatives, I’ve acquired them the honest way, through laboratory experience.

Though I find it problematic to label any type of hiring as “diversity hiring” —primarily because it tacitly implies to candidates and everyone else involved that they are getting the job not because of their merits but because of their demographics or to fill some quota—nomenclature is not my main critique of these initiatives. Though there are a number of big issues with how employers approach hiring, in this post I’ll talk about the one that’s at once the most subtle and the most damaging: systematically ignoring the technical interview practice gap between traditional and non-traditional candidates. I’ll explain exactly what the interview practice gap is and why it’s so bad in a moment, but first, allow me to tell you a little story.

The benefit of attending an elite CS university

I was fortunate enough to attend MIT for undergrad. Probably the greatest gift that MIT gave me was a big stamp on my forehead that, to this day, makes strangers think I’m smart. But there was another, more relevant gift that gave me a serious advantage over students who did not attend an elite computer science institution: boundless access to technical interview practice.

Not only was there a multi-week course during the month-long break between Fall and Spring semesters that was dedicated exclusively to passing technical interviews, but all of my peers were going through exactly the same thing at the same time. Everyone was interviewing at FAANG for internships and new grad positions, which meant that we could all practice with each other, share our successes and failures, and, over time, internalize just how much of technical interviewing is a numbers game. We learned that bombing a Google interview did not mean that you weren’t meant to be an engineer. It just meant that you needed to work some more problems, do more mock interviews, and try again at Facebook.

Practice, practice, practice

My anecdotal experience aside, we at interviewing.io have actual data. As I mentioned earlier, we’ve hosted close to 100K interviews on our platform. It has taught us two very important things. This interview style is controversial, in part because it’s not entirely similar to the work software engineers do every day but also because 1) like standardized testing, it’s a learned skill, and 2) unlike standardized testing, interview results are not consistent or repeatable—the same candidate can do well in one interview and fail another one the same day.

First, regardless of how strong you are technically, practice really matters. In a recent study, we looked at how people who got jobs at FAANG performed in practice vs. those who did not. Below is a graph of average technical score in interviews and the portion of people with each score who passed Facebook’s interview process.

You can see that technical ability did not obviously associate with interview success. So what did? It turned out that the number of practice interviews people completed (either on interviewing.io or off) had a much bigger bearing, as you can see below. Surprisingly, no other factors we included in our model (seniority, gender, degree, etc.) mattered at all.

Secondly, technical interview performance from interview to interview is fairly inconsistent, even among strong candidates. Notably, as above, consistency appears to have nothing to do with seniority, pedigree, or anything else. In fact, only about 20% of interviewees perform consistently from interview to interview. Why does this matter? Once you’ve done a few traditional technical interviews, the volatility and lack of determinism in the process is something you figure out anecdotally and kind of accept. And if you have the benefit of having friends who’ve also been through it, it only gets easier. But what if you don’t?

In a previous post, we talked about how women quit interview practice 7 times more often than men after just one bad interview. It’s not too much of a leap to say that this is probably happening to any number of groups who are underrepresented/underserved by the current system. In other words, though it’s a broken process for everyone, the flaws within the system hit these groups the hardest—just because they haven’t had the chance to internalize exactly how much of technical interviewing is a game. Sadly, since we’ve started running our Fellowship program for engineers from underrepresented backgrounds, we’ve seen this effect firsthand.

The technical interview practice gap

So what does this have to do with the practice gap? As you can see in the graph above, there was a pretty meaningful bump in performance (almost 2X as likely to pass!) after at least five practice interviews. Five interviews is not a lot for someone who’s actively practicing and knows how the interview prep game works. But it’s a far cry from what companies who are looking to boost their diversity numbers offer their candidates, and it’s equally far from what bootcamps, an increasingly important source of candidates from underrepresented backgrounds, provide their students.

What employers are doing wrong

I won’t name any specific companies here, but tech giants (many with good intentions and with marching orders to boost their diversity numbers) often do a considerable amount of outreach and pre-interview engagement with candidates from underrepresented backgrounds. This outreach is usually an info session where one engineer speaks to a virtual room of candidates from underrepresented backgrounds. The engineer tells them what to expect in technical interviews, encourages them to learn how to articulate their thought process out loud while solving a problem, and recommends resources like Cracking the Coding Interview. Some companies even go so far as to offer their underrepresented candidates a mock interview or two.

Unfortunately, for candidates who are unfamiliar with the process, neither of these interventions is nearly enough.

What bootcamps are doing wrong

Because they’re cheaper and much faster than a four-year program, bootcamps seem like a rational choice when compared to the price of attending a top university. Since 2013, bootcamp enrollment has grown 9X, with a total of 20,316 graduates in 2018. Though these numbers represent enrollment across all genders and the raw number of grads lags behind CS programs, the portion of women graduating from bootcamps is also on the rise and graduation from online programs has actually reached gender parity (as compared to only 20% in traditional CS programs).

Despite these encouraging numbers, things start to go sideways when it’s time for students to start interviewing for jobs, and disappointingly, what bootcamps provide their students isn’t much better than the tech giants with good intentions.

Again, without naming names, I’ve interacted in some capacity with almost every major bootcamp over the last few years, trying to sell them on providing interview practice for their students, and I’ve also witnessed how recent bootcamp grads perform in mock interviews on interviewing.io. Based on these observations, bootcamps seem to have two big limitations: not nearly enough time spent on data structures and algorithms in the curriculum and lack of interview prep support both during and after the program.

Discussing bootcamp curriculum is out of the scope of this post, but I will say a few words about how bootcamps handle interview preparation. Bootcamps, by and large, have pretty thin margins and can’t invest much in paying for interview prep or hiring engineers to do it. As such, if they do offer mock interviews with professionals, just like tech companies, they tend to cap them at one or two, which we know isn’t nearly enough.

If they do not offer mock interviews with professionals (and most do NOT), they instead tend to focus on peer to peer practice (students interviewing other students), or they solicit volunteers from their pool of recent alums. While peer interviews are not entirely useless, they’re a far cry from working with someone who knows what the interview process is like. For the price tag that bootcamps charge their students, it’s appalling that they’re unable to invest resources into the last mile: actually getting their students ready for interviews.

That’s not to say that bootcamp grads are not viable employees. If that were true, I wouldn’t be advocating bootcamps as a means of increasing diversity in software engineering. What’s heartbreaking is that bootcamp students have all the makings of good engineers—they just need to spend an extra 3-6 months after graduation drilling by themselves on the skills their programs should have offered them.

While I have less direct experience with curriculum and career centers at non-elite universities (another contributor to the practice gap), from speaking with students who attend these schools, it’s clear that access to consistent interview prep is hard to come by, especially when compared to what’s available at their more elite counterparts.

The cost of falling short

Why are bootcamps, universities, and employers all falling short and exacerbating the practice gap? With both employers and bootcamps, the teams responsible for facilitating interview info sessions or mock interviews typically consist of career coaches who don’t come from a technical background and therefore don’t have a good grasp of the difference between preparing for behavioral interviews and technical ones … and the big gulf between the two.

Behavioral interviews aren’t easy, sure, but they don’t require you to get used to a completely different mode of communication while simultaneously solving hard problems and turning them into code as someone breathes down your neck. Nor do behavioral interviews require hours and hours of grinding and drilling.

Preparing for technical interviews (and other analytical, domain-specific interviews… see the Tweet about management consulting below) is much more like studying for a math test than it is about practicing presenting yourself and telling your story. But until you’ve done it, you won’t really get it.

The most heartbreaking effect of the practice gap is this. Companies, under fire for hiring exclusively from top schools, expand their recruiting efforts and start targeting a broader range of schools and programs. One of the boldest programs of this type interviewed every CS student at each HBCU to develop a pipeline of African American candidates. Sadly, because of the interview practice gap, good-intentioned programs like this are destined to fail. A 2016 piece in Bloomberg (“Why Doesn’t Silicon Valley Hire Black Coders?”) covered this problem extensively:

When they started interviewing seniors, companies found—as Pratt did at Howard—that many were underprepared. They hadn’t been exposed to programming before college and had gaps in their college classes. The companies were coming into the process too late. So many of them have created programs geared toward freshmen and sophomores.

Then, when the initiative inevitably fails, companies walk away with their conscious biases about the “superiority” of top schools confirmed. Or even worse, this confirms their unconscious biases about race and/or gender, when the real problem isn’t the students but their lack of access to interview practice.

How you can (actually!) help

A few years ago, we ran a study that examined interview performance by school tier. Among students who were actively and regularly practicing technical interviewing, there was no difference in performance between elite schools and non-elite schools!

::iio-image{alt="Chart showing new grad interview performance by school tier" height="529" src="https://strapi-iio.s3.us-west-2.amazonaws.com/new_grad_interview_performance_008804d4ed.png" title="New Grad Interview Performance by School Tier" width="790"}

All that is to say that with at least 5 professional mock interviews per candidate, we could close the practice gap, level the playing field in software engineering, and hit our representation goals. This is a bold statement, but in my decade plus of operating in this space, I have not seen anything as effective at closing the gap in tech as getting everyone technical interview practice. If you’re an employer, I urge you to consider the following:

• Create a program where you sponsor interview practice for your candidates, especially those who aren’t socialized in technical interviews. I don’t care if it’s through interviewing.io or through someone else. Just do it.
• One mock interview is not enough. An info session is not enough. You need at least five interviews to move the needle, and they need to happen systematically and on a schedule. But you know what? It’s a few hundred dollars per candidate. Compared to the cost of sourcing and interviewing them, it’s peanuts.
• You may want to think about routing some of your D&I budget toward this purpose. If you’re spending any appreciable amount of money on initiatives like unconscious bias training, which has been shown over and over not to work, you could make a much bigger dent by relocating that budget toward interview practice for your candidates.

There is a lot of loud conversation in Silicon Valley about fixing representation in tech. It’s time to stop talking and instead do something that actually moves the needle. If interviewing.io had the means to give five mock interviews to every student in the US who doesn’t have the ability to pay for them, we’d do it right now.

My years of experience have shown me that there are countless members of underrepresented groups who are ready and able to be outstanding engineers, but who have been unable to pass their technical interview. We at interviewing.io are determined to make a difference, and we want to give people the same gift that MIT once gave me: the knowledge that interview practice is everything and that failing an interview doesn’t mean you’re not cut out for this career.

If you want to help us, get in touch.