The technical interview practice gap, and how it keeps underrepresented groups out of software engineering

I’ve been hiring engineers in some capacity for the past decade, and five years ago I founded interviewing.io, a technical recruiting marketplace that provides engineers with anonymous mock interviews and then fast-tracks top performers—regardless of who they are or how they look on paper—at top companies.

We’ve hosted close to 100K technical interviews on our platform and have helped thousands of engineers find jobs. Since last year, we’ve also been running a Fellowship program specifically for engineers from underrepresented backgrounds.

All that is to say that even though I have strong opinions about “diversity hiring” initiatives, I’ve acquired them the honest way, through laboratory experience.

Though I find it problematic to label any type of hiring as “diversity hiring” —primarily because it tacitly implies to candidates and everyone else involved that they are getting the job not because of their merits but because of their demographics or to fill some quota—nomenclature is not my main critique of these initiatives. Though there are a number of big issues with how employers approach hiring, in this post I’ll talk about the one that’s at once the most subtle and the most damaging: systematically ignoring the technical interview practice gap between traditional and non-traditional candidates. I’ll explain exactly what the interview practice gap is and why it’s so bad in a moment, but first, allow me to tell you a little story.

The benefit of attending an elite CS university

I was fortunate enough to attend MIT for undergrad. Probably the greatest gift that MIT gave me was a big stamp on my forehead that, to this day, makes strangers think I’m smart. But there was another, more relevant gift that gave me a serious advantage over students who did not attend an elite computer science institution: boundless access to technical interview practice.

Not only was there a multi-week course during the month-long break between Fall and Spring semesters that was dedicated exclusively to passing technical interviews, but all of my peers were going through exactly the same thing at the same time. Everyone was interviewing at FAANG for internships and new grad positions, which meant that we could all practice with each other, share our successes and failures, and, over time, internalize just how much of technical interviewing is a numbers game. We learned that bombing a Google interview did not mean that you weren’t meant to be an engineer. It just meant that you needed to work some more problems, do more mock interviews, and try again at Facebook.

Practice, practice, practice

My anecdotal experience aside, we at interviewing.io have actual data. As I mentioned earlier, we’ve hosted close to 100K interviews on our platform. It has taught us two very important things. This interview style is controversial, in part because it’s not entirely similar to the work software engineers do every day but also because 1) like standardized testing, it’s a learned skill, and 2) unlike standardized testing, interview results are not consistent or repeatable—the same candidate can do well in one interview and fail another one the same day. 

First, regardless of how strong you are technically, practice really matters. In a recent study, we looked at how people who got jobs at FAANG performed in practice vs. those who did not. Below is a graph of average technical score in interviews and the portion of people with each score who passed Facebook’s interview process.

You can see that technical ability did not obviously associate with interview success. So what did? It turned out that the number of practice interviews people completed (either on interviewing.io or off) had a much bigger bearing, as you can see below. Surprisingly, no other factors we included in our model (seniority, gender, degree, etc.) mattered at all.

Practice matters way more than raw technical ability

Secondly, technical interview performance from interview to interview is fairly inconsistent, even among strong candidates. Notably, as above, consistency appears to have nothing to do with seniority, pedigree, or anything else. In fact, only about 20% of interviewees perform consistently from interview to interview. Why does this matter? Once you’ve done a few traditional technical interviews, the volatility and lack of determinism in the process is something you figure out anecdotally and kind of accept. And if you have the benefit of having friends who’ve also been through it, it only gets easier. But what if you don’t?

In a previous post, we talked about how women quit interview practice 7 times more often than men after just one bad interview. It’s not too much of a leap to say that this is probably happening to any number of groups who are underrepresented/underserved by the current system. In other words, though it’s a broken process for everyone, the flaws within the system hit these groups the hardest—just because they haven’t had the chance to internalize exactly how much of technical interviewing is a game. Sadly, since we’ve started running our Fellowship program for engineers from underrepresented backgrounds, we’ve seen this effect firsthand.

The practice gap

So what does this have to do with the practice gap? As you can see in the graph above, there was a pretty meaningful bump in performance (almost 2X as likely to pass!) after at least five practice interviews. Five interviews is not a lot for someone who’s actively practicing and knows how the interview prep game works. But it’s a far cry from what companies who are looking to boost their diversity numbers offer their candidates, and it’s equally far from what bootcamps, an increasingly important source of candidates from underrepresented backgrounds, provide their students.

What employers are doing wrong

I won’t name any specific companies here, but tech giants (many with good intentions and with marching orders to boost their diversity numbers) often do a considerable amount of outreach and pre-interview engagement with candidates from underrepresented backgrounds. This outreach is usually an info session where one engineer speaks to a virtual room of candidates from underrepresented backgrounds. The engineer tells them what to expect in technical interviews, encourages them to learn how to articulate their thought process out loud while solving a problem, and recommends resources like Cracking the Coding Interview. Some companies even go so far as to offer their underrepresented candidates a mock interview or two.

Unfortunately, for candidates who are unfamiliar with the process, neither of these interventions is nearly enough.

What bootcamps are doing wrong

Because they’re cheaper and much faster than a four-year program, bootcamps seem like a rational choice when compared to the price of attending a top university. Since 2013, bootcamp enrollment has grown 9X, with a total of 20,316 graduates in 2018. Though these numbers represent enrollment across all genders and the raw number of grads lags behind CS programs, the portion of women graduating from bootcamps is also on the rise and graduation from online programs has actually reached gender parity (as compared to only 20% in traditional CS programs).

Despite these encouraging numbers, things start to go sideways when it’s time for students to start interviewing for jobs, and disappointingly, what bootcamps provide their students isn’t much better than the tech giants with good intentions.

Again, without naming names, I’ve interacted in some capacity with almost every major bootcamp over the last few years, trying to sell them on providing interview practice for their students, and I’ve also witnessed how recent bootcamp grads perform in mock interviews on interviewing.io. Based on these observations, bootcamps seem to have two big limitations: not nearly enough time spent on data structures and algorithms in the curriculum and lack of interview prep support both during and after the program.

Discussing bootcamp curriculum is out of the scope of this post, but I will say a few words about how bootcamps handle interview preparation. Bootcamps, by and large, have pretty thin margins and can’t invest much in paying for interview prep or hiring engineers to do it. As such, if they do offer mock interviews with professionals, just like tech companies, they tend to cap them at one or two, which we know isn’t nearly enough.

If they do not offer mock interviews with professionals (and most do NOT), they instead tend to focus on peer to peer practice (students interviewing other students), or they solicit volunteers from their pool of recent alums. While peer interviews are not entirely useless, they’re a far cry from working with someone who knows what the interview process is like. For the price tag that bootcamps charge their students, it’s appalling that they’re unable to invest resources into the last mile: actually getting their students ready for interviews.

That’s not to say that bootcamp grads are not viable employees. If that were true, I wouldn’t be advocating bootcamps as a means of increasing diversity in software engineering. What’s heartbreaking is that bootcamp students have all the makings of good engineers—they just need to spend an extra 3-6 months after graduation drilling by themselves on the skills their programs should have offered them.

While I have less direct experience with curriculum and career centers at non-elite universities (another contributor to the practice gap), from speaking with students who attend these schools, it’s clear that access to consistent interview prep is hard to come by, especially when compared to what’s available at their more elite counterparts.

The cost of falling short

Why are bootcamps, universities, and employers all falling short and exacerbating the practice gap? With both employers and bootcamps, the teams responsible for facilitating interview info sessions or mock interviews typically consist of career coaches who don’t come from a technical background and therefore don’t have a good grasp of the difference between preparing for behavioral interviews and technical ones … and the big gulf between the two.

Behavioral interviews aren’t easy, sure, but they don’t require you to get used to a completely different mode of communication while simultaneously solving hard problems and turning them into code as someone breathes down your neck. Nor do behavioral interviews require hours and hours of grinding and drilling.

Preparing for technical interviews (and other analytical, domain-specific interviews… see the Tweet about management consulting below) is much more like studying for a math test than it is about practicing presenting yourself and telling your story. But until you’ve done it, you won’t really get it.

The most heartbreaking effect of the practice gap is this. Companies, under fire for hiring exclusively from top schools, expand their recruiting efforts and start targeting a broader range of schools and programs. One of the boldest programs of this type interviewed every CS student at each HBCU to develop a pipeline of African American candidates. Sadly, because of the interview practice gap, good-intentioned programs like this are destined to fail. A 2016 piece in Bloomberg (“Why Doesn’t Silicon Valley Hire Black Coders?”) covered this problem extensively:

When they started interviewing seniors, companies found—as Pratt did at Howard—that many were underprepared. They hadn’t been exposed to programming before college and had gaps in their college classes. The companies were coming into the process too late. So many of them have created programs geared toward freshmen and sophomores.

Then, when the initiative inevitably fails, companies walk away with their conscious biases about the “superiority” of top schools confirmed. Or even worse, this confirms their unconscious biases about race and/or gender, when the real problem isn’t the students but their lack of access to interview practice.

How you can (actually!) help

A few years ago, we ran a study that examined interview performance by school tier. Among students who were actively and regularly practicing technical interviewing, there was no difference in performance between elite schools and non-elite schools!

With practice, the practice gap goes away

All that is to say that with at least 5 professional mock interviews per candidate, we could close the practice gap, level the playing field in software engineering, and hit our representation goals. This is a bold statement, but in my decade plus of operating in this space, I have not seen anything as effective at closing the gap in tech as getting everyone technical interview practice. If you’re an employer, I urge you to consider the following:

  • Create a program where you sponsor interview practice for your candidates, especially those who aren’t socialized in technical interviews. I don’t care if it’s through interviewing.io or through someone else. Just do it.
  • One mock interview is not enough. An info session is not enough. You need at least five interviews to move the needle, and they need to happen systematically and on a schedule. But you know what? It’s a few hundred dollars per candidate. Compared to the cost of sourcing and interviewing them, it’s peanuts.
  • You may want to think about routing some of your D&I budget toward this purpose. If you’re spending any appreciable amount of money on initiatives like unconscious bias training, which has been shown over and over not to work, you could make a much bigger dent by relocating that budget toward interview practice for your candidates.

There is a lot of loud conversation in Silicon Valley about fixing representation in tech. It’s time to stop talking and instead do something that actually moves the needle. If interviewing.io had the means to give five mock interviews to every student in the US who doesn’t have the ability to pay for them, we’d do it right now.

My years of experience have shown me that there are countless members of underrepresented groups who are ready and able to be outstanding engineers, but who have been unable to pass their technical interview. We at interviewing.io are determined to make a difference, and we want to give people the same gift that MIT once gave me: the knowledge that interview practice is everything and that failing an interview doesn’t mean you’re not cut out for this career.

If you want to help us, get in touch.

EDIT: If you’re a software engineer (or aspiring software engineer) who can’t afford interview practice, we have a solution. Sign up for our deferral program, practice now with professional interviewers from FAANG, and only pay us when you find a job.

8 thoughts on “The technical interview practice gap, and how it keeps underrepresented groups out of software engineering”

  1. This is totally misleading! I dropped out of high school and got a GED, then went to 2 different community colleges. I had excellent access to interview prep materials and I spent a lot of my time to work hard and do interview problems online. I couldn’t even afford paid services most of my interview prep. I also work with minorities and most of the time it’s not your cultural background but instead your financial background that makes accessibility an issue. If you’re poor you’re going to have to work a FT job and not have time to prep.

    The free resource list is massive and very accessible as long as you have internet and a computer.

    1. I could not disagree with you more, first of all this article is fantastic and directly aligns with my personal experience coming from a bootcamp background and now a few years later interviewing with the big companies.

      You seem to think that this article is stating the ONLY way to get the requisite amount of good practice is to go to a prestigious university. This is not the takeaway that I took from the article. My interpretation is, and only confirms my years of direct experience in software development, that going to college – good, bad or ugly – does nothing to predict the performance of an engineer, and interviewing just follows that pattern. That anyone can perform like an MIT or Stanford grad by dedicating time and effort to practice.

      If all you can access are free resources, as long as you practice, you will be on level terms with a Ivy Leaguer.

      I found this article to be above all else, incredibly heartening. This is something I’ve suspected, and I find it empowering to see data support the fact that we are all capable of carving our own destiny after all.

  2. It seems to me like the better solution here is to stop doing these types of interviews entirely. Like you said, they’re a game that filters out people based primarily on how much they prepared for it. Think about all the time a student at an “elite” CS school spends studying and practicing for these technical interviews. Just imagine how much better an employee that person would be if they spent that time studying something that was actually *meaningful*, a real skill that would help them on the job.

    Suggesting that companies and schools provide more study and training for this metagame only makes the problem worse. Instead, why not suggest better ways of interviewing candidates and judging their technical skills in a way that’s consistent, repeatable, and fair? One that candidates can approach honestly, without having to play games in order to get fair consideration? Your suggestion just creates an arms race that benefits nobody. When interview practice becomes more common, schools will increase their interview practice so they can continue to claim that they’re better than everyone else. Like a great philosopher once said, “the only way to win is not to play the game”.

    There are many, many companies out there hiring technical roles that *don’t* do these sorts of interviews. Their ratios of good hires to bad hires doesn’t seem that different than the supposed “top tech companies” that play these games. Perhaps that could be a topic for a future blog article: do the downsides of this style of interviewing (filtering candidates based on something unrelated to job performance) outweigh the supposed benefits?

    1. I completely agree, and we’ve written about this in the past: https://blog.interviewing.io/you-cant-fix-diversity-in-tech-without-fixing-the-technical-interview/ We even did a dive into why interviews are the way they are and why they persist this way.

      That said, no matter what Aline Lerner thinks or interviewing.io thinks, the giants will keep doing a process that’s net positive for them, and people will keep wanting to work at the giants. So while I hope with my whole heart that interviews get better and especially that small companies stop “cargo culting” FAANG’s practices and carve their own path (and many do as you said!), for now, if you want to get into FAANG, this is what you gotta do.

      1. I work at a FAANG company and every time I’ve conducted interviews, I’ve been able to choose the questions and format myself with no restrictions, even being early career! I’ve always been frustrated by the process but never knew what to do to fix it. This article definitely inspires me to do a lot more work to figure out a more effective process next time I conduct an interview. I’m also a woman from a non-traditional background, so I’ve experienced the practice gap firsthand.

  3. Are the phone screens in your data technical screens? In my experience the phone screen happens with a recruiter, so data structures and algorithms don’t come into play until the next step in the process.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top