logo blog
better interviewing through data


Announcing the Technical Interview Practice Fellowship

Posted on July 23rd, 2020.

I started because I was frustrated with how inefficient and unfair hiring was and how much emphasis employers placed on resumes.

But the problem is bigger than resumes. We’ve come to learn that interview practice matters just as much. The resume gets you in the door, and your interview performance is what gets you the offer. But, even though technical interviews are hard and scary for everyone — many of our users are senior engineers from FAANG who are terrified of getting back out there and code up the kinds of problems they don’t usually see at work while someone breathes down their neck — interview prep isn’t equitably distributed.

This inequity never really sat right with me (that’s why exists), but when we started charging for interview practice post-COVID, it really didn’t sit right with me.

As you may have read, if you follow news, COVID-19 turned our world upside down. In its wake, the pandemic left a deluge of hiring slowdowns and freezes. For a recruiting marketplace, this was an existential worst nightmare — in a matter of weeks, we found ourselves down from 7-figure revenue to literally nothing. Companies didn’t really want or need to pay for hiring anymore, and we were screwed.

Then, we pivoted and started charging our users, who had previously been able to practice on our platform completely for free (albeit with some strings, more on that in a moment). While this pivot was the right thing to do — without it, we would have had to shut down the company, unable to provide any practice at all — charging people, especially those from underrepresented backgrounds, didn’t sit right with us, and in our last post announcing our model, we made the following promises:

  • We’d ALWAYS have a free tier 
  • We’d immediately start working on a fellowship for engineers from underrepresented backgrounds or in a visa crisis/experiencing financial hardship ← That’s what this post is about!
  • We’d find a way to let people defer their payments

We launched with a free tier, and it’s still there and going strong. We’re still working on deferred payments and are in the thick of user research and price modeling.

But, the rest of this post is about the 2nd promise. To wit, I’m so proud to tell you that we’ve officially launched the first (pilot) cohort of the Technical Interview Practice Fellowship. This cohort will be focused on engineers from backgrounds that are underrepresented in tech. We are acutely aware, of course, that our first cohort couldn’t capture everyone who’s underrepresented, that gender and race isn’t enough, and that we need to do more for our users who can’t afford our price tags, regardless of who they are or where they come from.

Our hope is to expand this Fellowship to anyone who needs it.

We’re also working on the much harder problem of how to navigate the visa situation we’re in right now (different than when we wrote the first post, sadly… but especially important to me, given that I’m an immigrant myself).

What is the Fellowship, and why does it exist?

Before we tell you a little bit about the Fellows in our inaugural cohort and what the Fellowship entails, a quick word about why this matters.

In order to get a job as a software engineer, it’s not enough to have a degree in the field from a top school. However you learned your coding skills, you also have to pass a series of rigorous technical interviews, focusing on analytical problem solving, algorithms, and data structures.

This interview style is controversial, in part because it’s not entirely similar to the work software engineers do every day but also because 1) like standardized testing, it’s a learned skill and 2) unlike standardized testing, interview results are not consistent or repeatable — the same candidate can do well in one interview and fail another one in the same day. According to our data, only about 25% of candidates are consistent in their performance from interview to interview, and women quit 7X more often than men after a poor performance.

To account for both of these limitations, the best strategy to maximize your chances of success is to practice a lot so you can 1) get better and 2) accept that the results of a single interview are not the be-all and end-all of your future aptitude as a software engineer and that it’s ok to keep trying.

The main problem created by modern interview techniques is that, despite interview practice being such a critical prerequisite to success in this field, access to practice isn’t equitably distributed. We want to fix this, and we’re well equipped to do so. Based on our data, engineers are twice as likely to pass a real interview after they’ve done 3-5 practice sessions on our platform.

Our Fellows will get these practice sessions completely for free. These will be 1:1 hour-long sessions with senior engineers from a top company who have graciously volunteered their time and expertise. Huge thank you and a big shout-out to them all.

After each session, Fellows will get actionable feedback that will help them in their upcoming job search, and we will be helping Fellows connect with top companies as well.

Note: We’d like to be able to offer even more support – and are actively seeking more partners to do so. Please see the How you can help section below if you or your organization would like to get involved!

Why now?

The world seems to be in a place, now more than ever, to have the conversation about race, gender, socioeconomic, and other kinds of equity, in hiring. This is our small part of that conversation.

Who are the Fellows?

After opening up our application process, we close to 1,000 submissions in a week, and (though it was really, really hard) we culled those down to 56 Fellows.

Our first cohort is:

  • 82% Black, Latinx, and/or Indigenous
  • 53% women
  • 55% senior (4+ years of experience) & 45% junior (0-3 years of experience)

Here are some of their (anonymized) stories. There were a lot of stories like these.

My goal is to keep pressing as well as to share and give to underrepresented communities because the journey in tech can be isolating. Often I am the only one. It is critical that there are more people that look like me that are engineers *and* ascend the leadership ladder.

My parents immigrated from [redacted] to The Bronx without a formal education. I’m the first individual in my household to graduate from college and I’m the only Software Engineer in my family. I grew up in a poor neighborhood where many individuals had limited economic and educational opportunity. I aim to make the path to become a Software Engineer easier for those who were in my situation.

My journey to becoming a software engineer almost never happened. Throughout my undergraduate studies I was faced with having to drop out multiple times, due to the immigration status of my parents…. I was tasked with assisting in my family’s living situation and paying for school. I worked full time and started my own construction company in order to take care of my family and studies. It was always tough having to work 8-10 hours a day and then going to class or doing homework… Becoming a software engineer was always a goal of mine, and realizing that goal was well worth the struggle, given the struggle my parents went through to bring us here in the first place.

I spent 5 years in public education working directly with marginalized communities in the struggle for equity. My journey through software engineering is a continuation of this spirit of advocacy and changemaking. Software engineering is a tool to be put at the service of advocacy.

What can I do to help?

There are a number of ways you can help and get involved!

Help sponsor future Fellowship cohorts & create scholarships for underrepresented engineers!

Every Fellow in this first cohort represents at least 100X who are not. We have the tech to scale the hell out of this program, and all we need is backing and resources from people or organizations who recognize there’s a need (donations are tax-deductible). Please email if you’d like to get involved or want more information.

Hire through us!

Despite mounting evidence that resumes are poor predictors of aptitude, companies were obsessed with where people had gone to school and worked previously. On, software engineers, no matter where they come from or where they’re starting, can book anonymous mock interviews with senior interviewers from top companies. We use data from these interviews to identify top performers much more reliably than a resume, and fast-track them to real job interviews with employers on our platform through the same anonymous, fair process. Because we use data, not resumes, our candidates end up getting hired consistently by companies like Facebook, Uber, Twitch, Lyft, Dropbox, and many others, and 40% of the hires we’ve made to date have been candidates from non-traditional backgrounds. Many of our candidates have literally been rejected based on their resumes by the same employer who later hired them when they came through our anonymous platform (one notable candidate was rejected 3 times from a top-tier public company based on his resume before he got hired at that same company through our anonymous interview format).

Please email to get rolling.

Buy an individual practice session for someone who can’t afford it

If you know individual engineers who need interview practice but can’t afford it, use our handy interview gifting feature. Interviews are $100 each. They’re not cheap, but we have to price them that way to pay for interviewer time (interviewers are senior FAANG engineers) and cover our costs. Sadly that means practice interviews are not affordable to everyone. Even if you can’t get involved to help us fund interviews at scale, if you know someone who needs practice but can’t afford it, you can buy them an anonymous mock interview or two individually. It’s the best gift you can give to an engineer who’s starting their job search.


Uncategorized is finally out of beta. Anonymous technical interview practice for all!

Posted on June 4th, 2020.

First, for the brevity-minded, a TL;DR:

  • is now open to all engineers in North America and the UK, regardless of seniority level
  • We have both free and paid, premium mock interviews
  • No more limits on how many practice interviews you can do
  • You can now give (or receive!) practice interviews as gifts

I started 5 years ago. After working as both an engineer and a recruiter, my frustration with how inefficient and unfair hiring had reached a boiling point. What made me especially angry was that despite mounting evidence that resumes are poor predictors of aptitude, employers were obsessed with where people had gone to school and worked previously. In my mind, any great engineer, regardless of how they look on paper, should have the opportunity to get their foot in the door wherever they choose.

So, we set out to build a better system. On, software engineers can book anonymous mock interviews with senior engineers from companies like Facebook, Google, and others, and if they do well in practice, get fast-tracked with top employers regardless of how they look on paper. Fast-tracking means that you bypass resume screens, scheduling emails, and recruiter calls, and go straight to the technical interview (which, by the way, is still anonymous1) at companies of your choice. Because we use interview data, not resumes, our candidates end up getting hired consistently by companies like Facebook, Uber, Twitch, Lyft, Dropbox, and many others, and 40% of the hires we’ve made to date have been candidates from non-traditional backgrounds. What’s nuts is that many of our candidates have literally been rejected based on their resumes by the same employer who later hired them when they came through One notable candidate was rejected three times from a top-tier public company based on his resume before he got hired at that same company through us.

Over the past 5 years, we’ve hosted over 50,000 technical interviews (both practice and real) on our platform. Our YouTube channel, where you can watch other people interview, has gotten over 3.5M views, and, most importantly, we have helped thousands of engineers get great jobs.

All practice interviews are completely anonymous and include actionable, high-fidelity feedback

Despite that for our entire, multi-year existence, we’ve been in beta. Over the past year or so, this increasingly inaccurately named “beta” became kind of a smoke screen. Our product was stable, we had plenty of interviewers, but sadly, we couldn’t serve many of the people who needed us most. Because we made money by charging companies for hires, despite a growing waitlist of >180,000 engineers, we could only serve the ones whom we had a shot at placing, i.e. engineers who 1) were located in a city where we had customers and 2) had 4 or more years of experience⁠—sadly, despite our best efforts, employers across the board were not willing to pay for junior hires.

Then, COVID-19 happened and with it, a deluge of hiring slowdowns and freezes. In a matter of weeks, we found ourselves down from 7-figure revenue to literally nothing. Companies didn’t really want or need to pay for hiring anymore.

But these hiring slowdowns freezes weren’t just affecting us. In parallel, we saw a growing sea of layoffs, and we realized that, soon, more and more candidates would be vying for a shrinking number of jobs. On top of that, because a disproportionate amount of layoffs targeted recruiters, an overworked in-house skeleton recruiting team would go back to relying on resumes and other old-school proxies, unintentionally marginalizing non-traditional candidates once again. We also realized that many of the folks getting laid off would be here on visas, which meant that they’d have a paltry 60 days to find their next job or risk deportation.

So, we made a hard call. You may know that historically, we’ve offered completely free mock interviews. What you may not know, is that we pay our professional interviewers, as it’s the only way to ensure that we have seasoned, experienced engineers delivering a consistent, realistic, and high-quality experience. This is often our largest expense.

Since we previously funded practice by charging companies for hires, we had to find another revenue stream to continue. In the face of either shutting down the company, unable to provide any practice at all, or to begin to charge for premium practice interviews, we made the choice to launch a paid tier. But, because charging for practice felt anathema to our mission, we knew we needed some ground rules in place. I started this company to fix hiring, after all, and that’s why the people who work at are here, too. 

After 50,000 interviews, our data shows that where someone goes to school has no correspondence to their interview performance. Despite that, we are all too aware that just because aptitude is uniformly distributed among populations, resources are not. We understand that paying for interviews will be prohibitive to many of the people who need help most, so our ground rules and goals for this pivot were as follows:

  • We’d ALWAYS have a free tier
  • We’d immediately start working on a fellowship for engineers from underrepresented backgrounds or in a visa crisis experiencing financial hardship
  • We’d find a way to let people defer their payments

There is an upside to our new model though. Now that we’re no longer strictly beholden to employers, we’re able to open up to engineers of all seniority levels and many more locations. And because our revenue isn’t coming from placements but directly from practice, we don’t have to constrain our users to a limited number of practice interviews.

So, as of today, is open to engineers of all experience levels in North America and the UK. Engineers who sign up can book mock interviews 24 hours out, and top performers will still get fast-tracked for great jobs.2

 You can book either free (peer practice with other users) or premium interviews with experienced interviewers from FAANG, as early as 24 hours out

Here’s how it works:

  • If you’re able to pay, you can now book unlimited interviews with professional interviewers from FAANG and other top companies. We’re no longer constraining you to 2 or 3, and we have the interviewer supply to make this work. Interviews cost between $100 and $225.
  • If you can’t pay, there’s a free tier where you can do peer practice with other users.

What about the goals and ground rules above? We’ve already made some headway against these. The free tier was there from day one. Moreover, a number of our professional interviewers have stepped up and volunteered their time to help the people who need it most prepare for free (if you’d like to volunteer your time to help engineers from underrepresented groups practice interviewing, please email with the subject line ‘Volunteer’), and a formal fellowship is in the works. Lastly, we’re working on a deferred payment plan, where users who buy premium interviews will not have to pay us for them until they find their next job.

COVID-19 has changed a lot of things for our team (that’s us above doing the remote work thing). But not how much we want to fix hiring.

Look, whether you got laid off, are a student who’s reeling from your internship being withdrawn, lost your visa, whether you’re fortunate enough to still be employed but worry about your job stability, or if you just want to help making hiring fairer, we hope you’ll take us for a spin. Technical interviews are hard, and hiring is broken. And whether you’re new to interviewing or are just rusty after being off the market, and whether you can pay or not, we have your back. 

The coming months are going to be hard. We know that in the current climate more people than ever are feeling helpless, and the world feels like it’s burning. And it might not even seem like technical interviews matter that much. But they do to us… because this is our way of creating a world where access to opportunity isn’t determined by who you are or where you come from but by what you can do.

P.S. If you don’t need practice for yourself but you or your organization want to help others get awesome at technical interviews, check out our new gifting feature.



The Eng Hiring Bar: What the hell is it?

Posted on March 31st, 2020.

Recursive Cactus has been working as a full-stack engineer at a well-known tech company for the past 5 years, but he’s now considering a career move.

Over the past 6 months, Recursive Cactus (that’s his anonymous handle on has been preparing himself to succeed in future interviews, dedicating as much as 20-30 hours/week plowing through LeetCode exercises, digesting algorithms textbooks, and of course, practicing interviews on our platform to benchmark his progress.

Recursive Cactus’s typical weekday schedule

6:30am – 7:00amWake up
7:00am – 7:30amMeditate
7:30am – 9:30amPractice algorithmic questions
9:30am – 10:00amCommute to work
10:00am – 6:30pmWork
6:30pm – 7:00pmCommute from work
7:00pm – 7:30pmHang out with wife
7:30pm – 8:00pmMeditate
8:00pm – 10:00pmPractice algoirthmic questions

Recursive Cactus’s typical weekend schedule

8:00am – 10:00amPractice algorithmic questions
10:00am – 12:00pmGym
12:00pm – 2:00pmFree time
2:00pm – 4:00pmPractice algorithmic questions
4:00pm – 7:00pmDinner with wife & friends
7:00pm – 9:00pmPractice algorithmic questions

But this dedication to interview prep has been taking an emotional toll on him, his friends, and his family. Study time crowds out personal time, to the point where he basically has no life beyond work and interview prep.

“It keeps me up at night: what if I get zero offers? What if I spent all this time, and it was all for naught?”

We’ve all been through the job search, and many of us have found ourselves in a similar emotional state. But why is Recursive Cactus investing so much time, and what’s the source of this frustration?

He feels he can’t meet the engineer hiring bar (aka “The Bar”), that generally accepted minimum level of competency that all engineers must exhibit to get hired.

To meet “The Bar,” he’s chosen a specific tactic: to look like the engineer that people want, rather than just be the engineer that he is.

It seems silly to purposefully pretend to be someone you’re not. But if we want to understand why Recursive Cactus acts the way he does, it helps to know what “The Bar” actually is. And when you think a little more about it, you realize there’s not such a clear definition.

Defining “The Bar”

What if we look at how the FAANG companies (Facebook, Amazon, Apple, Netflix, Google) define “The Bar?” After all, it seems those companies receive the most attention from pretty much everybody, job seekers included.

Few of them disclose specific details about their hiring process. Apple doesn’t publicly share any information. Facebook describes the stages of their interview process, but not their assessment criteria. Netflix and Amazon both say they hire candidates that fit their work culture/leadership principles. Neither Netflix nor Amazon describes exactly how they assess against their respective principles. However, Amazon does share how interviews get conducted as well as software topics that could be discussed for software developer positions.

The most transparent of all FAANGs, Google publicly discloses its interview process with the most detail, with Laszlo Bock’s book Work Rules! adding even more insider color about how their interview process came to be.

And speaking of tech titans and the recent past, Aline (our founder) mentioned the 2003 book How Would You Move Mount Fuji? in a prior blog post, which recounted Microsoft’s interview process when they were the pre-eminent tech behemoth of the time.

In order to get a few more data points about how companies assess candidates, I also looked at Gayle Laakmann McDowell’s “Cracking the Coding Interview”, which is effectively the Bible of interviewing for prospective candidates, as well as Joel Spolsky’s Guerilla Guide to Interviewing 3.0, written by an influential and well-known figure within tech circles over the past 20-30 years.

Definitions of “The Bar”

Source Assessment Criteria
AppleNot publicly shared
AmazonAssessed against Amazon’s Leadership principles
FacebookNot publicly shared
NetflixNot publicly shared
Google1. General cognitive ability
2. Leadership
3. “Googleyness”
4. Role-related knowledge
Cracking the Coding Interview – Gayle Laakmann McDowell– Analytical skills
– Coding skills
– Technical knowledge/computer science fundamentals
– Experience
– Culture fit
Joel Spolsky– Be smart
– Get things done
Microsoft (circa 2003)– “The goal of Microsoft’s interviews is to assess a general problem-solving ability rather than a specific competency.”
– “Bandwidth, inventiveness, creative problem-solving ability, outside-the-box thinking”
– “Hire for what people can do rather than what they’ve done”
– Motivation
Defining “Intelligence”

It’s not surprising that coding and technical knowledge would be part of any company’s software developer criteria. After all, that is the job.

But beyond that, the most common criteria shared among all these entities is a concept of intelligence. Though they use different words and define the terms slightly differently, all point to some notion of what psychologies call “cognitive ability.”

SourceDefinition of cognitive ability
Google“General Cognitive Ability. Not surprisingly, we want smart people who can learn and adapt to new situations. Remember that this is about understanding how candidates have solved hard problems in real life and how they learn, not checking GPAs and SATs.”
Microsoft (circa 2003)“The goal of Microsoft’s interviews is to assess a general problem-solving ability rather than a specific competency… It is rarely dear what type of reasoning is required or what the precise limits of the problem are. The solver must nonetheless persist until it is possible to bring the analysis to a timely and successful conclusion.”
Joel Spolsky“For some reason most people seem to be born without the part of the brain that understands pointers.” 
Gayle Laakmann McDowell“If you’re able to work through several hard problems (with some help, perhaps), you’re probably pretty good at developing optimal algorithms. You’re smart.”

All these definitions of intelligence resemble early 19th-century psychologist Charles Spearman’s theory of intelligence, the most widely acknowledged framework for intelligence. After performing a series of cognitive tests on school children, Spearman found that those who did well in one type of test tended to also perform well in other tests. This insight led Spearman to theorize that there exists a single underlying general ability factor (called “g” or “g factor”) influencing all performance, separate from specific, task-specific abilities (named “s”).

If you believe in the existence of “g” (many do, some do not… there exist different theories of intelligence), finding candidates with high measures of “g” aligns neatly with the intelligence criteria companies look for.

While criteria like leadership and culture fit matter to companies, “The Bar” is not usually defined in those terms. “The Bar” is defined as having technical skills but also (and perhaps more so) having general intelligence. After all, candidates aren’t typically coming to to specifically practice leadership and culture fit.

The question then becomes how you measure these two things. Measuring technical skills seems tough but doable, but how do you measure “g?”

Measuring general intelligence

Mentioned in Bock’s book, Frank Schmidt’s and John Hunter’s 1998 paper “The Validity and Utility of Selection Methods in Personnel Psychology” attempted to answer this question by analyzing a diverse set of 19 candidate selection criteria to see which predicted future job performance the best. The authors concluded general mental ability (GMA) best predicted job performance based on a statistic called “predictive validity.”

In this study, a GMA test referred to an IQ test. But for Microsoft circa 2003, puzzle questions like “How many piano tuners are there in the world?” appear to have taken the place of IQ tests for measuring “g”. Their reasoning:

“At Microsoft, and now at many other companies, it is believed that there are parallels between the reasoning used to solve puzzles and the thought processes involved in solving the real problems of innovation and a changing marketplace. Both the solver of a puzzle and a technical innovator must be able to identify essential elements in a situation that is initially ill-defined.”

– “How Would You Move Mount Fuji?” – page 20

Fast forward to today, Google denounces this practice, concluding that “performance on these kinds of questions is at best a discrete skill that can be improved through practice, eliminating their utility for assessing candidates.”

So here we have two companies who test for general intelligence, but who also fundamentally disagree on how to measure it.

Are we measuring specific or general intelligence?

But maybe as Spolsky and McDowell have argued, the traditional algorithmic and computer science-based interview questions are themselves effective tests for general intelligence. Hunter & Schmidt’s study contains some data points that could support this line of reasoning. Among all single-criteria assessment tools, work sample tests possessed the highest predictive validity. Additionally, when observing the highest validity regression result of two-criteria assessment tool (GMA test plus work sample test), the standardized effect size on the work sample rating was larger than that of the GMA rating, suggesting a stronger relationship with future job performance.

If you believe algorithmic exercises function as work sample tests in interviews, then the study suggests traditional algorithm-based interviews could predict future job performance, maybe even more than a GMA/IQ test.

Recursive Cactus doesn’t believe there’s a connection.

There’s little overlap between the knowledge acquired on the job and knowledge about solving algorithmic questions. Most engineers rarely work with graph algorithms or dynamic programming. In application programming, lists and dictionary-like objects are the most common data structures. However, interview questions involving those are often seen as trivial, hence the focus on other categories of problems.

– Recursive Cactus

In his view, algorithms questions are similar to Microsoft’s puzzle questions: you learn how to get good at solving interview problems, which to him don’t ever show up in actual day-to-day work, which, if true, wouldn’t actually fit with the Schmidt & Hunter study.

Despite Recursive Cactus’s personal beliefs, interviewers like Spolsky still believe these skills are vital to being a productive programmer.

A lot of programmers that you might interview these days are apt to consider recursion, pointers, and even data structures to be a silly implementation detail which has been abstracted away by today’s many happy programming languages. “When was the last time you had to write a sorting algorithm?” they snicker.

Still, I don’t really care. I want my ER doctor to understand anatomy, even if all she has to do is put the computerized defibrillator nodes on my chest and push the big red button, and I want programmers to know programming down to the CPU level, even if Ruby on Rails does read your mind and build a complete Web 2.0 social collaborative networking site for you with three clicks of the mouse.

– Joel Spolsky

Spolsky seems to concede that traditional tech interview questions might not mimic actual work problems, and therefore wouldn’t act as work samples. Rather, it seems he’s testing for general computer science aptitude, which is general in a way, but specific in other ways. General intelligence within a specific domain, one might say.

That is, unless you believe intelligence in computer science is general intelligence. McDowell suggests this:

There’s another reason why data structure and algorithm knowledge comes up: because it’s hard to ask problem-solving questions that don’t involve them. It turns out that the vast majority of problem-solving questions involve some of these basics.

– Gayle Laakmann McDowell

This could be true assuming you view the world primarily through computer science lenses. Still, it seems pretty restrictive to suggest people who don’t speak the language of computer science would have more difficulty solving problems.

At this point, we’re not really talking about measuring general intelligence as Spearman originally defined it. Rather, it seems we’re talking about specific intelligence, defined or propagated by those grown or involved in traditional computer science programs, and conflating that with general intelligence (Spolsky, McDowell, Microsoft’s Bill Gates, and 4 of 5 FAANG founders studied computer science at either some Ivy League university or Stanford).

Maybe when we’re talking about “The Bar,” we’re really talking about something subjective, based on whoever is doing the measuring, and whose definition might not be consistent from person-to-person.

Looking at candidate assessment behavior from interviewers on the platform, you can find some evidence that supports this hypothesis.

“The Bar” is subjective

On the platform, people can practice technical interviews online and anonymously, with interviewers from top companies on the other side. Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from companies like Google, Facebook, Dropbox, Airbnb, and more. Check out our interview showcase to see how this all looks and to watch people get interviewed. After every interview, interviewers rate interviewees on a few different dimensions: technical skills, communication skills, and problem-solving skills. Each dimension gets rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!”. You can see what our feedback form looks like below:

If you do well in practice, you can bypass applying online/getting referrals/talking to recruiters and instead immediately book real technical interviews directly with our partner companies (more on that in a moment).

When observing our most frequent practice interviewers, we noticed differences across interviewers in the percent of candidates that person would hire, which we call the passthrough rate. Passthrough rates ranged anywhere between 30% and 60%. At first glance, certain interviewers seemed to be a lot stricter than others.

Because interviewees and interviewers are anonymized and matched randomly[1], we wouldn’t expect the quality of candidates to vary much across interviewers, and as a result, wouldn’t expect interviewee quality to explain the difference. Yet even after accounting for candidate attributes like experience level, differences in passthrough rates persist[2].

Maybe some interviewers choose to be strict on purpose because their bar for quality is higher. While it’s true that candidates who practiced with stricter interviewers tended to receive lower ratings, they also tended to perform better on their next practice.

This result could be interpreted in a couple of ways:

  • Stricter interviewers might systematically underrate candidates
  • Candidates get so beat up by strict interviewers that they tended to improve more between practices, striving to meet their original interviewer’s higher bar

If the latter were true, you would expect that candidates who practiced with stricter interviewers would perform better in real company interviews. However, we did not find a correlation between interviewer strictness and future company interview passthrough rate, based on real company interviews conducted on our platform[3].

Interviewers on our platform represent the kinds of people a candidate would encounter in a real company interview, since those same people also conduct phone screens and onsites at the tech companies you’re all applying to today. And because we don’t dictate how interviewers conduct their interviews, these graphs could be describing the distribution of opinions about your interview performance once you hang up the phone or leave the building.

This suggests that, independent of your actual performance, whom you interview with could affect your chance of getting hired. In other words, “The Bar” is subjective.

This variability across interviewers led us to reconsider our own internal definition of “The Bar,” which determined which candidates were allowed to interview with our partner companies. Our definition strongly resembled Spolsky’s binary criteria (“be smart”), heavily weighing an interviewer’s Would Hire opinion way more than our other 3 criteria, leading to the bimodal, camel-humped distribution below.

While our existing scoring system correlated decently with future interview performance, we found that an interviewer’s Would Hire rating wasn’t as strongly associated with future performance as our other criteria were. We lessened the weight on the Would Hire rating, which in turn improved our predictive accuracy[4]. Just like in “Talledega Nights” when Ricky Bobby learned there existed places other than first place and last place in a race, we learned that it was more beneficial to think beyond the binary construct of “hire” vs. “not hire,” or if you prefer, “smart” vs. “not smart.”

Of course, we didn’t get rid of all the subjectivity, since those other criteria were also chosen by the interviewer. And this is what makes assessment hard: an interviewer’s assessment is itself the measure of candidate ability.

If that measurement isn’t anchored to a standard definition (like we hope general intelligence would be), then the accuracy of any given measurement becomes less certain. It’s as if interviewers used measuring sticks of differing lengths, but all believed their own stick represented the same length, say 1 meter.

When we talked to our interviewers to understand how they assessed candidates, it became even more believable that different people might be using measuring sticks of differing lengths. Here are some example methods of how interviewers rated candidates:

  • Ask 2 questions. Pass if answer both
  • Ask questions of varying difficulty (easy, medium, hard). Pass if answers a medium
  • Speed of execution matters a lot, pass if answers “fast” (“fast” not clearly defined)
  • Speed doesn’t matter much, pass if have a working solution
  • Candidates start with full points. When candidates make mistakes, start docking points

Having different assessment criteria isn’t necessarily a bad thing (and actually seems totally normal). It just introduces more variance to our measurements, meaning our candidates’ assessments might not be totally accurate.

The problem is, when people talk about “The Bar,” that uncertainty around measurement usually gets ignored.

You’ll commonly see people advising you only to hire the highest quality people.

A good rule of thumb is to hire only people who are better than you. Do not compromise. Ever.

– Laszlo Bock

Don’t lower your standards no matter how hard it seems to find those great candidates.

– Joel Spolsky

In the Macintosh Division, we had a saying, “A player hire A players; B players hire C players”–meaning that great people hire great people.

– Guy Kawasaki

Every person hired should be better than 50 percent of those currently in similar roles – that’s raising the bar.

– Amazon Bar Raiser blog post

All of this is good advice, assuming “quality” could be measured reliably, which as we’ve seen so far, isn’t necessarily the case.

Even when uncertainty does get mentioned, that variance gets attributed to the candidate’s ability, rather than the measurement process or the person doing the measuring.

[I]n the middle, you have a large number of “maybes” who seem like they might just be able to contribute something. The trick is telling the difference between the superstars and the maybes, because the secret is that you don’t want to hire any of the maybes. Ever.

If you’re having trouble deciding, there’s a very simple solution. NO HIRE. Just don’t hire people that you aren’t sure about.

– Joel Spolsky

Assessing candidates isn’t a fully deterministic process, yet we talk about it like it is.

Why “The Bar” is so high

“Compromising on quality” isn’t really about compromise, it’s actually about decision-making in the face of uncertainty. And as you see from the quotes above, the conventional strategy is to only hire when certain.

No matter what kind of measuring stick you use, this leads to “The Bar” being set really high. Being really certain about a candidate means minimizing the possibility of making a bad hire (aka “false positives”). And companies will do whatever they can to avoid that.

A bad candidate will cost a lot of money and effort and waste other people’s time fixing all their bugs. Firing someone you hired by mistake can take months and be nightmarishly difficult, especially if they decide to be litigious about it.

– Joel Spolsky

Hunter and Schmidt quantified the cost of a bad hire: “The standard deviation… has been found to be at minimum 40% of the mean salary,” which in today’s terms would translate to $40,000 assuming a mean engineer salary of $100,000/year.

But if you set “The Bar” too high, chances are you’ll also miss out on some good candidates (aka “false negatives”). McDowell explains why companies don’t really mind a lot of false negatives:

“From the company’s perspective, it’s actually acceptable that some good candidates are rejected… They can accept that they miss out on some good people. They’d prefer not to, of course, as it raises their recruiting costs. It is an acceptable tradeoff, though, provided they can still hire enough good people.”

In other words, it’s worth holding out for a better candidate if the difference in their expected output is large, relative to the recruiting costs from continued searching. Additionally, the costs of HR or legal issues downstream from potentially problematic employees also tilt the calculation toward keeping “The Bar” high.

This is a very rational cost-benefit calculation. But has anyone ever done this calculation before? If you have done it, we’d love to hear from you. Otherwise, it seems difficult to do.

Given that nearly everyone is using hand-wavy math, if we do the same, maybe we can convince ourselves that “The Bar” doesn’t have to be set quite so high.

As mentioned before, the distribution of candidate ability might not be so binary, so Spolsky’s nightmare bad hire scenario wouldn’t necessarily happen with all “bad” hires, meaning the expected difference in output between “good” and “bad” employees might be lower than perceived.

Recruiting costs might be higher than perceived because finding and employing +1 standard deviation employees gets increasingly difficult. By definition, fewer of those people exist as your bar rises. Schmidt and Hunter’s “bad hire” calculation only compares candidates within an applicant pool. The study does not consider the relative cost of getting high-quality candidates into the applicant pool to begin with, which tends to be the more significant concern for many of today’s tech recruiting teams. And when you consider that other tech companies might be employing the same hiring strategy, competition would increase the average probability that offers get rejected, extending the time to fill a job opening.

Estimating the expected cost of HR involvement is also difficult. No one wants to find themselves interacting with HR. But then again, not all HR teams are as useless as Toby Flenderson.

Taken together, if the expected output between “good” and “bad” candidates were less than expected, and the recruiting costs were higher than perceived, it would make less sense to wait for a no-brainer hire, meaning “The Bar” might not have to be set so high.

Even if one does hire an underperformer, companies could adopt the tools of training and employee management to mitigate the negative effects from some disappointing hires. After all, people can and do become more productive over time as they acquire new skills and knowledge.

Employee development seems to rarely get mentioned in conjunction with hiring (Laszlo Bock makes a few connections here and there, but the topics are mostly discussed separated). But when you add employee development into the equation above, you start to see the relationship between hiring employees and developing employees. You can think of it as different methods for acquiring more company output from different kinds of people: paying to train existing employees versus paying to recruit new employees.

You can even think of it as a trade-off. Instead of developing employees in-house, why not outsource that development? Let others figure out how to develop the raw talent, and later pay recruiters to find them when they get good. Why shop the produce aisle at Whole Foods and cook at home when you can just pay Caviar to deliver pad thai to your doorstep? Why spend time managing and mentoring others when you can spend that time doing “real work” (i.e. engineering tasks)?

Perhaps “The Bar” is set high because companies don’t develop employees effectively, which puts more pressure on the hiring side of the company to yield productive employees.

Therefore, companies can lower their risk by shifting the burden of career development onto the candidates themselves. In response, candidates like Recursive Cactus have little choice but to train themselves.

Initially, I thought Recursive Cactus was a crazy outlier in terms of interview preparation. But apparently, he’s not alone.

Candidates are training themselves

Last year we surveyed our candidates about how many hours they spent preparing for interviews. Nearly half of the respondents reported spending 100 hours or more on interview preparation[5].

We wondered whether hiring managers and recruiters had the same expectations for candidates they encounter, Aline asked a similar question on Twitter, and results suggest they vastly underestimate the work and effort candidates endure prior to meeting with a company.

Decision makers clearly underestimate the amount of work candidates put into job hunt preparation. The discrepancy seems to reinforce the underlying and unstated message pervading all these choices around how we hire: If you’re not one of the smart ones (whatever that means), it’s not our problem. You’re on your own.

“The Bar” revisited

So this is what “The Bar” is. “The Bar” is a high standard set by companies in order to avoid false positives. It’s not clear whether companies have actually done the appropriate cost-benefit analysis when setting it, and it’s possible it can be explained by an aversion to investing in employee development.

“The Bar” is in large part meant to measure your general intelligence, but the actual instruments of measurement don’t necessarily follow the academic literature that underlies it. You can even quibble about the academic literature[6]. “The Bar” does measure specific intelligence in computer science, but that measurement might vary depending on who conducts your interview.

Despite the variance that exists across many aspects of the hiring process, we talk about “The Bar” as if it were deterministic. This allows hiring managers to make clear binary choices but discourages them to think critically about whether their team’s definition of “The Bar” could be improved.

And that helps us understand why Recursive Cactus spends so much time practicing. He’s training himself partially because his current company isn’t developing his skills. He’s preparing for the universe of possible questions and interviewers he might encounter because hiring criteria varies a lot, which cover topics that won’t necessarily be used in his day-to-day work, all so he can resemble someone that’s part of the “smart” crowd.

That’s the system he’s working within. Because the system is the way it is, it’s had significant impact on his personal life.

My wife’s said on more than one occasion that she misses me. I’ve got a rich happy life, but I don’t feel I can be competitive unless I put everything else on hold for months. No single mom can be doing what I’m doing right now.

– Recursive Cactus

This impacts his current co-workers too, whom he cares about a lot.

This process is sufficiently demanding that I’m no longer operating at 100% at work. I want to do the best job at work, but I don’t feel I can do the right thing for my future by practicing algorithms 4 hours a day and do my job well.

I don’t feel comfortable being bad at my job. I like my teammates. I feel a sense of responsibility. I know I won’t get fired if I mail it in, but I know that it’s them that pick up the slack.

– Recursive Cactus

It’s helpful to remember that all the micro decisions made around false positives, interview structure, brain teasers, hiring criteria, and employee development add up to define a system that, at the end of the day, impacts people’s personal lives. Not just the lives of the job hunters themselves, but also all the people that surround them.

Hiring is nowhere near a solved problem. Even if we do solve it somehow, it’s not clear we would ever eliminate all that uncertainty. After all, projecting a person’s future work output after spending an hour or two with them in an artificial work setting seems kinda hard. While we should definitely try to minimize uncertainty, it might be helpful to accept it as a natural part of the process.

This system can be improved. Doing so requires not only coming up with new ideas, but also revisiting decades-old ideas and assumptions, and expanding upon that prior work rather than anchoring ourselves to it.

We’re confident that all of you people in the tech industry will help make tech hiring better. We know you can do it, because after all, you’re smart.

[1] There does exist some potential for selection bias, particularly around the time frames when people choose to practice. Cursory analysis suggests there’s not much of a relationship, but we’re currently digging in deeper (hinthint: could be a future blog post). You can also choose between the traditional algorithmic interview vs. a systems design interview, but the vast majority opt for the traditional interview. The passthrough rates shown are for the traditional interview.
[2] You might be wondering about the relative quality of candidates on While it’s hard to pin down the true distribution of quality (which is the underlying question of this blog post), on average our practice interviewers have communicated to us that the quality of candidates on tends to be similar to the quality of candidates they encounter during their own company’s interview process, particularly during the phone screen process.
[3] This only includes candidates who have met our internal hiring bar and attended a company interview on our site. This does not represent the entire population of candidates who have interviewed with an interviewer.
[4] For those of you that have used before, you may remembere that we had an algorithm that adjusted for interviewer strictness. Upon further inspection, we found this algorithm also introduced variance to candidate scores in really unexpected ways. Because of this, we don’t rely on this algorithm as heavily.
[5] Spikes at 100 and 200 hours occurred because of an error in the labeling and max value of the survey question. The 3 survey questions asked were the following: 1) In your most recent job search, how many hours did you spend preparing for interviews? 2) How many hours did you spend on interview preparation before signing up for 3) How many hours did you spend on interview preparation after signing up for (not including time using itself)? Each question had a max value of 100 hours, but many respondents responded to questions 2) and 3) where their sum exceeded 100. The distribution here shows the sum of 2) and 3). The median from question 1) responses was 94, nearly identical to the median of the sum of 2) and 3), so we used the sum to observe the shape of the distribution beyond 100 hours. Key lessons: assume a larger max value than you’d expect, and double-check your survey.
[6]I found the study a little hard to reason about, mainly because I’m not a psychologist, so techniques like meta analysis were a little foreign to me even if the underlying statistical tools were familiar. It’s not a question of whether the tools are valid, it’s that reasoning about the study’s underlying data was difficult. Similar to spaghetti code, the validation of the underlying datasets is spread across decades of prior academic papers, which makes it difficult to follow. It’s likely this is the nature of psychology, where useful data is harder to acquire, at least compared to the kinds of data we deal with in tech. Beyond that, I also had other questions about their methodology, which this article asks in far greater detail than I could.



No engineer has ever sued a company because of constructive post-interview feedback. So why don’t employers do it?

Posted on February 6th, 2020.

One of the things that sucks most about technical interviews is that they’re a black box—candidates (usually) get told whether they made it to the next round, but they’re rarely told why they got the outcome that they did. Lack of feedback, or feedback that doesn’t come right away, isn’t just frustrating to candidates. It’s bad for business. We did a whole study on this. It turns out that candidates chronically underrate and overrate their technical interview performance, like so:

Where this finding starts to get actionable is that there’s a statistically significant relationship between whether people think they did well in an interview and whether they’d want to work with you. In other words, in every interview cycle, some portion of interviewees are losing interest in joining your company just because they don’t think they did well, even when they actually did. It makes sense… when you suspect you might not have done well, you’re prone to embark on a painful bout of self-flagellation, and to make it stop, you’ll rationalize away the job by telling yourself that you totally didn’t want to work there anyway.

Practically speaking, giving instant feedback to successful candidates can do wonders for increasing your close rate.

In addition to making candidates you want today more likely to join your team, feedback is crucial for the candidates you might want down the road. Technical interview outcomes are highly non-deterministic. According to our data, only about 25% of candidates perform consistently from interview to interview. Why does this matter? If interview outcomes are erratic, it means that the same candidate you reject this time might be someone you want to hire in 6 months. It’s in your interest to forge a good relationship with them now and be cognizant of and humble about the flaws in your hiring process.

I thought this tweet captured my sentiments particularly well.

Danny Trinh's tweet

So, despite the benefits, why do most companies persist in giving slow feedback or none at all? I surveyed founders, hiring managers, recruiters and labor lawyers (and also put out some questions to the Twitterverse) to understand why anyone who’s ever gone through interviewer training has been told in no uncertain terms to not give feedback.

As it turns out, feedback is discouraged primarily because companies are scared of getting sued… and because interviewers fear defensive candidate backlash. In some cases, giving feedback is avoided just because companies view it as a no-upside hassle.

The sad truth is that hiring practices have not caught up with market realities. Many of the hiring practices we take for granted today originated in a world where there was a surplus of candidates and a shortage of jobs. This extends to everything from painfully long take-home assignments to poorly written job descriptions. And post-interview feedback is no exception. As Gayle Laakmann McDowell, author of Cracking the Coding Interview, explains on Quora:

“Companies are not trying to create the most perfect process for you. They are trying to hire—ideally efficiently, cheaper, and effectively. This is about their goals, not yours. Maybe when it’s easy they’ll help you too, but really this whole process is about them… Companies do not believe it helps them to give candidates feedback. Frankly, all they see is downside.”

Look, I’m guilty of this, too. Here’s a rejection email I wrote when I was head of technical recruiting at TrialPay. This email makes me want to go back in time and punch myself in the face and then wish myself the best in my future endeavors to not get punched in the face.

Bad rejection email

These kinds of form letter rejections (which I guess is better than just leaving the person hanging) make a lot of sense when you have a revolving door of disposable candidates. They are completely irrational in this brave new world where candidates have more leverage than companies. But, because HR is fundamentally a cost center tasked with risk mitigation (rather than a profit center tasked with, you know, making stuff better), and because engineers on the ground only have so many cognitive cycles to tackle hard stuff outside their job descriptions, we continue to march forward on autopilot, perpetuating outdated and harmful practices like this one.

In this hiring climate, companies should move toward practices that give candidates a better interview experience. Is fear of litigation and discomfort legit enough to keep companies from giving feedback? Does optimizing for fear and a few bad actors in lieu of candidate experience make sense in the midst of a severe engineering shortage? Let’s break it down.

Does the fear of getting sued even make sense?

While researching this piece, I spoke to a few labor lawyers and ran some Lexis Nexis searches to see just how often a company’s constructive feedback (i.e. not “durrrr we didn’t hire you because you’re a woman”) to a rejected eng candidate has resulted in litigation.


As some of my lawyer contacts pointed out, a lot of cases get settled out of court, and that data is much harder to get. But in this market, creating poor candidate experience to hedge against something that is highly unlikely seems… irrational at best and destructive at worst.

What about candidates getting defensive?

At some point, I stopped writing trite rejection emails like the one above, but I was still beholden to my employer’s rules about written feedback.2 As an experiment, I tried giving candidates verbal feedback over the phone.

For context, I had a unique, hybrid role at TrialPay. Though my title was Head of Technical Recruiting, which meant I was accountable for normal recruiter stuff like branding and sourcing and interview process logistics, my role had one unique component. Because I had previously been a software engineer, to take the heat off the long-suffering eng team, I was often the first line of defense for technical interviews and conducted something like 500 of them that year.

After doing a lot of interviews day in and day out, I became less shy about ending them early when it was clear that a candidate wasn’t qualified (e.g. they couldn’t get through the brute force solution to the problem, let alone optimize). Did ending interviews early cause candidates to fly off the handle or feel particularly awkward, as many people suspect?

Defensive candidates tweet

In my experience, cutting things off and saying nothing about why is a lot more awkward and leads to more defensiveness than letting candidates know what the deal is. Some candidates will get defensive (at which point you can politely end the call), but if you offer constructive feedback—let them know what went wrong, make some recommendations about books to read, point them to problem repositories like Leetcode3, etc.—most will be grateful. My personal experience with giving feedback has been overwhelmingly positive. I used to love mailing books to candidates, and I formed lasting relationships with many. Some became early users a few years later.

Anyway, the way to avoid negative reactions and defensiveness from candidates is to practice giving feedback in a way that’s constructive. We’ll cover this next.

So if giving feedback isn’t actually risky and has real upsides, how does one do it?

When I started, it was the culmination of what I had started experimenting with at TrialPay. It was clear to me that feedback is a Good Thing and that candidates liked it… which in this market means it’s also good for companies. But, we still had to grapple with prospective customers’ (pretty irrational) fears about the market being flooded with defensive candidates with a lawyer on speed dial.

For context, is a hiring marketplace. Before talking to companies, engineers can practice technical interviewing anonymously, and if things go well, unlock our jobs portal, where they can bypass the usual top-of-funnel cruft (applying online, talking to recruiters or “talent managers,” finding friends who can refer them) and book real technical interviews with companies like Microsoft, Twitter, Coinbase, Twitch, and many others… often as early as the next day.

The cool thing is that both practice and real interviews with companies take place within the ecosystem, and wrt feedback, you’ll see why this matters in a moment.

Before we started working with employers, we spent some time building out our practice platform and getting the mechanics right. For practice interviews, our post-interview feedback forms looked like this:

The feedback form that an interviewer fills out

The feedback form that an interviewer fills out

After each practice interview, interviewers fill out the form above. Candidates fill out a similar form rating their interviewer. When both parties fill out their forms, they can see each other’s responses.

If you’re curious, you can watch people practicing and read real feedback that they got in our public showcase. Here’s a snapshot:

An interviewer's rubric after an interview

Check out our showcase. It’s cool.

When we started letting employers hire on our platform, we just recycled this post-interview feedback format, told them they should leave feedback to help us calibrate and because it’s good for candidate experience, and fervently hoped that they wouldn’t have an issue with it.

To our surprise and delight, employers were eminently willing to leave feedback. On our platform, candidates were able to see whether they passed or not and exactly how they did, just a few minutes after the interview was over, stopping the rising tide of post-interview anxiety and self-flagellation in its tracks, and, as we’ve said, increasing the likelihood that a great candidate will accept an offer.

A real, successful company interview on

A real, successful company interview on

And if a candidate failed an interview, they got to see exactly why they failed and what they needed to work on, probably for the first time ever in interview history.

A real, failed company interview on

A real, failed company interview on

Anonymity makes it easier to give feedback

On, interviews are anonymous: an employer knows literally nothing about the candidate before and during the interview (employers can even enable our real-time voice masking feature). Candidates’ identities are only revealed after a successful interview, i.e. after the employer has already submitted feedback.

We insist on anonymity because about 40% of our top-performing candidates are non-traditional, and we don’t want lack of pedigree or an unusual background to open them up to bias. Because interviews are anonymous, it’s often impossible to discriminate on the basis of age or gender or background. Therefore, feedback has to be constructive, by design, because the only info the interviewer has to go on is how well the candidate is performing in the interview. In addition to helping candidates get a fair shot, this anonymity provides something of a safety net for employers—it’s harder to build a discrimination case out of the feedback when the employer doesn’t know the identity of the candidate.

In many other contexts, anonymity can be destructive because of reduced accountability. But in the interview process, we’ve discovered over and over that anonymity sets free the better angels of our nature and creates a kinder, more inclusive interview experience for candidates and employers alike.

Building post-interview feedback into your eng hiring process

So, how can you fold these learnings into your process? Of course, the easiest way to get your feet wet is to start using We’ll get you candidates you wouldn’t be sourcing without us, and we’ll empower you to give them the best interview experience you can.

But even if you don’t use us, based on how unlikely it is that you’re going to get sued or deal with angry candidates, we strongly recommend having your interviewers provide constructive feedback (like in the examples above) after every interview for all candidates, whether they pass or not, over email.

Here are a few strategies for delivering constructive feedback:

  1. Be clear that it’s a no-go. Ambiguity is psychologically difficult in a stressful situation. For instance: Thank you for interviewing with us. Unfortunately, you didn’t pass the interview.
  2. After you make it clear that it’s a no-go, tell them something nice. Find something about their performance—an answer they gave, or the way they thought through a problem, or how they asked the right questions—and share it with them. They’ll be more receptive to the rest of your feedback once they know that you’re on their side. For instance: Despite the fact that it didn’t work out this time, you did {x,y,z} really well, and I think that you can do much better in the future. Here are some suggestions for what you can work on.
  3. When you give suggestions, be specific and constructive. Don’t tell them that they royally screwed the whole operation and need to rethink their line of work. Instead, focus on specific things they can work on. Or, to put it another way, “Hey, familiarize yourself with big O notation. It’s not as scary as it sounds bc it comes up a lot in these kinds of interviews.”4 doesn’t say “you’re dumb and your work experience is dumb and you should feel bad” or “you seem like an asshole.” It says you should familiarize yourself with big O notation.
  4. Make recommendations. Is there a book they could read? If they’re promising but just lack knowledge, it’s a really nice gesture to ship said book to them.
  5. If you think the candidate is on their way to becoming a great engineer (especially if they take your recommendations and advice!), let them know that they can contact you again in a few months. You’ll build goodwill with someone who, even if they don’t work with you in the future, will talk about you to others. And when they do improve, you’ll be in a better position to bring them onto your team.
1If you know of such a case, please tell me, and I’ll update this post/associated content accordingly ASAP.
2This is a rule pretty much every company has, ever. It’s not just TrialPay, which was a great place to work and whose defensive HR policies, like every other company’s, were in no way indicative of their workplace culture.
3Whether algorithmic problems of the type you’d find on Leetcode are the best way to interview is a question worth asking… and I’ve since come to feel pretty strongly that many of them are not. But that’s out of scope for this piece.
4I recently discovered an amazing online book that makes big O approachable, practical, and not scary (all without talking down to the reader): Grokking Algorithms by Aditya Bhargava.



We ran the numbers, and there really is a pipeline problem in eng hiring.

Posted on December 3rd, 2019.

If you say the words “there’s a pipeline problem” to explain why we’ve failed to make meaningful progress toward gender parity in software engineering, you probably won’t make many friends (or many hires). The pipeline problem argument goes something like this: “There aren’t enough qualified women out there, so it’s not our fault if we don’t hire them.”

Many people don’t like this reductive line of thinking because it ignores the growing body of research that points to unwelcoming environments that drive underrepresented talent out of tech: STEM in early education being unfriendly to children from underrepresented backgrounds, lack of a level playing field and unequal access to quality STEM education (see this study on how few of California’s schools offer AP Computer Science for instance), hostile university culture in male-dominated CS programs, biased hiring practices, and ultimately non-inclusive work environments that force women and people of color to leave tech at disproportionately high rates.1

However, because systemic issues can be hard to fix (they can take years, concerted efforts across many big organizations, and even huge socioeconomic shifts), the argument against the pipeline problem tends to get reduced to “No, the candidates are there. We just need to fix the bias in our process.”

This kind of reductive thinking is also not great. For years, companies have been pumping money and resources into things like unconscious bias training (which has been shown not to work), anonymizing resumes, and all sorts of other initiatives, and the numbers have barely moved. It’s no wonder tech eventually succumbs to a “diversity fatigue” that comes from trying to make changes and not seeing results.

We ran the numbers and learned that there really IS a pipeline problem in hiring — there really aren’t enough women to meet demand… if we keep hiring the way we’re hiring. Namely, if we keep over-indexing on CS degrees from top schools, and even if we remove unconscious bias from the process entirely, we will not get to gender parity. And yes, there is a way to surface strong candidates without relying on proxies like a college degree. We’ll talk about that toward the end.

Our findings ARE NOT meant to diminish the systemic issues that make engineering feel unwelcome to underrepresented talent, nor to diminish our ability to work together as an industry to effect change — to enact policy changes like access to CS in public schools, for instance. Our findings ARE meant to empower those individuals already working very hard to make hiring better who find themselves frustrated because, despite their efforts, the numbers aren’t moving. To those people, we say, please don’t lose sight of the systemic problems, but in the short term, there are things you can do that will yield results. We hope that, over time, by addressing both systemic pipeline issues and biases, we will get to critical mass of women from all backgrounds in engineering positions, and that these women, in turn, will do a lot to change the pipeline problem by providing role models and advocates and by changing the culture within companies.

Lastly, an important disclaimer before we proceed. In this post, we chose to focus on gender (and not on race). This decision was mostly due to the dearth of publicly available data around race and intersectionality in CS programs/bootcamps/MOOCs.2 While this analysis does not examine race and intersectionality, it is important to note that we recognize: 1) Not all women have the same experience in their engineering journey, and 2) tech’s disparities by gender is no more important than the lack of representation of people of color in engineering. We will revisit these subjects in a future post.

The percentage of women engineers is low and likely worse than reported

It’s very hard to have a cogent public conversation about diversity when there is no standardization of what statistic means what. As this post is about engineering specifically, we needed to find a way to get at how many women engineers are actually in the market and work around two big limitations in how FAAMNG (Facebook, Amazon, Apple, Microsoft, Google, and Netflix) report their diversity numbers.

The first limitation is that FAAMNG’s numbers are global. Why does this matter? It turns out that other countries, especially those where big companies have their offshore development offices, tend to have a higher percentage of female developers.3 In India, for instance, about 35% of developers are women; in the U.S., it’s 16%. Why are these numbers reported globally? The cynic in me says that it’s likely because the U.S. numbers, on their own, are pretty dismal, and these companies know it.4 To account for this limitation and get at the U.S. estimate, we did some napkin math and conservatively cut each company’s female headcount by 20%.

The second limitation is that reported numbers are based on “technical roles,” which Facebook at least defines very broadly: “A position that requires specialization and knowledge needed to accomplish mathematical, engineering, or scientific related duties.” I expect the other giants use something similar. What are the implications of this fairly broad definition? Given that product management and UX design count as technical roles, we did some more napkin math and removed ~20% to correct for PMs and designers.4

With these limitations in mind, below is a graph comparing the makeup of the U.S. population to its representation in tech at FAAMNG companies where said data was available, as well as an estimate of women in engineering specifically.

There's still a lot of work to do if we want to reach gender parity in engineering


If we want to reach gender parity in engineering, especially when we correct for women in the U.S. (and whether they’re actually software engineers), you can see that we have a long way to go.

Is it a pipeline problem?

So, are there just not enough qualified women in the hiring pool? It turns out that we’re actually hiring women at pretty much the same rate that women are graduating with CS degrees from four-year universities — out of the 71,420 students who graduated with a CS degree in 2017, 13,654, or ~20%, were women.5 So maybe we just need more women to get CS degrees?

Top tech companies and their non-profit arms have been using their diversity and inclusion budgets to bolster education initiatives, in the hopes that this will help them improve gender diversity in hiring. Diversity initiatives started taking off in earnest in 2014, and in 4 years, enrollment in CS programs grew by about 60%. It’s not anywhere near enough to get to gender parity.

And even if we could meaningfully increase the number of women enrolling in CS programs overall, top companies have historically tended to favor candidates from elite universities (based on some targeted LinkedIn Recruiter searches, 60% of software engineers at FAAMNG hold a degree from a top 20 school). You can see enrollment rates of women in 3 representative top computer science programs below. Note that while the numbers are mostly going up, growth is linear and not very fast.

Growth in women's undergraduate computer science enrollment at top schools

Sources: UC-Berkeley, MIT (1 and 2), and Stanford enrollment data

To see if it’s possible to reach gender parity if we remove unconscious bias but keep hiring primarily from top schools, let’s build a model. For the purposes of this model let’s focus solely on new jobs — if companies want to meet their diversity goals, at a minimum they need to achieve parity on any new jobs they’ve created. Based on the US BLS’s projections, the number of software engineering jobs is estimated to increase by 20% by 2028 (or about 1.8% annually). Today, the BLS estimates there are about 4 million computer-related jobs. This projects to about 70,000 new jobs created this year, increasing to 85,000 new jobs created in 2028.

If the goal is to hit gender parity in the workforce, our goal should be to have 50% of these new seats filled by women.

To see if this is possible, let’s project the growth of the incoming job pool over the same timeframe. Based on NCES’ 2017 survey, computer science graduates have grown annually anywhere between 7% and 11% this decade. Let’s optimistically assume this annual growth rate persists at 10%. Let’s also assume that the
percentage of graduates who are women remains at 20%, which has been true for the last 15 years. But, there are some gotchas.

First, there’s no guarantee that the seats earmarked for women actually get filled by women, particularly in a world where male CS graduates will continue to outnumber females 4-to-1. Not all of these jobs will be entry-level, so some portion of these jobs will be pulling from an already female-constrained pool of senior candidates. Finally, there’s no guarantee that traditional 4-year colleges will be able to support the projected influx of computer science candidates, particularly from the top-tier universities that companies usually prefer. Below, we graph the net new seats we’d need to fill if women held half of software engineering jobs (blue line) vs. how many women are actually available to hire if we keep focusing largely on educational pedigree in our recruiting efforts (red line). As you can see, it’s not possible to hit our goals, whether or not we’re biased against women at any point in the hiring process.6


So if the pipeline is at least partially to blame, what can we do?

You saw above that enrollment in undergraduate computer science programs among women is growing linearly. Rising tuition costs coupled with 4-year universities’ inability to keep up with demand for computer science education have forced growing numbers of people to go outside the system to learn to code.

Below is a graph of the portion of developers who have a bachelor’s degree in computer science and the portion of developers who are at least partially self-taught, according to the Stack Overflow Developer Surveys from 2015 to 2019. As you can see, in 2015, the numbers were pretty close, and then, with the emergence of MOOCs, there was a serious spurt, with more likely to come.

Education sources for software engineers

Sources: Stack Overflow Developer Surveys for 2015, 2016, 2018, and 2019

The rate of change in alternative, more affordable education is rapidly outpacing university enrollment. Unlike enrollment in traditional four-year schools, enrollment in MOOCs and bootcamps is growing exponentially.

In 2015 alone, over 35 million people have signed up for at least one MOOC course, and in 2018 MOOCs collectively had over 100M students. Of course, many people treat MOOCs as a supplement to their existing educational efforts or career rather than relying on MOOCs entirely to learn to code. This is something we factored into our model.

Despite their price tag (most charge on the order of $10-20K), bootcamps seem like a rational choice when compared to the price of top colleges. Since 2013, bootcamp enrollment has grown 9X, with a total of 20,316 grads in 2018. Though these numbers represent enrollment across all genders7 and the raw number of grads lags behind CS programs (for now), below you can see that the portion of women graduating from bootcamps is also on the rise and that graduation from online programs has actually reached gender parity (as compared to 20% in traditional CS programs).

Source: Course Report’s Data Dive: Gender in Coding Bootcamps

Source: Course Report’s Data Dive: Gender in Coding Bootcamps

Of course, one may rightfully question the quality of grads from alternative education programs. We factored in bootcamp placement rates in building our updated model below.

Outside of alternative education programs, the most obvious thing we can do to increase the supply of qualified women engineers is to expand our pipeline to include strong engineers who don’t hail from top schools or top companies.

In previous posts, we looked at the relationship between interview performance and traditional credentialing and found that participation in MOOCs mattered almost twice as much for interview performance than whether the candidate had worked at a top company. And top school was least predictive of performance and sometimes not at all. And some of my earlier research indicates that the most predictive attribute of a resume is the number of typos and grammatical errors (more is bad), rather than top school or top company. In this particular study, experience at a top company mattered a little, and a degree from a top school didn’t matter at all.

But, even if lower-tier schools and alternative programs have their fair share of talent, how do we surface the most qualified candidates? After all, employers have historically leaned so hard on 4-year degrees from top schools because they’re a decent-seeming proxy. Is there a better way?

But culling non-traditional talent is hard… that’s why we rely on pedigree and can’t change how we hire!

In this brave new world, where we have the technology to write code together remotely, and where we can collect data and reason about it, technology has the power to free us from relying on proxies, so that we can look at each individual as an indicative, unique bundle of performance-based data points. At, we make it possible to move away from proxies by looking at each interviewee as a collection of data points that tell a story, rather than a largely signal-less document a recruiter looks at for 10 seconds and then makes a largely arbitrary decision before moving on to the next candidate.

Of course, this post lives on our blog, so I’ll take a moment to plug what we do. In a world where there’s a growing credentialing gap and where it’s really hard to figure out how to separate a mediocre non-traditional candidate from a stellar one, we can help. helps companies find and hire engineers based on ability, not pedigree. We give out free mock interviews to engineers, and we use the data from these interviews to identify top performers, independently of how they look on paper. Those top performers then get to interview anonymously with employers on our platform (we’ve hired for Lyft, Uber, Dropbox, Quora, and many other great, high-bar companies). And this system works. Not only are our candidates’ conversion rates 3X the industry standard (about 70% of our candidates ace their phone screens, as compared to 20-25% in a typical, high-performing funnel), about 40% of the hires made by top companies on our platform have come from non-traditional backgrounds. Because of our completely anonymous, skills-first approach, we’ve seen an interesting phenomenon happen time and time again: when an engineer unmasks at the end of a successful interview, the company in question realizes that the student who just aced their phone screen was one whose resume was sitting at the bottom of the pile all along (we recently had someone get hired after having been rejected by that same company 3 times based on his resume!).

Frankly, think of how much time and money you’re wasting competing for only a small pool of superficially qualified candidates when you could be hiring overlooked talent that’s actually qualified. Your CFO will be happier, and so will your engineers. Look, whether you use us or something else, there’s a slew of tech-enabled solutions that are redefining credentialing in engineering, from asynchronous coding assessments like CodeSignal or HackerRank to solutions that vet candidates before sending them to you, like Triplebyte, to solutions that help you vet your inbound candidate pool, like Karat.

And using these new tools isn’t just paying lip service to a long-suffering D&I initiative. It gets you the candidates that everyone in the world isn’t chasing without compromising on quality, helps you make more hires faster, and just makes hiring fairer across the board. And, yes, it will also help you meet your diversity goals. Here’s another model.

How does changing your hiring practices improve the pipeline?

Above, you saw our take on status quo supply and demand of women engineers — basically how many engineers are available to hire using today’s practices versus how many we’d need to actually reach gender parity. Now, let’s see what it looks like when we include candidates without a traditional pedigree (yellow line).


As you can see, broadening your pipeline isn’t a magic pill, and as long as demand for software engineers continues to grow, it’s still going to be really hard, systemic changes to our society notwithstanding. If we do make these changes, however, the tech industry as a whole can accelerate its path toward gender parity and potentially get there within a decade.

What about specific companies? An interactive visualization.

So far we’ve talked about trends in the industry as a whole. But, how do these insights affect individual employers? Below is an interactive model where you visualize when Google, Facebook, or your company (where you can plug in your hiring numbers) will be able to hit their goals based on current hiring practices versus the more inclusive ones we advocate in this post. Unlike the industry as a whole, built into this visualization is the idea of candidate reach, as well as hire rates — one company can’t source and hire ALL the women (as much as they might want to). Of course, the stronger your brand, the higher your response rates will be.

We made some assumptions about response rates to sourcing outreach for both Google and Facebook. Specifically, we guessed a 60%-70% response rate for these giants based on the strength of their brand and their army of sourcers — when those companies reach out and tenaciously follow up, you’ll probably respond eventually.8 We also made some assumptions about their hire rates (5-10% of interviewed candidates). You can see both sets of assumptions below. And you can see that even with all the changes we propose, in our model, Google and Facebook will still not get to gender parity!

We also included a tab called “Your company” where you can play around with the sliders and see how long it would take your company to get to gender parity/whether it’s possible. There, we made much more conservative assumptions about response rates!

As you can see, for the giants, getting to gender parity is a tall order even with broadening your pipeline to include non-traditional candidates. And while it may be easier for smaller companies to get there without making drastic changes, when you’re small is exactly the right time to get fairer hiring into your DNA. It’s much harder to turn the ship around later on.


Regardless of whether you’re a giant or a small company, as long as hiring practices largely limits itself to top schools, the status quo will continue to be fundamentally inefficient, unmeritocratic, and elitist, and any hope of reaching gender parity will be impossible. Look, there are no easy fixes or band-aids when it comes to diversifying your workforce. Rather than continuing to look for hidden sources of women engineers (I promise, we’re not all hiding somewhere, just slightly out of reach) or trying to hire all the women from top schools, the data clearly shows that the only path forward is to improve hiring for everyone by going beyond top schools and hiring the best people for the job based on their ability, not how they look on paper.

I was recently in a pitch meeting where I got asked what’s mission is. I said that it’s to make hiring more efficient. The investors in the room were a bit surprised by this and asked, given that I care about hiring being fair, why that’s not the mission. First off, “fair” is hard to define and open to all manners of interpretation, whereas in an efficient hiring market, a qualified candidate, by definition, gets the job, with the least amount of pain and missteps. In other words, meritocracy is a logical consequence of efficiency. Secondly, and even more importantly, while I firmly believe that most people at companies want to do “the right thing”, it’s much easier to actually do the right thing in a big organization when it’s also cheaper, better, and faster.

All that’s to say that there are no shortcuts, and the most honorable (and most viable) path forward is to make hiring better for everyone and then hit your diversity goals in the process (or at least get closer to them). Software engineering is supposed to be this microcosm of the American dream — anyone can put in the work, learn to code, and level up, right? Until we own our very conscious biases about pedigree and change how we hire, that dream is a hollow pipe.

Appendix: Model Description and assumptions

To assess whether there exists a pipeline problem, we need to estimate the number of job openings that exist, as well as the number of recent female job market entrants that could feasibly fill those roles. If a pipeline problem does exist, the number of job openings would be greater than the number of female entrants.

For this analysis, we focused on new jobs created over the next 10 years and ignored openings from existing jobs due to attrition. Unfortunately, engineering does have a significantly higher attrition rate for women than other industries, so likely the numbers are worse than they appear in our models.9

That said, if a company wants to meet its diversity goals, it seems reasonable to expect them to do so with jobs that don’t yet exist, rather than on existing jobs whose pool of candidates we know are dominated by men.

Demand: Projected new jobs created

Tech industry net new jobs created = (# tech industry jobs prior year) x (annual growth rate)


Supply: Projected new women in job pool from top tier universities

New women in job pool = (# CS graduates prior year) x (annual CS graduate growth rate) x (% CS graduates that are women) * (% CS graduates from top tier schools)


Supply: Projected new women in job pool beyond top tier universities

This represents female bootcamp graduates plus female CS graduates not from top schools.

New women in job pool from beyond top schools =
(# bootcamp graduates prior year) x (%  bootcamp graduates that are women) x (% annual bootcamp graduate growth)
+ (# CS graduates prior year) x (annual CS graduate growth rate) x (% women in CS grads) x (% CS graduates not from top tier schools)

Bootcamp assumptions:

  • Current # bootcamp graduates: 20,000 (Course Report)
  • % bootcamp graduates that are women: 40% (Switchup)
  • Bootcamp graduates growth rate: 10% (Course Report)
  • Placement rate: 50% (Switchup)
  • % CS graduates not from top tier schools: 75% (see assumption from “Supply: Projected new women in job pool from top tier universities”)

Assumptions for CS graduates beyond top tier universities are the same as those found under “Supply: Projected new women in job pool from top tier universities”, but taking the remaining 75% of CS graduates excluded there.

Company-specific Demand: Projected number of candidates needed to source for job openings

Number of women to source = (# Engineers employed prior year) x (% annual growth rate) x (% diversity goal) x (1 / hire rate) x (1 / sourcing response rate)

In practice, companies typically need to contact many people for any single job opening, since there is plenty of inherent variability in the sourcing and interview process. This line describes how many people your company would have to reach to fill all new job openings created, based on assumptions about your company’s hiring practices.

1The Kapor Center has some great, data-rich reports on attrition in tech as well as systemic factors that contribute to the leaky pipeline. For a detailed study of attrition in tech, please see the Tech Leavers Study. For a comprehensive look at systemic factors that contribute to the leaky tech pipeline (and a series of long-term solutions), please see this comprehensive report. For a survey of the deplorable state of computer science education in California schools, please see this report.
2For instance, MIT keeps their race stats behind a login wall.
3According to a HackerRank study, “India, the United Arab Emirates, Romania, China, Sri Lanka, and Italy are the six countries with the highest percentage of women developers, [whereas the] U.S. came in at number 11.”.
4We assumed a 1:7 ratio of PMs to engineers and a 1:7 ratio of designers to engineers on product teams. Removing PMs and designers from our numbers does not mean to imply that their representation in tech doesn’t matter but rather to scope this post specifically to software engineers.).
5The National Center for Education Statistics doesn’t yet list graduation rates beyond 2017… the new numbers might be a bit higher, as you’ll see when you look at enrollment numbers for CS a bit further down in the post..
6This is not independent of the idea that deep, systemic issues within the fabric of our society (such as the ones we mention at the beginning of the post) are keeping women from entering engineering in droves. But, as I mentioned in the intro, laying the blame at the feet of these systemic issues entirely paralyzes us and prevents us from fixing the things we can fix.
7We couldn’t find a public record of women’s numbers by year and so are relying on graduation rates from Course Report as a proxy.
8These rates might seem high to recruiters reading this post. They might be high. We did try to correct for 2 things, both of which made our estimates higher: 1) this includes university and junior candidates, who tend to be way more responsive, and 2) this isn’t per sourcing attempt but over the lifetme of a candidate, so it includes followups, outreach to candidates who had previously been declined, and so on. However, if this still seems off, please write in and tell us!
9There are two good sources that look at attrition among women in tech. One is Women in Tech: The Facts, published by the National Center for Women&IT. The other is the excellent Tech Leavers Study by the Kapor Center.

Thank you to the data & eng team for all the data modeling, projections, and visualizations, as well as everyone who proofread the myriad long drafts.



3 exercises to craft the kind of employer brand that actually makes engineers want to work for you

Posted on May 14th, 2019.

If I’m honest, I’ve wanted to write something about employer brand for a long time. One of the things that really gets my goat is when companies build employer brand by over-indexing on banalities (“look we have a ping pong table!”, “look we’re a startup so you’ll have a huge impact”, etc.) instead of focusing on the narratives that make them special.

Hiring engineers is really hard. It’s hard for tech giants, and it’s hard for small companies… but it’s especially hard for small companies people haven’t quite heard of, and they can use all the help they can get because talking about impact and ping pong tables just doesn’t cut it anymore.

At, making small companies shine is core to our business, and I’ll share some of what we’ve learned about branding as a result. I’ll also walk you through 3 simple exercises you can do to help you craft and distill your employer brand… in a way that highlights what’s actually special about you and will make great engineers excited to learn more.

If I have my druthers, this will be one of three posts on brand. This first one will focus on how to craft your story, the second will show you how to use that story to write great job descriptions, and the final one will focus on how to take the story you’ve created and get it out into the world in front of the kinds of people you’d want to hire.

So, onto the first post!

What is employer brand, and why does it matter?

Companies like Google have built a great brand, and this brand is largely what makes it possible for them to hire thousands of good engineers every year — when you think of Google (or, more recently, Microsoft!), you might think of an army of competent, first-principles thinkers competently building products people love… or maybe a niche group of engineers working on moonshots that will change the world.

This is employer brand. Put simply, it’s what comes to mind when prospective candidates think about your company. Employer brand encompasses every aspect of your narrative: whether people use/like your product, the problem you’re trying to solve, your founding story, your mission, your culture, and what generally what it’s like to work for you. Put another way, all of these attributes (and others besides!) coalesce into the visceral feeling people get when they imagine working for you.

Brand is the single most important thing for getting candidates in the door — even if you have a stellar personal network, in most cases, that’ll usually only last you until your first 30 hires or so — after that, your networks begin to sort of converge on one another. Even so, despite how important brand is for hiring, building it is one of the psychologically hardest things to do at the beginning of your journey because the opportunity cost of spending your time on ANYTHING is staggering, and it’s really hard to justify writing blog posts and hosting events and speaking at conferences when you have to build product and make individual hires and do 50 kabillion other things.

But, until you build a brand and get it out in the world, you’re going to be hacking through the jungle with a proverbial machete, making hires one by one, trying to charm each one by telling them your story. And once you have a brand, all of a sudden, sourcing is going to feel really, really different (just like it feels when you’ve found product market fit!).

Over time, if you continue to tell your own story, you, too, will see how much easier sourcing and hiring can be. So, let’s talk about how to craft the right narrative and then proudly shout it from the rooftops.

Why knows about branding

I mentioned earlier that a lot of what we do at is help our customers put their best foot forward and present themselves to engineers in a way that’s authentic and compelling. I’ll show you some examples of good copy in a moment, but here’s a bit of our flow to put it in context.
When engineers do really well in practice, they unlock our jobs portal, where they can see a list of companies and roles, like so:

As you can see, companies simply describe who they are and what they do, and top-performing engineers just book real interviews with one click. Because our goal is to remove obstacles from engineers talking to other engineers, we don’t have talent managers or talent advocates or, as they’re often called, recruiters, on staff to talk to our candidates and try to convince them to interview at a certain company. As a result, we often find ourselves coaching companies on how to present themselves, given limited time and space. We do work with quite a few companies whose brands are household names, but a good chunk of our customers are smaller, fast-growing companies. What’s interesting is that while, on our platform, a household name can have 7X the engagement of a company no one’s heard of, companies no one’s heard of that have exceptional brand narratives aren’t far behind burgeoning unicorns! (We define engagement as the frequency with which candidates who look at our jobs portal then choose to visit that employer’s page.)

Candidate engagement as a function of employer brand

And that’s why having a brand story matters… and why we’re equipped to talk about it at some depth.

What constitutes brand

Below are some attributes that can make up a brand story. As you look at the list, think about what each of these corresponds to in your company, and then, think about which of these are the most unique to you. For instance, every early-stage startup can say they have a culture characterized by autonomy and the potential for huge impact. It’s become a trope that doesn’t differentiate anyone in any way anymore and is therefore probably not worth emphasizing. On the other hand, if you are solving a problem that a lot of your candidates happen to have or if you use a really cool tech stack that attracts some niche community, that’s really special and worth emphasizing.

  • Your product and whether people have heard of it/like it
  • Your growth numbers if they’re impressive
  • Your tech stack and how flashy and cool it is
  • Your founding story and mission… are you working on a problem that people care about personally? If not, are you disrupting some outdated, inefficient way to do things?
  • How hard are the problems you’re solving? Both technically and otherwise?
  • How elite is your team?
  • What is it like to work for you, both with regard to overall culture and then eng culture specifically?1
    • Overall culture:
      • Are you known for kindness/work-life balance? Or grueling hours? (Either can be good depending on whom you want to attract.)
      • What portion of your employees have gone on to found startups?
      • Do you have a lot of organization/structure or are you chaos pirates?
    • Eng culture:
      • Are you more pragmatic/hacker-y vs. academic?
      • Do you subscribe to or actively reject any particular development methodologies (e.g. agile)?
      • Do you tend to roll your own solutions in house or do you try to use 3rd party tools wherever possible?
      • Do you open source parts of your code? Or regularly contribute to open source?

How to craft your story in 3 easy exercises!

There isn’t a single good formula for what to focus on or highlight when crafting this story, but there are a few exercises that we’ve seen be effective. You can do them in the order below, and by the end, you should have a concise, authentic, pithy narrative that you can shout proudly to the world.

We’ve found that it makes sense to do these exercises in the order below. First, you’re going to go with your gut and craft a high-level story. Then you’ll embellish it with details that make you unique and with anecdotes from your employees. And finally, you’ll edit it down into something crisp. As you work, you can use the list from the “What constitutes brand” section above as a reference. Note that, in general, your story plus details shouldn’t have more than 3 distinct ideas total or it’ll start to feel a bit all over the place.

Exercise 1 – The story itself

Imagine you have a smart, technical friend you respect but who doesn’t know anything about your company’s space. Quick, how would you describe what your company does and why it matters to them? Write it down (and target 5-6 sentences… but don’t worry too much about editing it yet… we’ll do that later).

If you’re feeling a bit stuck, here are some questions to get you started — think about how you might answer if your friend were asking each of these:

  • Why does the company exist/what does your product do, and why does that matter?
  • Why are you going to succeed where others have failed?
  • Why does the company matter to you personally?
  • What do you know that no one else does about your space?
  • What is your company doing that no one else is doing, and why does that matter?

As you do this exercise, note that when talking to your friend, you dispense with flowery language and explain things succinctly and clearly in simple terms! And that’s the point — the audience you’re selling to is not different than your friend, and your friend probably shares the same cynicism about disingenuous branding that they do!

Exercise 2 – The unique embellishments

Once you have the story you came up with above, which will likely be at a pretty high level, it’s time to drill down into the details that make you special. These details will likely be 2nd order, in the sense that they won’t be as broad or all-encompassing as the attributes that came to mind in the first exercise, but they might still be special and unique and worth noting.

Some examples of unique embellishments can be:

  • Your tech stack
    • Do you use any cutting-edge programming languages that one might not often see being used in production? If so, it might be a bit polarizing but attract the community around that language. More on the value of polarization when it comes to unique embellishments below.
  • Unfettered access to some type of user/a specific group you care about that your product impacts
    • Do you build products for Hollywood? Or for VCs? Or for schools? Some portion of your candidates, depending on their interests outside of work or their future career ambitions, are going to be really excited that they’ll get more direct access to users who operate in these worlds.
  • Unique lifestyle/work culture stuff like working remotely or pair programming
    • E.g. 37Signals and Pivotal respectively
  • Access to a ton of data/ability to work on massively distributed systems
    • E.g. even in its early days, Twitch had almost as much ingest as YouTube, and this was a meaningful selling point to candidates who wanted to work at scale but didn’t necessarily want to work at FAANG

The surprising value of polarization

Today’s job seeker is in equal parts jaded and savvy, and we’re currently in a labor market climate where talent has the power. The latter makes branding especially important, and by now, engineers have been told all manners of generalities about how much impact they’re going to have if they join a small startup and how whatever you’re working on is going to change the world… to the point where these things have become cliches that shows like Silicon Valley deride with impunity. To avoid cliches like this, think about what TRULY makes you special, and even if it’s a bit polarizing, own it. It’s your story, and the more honest you are about who you are and what you do, the more trust you’ll build with your audience and the more they’ll want to engage.

Another way to say this is that the most memorable stories might be shrouded in a bit of controversy. That’s not to say that you have to be controversial or contrive it when it isn’t there, but if you do operate in a space or have some aspect to your culture or tech stack that not everyone agrees with, you might find that the resulting self-selection among candidates can work to your advantage. Below are some examples of polarizing narratives.

  • Your work style. Some companies really value work-life balance, whereas others exalt burning the midnight oil. Some run on chaos and some take a more orderly approach. Some work by the book, and some choose more of a bulldoze your way to success and ask forgiveness rather than permission approach. An example of the latter is Uber — for a long time, their culture was known for a take-no-prisoners approach to getting things done, and this approach has a certain type of appeal for the right people.
  • Your tech stack. Certainly choosing your tech stack is, first and foremost, and engineering decision, but this decision has some implications for hiring, and choosing a fringe or niche stack or language can be a great way to attract the entire community around it. The more culty a language, the more fiercely passionate its acolytes will be about working for you (e.g. Rust… though by the time this guide comes out it might be something else!). Note that it doesn’t have to be thaaaat fringe of a language choice as long as there’s a tight-knit community around it, e.g. C#.
  • Your engineering culture. Do you subscribe to any particular type of methodology that might be controversial, e.g. are you super into TDD? Are you adamant about rolling all your own everything?

Note that there is no right or wrong here — to loosely paraphase Tolstoy, every startup is broken in its own way, and one saying we’ve heard is that, especially during the early days, The only thing you can do wrong is not own who you are — if you misrepresent how you work or make decisions, you’ll find yourself in one of two regrettable positions: either your hires will leave well before their time or you’ll have a bunch of people marching in different directions or completely paralyzed and unable to choose the right course of action on their own.

Exercise 3 – Your employees’ unique perspective

As your team grows beyond you, you will find that your employees’ reasons for working for you are likely different than the answers to the questions above. Talking to them (or, if you don’t want to put them on the spot, having another team member do so) can surface gold for your narrative. In particular, when I was a recruiter, one of the most useful exercises I did was asking my client to introduce me to a very specific handful of engineers. In particular, I was looking for people who 1) didn’t come to the company through personal referrals and 2) had a lot of good offers during their last job search. Why this mix of people? Because they’re the ones who, despite no personal connection to the company and despite other having other options, actively chose to work for you! You’d be surprised what stories I heard, and they’re rarely just about the company mission. For instance, one candidate I spoke to was really excited about the chance to closely interact with investors because he wanted to start his own company one day. Another was stoked at the chance to use Go in production.

Sometimes you’ll be surprised by what you’ll hear because the people working at your company might be there for very different reasons than you, but these anecdotes help flesh out your narrative and make it feel a bit more personal and real.

Once you have a few choice tidbits from employees, ask yourself whether each one is somehow charming or unusual and whether it’s a reason that a lot of people would find compelling about your company. If it’s all of these things, it should likely make it in your narrative. If it’s not particularly original (e.g. short commute) it may not be worth calling out in your primary narrative, but it’s well worth repeating and telling once you actually interact with candidates.

The finished product

So, what should the finished product look like? At a minimum, it’ll be some concise, compelling copy that you can use in your job descriptions. Hopefully, though, it’s more than that. Hopefully it becomes a consistent refrain you and your team use during calls, interviews, maybe investor pitches… a way to highlight all the things you’re most proud of about your company and the things that make you special… without having to reinvent the wheel every time.

Is brand the be-all and end-all of hiring? Not quite.

In closing, I’d like to leave you with a word or two of encouragement. Sure, as you saw in this post, brand matters. Having a great story will you get somewhere, but it won’t get you everywhere with candidates, and the truth is that the more established you are, the more candidates will come to you. But… there’s one piece of data we found in our interviewing and hiring adventures that flies in the face of brand completely proscribing your hiring destiny.

When we looked at how often candidates wanted to work at companies after interviewing there as a function of brand strength, its impact was not statistically significant. In other words, we found that brand strength didn’t matter at all when it came to either whether the candidate wanted to move forward or how excited the candidate was to work at the company. This was a bit surprising, so I decided to dig deeper. Maybe brand strength doesn’t matter overall but matters when the interviewer or the questions they asked aren’t highly rated? In other words, can brand buttress less-than-stellar interviewers? Not so, according to our data. Brand didn’t matter even when you corrected for interviewer quality. In fact, of the top 10 best-rated companies on our platform, half have no brand to speak of, 3 are mid-sized YC companies that command respect in Bay Area circles but are definitely not universally recognizable, and only 2 have anything approaching household name status.

So, what’s the takeaway here? Maybe the most realistic thing we can say is that while brand likely matters a lot for getting candidates in the door, once they’re in, no matter how well-branded you are, they’re yours to lose.

So, take heart.

Portions of this post will also appear in part in an upcoming, comprehensive Guide to Technical Recruiting and Hiring published by Holloway (where you can sign up if you’d like to read, review, or contribute to it).

1The 2019 Stack Overflow Developer Survey recently came out, and it turns out that in the US the most important thing for engineers is office/company culture… which realistically refers to the eng team culture because that’s engineers will spend most of their time. Anything you can do to call yours out (assuming, well, that it’s good) is going to be a win.



You probably don’t factor in engineering time when calculating cost per hire. Here’s why you really should.

Posted on April 24th, 2019.

Whether you’re a recruiter yourself or an engineer who’s involved in hiring, you’ve probably heard of the following two recruiting-related metrics: time to hire and cost per hire. Indeed, these are THE two metrics that any self-respecting recruiting team will track. Time to hire is important because it lets you plan — if a given role has historically taken 3 months to fill, you’re going to act differently when you need to fill it again than if it takes 2 weeks. And, traditionally, cost per hire has been a planning tool as well — if you’re setting recruiting budgets for next year and have a headcount in mind, seeing what recruiting spent last year is super helpful.

But, with cost per hire (or CPH, as I’ll refer to it from now on in this post) in particular, there’s a problem. CPH is typically blended across ALL your hiring channels and is confined to recruiting spend alone. Computing one holistic CPH and confining it to just the recruiting team’s spend hides problems with your funnel and doesn’t help compare the quality of all your various candidate sources. And, most importantly, it completely overlooks arguably the most important thing of all — how much time your team is actually spending on hiring. Drilling down further, engineering time, specifically, despite being one of the most expensive resources, isn’t usually measured as part of the overall cost per hire. Rather, it’s generally written off as part of the cost of doing business. The irony, of course, is that a typical interview process puts the recruiter call at the very beginning of the process precisely to save eng time, but if we don’t measure eng time spent and quantify, then we can’t really save it.

For what it’s worth, the Twitterverse (my followers are something like 50/50 engineers and recruiters) seems to agree. Here are the results (and some associated comments) of a poll I conducted on this very issue:

And yet, most of us don’t do it. Why? Is it because it doesn’t measure the things recruiters care about? Or is it because it’s hard? Or is it because we can’t change anything, so why bother? After all, engineers need to do interviews, both phone screens and onsites, and we already try to shield them as much as possible by having candidates chat with recruiters or do coding challenges first, so what else can you do?

If you’d like to skip straight to how to compute a better, more inclusive CPH, you can skip down to our handy spreadsheet. Otherwise read on!

I’ve worked as both an engineer and an in-house recruiter before founding, so I have the good fortune of having seen the limitations of measuring CPH, from both sides of the table. As such, in this post, I’ll throw out two ways that we can make the cost per hire calculation more useful — by including eng time and by breaking it out by candidate source — and try to quantify exactly why these improvements are impactful… while building better rapport between recruiting and eng (where, real talk, relationships can be somewhat strained). But first, let’s talk about how CPH is typically calculated.

How is CPH typically calculated, and why does it omit eng time?

As I called out above, the primary purpose of calculating cost per hire is to plan the recruiting department’s budget for the next cycle. With that in mind, below is the formula that you’ll find if you google how to calculate cost per hire (pulled from Workable):

To figure out your CPH, you add up all the external and internal costs incurred during a recruiting cycle and divide by the number of hires.

“External” refers to any money paid out to third parties. Examples include job boards, tools (e.g. sourcing, assessment, your ATS), agency fees, candidate travel and lodging, and recruiting events/career fairs.

“Internal” refers to any money you spend within your company: recruiting team salaries, as well as any employee referral bonuses paid out over the course of the last cycle.

Note that internal costs don’t include eng salaries, as engineering and recruiting teams typically draw from different budgets. Hiring stuff is the domain of the recruiting team, and they pay for it out of their pockets… and engineers pay for… engineering stuff.

What’s problematic is that, while being called “cost per hire” this metric actually tells us what recruiting spends rather than what’s actually being spent as a whole. While tracking recruiting spend makes sense for budget planning, this metric, because of its increasingly inaccurate name, often gets pulled into something it ironically wasn’t intended for: figuring out how much the company is actually spending to make hires.

Why does factoring in engineering time matter?

As you saw above, not only is this the way we compute CPH inaccurate because it doesn’t factor in any time or resource expenditure outside the recruiting team (with eng being the biggest one). But, does engineering time really matter?

Yes, it matters a lot, for the following three reasons:

  1. Way more eng time than recruiting time goes into hiring (as you’ll see in this post!)
  2. Eng time is more expensive
  3. Eng time expenditure can vary wildly by channel

To establish that these things are (probably) true, let’s look at a typical eng hiring funnel.1 For the purposes of this exercise, we’ll start the funnel at the recruiter screen and assume that the costs of sourcing candidates are fixed.2

The green arrows are conversion rates between each step (e.g. 50% of people who get offers accept and get hired). The small gray text at the bottom of each box is how long that step takes for an engineer or recruiter (or both, in the case of an onsite). And the black number is how many times that needs to happen to ultimately make 1 hire, based on the green-arrow conversion rates.

So, with that in mind, to make one hire, let’s see how much time both eng and recruiting need to spend to make 1 hire and how much that time costs. Note that I’m assuming $100/hour is a decent approximation for recruiting comp and $150/hour is a decent approximation for eng comp.

Is eng time spent on recruiting really that costly?

Based on the funnel above, here’s the breakdown of time spent by both engineering and recruiting to make 1 hire. The parentheticals next to each line of time spent are based on how long that step takes times the number of times it needs to happen.

RECRUITING – 15 total hours
10 hours of recruiter screens (20 screens needed * 30 min per screen)
4 hours of onsites (4 onsites needed * 1 hour per onsite)
1 hour of offers (2 offer calls needed * 30 min per offer call)

To make 1 hire, it takes 15 recruiting hours or $1500.

ENGINEERING – 40 total hours
16 hours of phone screens (16 screens needed * 1 hour per screen)
24 hours of onsites (4 onsites needed * 6 hours per onsite)

For 1 hire, that’s a total of 40 eng hours, and on the face of it, it’s $6,000 of engineering time, but there is one more subtle multiplier on eng time that doesn’t apply to recruiting time that we need to factor in. Every time you interrupt an engineer from their primary job, which is solving problems with code, it takes time to refocus and get back into it. If you’re an engineer, you know this deep in your bones. And if you’re not, interruptions are very likely something you’ve heard your engineering friends decry… because they’re so painful and detrimental to continued productivity. Back when I was writing code on a regular basis, it would take me 15 minutes of staring at my IDE (or, if I’m honest, occasionally reading Hacker News or Reddit) to let my brain ease back into doing work after coming back from an interview. And it would take me 15 minutes before an interview to read a candidate’s resume and get in the mindset of whatever coding or design question I was going to ask. I expect my time windows are pretty typical, so it basically ends up being a half hour of ramping up and back down for every hour spent interviewing.

Therefore, with ramp-up and ramp-down time in mind, it’s more like $9,000 in eng hours.3

Ultimately, for one hire, we’re paying a total of $10,500, but eng incurs 6X the cost that recruiting does during the hiring process.

Why does breaking out cost per hire by source matter?

So, hopefully, I’ve convinced you that engineering time spent on hiring matters and that it’s the biggest cost you incur. But, if there’s nothing we can do to change it, and it’s just the cost of doing business, then why factor it in to CPH calculations? It turns out that eng time spent IS a lever you can pull, and its impact becomes clear when you think about cost per hire by candidate source.

To make that more concrete, let’s take a look at 2 examples. In both cases, we’ll pretend that one of our candidate sources has a different conversion rate than the overall rate at some step in the funnel. Then we’ll change up the conversion rate at one step in the funnel and try to guess that the financial implications of that are… and then actually calculate it. You might be surprised by the results.

What happens when you increase TPS to onsite conversion to 50%?

As you can see in the funnel above, a decent TPS to onsite conversion rate is 25%. Let’s say one of your sources could double that to 50% (by doing more extensive top-of-funnel filtering, let’s say). What do you think this will do to cost per hire?

In this model, we’re spending a total of 10 recruiting hours (worth $1000) and 32 eng hours (worth $7200).4 Unlike in the first example, we’re now paying a total of $8200 to make a hire.

In this case, you’ve reduced your recruiting time spent by 30% and your eng time spent by 20%, ultimately saving $2300 per hire. If one of your sources can get you this kind of efficiency gain, you probably want to invest more resources into it. And though doubling conversion from tech screen to onsite sounds great and perhaps something you would have known already about your source, without computing the cost per hire for this channel, it’s not intuitively clear just how much money a funnel improvement can save you, end to end.

What happens when you cut your offer acceptance rate in half?

Another possibility is that one of your sources does pretty well when it comes to candidate quality all the way to offer, but for some reason, those candidates are twice as hard to close. In this scenario, you double both the eng and recruiting time expenditure and ultimately pay an extra $7500 per hire for this source (which you’ll likely want to deallocate resources from here on out).5

In either of the examples above, until you break out CPH by source and see exactly what each is costing you, it’s a lot harder to figure out how to optimize your spend.

How to actually measure cost per hire (and include eng time of course!)

The usual way to calculate cost per hire is definitely useful for setting recruiting budget, as we discussed above, but if you want to figure out how much your whole company is actually spending on hiring, you need to factor in the most expensive piece — engineering time.

To do this, we propose a different metric, one that’s based on time spent by your team rather than overall salaries and fixed costs. Let’s call it “cost per hire prime” or CPH prime.

CPH prime doesn’t factor in fixed costs like salaries or events, which you can still do using the formula above… but it is going to be instrumental in helping you get a handle on what your spend actually looks like and will help you compare different channels.

To make your life easier, we’ve created a handy spreadsheet for you to copy and then fill in your numbers, like so:

As you can see, once you fill them the highlighted cells with your own conversion numbers (and optionally your hourly wages if yours differ much from our guesses), we’ll compute CPH prime for you.

And because we’re a business and want you to hire through us, we’ve included the average savings for companies hiring through our platform. We provide two big value-adds: we can pretty drastically improve your TPS to onsite conversion — about 65% of our candidates pass the tech screen at companies on average. From there, they get offers and accept them at the same rate as you’d see in your regular funnel.

Closing thoughts on building bridges between eng and recruiting

So, why does being cognizant of eng time in your CPH calculations matter? I’ve already kind of beaten it into the ground that it’s the biggest cost sink. However, there’s another, more noble reason, to care about eng time. In my career, having sat on all different sides of the table, I’ve noticed one unfortunate, inalienable truth: engineering and recruiting teams are simply not aligned.

Engineers tend to harbor some resentment toward recruiters because recruiters are the arbiters of how eng spends their time when it comes to hiring without a set of clear metrics or goals that help protect that time.

Recruiters often feel some amount of resentment toward engineers who tend to be resistant to interruptions, toward putting in the time to provide meaningful feedback about candidates so that recruiting can get better, and toward changes in the process.

In our humble opinion, much of the resentment on both sides could be cured by incorporating recruiting and engineering costs together in a specific, actionable way that will reduce the misalignment we’re seeing. Recruiters tend to hold the cards when it comes to hiring practices, so we’d love to see them take the lead to reach across the aisle by proactively factoring in eng time spent during hiring and ultimately incorporating recruiting and eng costs together in one metric that matters. Once that’s in place, recruiting can use the data they gather to make better decisions about how to use eng time, and in the process, rebuild much of the rapport and love that’s lost between the two departments.

1We’re basing these numbers on a mix of ATS reporting (Lever’s recruiting metrics report in particular) and what we’ve heard from our customers.

2We’re assuming sourcing costs are fixed for purposes of simplicity and because this post is largely about the importance of eng time factored in to the funnel. Of course, if you have channels that reduce sourcing time significantly, you’ll want to weigh that when deciding its efficacy.

3Really though, the value of an hour of work for an engineer is intangible and much higher than an hourly wage. There ARE inefficiencies and overhead to having a larger staff, not every hour is effective, and most likely it’s your best people who are conducting interviews. The reality is that the money spent on salaries is probably only a fraction of the true cost to the company, particularly for engineers (as opposed to recruiters).

4Here’s us showing our work in figuring out how much recruiting and eng time it takes to make a hire when your TPS to onsite conversion rate is 50%:
RECRUITING – 15 total hours or $1500
5 hours of recruiter screens (10 screens needed * 30 min per screen)
4 hours of onsites (4 onsites needed * 1 hour per onsite)
1 hour of offers (2 offer calls needed * 30 min per offer call)
ENGINEERING – 32 total hours or $7200
8 hours of phone screens (8 screens needed * 1 hour per screen)
24 hours of onsites (4 onsites needed * 6 hours per onsite)

5Here’s us showing our work in figuring out how much recruiting and eng time it takes to make a hire when you cut your offer acceptance rate in half:
RECRUITING – 30 total hours or $3000
20 hours of recruiter screens (40 screens needed * 30 min per screen)
8 hours of onsites (8 onsites needed * 1 hour per onsite)
2 hours of offers (4 offer calls needed * 30 min per offer call)
ENGINEERING – 80 total hours or $18,000
32 hours of phone screens (32 screens needed * 1 hour per screen)
48 hours of onsites (8 onsites needed * 6 hours per onsite)



Can fake names create bias? An exploration into’s random name generator

Posted on March 7th, 2019.

Hello everyone, my name is Atomic Artichoke, and I’m the newest employee of the team, having joined a couple months ago as a Data Scientist.

Atomic Artichoke isn’t my real name, of course. That’s the pseudonym the platform gave me, right before I took my final interview with the company. If you’ve never used before (and hey, if you haven’t already, why not sign up now?), it’s a platform where you can practice technical interviewing anonymously with experienced engineers (and do real job interviews anonymously too).

On signup, creates an anonymous handle for you

On signup, creates an anonymous handle for you

When it’s time to interview, you and your partner meet in a collaborative coding environment with voice, text chat, and a whiteboard (check out recordings of real interviews to see this process in action). During interviews, instead of your name, your partner will see your pseudonym, like so:

During interviews, you are identified by your anonymized pseudonym

Within the platform, you are identified by your anonymized pseudonym

In my opinion, “Atomic Artichoke” is a pretty cool name. It sounds like a Teenage Mutant Ninja Turtles villain, and alliterative phrases are always cool. However, I had some reservations about that handle, because I feel like the pseudonym represented me in ways with which I didn’t identify. I don’t know how to eat or cook an artichoke, I never really understood atoms much, and I possess no mutant superpowers.

But I wondered, how did the interviewer perceive me? Did this person think “Atomic Artichoke” was a cool name? If so, did that name influence his or her perception of me in any way? More importantly, did my pseudonym have any influence in me getting hired? If I had a different, less cool name, would I have gotten this job?

I know, it’s a silly question. I’d like to think I was hired because of my skills, but who really knows? I was curious, so I wasted invested a few days to investigate.

What we already know about names in the hiring process

You might be asking, “Why does have pseudonyms, anyway?” Anonymity. We want candidates to be assessed on their actual skills, not on proxies of skill like the colleges they’ve attended, the notoriety of their social circles, or prior companies they’ve worked at. If a hiring manager knows a person’s name and knows how to use the Internet, it’s easy to find this information.

I’m not the first to wonder about names and hiring. Plenty of academic literature exists exploring the impact of name choice on various life outcomes. I’ll briefly touch on a handful of those perspectives.

As you can see, academic opinions differ. However, in the case that name-based bias actually exists, maybe we can implement a cheap-enough solution to eliminate the bias completely. Randomly-generated pseudonyms fits that bill nicely.

But as I wondered before, maybe the pseudonym name generator creates a different kind of bias, leaving us in a similarly biased place that using real people’s names leaves us. I first needed to understand how pseudonyms get generated, so I dug into some code.

Exploring code

After dusting off what little Javascript knowledge I acquired 6 years ago, I found the 13 lines of code that generates pseudonyms. Mechanics-wise it’s simple: there are two lists, one containing adjectives and one containing nouns. The pseudonym generator randomly pulls one adjective and one noun from each list, and mashes them together in that order, with a space in between. The generator outputs some sweet sounding pseudonyms like:

  • Serpentine Gyroscope
  • Moldy Parallelogram
  • Frumious Slide Rule
  • Supersonic Llama

But they can also come up with less memorable, more commonplace, and more boring phrases like:

  • Ice Snow
  • Warm Wind
  • Red Egg1
  • Infinite Avalanche

After running through a few example pseudonyms, anecdotally I felt the first list was more attractive to me than the second. It sparked more joy in me, one could say. I just couldn’t articulate why.

That’s when I noticed that certain themes kept recurring. For example, there were multiple Alice in Wonderland references, a bunch of animals, and many types of foods listed. At first glance the chosen words seemed odd. But after getting to know my co-workers better, the list of words began to make a lot more sense.

The co-worker sitting across from me is a huge Alice in Wonderland fan. Our founders seem to love animals, since they bring their dogs to work most days. Finally, food and restaurant discussions fuel most lunchtime arguments. Just in my first month, I had heard more discussion about chicken mole and Olive Garden than I ever had in my life.

While it’s true the pseudonym generator chooses words randomly, the choice of which words get onto the list isn’t necessarily random. If anything, the choice of words reflects the interests of the people who built the application. Might it be possible that the first list appealed to me because they reference math concepts, and I happen to like math-y things?

The hypothesis

This insight helped me craft my hypothesis more concretely: all else equal, do some candidates receive better ratings on interviews, because interviewers happen to associate positively with users whose pseudonyms reference the interviewers’ personal interests?

This hypothesis rests upon the assumption that people are drawn to stuff that’s similar to themselves. This seems intuitive: when individuals share common interests or backgrounds with others, chances are they’ll like each other. Therefore, is it possible that interviewers like certain candidates more because they find commonality with them, even though we manufactured that commonality? And did that likability translate to better interview ratings?

To test this, I categorized users into one of the following 6 categories based on the noun part of their pseudonym, which will be called Noun Category going forward.

  • Animal
  • Fantasy
  • Food
  • History
  • Object
  • Science

These broad categories aimed to differentiate among interest areas that might appeal differently to different interviewers. Among these 6 groups, I wanted to observe differences in interview performance. And knowing the pseudonym generator assigns names randomly, we would not expect to find a difference.

To proxy for interview performance, I used the “Would You Hire” response from the interviewer on the interviewee, which is the first item on the interviewer’s post-interview questionnaire.

An interviewer's rubric after an interview

An interviewer’s rubric after an interview

These two pieces of data led to a clear, testable null hypothesis: there should exist no relationship between Noun Category and the Would You Hire response. If we reject this null hypothesis, we would have evidence suggesting our pseudonyms can impact hiring decisions.

Data analysis and interpretation

I pulled data on a sample of a few thousand candidates’ first interview on our platform, and performed a Chi-Squared test against the observed frequencies of the 6 “Noun Categories” and 2 “Would You Hire” interviewer responses. Each cell of the 6 x 2 matrix contained at least 40 observations.

Below are the mean percentage of candidates who received a Yes from their interviewer, broken out by Noun Category. While most of the categories seemed to clump around a similar pass rate, the History group seemed to under-perform while the Fantasy group over-performed.

“Would You Hire” pass rate, by Noun Category

The Chi-Square test rejected the null hypothesis at a 5% significance level.

These results suggest a relationship might exists between Noun Category and an interviewer’s Would You Hire response. Which again, should not occur because a candidate’s Noun Category was randomly assigned!2

What next?

While this analysis doesn’t predict outcomes for specific individuals, the result suggests it isn’t totally crazy to believe I may gotten lucky on my interview. Maybe I don’t suffer from imposter syndrome, maybe I am an imposter. How depressing.

So what now? Fortunately (or unfortunately) for my new company, if we want to eliminate this bias, I can suggest potential next steps.

One solution might be to pander to an interviewer’s interests. We could randomly generate a new pseudonym for candidates every time they meet a different interviewer, ensuring that pseudonym creates positive associations with the interviewer. Similarly, we could generate more pseudonyms referencing Lord of the Rings and Warcraft, if we know our interviewer pool tends to be fantasy-inclined.

An alternative solution might be to give candidates pseudonyms with no meaning at all. For example, we could generate random strings, similar to what password managers generate for you. This would eliminate any real world associations, but we’d lose some whimsy and human readability that the current pseudonyms provide.

Yet another alternative solution could be to do more analysis before acting. The analysis didn’t quantify the magnitude of the bias, so we could construct a new sample to test a more specific hypothesis about bias size. It’s possible the practical impact of the bias isn’t huge, and we should focus our energy elsewhere.

Zooming out

On the face of it, this pseudonym bias seems trivial, and in the universe of all biases that could exist, that’s probably true. However, it makes me wonder how many other hidden biases might exist elsewhere in life.

I think that’s why I was hired. I’m obsessed with bias. Though I’ll be doing normal business-y Data Scientist stuff, my more interesting responsibilities will be poking at all aspects of the hiring market and examining the myriad of factors, mechanisms, and individuals that make the hiring market function, and perhaps not function effectively for some people.

Going a step further than identifying hiring biases, I’d like to shift discussions toward action. It’s great that the tech industry talks about diversity more, but I think we can facilitate more discussions around which concrete actions are being taken, and whether those actions actually achieve our goals, whatever those goals may be.

I think it all starts with being introspective about ourselves, and investigating whether something as innocuous as a randomly generated phrase could ever matter.

Atomic Artichoke
(Ken Pascual)

1This is the shortest pseudonym possible on

2This is not entirely true. Users can re-generate a random pseudonym as often as they want, meaning a user can choose their name if they re-generate a lot. However, there’s no evidence this happens often, because we found no significant difference in the observed and theoretical randomized distribution of Noun Categories.



There is a real connection between technical interview performance and salary. Here’s the data.

Posted on February 26th, 2019.

At the end of the day, money is a huge driver for the decisions we make about what jobs to go after. In the past, we’ve written about how to negotiate your salary, and there are a lot of labor statistics and reports out there looking at salaries in the tech industry as a whole. But as with many things in eng hiring, there’s very little concrete data on whether technical interview performance plays a role in compensation offers.

So we set out to gather the data and asked our users who had gone on to successfully get jobs after using our platform to share their salary info. With our unique dataset of real coding interviews, we could ask questions like:

  • Does interview performance matter when it comes to compensation packages?
  • Do engineers who prioritize other parts of a role over compensation (e.g. values alignment) end up with lower salaries?
  • What else seems to matter in getting a higher salary?

To be clear, this is an exploration of past average interview performance and its connection with current salary, versus looking at how someone did in an interview and then what salary they got when they took that specific job. In other words, we haven’t paired job interviews with the salary for that same job. We believe that looking at these more general measures is more informative than trying to match single interviews and job offers, given how volatile individual interview performance can be. But our interviewing platform allowed us to look at performance across multiple interviews for respondents, which gave us more stability and more data.

The setup

On the platform, people can practice technical interviews online and anonymously, with real engineers on the other side.

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question. Check out our recordings page to see this process in action.

Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from top companies like Google, Facebook, Dropbox, Airbnb, and more.

After every interview, interviewers rate interviewees on a few different dimensions: technical skills, communication skills, and problem solving skills. These each get rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!”. On our platform, a score of 3 or above has generally meant that the person was good enough to move forward. You can see what our feedback form looks like below:

With this in mind, we surveyed users about their current roles, including salary, bonuses, and how satisfied they felt in their job and then tied their comp back to how they did in interviews on our platform. We ended up with responses from 494 engineers1, and because compensation packages are so complex and vary from company to company, we analyzed the data in several different ways, looking at annual salary numbers, bonuses, and equity. Then we tied compensation data to performance in technical interviews to see whether it matters, and if so, how much.

The results

We looked at the relationships between interview performance (technical skills, communication ability, and problem solving ability) and the following: base salary, bonuses, and equity. In all cases, we corrected for location (being in the Bay Area means a higher salary) and experience (senior engineers make senior salaries), and where we could, we corrected for company size (bigger companies can generally pay bigger salaries).

The mean yearly salary for all survey participants was around $130k, and 57% of them reported a yearly bonus. For that group, the average yearly bonus was $20k. For people who reported a dollar amount for equity, the average was $54k. Below is the distribution of experience level/seniority of survey respondents.

Salary vs. Technical Interview Performance

Here’s what we found.

Better technical skills correlate with higher compensation

As probably comes as no surprise, people who score higher on technical skills during interviews do make more money. First, let’s look at base salary.2

Salary vs. Technical Interview Performance

Bonuses, too, correlate with technical skills, with an additional point in performance potentially worth about 10k:3

Bonus vs. Technical Interview Performance

The relationship between compensation and other interviewing skills

We also looked at the two other ratings that interviewers give after interviews: communication and problem solving. Better communication scores had a small but statistically significant correlation with salaries (r = .15, p < .01), but we found no significant relationship for problem solving scores in isolation:

Bonus vs. Technical Interview Performance

We also didn’t see a relationship between either communication ability or problem solving ability when it came to bonuses.

The non-relationships didn’t surprise us too much, to be honest, because with a relatively small sample size it’s notoriously difficult to get subcomponents of ratings to show a relationship to something distal and complicated like salaries. It’s very possible these relationships do exist, and with many determiners besides actual interview performance, like seniority and market salary norms, we’d like to repeat this survey at a bigger scale to inform this question.

What else?

We asked engineers whether they felt satisfied with their role, and found that engineers who felt satisfied earned an average $14k more than engineers who felt dissatisfied.4

We also looked at people’s perceptions of their own performance. In a previous post, we explored how people rated their own technical performance after an interview compared to how the interviewer rated them and found that even experienced engineers aren’t great at guessing how they did. For this project, we were curious about whether overconfident engineers might net higher salaries (perhaps they negotiate harder!). So we also looked at people who rated their performance higher than their actual interview score — but found no difference in their compensation packages.

Another thing we were curious about was whether people who valued money over other factors while making a job decision would have higher salaries. So we asked people to rank the most important variables in their job decisions. 32% of respondents said that a compensation package was the most important part of their decision; the next highest response was “matches my interests and values”. But these question didn’t have any predictive value for the actual salary amount: people who said money matters the most didn’t have significantly different salaries from people who say money matters the least. It’s possible that with salaries being impacted by so many outside factors, like location and role type, candidates don’t truly have a lot of negotiating power over that salary number.

We looked at equity as well, and the average reported equity package size was 54k. We did not find any significant association between interview performance and reported equity packages. That said, enormous amounts of research have documented various salary gaps based on gender, race, and other important sociocultural and demographic factors, and we hope to repeat this analysis when we have more data.

What do these findings mean for you?
Interview performance doesn’t just get you in the door or not: it can have a demonstrable connection to your eventual compensation. For instance, doing just a point better in your technical interview could be worth 10k or more, and with bonus, it could add 20k to your annual comp.

Given how much technical interview performance matters, we’d be remiss if we didn’t suggest signing up for free, anonymous mock interviews on our platform. So, please go do that.

And, if you’re curious about what our salary survey looked like or want to participate and contribute to v2 of this post, please do so too!

1Our salary survey ended up with 494 respondents, but because some people filled out our survey but had not yet done an interview on our platform, only a subset of folks had both salary data and interview data: N = 234, or 47% of our salary sample. Therefore in all the analyses where we compare salaries to interview performance, it’s only for this subgroup.

2We ran both a correlation between these factors and a regression to correct for confounding factors like seniority and location. For the correlation, r = .22 and p < 0.001. For the regression, F = 16.06 and p < .001.

3As with base salary above, we ran both a correlation between these factors and a regression to correct for confounding factors like seniority and location. For the correlation, r = .17 and p < 0.05. For the regression, F = 1.63 and p < .05.

4Looking at it as a binary, satisfied engineers earned significantly more than nonsatisfied engineers in a predictive test, F = 5.2398, p < 0.02, and satisfaction and salary amount are also positively correlated.



Impostor syndrome strikes men just as hard as women… and other findings from thousands of technical interviews

Posted on October 30th, 2018.

The modern technical interview is a rite of passage for software engineers and (hopefully!) the precursor to a great job. But it’s also a huge source of stress and endless questions for new candidates. Just searching “how do I prepare for a technical interview” turns up millions of Medium posts, coding bootcamp blogs, Quora discussions, and entire books.

Despite all this conversation, people struggle to know how they’re even doing in interviews. In a previous post, we found that a surprisingly large number of’s users consistently underestimate their performance, making them more likely to drop out of the process and ultimately harder to hire. Now, and with considerably more data (over 10k interviews led by real software engineers!), we wanted to go deeper: what seems to make candidates worse at gauging their own performance?

We know some general facts that make accuracy a challenge: people aren’t always great at assessing or even remembering their performance on difficult cognitive tasks like writing code.1 Technical interviews can be particularly hard to judge if candidates don’t have much experience with questions with no single right answer. Since many companies don’t share any kind of detailed post-interview feedback (beyond a yes/no) with candidates for liability reasons, many folks never get any sense of how they did, what they did well, or what could have been better.2, 3 Indeed, pulling back the curtain on interviewing, across the industry, was one of the primary motivators for building!

But to our knowledge there’s little data out there looking specifically at how people feel after real interviews on this scale, across different companies–so we gathered it, giving us the ability to test interesting industry assumptions about engineers and coding confidence.

One big factor we were interested in was impostor syndrome. Impostor syndrome resonates with a lot of engineers,4 indicating that many wonder whether they truly match up to colleagues and discount even strong evidence of competence as a fluke. Impostor syndrome can make us wonder whether we can count on the positive performance feedback that we’re getting, and how much our opportunities have come from our own effort, versus luck. Of particular interest to us was whether this would show up for women on our platform. There’s a lot of research evidence that candidates from underrepresented backgrounds experience a greater lack of belonging that feeds impostor syndrome,5 and this could show up as inaccuracy about judging your own interview performance.

The setup is a platform where people can practice technical interviewing anonymously, and if things go well, get jobs at top companies in the process. We started it because resumes uck and because we believe that anyone, regardless of how they look on paper, should have the opportunity to prove their mettle.

When an interviewer and an interviewee match on, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question (feel free to watch this process in action on our interview recordings page).  After each interview, people leave one another feedback, and each party can see what the other person said about them once they both submit their reviews.

Here’s an example of an interviewer feedback form:

Feedback form for interviewers

Feedback form for interviewers

Immediately after the interview, candidates answered a question about how well they thought they’d done on the same 1-4 scale:

Feedback form for interviewees

Feedback form for interviewees

For this post, we looked at over 10k technical interviews led by real software engineers from top companies. In each interview, a candidate was rated by an interviewer on their problem-solving ability, technical ability, and communication skills, as well as whether the interviewer would advance them to the next round. This gave us a measure of how different someone’s self-rating was from the rating that the interviewer actually gave them, and in which direction. In other words, how skewed was their estimation from their true performance?

Going in, we had some hunches about what might matter:

  • Gender. Would women be harder on their coding performance than men?
  • Having been an interviewer before. It seems reasonable that having been on the other side will pull back the curtain on interviews.
  • Being employed at a top company. Similar to above.
  • Being a top-performing interviewee on — people who are better interviewees overall might have more confidence and awareness of when they’ve gotten things right (or wrong!)
  • Being in the Bay Area or not. Since tech is still so geographically centered on the Bay Area, we considered that folks who live in a more engineering-saturated culture could have greater familiarity with professional norms around interviews.
  • Within the interview itself, question quality and interviewer quality. Presumably, a better interviewer is also a better communicator, whereas a confusing interviewer might throw off a candidates’ entire assessment of their performance. We also looked at whether it was a practice interview, or for a specific company role.
  • For some candidates, we could also look at few measures of their personal brand within the industry, like their number of GitHub and Twitter followers. Maybe people with a strong online presence are more sure of themselves when they interview?

So what did we find?

Women are just as accurate as men at assessing their technical ability

Contrary to expectations around gender and confidence, we didn’t find a reliable statistically significant gender difference in accuracy. At first, it looked like female candidates were more likely to underestimate their performance, but when we controlled for other variables, like experience and rated technical ability, it turned out the key differentiator was experience. More experienced engineers are more accurate about their interview performance, and men are more likely to be experienced engineers, but experienced female engineers are just as accurate about their technical ability.

Based on previous research, we hypothesized that impostor syndrome and a greater lack of belonging could result in female candidates penalizing their interview performance, but we didn’t find that pattern.6 However, our finding echoes a research project from the Stanford Clayman Institute for Gender Research, which looked at 1,795 mid-level tech workers from high tech companies. They found that women in tech aren’t necessarily less accurate when assessing their own abilities, but do have significantly different ideas about what success requires (e.g., long working hours and risk-taking). In other words, women in tech may not doubt their own abilities but might have different ideas about what’s expected. And a survey from Harvard Business Review  asking over a thousand professionals about their job application decisions also made this point. Their results emphasized that gender gaps in evaluation scenarios could be more about different expectations for how scenarios like interviews are judged.

That said, we did find one interesting difference: women went through fewer practice interviews overall than men did. The difference was small but statistically significant, and harkens back to our earlier finding that women leave roughly 7 times as often as men do, after a bad interview.

But in that same earlier post, we also found that masking voices didn’t impact interview outcomes. This whole cluster of findings affirms what we suspected and what the folks doing in-depth studies of gender in tech have found: it’s complicated. Women’s lack of persistence in interviews can’t be explained only by impostor syndrome about their own abilities, but it’s still likely that they’re interpreting negative feedback more severely and making different assumptions about interviews.

Here’s the distribution of accuracy distance for both female and male candidates on our platform (zero indicates a rating that matches the interviewer’s score, while negative values indicate underestimated score, and positive values indicate an overestimated score). The two groups look pretty much identical:

Accuracy by gender

What else didn’t matter?

Another surprise: having been an interviewer didn’t help. Even people who had been interviewers themselves don’t seem to get an accuracy boost from that. Personal brand was another non-finding. People with more GitHub followers weren’t more accurate than people with few to no GitHub followers. Nor did interviewer rating matter (i.e. how well an interviewer was reviewed by their candidates), although to be fair, interviewers are generally rated quite highly on the site.

So what was a statistically significant boost to accurate judgments of interview performance? Mostly, experience.

Experienced engineers have a better sense for how well they did in interviews, compared with engineers earlier in their careers.7 But it doesn’t seem to just be that you’re better at gauging your interview performance because you’re better at writing code; although there is a small lift from this, with higher rated engineers being more accurate. But when you look at junior engineers, even top-performing junior candidates struggled to accurately assess their performance.8  

experienced versus juniors

Our data mirrors a trend seen in Stack Overflow’s 2018 Developer survey. They asked respondents several questions about confidence and competition with other developers, and noted that more experienced engineers feel less competitive and more confident.9 This isn’t necessarily surprising: experience is correlated with skill level, after all, and highly skilled people are likely to be more confident. But our analysis let us control for performance and code skill within career groups, and we still found that experienced engineers were better at predicting their interview scores. There are probably multiple factors here: experienced engineers have been through more interviews, have led interviews themselves, and have a stronger sense of belonging, all of which may combat impostor syndrome.

Insider knowledge and context also seems to help: Being in the Bay Area and being at a top company both made people more accurate. Like the experienced career group, engineers who seem more likely to have contextual industry knowledge are also more accurate. We found small but statistically significant lifts from factors like being located in the Bay Area and working at a top company. However, the lift from working at a top company seems to mostly measure a lift from overall technical ability: being at a top company is essentially a proxy measure for being a more experienced, higher quality engineer.

Finally, as you get better at interviewing and move into company interviews, you do get more accurate. People were more accurate about their performance in company interviews compared to practice interviews, and their overall ranking on the site also predicted improved accuracy: also gives users an overall ranking, based on their performance over multiple interviews and weighted toward more recent measures. People who scored in the top 25% were more likely to be accurate about their interview performance.

In general, how are people at gauging their interview performance overall? We’ve looked at this before, with roughly a thousand interviews, and now, with ten thousand, the finding continues to hold up. Candidates were accurate about how they did in only 46% of interviews, and underestimated themselves in 35% of interviews (and the remaining 19%, of course, are the overestimators). Still, candidates are generally on the right track — it’s not like people who score a 4 are always giving themselves a 1.10 Self-ratings are statistically significantly predictive for actual interview scores (and positively correlated), but that relationship is noisy.

The implications

Accurately judging your own interview performance is a skill in its own right and one that engineers need to learn from experience and context in the tech industry. But we’ve also learned that many of the assumptions we made about performance accuracy didn’t hold up to scrutiny — female engineers had just as accurate a view of their own skills as male ones, and engineers who had led more interviews or were well known on GitHub weren’t particularly better at gauging their performance.

What does this mean for the industry as a whole? First off, impostor syndrome appears to be the bleary-eyed monster that attacks across gender ability, and how good you are, or where you are, or how famous you are isn’t that important. Seniority does help mitigate some of the pain, but impostor syndrome affects everyone, regardless of who they are or where they’re from. So, maybe it’s time for a kinder, more empathetic interviewing culture. And a culture that’s kinder to everyone, because though marginalized groups who haven’t been socialized in technical interviewing are hit the hardest by shortcomings in the interview process, no one is immune to self-doubt.

We’ve previously discussed what makes someone a good interviewer, and empathy plays a disproportionately large role. And we’ve seen that providing immediate post-interview feedback is really important for keeping candidates from dropping out. So, whether you’re motivated by kindness and ideology or cold, hard pragmatism, a bit more kindness and understanding toward your candidates is in order.

Cat Hicks, the author of this guest post, is a researcher and data scientist with a focus on learning. She’s published empirical research on learning environments, and led research on the cognitive work of engineering teams at Google and She holds a PhD in Psychology from UC San Diego.

1Self-assessment has been explored in a number of domains, and often used to measure learning. One important criticism is that it’s highly impacted by people’s motivation and emotional state at the time of asking. See: Sitzmann, T., Ely, K., Brown, K. G., & Bauer, K. N. (2010). Self-assessment of knowledge: A cognitive learning or affective measure?. Academy of Management Learning & Education, 9(2), 169-191.

2Designing a good technical interview is no small task on the interviewer side. For an informal discussion of this, see this post.

3For some anecdotal conversation about interview self-assessment, see this one

4E.g., this article and this one.

5Some examples of further reading in social science research:
Good, C., Rattan, A., & Dweck, C. S. (2012). Why do women opt out? Sense of belonging and women’s representation in mathematics. Journal of personality and social psychology, 102(4), 700.
Master, A., Cheryan, S., & Meltzoff, A. N. (2016). Computing whether she belongs: Stereotypes undermine girls’ interest and sense of belonging in computer science. Journal of Educational Psychology, 108(3), 424.

6One complication for our dataset is the representation of experienced female engineers: we simply didn’t have very many, which is true to the demographics of the tech industry, but also means that selection biases in the small group of experienced female engineers we do have are more likely to be present, and this isn’t the be-all and end-all of exploring for group differences. We’d like to continue looking at interviews with female participants to explore this fully.

7These effects and the previous non-findings were all explored in a linear mixed model. Significant results for the individual effects are all p<.05

8Experienced engineers have an average skew of -.14; Junior engineers have an average skew of -.22, New Grads have an average skew of -.25.

9See also:

10Another wrinkle with the design behind this data is that there’s a floor and a ceiling on the scale: people who always score a 4, for example, can’t ever overrate themselves, because they’re already at the top of the scale. We dealt with this a couple of ways: by excluding people at the floor and ceiling and re-running analyses on the middle subset, and by binning skew into either accurate or not and looking at that. The findings hold up across this.



Exactly what to say when recruiters ask you to name the first number… and other negotiation word-for-words

Posted on August 16th, 2018.

There are a lot of resources out there that talk about salary negotiation but many tend to skew a bit theoretical. In my experience, one of the hardest things about negotiating your salary is knowing what to say in tough, ambiguous situations with a power balance that’s not in your favor. What’s OK? What’s rude? What are the social norms? And so on.

Before I started, I’ve worked as a software engineer, an in-house recruiter, and an agency recruiter, so I’ve literally been on all sides of the negotiating table. For the last few years, I’ve been guest-lecturing MIT’s 6.UAT, a class about technical communication for computer science majors. Every semester, negotiation is one of the most-requested topics from students. In this post, I’m sharing the content of that lecture, which is targeted toward students, but has served seasoned industry folks just as well. You’re never too young or too old to advocate for yourself.

Btw, if you don’t like reading and prefer long, rambly diatribes in front of an unsightly glass wall, I covered most of this material (and other stuff) in a webinar I did with the fine people at Udacity (where I used to run hiring) a few months ago. So, pick your poison.

Why negotiate at all, especially if I’m junior?

If you’re early in your career, you might say that negotiation isn’t worth the hassle — after all, junior roles have pretty narrow salary bands. There are a few reasons this view is short-sighted and wrong. First, though it’s pretty unlikely in the grand scheme of things, if you’re applying to a startup, there might come a magical day when your equity is worth something. This is especially true if you’re an early employee — with a good exit, a delta of a few tenths of a percent might end up being worth a down payment on a home in San Francisco.

But, let’s get real, your equity is likely worthless (except’s equity… that’s totes gonna be worth something), so let me give you a better, more immediate reason to learn to haggle early in your career, precisely because that’s when the stakes are low. Humans are frighteningly adaptable creatures. Scared of public speaking? Give 3 talks. The first one will be gut-wrenchingly horrific, the stuff of nightmares. Your voice will crack, you’ll mumble, and the whole time, you’ll want to vomit. The next one will be nerve-wracking. The last one will mostly be OK. And after that, you’ll be just fine. Same thing applies to approach anxiety, mathematical proofs, sex, and, you guessed it, salary negotiation!

So, make all the awkward, teeth-cringing mistakes now, while it doesn’t matter, and where failure will cost you $5K or $10K a year. Because the further along you get in your career, the bigger the upside will be… and the bigger the downside will be for not negotiating. Not only will the salary bands be wider for senior roles, but as you get more senior, more of your comp comes from equity, and equity has an even wider range for negotiating. Negotiating your stock well can make 6-figure differences and beyond (especially if you apply some of these same skills to negotiating with investors over term sheets, should you ever start your own company)… so learn these skills (and fail) while you’re young, because the older you get, the harder it’s going to be to start and the more high-stakes it’s going to be.

So, below, as promised, I’ll give you a few archetypal, stress-inducing situations and what to say, word-for-word in each one. But first, let me address the elephant in the room…

Will my offer be rescinded if I try to negotiate?

As I mentioned earlier, this blog post is coming out of a lecture I give at MIT. Every semester, I start the negotiation portion of the lecture with the unshakeable refrain that no one will ever rescind your offer for negotiating. Last semester was different, though. I was just starting to feel my oats and get into my talk (the negotiation piece comes about halfway through) and smugly recited the bit about offers never being rescinded, followed by my usual caveat… “unless you act like a douche while negotiating.” Then, a hand shot up in the back of the room. Ah ha, I thought to myself, one of the non-believers. Ready to placate him, I called on the gentleman in the back.

“My offer got rescinded for negotiation.”

The class broke out into uproarious laughter. I laughed too. It was kind of funny… but it was also unnerving, and I wanted to get to the bottom of it.

“Were you a giant jerk when you negotiated?”

“Nope.” Shit, OK, what else can I come up with…

“Were you applying at a really small company with maybe one open role?” I asked, praying against hope that he’d say yes.


“Thank god.”

So, there’s the one exception I’ve found so far to my blanket statement. After working with hundreds and hundreds of candidates back when I was still a recruiter, I had never heard or seen an offer get rescinded (and none of my candidates acted like douches while negotiating, thank god), until then. So, if you’re talking to a super small company with one role that closes as soon as they find someone, yes, then they might rescind the offer.

But, to be honest, and I’m not just saying this because I was wrong in front of hundreds of bloodthirsty undergrads, an early startup punishing a prospective employee for being entrepreneurial is a huge red flag to me.

OK, so, now onto the bit where I tell you exactly what to say.1

What to say when asked to name the first number

There will come a time in every job search where a recruiter will ask you about your compensation expectations. This will likely happen very early in said search, maybe even during the first call you’ll ever have with the company.

I think this is a heinous, illaudable practice fraught with value asymmetry. Companies know their salary ranges and roughly what you’re worth to them before they ever talk to you (barring phenomenal performance in interviews which kicks you into a different band). And they know what cost of living is in your area. So they already have all the info they need about you, while you have none about them or the role or even your market value. Sure, there are some extenuating circumstances where you are too expensive, e.g. you’re like an L6 at Google and are talking to an early stage startup that can only afford to pay you 100K a year in base, but honestly even in that situation, if the job is cool enough and if you have the savings, you might take it anyway.

So, basically, telling them something will only hurt you and never help you. So don’t do it. Now, here’s exactly what to say when asked to name the first number.

At this point, I don’t feel equipped to throw out a number because I’d like to find out more about the opportunity first – right now, I simply don’t have the data to be able to say something concrete. If you end up making me an offer, I would be more than happy to iterate on it if needed and figure out something that works. I also promise not to accept other offers until I have a chance to discuss them with you.


What to say when you’re handed an exploding offer

Exploding offers, in my book, are the last bastion of the incompetent. The idea goes something like this… if we give a candidate an aggressive deadline, they’ll have less of a chance to talk to other companies. Game theory for the insipid.

Having been on the other side of the table, I know just how arbitrary offer deadlines often are. Deadlines make sense when there is a limited number of positions and applicants all come in at the same time (e.g. internships). They do not make any sense in this market, where companies are perpetually hiring all the time — therefore it’s an entirely artificial construct. Joel Spolsky, the creator of Trello and Stack Overflow, had something particularly biting to say on the matter of exploding offers many years ago (the full post, Exploding Offer Season, is really good):

“Here’s what you’re thinking. You’re thinking, well, that’s a good company, not my first choice, but still a good offer, and I’d hate to lose this opportunity. And you don’t know for sure if your number one choice would even hire you. So you accept the offer at your second-choice company and never go to any other interviews. And now, you lost out. You’re going to spend several years of your life in some cold dark cubicle with a crazy boss who couldn’t program a twenty out of an ATM, while some recruiter somewhere gets a $1000 bonus because she was better at negotiating than you were.”

Even in the case of internships, offer deadlines need not be as aggressive as they often are, and I’m happy to report that many college career centers have taken stands against exploding offers. Nevertheless, if you’re not a student or if your school hasn’t outlawed this vile practice, here’s exactly what to say if it ever happens to you.

I would very much appreciate having a bit more time. I’m very excited about Company X. At the same time, choosing where I work is extremely important to me. Of course, I will not drag things out, and I will continue to keep you in the loop, but I hope you can understand my desire to make as informed of a decision as possible. How about I make a decision by…?

The reverse used car salesman… or what to say to always get more

At the end of the day, the best way to get more money is to have other offers. I know, I know, interviewing sucks and is a giant gauntlet-slog, but in many cases, having just one other offer (so, I don’t know, spending a few extra days of your time spread over a few weeks) can get you at least $10K extra. It’s a pretty rational, clear-cut argument for biting the slog-bullet and doing a few more interviews.

One anecdote I’ll share on the subject goes like this. A few years ago, a close friend of mine who’s notoriously bad at negotiation and hates it with a passion was interviewing at one of the big 4 companies. I was trying to talk to him into getting out there just a little bit, for the love of god, and talk to at least one more company. I ended up introducing him to a mid-sized startup where he quickly got an onsite interview. Just mentioning that he had an onsite at this company to his recruiter from the bigco got him an extra $5K in his signing bonus.

Offers are, of course, better than onsites, but in a pinch, even onsites will do… because every onsite increases your odds of not accepting the offer from the company you’re negotiating with. So, let’s say you do have some offers. Do you reveal the details?

The answer is that it depends. If the cash parts of the offers you have are worth more than the one you have in hand, then you can reveal the details. If they’re worth more in total but less in cash, it’s a bit dicier because equity at smaller companies is kind of worthless… you can still use it as leverage if you tell the story that that equity is worth more to YOU, but that’s going to take a bit more finesse, so if you’ve never negotiated before, you might want to hold off.

If the cash part of your equity is not worth more, it’s sufficient to say you have offers and when pressed, you can simply say that you’re not sharing the details (it’s ok not to share the details).

But whether you reveal details or not, here’s the basic formula for getting more. See why I call it the reverse used car salesman?

I have the following onsites/offers, and I’m still interviewing at Company X and Company Y, but I’m really excited about this opportunity and will drop my other stuff and SIGN TODAY if…

So, “if” what? I propose listing 3 things you want, which will typically be:

  • Equity
  • Salary
  • Signing/relocation bonus

The reason I list 3 things above isn’t because I expect you’ll be able to get all 3, but this way, you’re giving the person you’re negotiating with some options. In my experience, you’ll likely get 2 out of the 3.

So, what amounts should you ask for when executing on the reverse used car salesman? It’s usually easier to get equity and bonuses than salary (taxed differently from the company’s perspective, signing bonus is a one-off rather than something that repeats every year). Therefore, it’s not crazy to ask for 1.5X-2X the equity and an extra 10-15% in salary. For the bonus portion, a lot depends on the size of the company, but if you’re talking to a company that’s beyond seed stage, you can safely ask for at least 20% of your base salary as a signing bonus.2

What if the company says no to all or most of these and are a big enough brand to where you don’t have much of a leg to stand on? You can still get creative. One of our users told me about a sweet deal he came up with — he said he’d sign today if he got to choose the team he could join and had a specific team in mind.

Other resources

As I mentioned at the beginning of this post, there are plenty of blog posts and resources on the internets about negotiation, so I’ll just mention two of my favorites. The first is a riveting, first-hand account of negotiation adventures from one of my favorite writers in this space, Haseeb Qureshi. In his post, Haseeb talks about how he negotiated for a 250K (total package) offer with Airbnb and what he learned along the way. It’s one of the most honest and thoughtful accounts of the negotiation process I’ve ever read.

The second post I’ll recommend is a seminal work in salary negotiation by Patrick McKenzie (patio11 on Hacker News, in case that’s more recognizable). I read it back when I was still an engineer, and it was one of those things that indelibly changed how I looked at the world. I still madly link anyone and everyone who asks me about negotiation to this piece of writing, and it’s still bookmarked in my browser.

If you’re an user and have a job offer or five that you’re weighing and want to know exactly what to say when negotiating in your own nuanced, unique situation, please email me, and I’ll whisper sweet, fiscal nothings in your ear like a modern-day Cyrano de Bergerac wooing the sweet mistress that is capitalism.3

1If you’re interviewing at, USE THESE ON ME. IT’LL BE GREAT. And while you’re at it, use these on me as well.

2Some of the larger tech companies offer huge signing bonuses to new grads (~100K-ish). Obviously this advice is not for that situation.

3An increasing number of our customers pay us on subscription, so we don’t get more money if you do.4 And for the ones who don’t, salary and recruiting fees typically come out of a different budget.

4In the early days of, we tried to charge a flat per-hire fee in lieu of a percentage of salary, precisely for this reason — we wanted to set ourselves up as an entirely impartial platform where lining up with our candidates’ best interests was codified into our incentive structure. Companies were pretty weirded out by the flat fee, so we went back to doing percentages, but these days we’re moving over as many of our customers to subscription as possible — it’s cheaper for them, better for candidates, and I won’t lie that I like to see that recurring revenue.



We looked at how a thousand college students performed in technical interviews to see if where they went to school mattered. It didn’t.

Posted on February 13th, 2018. is a platform where engineers practice technical interviewing anonymously. If things go well, they can unlock the ability to participate in real, still anonymous, interviews with top companies like Twitch, Lyft and more. Earlier this year, we launched an offering specifically for university students, with the intent of helping level the playing field right at the start of people’s careers. The sad truth is that with the state of college recruiting today, if you don’t attend one of very few top schools, your chances of interacting with companies on campus are slim. It’s not fair, and it sucks, but university recruiting is still dominated by career fairs. Companies pragmatically choose to visit the same few schools every year, and despite the career fair being one of the most antiquated, biased forms of recruiting that there is, the format persists, likely due to the fact that there doesn’t seem to be a better way to quickly connect with students at scale. So, despite the increasingly loud conversation about diversity, campus recruiting marches on, and companies keep doing the same thing expecting different results.

In a previous blog post, we explained why companies should stop courting students from the same five schools. Regardless of your opinion on how important that idea is (for altruistic reasons, perhaps), you may have been left skeptical about the value and practicality of broadening the college recruiting effort, and you probably concede that it’s rational to visit top schools, given limited resources — while society is often willing to agree that there are perfectly qualified students coming out of non-top colleges, they maintain that they’re relatively rare. We’re here to show you, with some nifty data from our university platform, that this not true.

To be fair, this isn’t the first time we’ve looked at whether where you went to school matters. In a previous post, we found that taking Udacity and Coursera programming classes mattered way more than where you went to school. And way back when, one of our founders figured out that where you went to school didn’t matter at all but that the number of typos and grammatical errors on your resume did. So, what’s different this time? The big, exciting thing is that these prior analyses were focused mostly on engineers who had been working for at least a few years already, making it possible to argue that a few years of work experience smoothes out any performance disparity that comes from having attended (or not attended a top school). In fact, the good people at Google found that while GPA didn’t really matter after a few years of work, it did matter for college students. So, we wanted to face this question head-on and look specifically at college juniors and seniors while they’re still in school. Even more pragmatically, we wanted to see if companies limiting their hiring efforts to just top schools means they’re going to get a higher caliber of candidate.

Before delving into the numbers, here’s a quick rundown of how our university platform works and the data we collect.

The setup

For students who want to practice on, the first step is a brief (~15-minute) coding assessment on Qualified to test basic programming competency. Students who pass this assessment, i.e. those who are ready to code while another human being breathes down their neck, get to start booking practice interviews.

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question. Check out our recordings page to see this process in action.

Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from top companies like Google, Facebook, Dropbox, Airbnb, and more.

After every interview, interviewers rate interviewees on a few different dimensions, including technical ability. Technical ability gets rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!”. On our platform, a score of 3 or above has generally meant that the person was good enough to move forward. You can see what our feedback form looks like below:


On our platform, we’re fortunate to have thousands of students from all over the U.S., spanning over 200 universities. We thought this presented a unique opportunity to look at the relationship between school tier and interview performance for both juniors (interns) and seniors (new grads). To study this relationship, we first split schools into the following four tiers, based on rankings from U.S. News & World Report:

  • “Elite” schools (e.g. MIT, Stanford, Carnegie Mellon, UC-Berkeley)
  • Top 15 schools (not including top tier, e.g. University of Wisconsin, Cornell, Columbia)
  • Top 50 schools (not including top 15, e.g. Ohio State University, NYU, Arizona State University)
  • The rest (e.g. Michigan State, Vanderbilt University, Northeastern University, UC-Santa Barbara)

Then, we ran some statistical significance testing on interview scores vs. school tier to see if school tier mattered, for both interns (college juniors) and new grads (college seniors), comprising a set of roughly 1000 students.

Does school have anything to do with interview performance?

In the graphs below, you can see technical score distributions for interviews with students in each of the four school tiers (see legend). As you recall from above, each interview is scored on a scale of 1 to 4, where 1 is the worst and 4 is the best.

First, the college juniors…

interns by tier

And then, the seniors…

New grads by tier

What’s pretty startling is that the shape of these distributions, for both juniors and seniors, is remarkably similar. Indeed, statistical significance testing revealed no difference between students of any tier when it came to interview performance.1 What this means is that top-tier students are achieving the same results as those in no-name schools. So the question becomes: if the students are comparable in skill, why are companies spending egregious amounts of money attracting only a subset of them?

Okay, so what are companies missing?

Besides missing out on great, cheaper-to-acquire future employees, companies are missing out on an opportunity to save time and money. Right now a ridiculous amount of money is being spent on university recruiting. We’ve previously cited the $18k price tag just for entry to the MIT career fair. In a study done by Lauren Rivera through the Harvard Business Review, she reveals that one firm budgeted nearly $1m just for social recruiting events on a single campus.

The higher price tag of these events also means it makes even less sense for smaller companies or startups to try and compete with high-profile, high-profit tech giants. Most of the top schools that are being heavily pursued already have enough recruiters vying for their students. Unwittingly, this pursuit seems to run contrary to most companies desires for high diversity and long-term sustainable growth.

Even when companies do believe talent is evenly distributed across school tiers, there are still reasons for why companies might recruit at top schools. There are other factors that help elevate certain schools in a recruiter’s mind. There are long-standing company-school relationships (for example, the number of alumni who work at the company currently). There are signaling effects too — companies get Silicon Valley bonus points by saying their eng team is comprised of a bunch of ex-Stanford, ex-MIT, ex- etc. etc. students.

A quick word about selection bias

Since this post appeared on Hacker News, there’s been some loud, legitimate discussion about how the pool of students on may not be representative of the population at large because we have a self-selected pool of students who decided to practice interviewing. Certainly, all the blog posts we publish are subject to this (very valid) line of criticism, and for this post in particular. As such, selection bias in our user pool might mean that 1) we’re getting only the worst students from top schools (because, presumably, the best ones don’t need the practice), or 2) we’re getting only the best/most motivated students for non-top schools, or both. Any subset of these is entirely possible, but we have a few reasons why we believe what we’ve published here might hold true regardless.

First off, in our experience, regardless of their background or pedigree, everyone is scared of technical interviewing. Case in point… before we started working on, we didn’t really have a product yet. So before investing a lot of time and heartache into this questionable undertaking, we wanted to test the waters to see if interview practice was something engineers really wanted, and more so, who these engineers that wanted practice were. So, we put up a pretty mediocre landing page on Hacker News… and got something like 7,000 signups the first day. Of these 7,000 signups, roughly 25% were senior (4+ years of experience) engineers from companies like Google and Facebook (this isn’t to say that they’re necessarily the best engineers out there… but just that the engineers the market seems to value the most still need our service).

Another data point comes from one of our founders. Every year, Aline does a guest lecture on job search preparedness for a technical communication course at MIT. This course is one way to fulfill the computer science major communication requirement, so enrollment tends to span the gamut of computer science students. Before every lecture, she sends out a survey asking students what their biggest pain points are in preparing for their job search. Every year, trepidation about technical interviewing is either at the top of the list of 2nd from the top.

And though this doesn’t directly address the issue of whether we’re only getting the best of the worst or the worst of the best (I hope the above has convinced you there’s more to it than that), here’s the distribution of school tiers among our users, which I expect mirrors the kinds of distributions companies see in their student applicant pool:

New grads by tier

So what can companies do?

As such, companies may never stop recruiting at top-tier schools entirely, but they ought to at least include schools outside of that very small circle in the search for future employees. The end result of the data is the same: for good engineers, school means a lot less than we think. The time and money that companies put in to compete for candidates within the same select few schools would be better spent creating opportunities that include everyone, as well as developing tools to vet students more fairly and efficiently.

As you saw above, we used a 15-minute coding assessment to cull our inbound student flow, and just a short challenge leveled the playing field between students from all walks of life. At the very least, we’d recommend employers do the same thing in their process. But, of course, we’d be remiss if we didn’t suggest one other thing.

At, we’ve proudly built a platform that grants the best-performing students access to top employers, no matter where they went to school or where they come from. Our university program, in particular, allows us to grant companies the privilege to reach an exponentially larger pool of students, for the same cost of attending one or two career fairs at top target schools. Want diverse, top talent without the chase? Sign up to be an employer on our university platform!

1Of course, this hinges on everyone completing a quick 15-minute coding challenge first, to ensure they’re ready for synchronous technical interviews. We’re excited about this because companies can replicate this step in their process as well!



What do the best interviewers have in common? We looked at thousands of real interviews to find out.

Posted on November 29th, 2017.

At, we’ve analyzed and written at some depth about what makes for a good interview from the perspective of an interviewee. However, despite the inherent power imbalance, interviewing is a two-way street. I wrote a while ago about how, in this market, recruiting isn’t about vetting as much as it is about selling, and not engaging candidates in the course of talking to them for an hour is a woefully missed opportunity. But, just like solving interview questions is a learned skill that takes time and practice, so, too, is the other side of the table. Being a good interviewer takes time and effort and a fundamental willingness to get out of autopilot and engage meaningfully with the other person.

Of course, everyone and their uncle has strong opinions about what makes someone a good interviewer, so instead of waxing philosophical, we’ll present some data and focus on analytically answering questions like… Does it matter how strong of an engineering brand your company has, for instance? Do the questions you ask actually help get candidates excited? How important is it to give good hints to your candidate? How much should you talk about yourself? And is it true that, at the end of the day, what you say is way less important than how you make people feel?1 And so on.

Before I delve into our findings, I’ll say a few words about and the data we collect.

The setup is an anonymous technical interviewing platform. On, people can practice technical interviewing anonymously, and if things go well, unlock real (still anonymous) interviews with companies like Lyft, Twitch, Quora, and more.

The cool thing is that both practice and real interviews with companies take place within the ecosystem. As a result, we’re able to collect quite a bit of interview data and analyze it to better understand technical interviewing. One of the most important pieces of data we collect is feedback from both the interviewer and interviewee about how they thought the interview went and what they thought of each other. If you’re curious, you can watch a real interview on our recordings page, and see what the feedback forms for interviewers and interviewees look like below — in addition to one direct yes/no question, we also ask about a few different aspects of interview performance using a 1-4 scale. We also ask interviewees some extra questions that we don’t share with their interviewers, one of which is their own take on how they thought they did.

Feedback form for interviewers

Feedback form for interviewers

Feedback form for interviewees

Feedback form for interviewees

In this post, we’ll be analyzing feedback and outcomes of thousands of real interviews with companies to figure out what traits the best interviewers have in common.

Before we get into the nitty-gritty of individual interviewer behaviors, let’s first put the value of a good interviewer in context by looking at the impact of a company’s brand on the outcome. After all, if brand matters a lot, then maybe being a good interviewer isn’t as important as we might think.

Brand strength

So, does brand really matter for interview outcomes? One quick caveat before we get into the data: every interview on the platform is user-initiated. In other words, once you unlock our jobs portal (you have to do really well in practice interviews to do so), you decide who you talk to. So, candidates talking to companies on our platform will be predisposed to move forward because they’ve chosen the company in the first place. And, as should come as no surprise to anyone, companies with a very strong brand have an easier time pulling candidates (on our platform and out in the world at large) than their lesser-known counterparts. Moreover, many of the companies we work with do have a pretty strong brand, so our pool isn’t representative of the entire branding landscape. However, all is not lost — in addition to working with very recognizable brands, we work with a number of small, up-and-coming startups, so we hope that if you, the reader, are coming from a company that’s doing cool stuff but that hasn’t yet become a household name, our findings likely apply to you. And, as you’ll see, getting candidates in the door isn’t the same as keeping them.

To try to quantify brand strength, we used three different measures: the company’s Klout Score (yes, that still exists), its Mattermark Mindshare Score, and its score on Glassdoor (under general reviews).2

When we looked at interview outcomes relative to brand strength, its impact was not statistically significant. In other words, we found that brand strength didn’t matter at all when it came to either whether the candidate wanted to move forward or how excited the candidate was to work at the company.

This was a bit surprising, so I decided to dig deeper. Maybe brand strength doesn’t matter overall but matters when the interviewer or the questions they asked aren’t highly rated? In other words, can brand buttress less-than-stellar interviewers? Not so, according to our data. Brand didn’t matter even when you corrected for interviewer quality. In fact, of the top 10 best-rated companies on our platform, half have no brand to speak of, 3 are mid-sized YC companies that command respect in Bay Area circles but are definitely not universally recognizable, and only 2 have anything approaching household name status.

So, what’s the takeaway here? Maybe the most realistic thing we can say is that while brand likely matters a lot for getting candidates in the door, once they’re in, no matter how well-branded you are, they’re yours to lose.

Choosing the question

If brand doesn’t matter once you’ve actually gotten a candidate in the door, then what does? Turns out, the questions you ask matter a TON. As you recall, feedback on is symmetric, which means that in addition to the interviewer rating the candidate, the candidate also rates the interviewer, and one of the things we ask candidates is how good the question(s) they got asked were.

Question quality was extremely significant (p < 0.002 with an effect size of 1.25) when it came to whether the candidate wanted to move forward with the company. This held both when candidates did well and when they did poorly.

While we obviously can’t share the best questions (these are company interviews, after all), we can look at what candidates had to say about the best and worst-rated questions on the platform.

The good

I liked the fact that questions were building on top of each other so that previous work was not wasted and
finding ways to improve on the given solution.

Always nice to get questions that are more than just plain algorithms.

Really good asking of a classic question, opened my mind up to edge cases and considerations that I never contemplated the couple of times I’ve been exposed to the internals of this data structure.

This was the longest interview I have ever done, and it is also the most enjoyable one! I really like how we started with a simple data structure and implemented algorithms on top of it. It felt like working on a simple small-scale project and was fun.

He chose an interesting and challenging interview problem that made me feel like I was learning while I was solving it. I can’t think of any improvements. He would be great to work with.

I liked the question — it takes a relatively simple algorithms problem (build and traverse a tree) and adds some depth. I also liked that the interviewer connected the problem to a real product at [Redacted] which made it feel like less like a toy problem and more like a pared-down version of a real problem.

This is my favorite question that I’ve encountered on this site. it was one of the only ones that seem like it had actual real-life applicability and was drawn from a real (or potentially real) business challenge. And it also nicely wove in challenges like complexity, efficiency, and blocking.

The bad

Question wasn’t straightforward and it required a lot of thinking/understanding since functions/data structures weren’t defined until a lot later. [Redacted] is definitely a cool company to work for, but some form of structure in interviews would have been a lot more helpful. Spent a long time figuring out what the question is even asking, and interviewer was not language-agnostic.

I was expecting a more technical/design question that showcases the ability to think about a problem. Having a domain-specific question (regex) limits the ability to show one’s problem-solving skills. I am sure with enough research one could come up with a beautiful regex expression but unless this is something one does often, I don’t think it [makes for] a very good assessment.

This is not a good general interview question. A good interview question should have more than one solution with simplified constraints.

Anatomy of a good interview question

  1. Layer complexity (including asking a warmup)
  2. No trivia
  3. Real-world components/relevance to the work the company is doing are preferable to textbook algorithmic problems
  4. If you’re asking a classic algorithmic question, that’s ok, but you ought to bring some nuance and depth to the table, and if you can teach the interviewee something interesting in the process, even better!

Asking the question

One of the other things we ask candidates after their interviews is how helpful their interviewer was in guiding them to the solution. Providing your candidate with well-timed hints that get them out of the weeds without giving away too much is a delicate art that takes a lot of practice (and a lot of repetition), but how much does it matter?

As it turns out, being able to do this well matters a ton. Being good at providing hints was extremely significant (p < 0.00001 with an effect size of 2.95) when it came to whether the candidate wanted to move forward with the company (as before, we corrected for whether the interview went well).

You can see for yourself what candidates thought of their interviewers when it came to their helpfulness and engagement below. Though this attribute is a bit harder to quantify, it seems that hint quality is actually a specific instance of something bigger, namely the notion of turning something inherently adversarial into a collaborative exercise that leaves both people in a better place than where they started.3

And if you can’t do that every time, then at the very least, be present and engaged during the interview. And no matter what the devil on your shoulder tells you, no good will ever come of opening Reddit in another tab.4

One of the most memorable, pithy conversations I ever had about interviewing was with a seasoned engineer who had spent years as a very senior software architect at a huge tech company before going back to what he’d always liked in the first place, writing code. He’d conducted a lot of interviews over a career spanning several decades, and after trying out a number of different interview styles, what he settled on was elegant, simple, and satisfying. According to him, the purpose of any interview is to “see if we can be smart together.” I like that so much, and it’s advice I repeat whenever anyone will listen.

The good

I liked that you laid out the structure of the interview at the outset and mentioned that the first question did not have any tricks. That helped set the pace of the interview so I didn’t spend an inordinate amount of time on the first one.

The interview wasn’t easy, but it was really fun. It felt more like making a design discussion with a colleague than an interview. I think the question was designed/prepared to fill the 45 minute slot perfectly.

I’m impressed by how quickly he identified the issue (typo) in my hash computation code and how gently he led me to locating it myself with two very high-level hints (“what other tests cases would you try?” and “would your code always work if you look for the the pattern that’s just there at the beginning of the string?”). Great job!

He never corrected me, instead asked questions and for me to elaborate in areas where I was incorrect – I very much appreciate this.

The question seemed very overwhelming at first but the interviewer was good at helping to break it down into smaller problems and suggest we focus on one of those first.

The bad

[It] was a little nerve-wracking hearing you yawn while I was coding.

What I found much more difficult about this interview was the lack of back and forth as I went along, even if it was simple affirmation that “yes, that code you just wrote looks good”. There were times when it seemed like I was the only one who had talked in the past five minutes (I’m sure that’s an exaggeration). This made it feel much more like a performance than like a collaboration, and my heart was racing at the end as a result.

While the question was very straightforward, and [he] was likely looking for me to blow through it with no prompting whatsoever in order to consider moving forward in an interview process, it would have been helpful to get a discussion or even mild hinting from him when I was obviously stuck thinking about an approach to solve the the problem. While I did get to the answer in the end, having a conversation about it would have made it feel more like a journey and learning experience. That would have also been a strong demonstration of the collaborative culture that exists while working with teams of people at a tech company, and would have sold me more vis-a-vis my excitement level.

If an interview is set to 45 minutes, the questions should fit this time frame, because people plan accordingly. I think that if you plan to have a longer interview you should notify the interviewee beforehand, so he can be ready for it.

One issue I had with the question though is what exactly he was trying to evaluate from me with the question. At points we talking about very nitty-gritty details about python linked list or array iteration, but it was unclear at any point if that was what he was judging me on. I think in the future he could outline at the beginning what exactly he was looking for with the problem in order to keep the conversation focused and ensure he is well calibrated judging candidates.

Try to be more familiar with all the possible solutions to the problem you choose to pose to the candidate. Try to work on communicating more clearly with the candidate.

Anatomy of a good interview

  1. Set expectations, and control timing/pacing
  2. Be engaged!
  3. Familiarity with the problem and its associated rabbit holes/garden paths
  4. Good balance of hints and letting candidate think
  5. Turn the interview into a collaborative exercise where both people are free to be smart together

The art of storytelling… and the importance of being human

Beyond choosing and crafting good questions and being engaged (but not overbearing) during the interview, what else do top-rated interviewers have in common?

The pervasive common thread I noticed among the best interviewers on our platform is, as above, a bit hard to quantify but dovetails well with the notion of being engaged and creating a collaborative experience. It’s taking a dehumanizing process and elevating it to an organic experience between two capable, thinking humans. Many times, that translates into revealing something real about yourself and telling a story. It can be sharing a bit about the company you work at and why, out of all the places you could have landed, you ended up there. Or some aspect of the company’s mission that resonated with you specifically. Or how the projects you’ve worked on tie into your own, personal goals.

The good

I like the interview format, in particular how it was primarily a discussion about cool tech, as well as an honest description of the company… the discussion section was valuable, and may be a better gauge of fit anyway. It’s nice to see a company which places value on that 🙂

The interviewer was helpful throughout the interview. He didn’t mind any questions on their company’s internal technology decisions, or how it’s structured. I liked that the interviewer gave me a good insight of how the company functions.

Extremely kind and very generous with explaining everything they do at [redacted]. Really interested in the technical challenges they’re working on. Great!

Interesting questions but the most valuable and interesting thing were the insights he gave me about [redacted]. He sounded very passionate about engineering in general, particularly about the challenges they are facing at [redacted]. Would love to work with him.

The bad

[A] little bit of friendly banter (even if it’s just “how are you doing”?) at the very beginning of the interview would probably help a bit with keeping the candidate calm and comfortable.

I thought the interview was very impersonal, [and] I could not get a good read on the goal or mission of the company.

And, as we wrote about in a previous post, one of the most genuine, human things of all is giving people immediate, actionable feedback. As you recall, during the feedback step that happens after each interview, we ask interviewees if they’d want to work with their interviewer. As it turns out, there’s a very statistically significant relationship (p < 0.00005)5 between whether people think they did well and whether they’d want to work with the interviewer. This means that when people think they did poorly, they may be a lot less likely to want to work with you. And by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did.

How can one mitigate these losses? Give positive, actionable feedback immediately (or as soon as possible)! This way people don’t have time to go through the self-flagellation gauntlet that happens after a perceived poor performance, followed by the inevitable rationalization that they totally didn’t want to work there anyway.

How to be human

  1. Talk about what your company does… and what specifically about it appealed to you and made you want to join
  2. Talk about what you’re currently working on and how that fits in with what you’re passionate about
  3. When you like a candidate, give positive feedback as quickly as you can to save them from the self-flagellation that they’ll likely go through otherwise… and which might make them rationalize away wanting to work with you
  4. And, you know, be friendly. A little bit of warmth can go a long way.

Becoming a better interviewer

Interviewing people is hard. It’s hard to come up with good questions, it’s hard to give a good interview, and it’s especially hard to be human in the face of conducting a never-ending parade of interviews. But, being a good interviewer is massively important. As we saw, while your company’s brand will get people in the door, once they’ve reached the technical interview, the playing field is effectively level, and you can no longer use your brand as a crutch to mask poor questions or a lack of engagement. And in this market, where the best candidates have a ton of options, when wielded properly, a good interview that elevates a potentially cold, transactional interaction into something real and genuine can become the selling point that gets great engineers to work for you, whether you’re a household name or a startup that just got its first users.

Given how important it is to do interviews well, what are some things you can do to get better right away? One thing I found incredibly useful for coming up with good, original questions is to start a shared doc with your team where every time someone solves a problem they think is interesting, no matter how small, they jot down a quick note. These notes don’t have to be fleshed out at all, but they can be the seeds for unique interview questions that give candidates insight into the day-to-day at your company. Turning these disjointed seeds into interview questions takes thought and effort — you have to prune out a lot of the details and distill the essence of the problem into something it doesn’t take the candidate a lot of work/setup to grok, and you’ll likely have to iterate on the question a few times before you get it right — but they payoff can be huge.

Another thing you can do to get actionable feedback like the kind you saw in this post (and then immediately level up) is to get on as an interviewer. If you interview people in our double-blind practice pool, no one will know who you are or which company you represent, which means that you get a truly unbiased take on your interviewing ability, which includes your question quality, how excited people would be to work with you, and how good you are at helping people along without giving away too much. It’s also a great way to go beyond your team, which can be pretty awkward, and try out new questions on a very engaged, high-quality user base. You’ll also get to keep replays of your interviews so you can revisit crucial moments and figure out exactly what you need to do to get better next time.

Become a better interviewer with honest, actionable feedback from candidates

Become a better interviewer with honest, actionable feedback from candidates

Want to hone your skills as an interviewer? Want to help new interviewers at your company warm up before they officially get added to your interview loops? You can sign up to our platform as an interviewer, or (especially for groups) ping us at

1“People will forget what you said, people will forget what you did, but people will never forget how you made them feel.” -Maya Angelou

2It’s important to call out that brand and engineering brand are two separate things that can diverge pretty wildly. For instance, Target has a strong brand overall but probably not the best engineering brand (sorry). Heap, on the other hand, is one of the better-respected places to work among engineers (both on and off), but it doesn’t have a huge overall brand. Both the Klout and Mattermark Mindshare scores aren’t terrible for quantifying brand strength, but they’re not amazing at engineering brand strength (they’re high for Target and low for Heap). The Glassdoor score is a bit better because reviewers tend to skew engineering-heavy, but it’s still not that great of a measure. So, if anyone has a better way to quantify this stuff, let me know. If I were doing it, I’d probably look at GitHub repos of the company and its employees, who their investors are, and so on and so forth. But that’s a project that’s out of scope for this post.

3If you’re familiar with Dan Savage’s campsite rule for relationships, I think there should be a similar for interviewing… leave your candidates in better shape than when you found them.

4Let us save you the time: Trump is bad, dogs are cute, someone ate something.

5This time with even more significance!



If you care about diversity, don’t just hire from the same five schools

Posted on October 24th, 2017.

EDIT: Our university hiring platform is now on Product Hunt!

If you’re a software engineer, you probably believe that, despite some glitches here and there, folks who have the technical chops can get hired as software engineers. We regularly hear stories about college dropouts, who, through hard work and sheer determination, bootstrapped themselves into millionaires. These stories appeal to our sense of wonder and our desire for fairness in the world, but the reality is very different. For many students looking for their first job, the odds of breaking into a top company are slim because they will likely never even have the chance to show their skills in an interview. For these students (typically ones without a top school on their resume), their first job is often a watershed moment where success or failure can determine which opportunities will be open to them from that point forward and ultimately define the course of their entire career. In other words, having the right skills as a student is nowhere near enough to get you a job at a top-tier tech company.

To make this point concrete, consider three (fictitious, yet indicative) student personas, similar in smarts and skills but attending vastly different colleges. All are seeking jobs as software engineers at top companies upon graduation.

Mason goes to Harvard. He has a mediocre GPA but knows that doesn’t matter to tech companies, where some of his friends already work. Come September, recent graduates and alums fly back to campus on their company’s dime in order to recruit him. While enjoying a nice free meal in Harvard Square, he has the opportunity to ask these successful engineers questions about their current work. If he likes the company, all he has to do is accept the company’s standing invitation to interview on campus the next morning.

Emily is a computer science student at a mid-sized school ranked in the top 30 for computer science. She has solid coursework in algorithms under her belt, a good GPA, and experience as an engineering intern at a local bank. On the day of her campus’s career fair, she works up the courage to approach companies – this will be her only chance to interact with companies where she dreams of working. Despite the tech industry being casual, the attire of this career fair is business formal with a tinge of sweaty. So after awkwardly putting together an outfit she would never wear again1, she locates an ancient printer on the far side of campus and prints 50 copies of her resume. After pushing through the lines in order to line up at the booths of tech companies, she gives her resume to every single tech company at the fair over the course of several hours. She won’t find out for two more weeks if she got any interviews.

Anthony goes to a state school near the town where he grew up. He is top in his class, as well as a self-taught programmer, having gone above and beyond his coursework to hack together some apps. His school’s career fair has a bunch of local, non-tech employers. He has no means of connecting with tech companies face-to-face and doesn’t know anyone who works in tech. So, he applies to nearly a hundred tech companies indiscriminately through their website online, uploading his resume and carefully crafted cover letter. He will probably never hear from them.

Career fair mania

The status quo in university recruiting revolves around career fairs and in-person campus recruiting, which have serious limitations. For one, they are extremely expensive, especially at elite schools. Prime real estate at the MIT career fair will run you a steep $18,000, for entry alone. That’s not counting the price of swag (which gets more exorbitant each year), travel, and, most importantly, the opportunity cost of attending engineers’ time. While college students command the lowest salaries, it’s not uncommon for tech companies to spend 50% more on recruiting a student than a senior engineer.

At elite schools, the lengths to which companies go to differentiate themselves is becoming more exorbitant with each passing year. In fact, students at elite colleges suffer from company overload because every major tech company, big and small, is trying to recruit them. All of this, while students at non-elite colleges are scrambling to get their foot in the door without any recruiters, let alone VPs of high-profile companies, visiting their campus.

Of course, due to this cost, companies are limited in their ability to visit colleges in person, and even large companies can visit around 15 or 20 colleges at most. This strategy overlooks top students at solid CS programs that are out of physical reach.

In an effort to overcome this, companies are attending conferences and hackathons out of desperation to reach students at other colleges. The sponsorship tier for the Grace Hopper Conference, the premier gathering for women in tech, tops out at $100,000, with the sponsorship tier to get a single interview booth starting at $30,000. Additionally, larger companies send representatives (usually engineers) to large hackathons in an effort to recruit students in the midst of a 48-hour all-nighter. However, the nature of in-person career fairs and events are that not all students will be present. Grace Hopper is famously expensive to attend as a student, especially when factoring in airfare and hotel.

This cost is inefficient at best, and prohibitive at worst, especially for small startups with low budget and brand. Career fairs serve a tiny portion of companies and a tiny portion of students, and the rest are caught in the pecuniary crossfire. Demand for talented engineers out of college who bring a different lived experience to tech has never been higher, yet companies are passing on precisely these students via traditional methods. Confounding the issue even further is the fundamental question of whether having attended a top school has much bearing on candidate quality in the first place (more on that in the section on technical screening below).

Homogeneity of hires

The focus of companies on elite schools has notable, negative implications for the diversity of their applicants. In particular, many schools that companies traditionally visit are notably lacking in diversity, especially when it comes to race and socioeconomic status. According to a survey of computer science students at Stanford, there were just fifteen Hispanic female and fifteen black female computer science majors in the 2015 graduating class total. In this analysis, the Stanford 2015 CS major was 9% Hispanic and 6% black. According to a 2015 analysis, the Harvard CS major was just 3% black and 5 percent Hispanic. Companies that are diversity-forward and constrained to recruiting at the same few schools end up competing over this small pool of diverse students. Meanwhile, there is an entire ocean of qualified, racially diverse students from less traditional backgrounds whom companies are overlooking.

The focus on elite schools also has meaningful implications on socioeconomic diversity. According to a detailed New York Times infographic, “four in 10 students from the top 0.1 percent attend an Ivy League or elite university, roughly equivalent to the share of students from poor families who attend any two- or four-year college.” The infographic highlights the rigid segmentation of students by class background in college matriculation.

Source: New York Times

The article finds that the few lower-income students who end up at elite colleges do about as well as their more affluent classmates but that attending an elite versus non-elite college makes a huge difference in future income.

The focus of tech companies on elite schools lends credence to this statistic, codifying the rigidity with which students at elite college are catapulted into the 1 percent, while others are left behind. Career-wise, it’s that first job or internship you get while you’re still in school that can determine what opportunities you have access to in the future. And yet, students at non-elite colleges have trouble accessing these very internships and jobs, or even getting a meager first round interview, contributing to the lack of social mobility in our society not for lack of skills but for lack of connections. This sucks. A lot.

The technical screen

Let’s return to our three students. Let’s say that Emily, the student who attended her college’s career fair, gets called back by one or two companies for a first round interview if her resume meets the criteria that companies are looking for. Not having an internship at a top tech company already — quite the catch-22 — puts her at a disadvantage. Anthony has little to no chance of hearing back from employers via his applications online, but let’s say that by some miracle lands a phone screen with one of the tech giants (his best shot, as there are more recruiters to look through the resume dump on the other end).

What are their experiences when it comes to prepping for upcoming technical interviews?

Mason, the Harvard student, attends an event on campus with Facebook engineers teaching him how to pass the technical interview. He also accepts a few interviews at companies he’s less excited with for practice, and just in case. While he of course needs be sharp and prepare in order to get good at these sorts of algorithmic problems, he has all of the resources he could ask for and more at his disposal. Unsurprisingly, his Facebook interview goes well.

Emily’s school has an informal, undergraduate computer science club in which they are collectively reading technical interviewing guides and trying to figure out what tech companies want from them. She has a couple interviews lined up, but all of which are for jobs she’s desperate to get. They trade tips after interviews but ultimately have a shaky understanding of they did right and wrong in the absence of post-interview feedback from companies. Only a couple of alumni from their school have made it to top tech companies in the past, and so they lack the kinds of information that Mason has on what companies are looking for. (E.g. Don’t be afraid to take hints, make sure to explain your thought process, what the heck is this CoderPad thing anyway…)

Anthony doesn’t know anyone who has a tech job like the one he’s interviewing for, and only one of his friends is also interviewing. He doesn’t know where to start when it comes to getting ready for his upcoming interview at GoogFaceSoft. He only has one shot at it with no practice interviews lined up. He prepares by googling “tech interview questions” and stumbles upon a bunch of unrealistic interview questions, many of them behavioral or outdated. He might be offered the interview and be fit for the job, but he sure doesn’t know how to pass the interview.

For students who may be unfamiliar with the art of the technical interview, algorithmic interviews can be mystifying, leading to an imbalance of information on how to succeed. Given that technical interviewing is a game, it is important that everyone knows the rules, spoken and unspoken. There are many practice resources available, but no amount of reading and re-reading Cracking the Coding Interview can prepare you for that moment when you are suddenly in a live, technical phone screen with another human.

We built a better way to hire

Ultimately, as long as university hiring relies on a campus-by-campus approach, the status quo will continue to be fundamentally inefficient and unmeritocratic. No company, not even the tech giants, can cover every school or every resume submitted online. And, in the absence of any meaningful information on a student’s resume, companies default to their university as the only proxy. This approach is inefficient at best and, at worst, it’s the first in a series of watershed moments that derail the promise of social mobility for the non-elite, taking with them any hope of promoting diversity among computer science students.

Because this level of inequity, placed for maximum damage right at the start of people’s careers, really pissed us off, we decided to do something about it.’s answer to the unfortunate status quo is a university-specific hiring platform. If you’re already familiar with how core works, you’ll see that the premise is exactly the same. We give out free practice to students, and use their performance in practice to identify top performers, completely independently of their pedigree. Those top performers then get to interview with companies like Lyft and Quora on our platform. In other words, we’re excited to provide students with pathways into tech that don’t involve going to an elite school or knowing someone on the inside. So far, we’ve been very pleased with the results. You can see our student demographics and where they’re coming from below. Students from all walks of life, whether they’re from MIT or a school you’d never visit, are flocking to the platform, and we couldn’t be prouder.

school tier distribution evaluates students based on their coding skills, not their resume. We are open to students regardless of their university affiliation, college major, and pretty much anything else (we ask for your class year to make sure you’re available when companies want you and that’s about it). Unlike traditional campus recruiting, we attract students organically (getting free practice with engineers from top companies is a pretty big draw) from schools big and small from across the country.

student heatmap

We’re also proud that almost 40 percent of our university candidates come from backgrounds that are underrepresented in tech.

student heatmap

Because of our completely blind, skills-first approach, we’ve seen an interesting phenomenon happen time and time again: when a student unmasks at the end of a successful interview, the company in question realizes that the student who just aced their technical phone screen was one whose resume was sitting at the bottom of the pile all along.

In addition to identifying top students who bring a different lived experience to tech, we’re excited about the economics of our model. With, a mid-sized startup can staff their entire intern class for the same cost as attending 1-2 career fairs at top schools… with a good chunk of those interns coming from underrepresented backgrounds. Want to hire interns and new grads in the most efficient, fair way possible? Sign up to be an employer on our university platform!

Meena runs’s university hiring platform. We help companies hire college students from all over the US, with a focus on diversity. Prior to joining, Meena was a software engineer at Clever, and before that, Meena was in college on the other side of the engineer interviewing equation.

1At least her school didn’t send out this.



We analyzed thousands of technical interviews on everything from language to code style. Here’s what we found.

Posted on June 13th, 2017.

Note: Though I wrote most of the words in this post, the legendary Dave Holtz did the heavy lifting on the data side. See more of his work on his blog.

If you’re reading this post, there’s a decent chance that you’re about to re-enter the crazy and scary world of technical interviewing. Maybe you’re a college student or fresh grad who is going through the interviewing process for the first time. Maybe you’re an experienced software engineer who hasn’t even thought about interviews for a few years. Either way, the first step in the interviewing process is usually to read a bunch of online interview guides (especially if they’re written by companies you’re interested in) and to chat with friends about their experiences with the interviewing process (both as an interviewer and interviewee). More likely than not, what you read and learn in this first, “exploratory” phase of the interview process will inform how you choose to prepare moving forward.

There are a few issues with this typical approach to interview preparation:

  • Most interview guides are written from the perspective of one company. While Company A may really value efficient code, Company B may place more of an emphasis on high-level problem-solving skills. Unless your heart is set on Company A, you probably don’t want to give too much weight to what they value.
  • People lie sometimes, even if they don’t mean to. In writing, companies may say they’re language agnostic, or that it’s worthwhile to explain your thought process, even if the answer isn’t quite right. However, it’s not clear if this is actually how they act! We’re not saying that tech companies are nefarious liars who are trying to mislead their applicant pool. We’re just saying that sometimes implicit biases sneak in and people aren’t even aware of them.
  • A lot of the “folk knowledge” that you hear from friends and acquaintances may not be based in fact at all. A lot of people assume that short interviews spell doom. Similarly, everyone can recall one long interview after which they’ve thought to themselves, “I really hit it off with that interviewer, I’ll definitely get passed onto the next stage.” In the past, we’ve seen that people are really bad at gauging how they did in interviews. This time, we wanted to look directly at indicators like interview length and see if those actually matter.

Here at, we are uniquely positioned to approach technical interviews and their outcomes in a data-driven way. This time, we’ve opted for a quick (if not dirty) and quantitative analysis. In other words, rather than digging deep into individual interviews, we focused on easily measurable attributes that many interviews share, like duration and language choice. In upcoming posts, we’ll be delving deeper into the interview content itself. If you’re new to our blog and want to get some context about how works and what interview data we collect, please take a look at the section called “The setup” below. Otherwise, please skip over that and head straight for the results!

The setup is a platform where people can practice technical interviewing anonymously, and if things go well, unlock the ability to interview anonymously, whenever they’d like, with top companies like Uber, Lyft, and Twitch. The cool thing is that both practice interviews and real interviews with companies take place within the ecosystem. As a result, we’re able to collect quite a bit of interview data and analyze it to better understand technical interviews, the signal they carry, what works and what doesn’t, and which aspects of an interview might actually matter for the outcome.

Each interview, whether it’s practice or real, starts with the interviewer and interviewee meeting in a collaborative coding environment with voice, text chat, and a whiteboard, at which point they jump right into a technical question. Interview questions tend to fall into the category of what you’d encounter in a phone screen for a back-end software engineering role. During these interviews, we collect everything that happens, including audio transcripts, data and metadata describing the code that the interviewee wrote and tried to run, and detailed feedback from both the interviewer and interviewee about how they think the interview went and what they thought of each other.

If you’re curious, you can see what the feedback forms for interviewers and interviewees look like below — in addition to one direct yes/no question, we also ask about a few different aspects of interview performance using a 1-4 scale. We also ask interviewees some extra questions that we don’t share with their interviewers, and one of the things we ask is whether an interviewee has previously seen the question they just worked on.

Feedback form for interviewers

Feedback form for interviewers

Feedback form for interviewees

Feedback form for interviewees

The results

Before getting into the thick of it, it’s worth noting that the conclusions below are based on observational data, which means we can’t make strong causal claims… but we can still share surprising relationships we’ve observed and explain what we found so you can draw your own conclusions.

Having seen the interview question before

“We’re talking about practice!” -Allen Iverson

First thing’s first. It doesn’t take a rocket scientist to suggest that one of the best ways to do better in interviews is to… practice interviewing. There are a number of resources out there to help you practice, ours among them. One of the main benefits of working through practice problems is that you reduce the likelihood of being asked to solve something you’ve never seen before. Balancing that binary search tree will be much less intimidating if you’ve already done it once or twice.

We looked at a sample of ~3000 interviews and compared the outcome to whether the interviewee had seen the interview question before. You can see the results in the plot below.


Unsurprisingly, interviewees who had seen the question were 16.6% more likely to be considered hirable by their interviewer. This difference is statistically significant (p < 0.001).1

Does it matter what language you code in?

“Whoever does not love the language of his birth is lower than a beast and a foul smelling fish.” -Jose Rizal

You might imagine that different languages lead to better interviews. For instance, maybe the readability of Python gives you a leg up in interviews. Or perhaps the fact that certain languages handle data structures in a particularly clean way makes common interview questions easier. We wanted to see whether or not there were statistically significant differences in interview performance across different interview languages.

To investigate, we grouped interviews on our platform by interview language and filtered out any languages that were used in fewer than 5 interviews (this only threw out a handful of interviews). After doing this, we were able to look at interview outcome and how it varied as a function of interview language.

The results of that analysis are in the chart below. Any non-overlapping confidence intervals represent a statistically significant difference in how likely an interviewee is to ‘pass’ an interview, as a function of interview language. Although we don’t do a pairwise comparison for every possible pair of languages, the data below suggest that generally speaking, there aren’t statistically significant differences between the success rate when interviews are conducted in different languages.2


That said, one of the most common mistakes we’ve observed qualitatively is people choosing languages they’re not comfortable in and then messing up basic stuff like array length lookup, iterating over an array, instantiating a hash table, and so on. This is especially mortifying when interviewees purposely pick a fancy-sounding language to impress their interviewer. Trust us, wielding your language of choice comfortably beats out showing off in a fancy-sounding language you don’t know well, every time.

Even if language doesn’t matter… is it advantageous to code in the company’s language of choice?

“God help me, I’ve gone native.” -Margaret Blaine

It’s all well and good that, in general, interview language doesn’t seem particularly correlated with performance. However, you might imagine that there could be an effect depending on the language that a given company uses. You could imagine a Ruby shop saying “we only hire Ruby developers, if you interview in Python we’re less likely to hire you.” On the flip side, you could imagine that a company that writes all of their code in Python is going to be much more critical of an interviewee in Python – they know the ins and outs of the language, and might judge the candidate for doing all sorts of “non-pythonic” things during their interview.

The chart below is similar to the chart which showed differences in interview success rate (as measured by interviewers being willing to hire the interviewee) for C++, Java, and Python. However, this chart also breaks out performance by whether or not the interview language is in the company’s stack. We restrict this analysis to C++, Java and Python because these are the three languages where we had a good mixture of interviews where the company did and did not use that language. The results here are mixed. When the interview language is Python or C++, there’s no statistically significant difference between the success rates for interviews where the interview language is or is not a language in the company’s stack. However, interviewers who interviewed in Java were more likely to succeed when interviewing with a Java shop (p=0.037).

So, why is it that coding in the company’s language seems to be helpful when it’s Java, but not when it’s Python or C++? One possible explanation is that the communities that exist around certain programming languages (such as Java) place a higher premium on previous experience with the language. Along these lines, it’s also possible that interviewers from companies that use Java are more likely to ask questions that favor those with a pre-existing knowledge of Java’s idiosyncrasies.


What about the relationship between what language you program in and how good of a communicator you’re perceived to be?

“To handle a language skillfully is to practice a kind of evocative sorcery.” -Charles Baudelaire

Even if language choice doesn’t matter that much for overall performance (Java-wielding companies notwithstanding), we were curious whether different language choices led to different outcomes in other interview dimensions. For instance, an extremely readable language, like Python, may lead to interview candidates who are assessed to have communicated better. On the other hand, a low-level language like C++ might lead to higher scores for technical ability. Furthermore, very readable or low-level languages might lead to correlations between these two scores (for instance, maybe they’re a C++ interview candidate who can’t explain at all what he or she is doing but who writes very efficient code). The chart below suggests that there isn’t really any observable difference between how candidates’ technical and communication abilities are perceived, across a variety of programming languages.

Furthermore, no matter what, poor technical ability seems highly correlated with poor communication ability – regardless of language, it’s relatively rare for candidates to perform well technically but not effectively communicate what they’re doing (or vice versa), largely (and fortunately) debunking the myth of the incoherent, fast-talking, awkward engineer.3

Interview duration

“It’s fine when you careen off disasters and terrifyingly bad reviews and rejection and all that stuff when you’re young; your resilience is just terrific.” -Harold Prince

We’ve all had the experience of leaving an interview and just feeling like it went poorly. Often, that feeling of certain underperformance is motivated by rules of thumb that we’ve either come up with ourselves or heard repeated over and over again. You might find yourself thinking, “the interview didn’t last long? That’s probably a bad sign… ” or “I barely wrote anything in that interview! I’m definitely not going to pass.” Using our data, we wanted to see whether these rules of thumb for evaluating your interview performance had any merit.

First, we looked at the length of the interview. Does a shorter interviewer mean you were such a trainwreck that the interviewer just had to stop the interview early? Or was it maybe the case that the interviewer had less time than normal, or had seen in just a short amount of time that you were an awesome candidate? The plot below shows the distributions of interview length (measured in minutes) for both successful and unsuccessful candidates. A quick look at this chart suggests that there is no difference in the distribution of interview lengths between interviews that go well and interviews that don’t — the average length of interviews where the interviewer wanted to hire the candidate was 51.00 minutes, whereas the average length of interviews where the interviewer did not was 49.95 minutes. This difference is not statistically significant.4


Amount of code written

“Brevity is the soul of wit.” -William Shakespeare

You may have experienced an interview where you were totally stumped. The interviewer asks you a question you barely understand, you repeat back to him or her “binary search what?”, and you basically write no code during your interview. You might hope that you could still pass an interview like this through sheer wit, charm, and high-level problem-solving skills. In order to assess whether or not this was true, we looked at the final character length of code written by the interviewee. The plot below shows the distributions of character length for both successful and unsuccessful. A quick look at this chart suggests that there is a difference between the two — interviews that don’t go well tend to have less code. There are two phenomena that may contribute to this. First, unsuccessful interviewers may write less code to begin with. Additionally, they may be more prone to delete large swathes of code they’ve written that either don’t run or don’t return the expected result.


On average, successful interviews had final interview code that was on average 2045 characters long, whereas unsuccessful ones were, on average, 1760 characters long. That’s a big difference! This finding is statistically significant and probably not very surprising.

Code modularity

“The mark of a mature programmer is willingness to throw out code you spent time on when you realize it’s pointless.” -Bram Cohen

In addition to just look at how much code you write, we can also think about the type of code you write. Conventional wisdom suggests that good programmers don’t recycle code – they write modular code that can be reused over and over again. We wanted to know if that type of behavior was actually rewarded during the interview process. In order to do so, we looked at interviews conducted in Python5 and counted how many function definitions appeared in the final version of the interview. We wanted to know if successful interviewees defined more functions — while having more function handlers is not the definition of modularity, in our experience, it’s a pretty strong signal of it. As always, it’s impossible to make strong causal claims about this – it might be the case that certain interviewers (who are more or less lenient) ask interview questions that lend themselves to more or fewer functions. Nonetheless, it is an interesting trend to investigate!

The plot below shows the distribution of the number of Python functions defined for both candidates who the interviewer said they would hire and candidates who the interviewer said they would not hire. A quick look at this chart suggests that there is a difference in the distribution of function definitions between interviews that go well and interviews that don’t. Successful interviewees seem to define more functions.


On average, successful candidates interviewing in Python define 3.29 functions, whereas unsuccessful candidates define 2.71 functions. This finding is statistically significant. The upshot here is that interviewers really do reward the kind of code they say they want you to write.

Does it matter if your code runs?

“Move fast and break things. Unless you are breaking stuff, you are not moving fast enough.” -Mark Zuckerberg
“The most effective debugging tool is still careful thought, coupled with judiciously placed print statements.” -Brian Kernighan

A common refrain in technical interviews is that interviewers don’t actually care if your code runs – what they care about is problem-solving skills. Since we collect data on the code interviewees run and whether or not that code compiles, we wanted to see if there was evidence for this in our data. Is there any difference between the percentage of code that compiles error-free in successful interviews versus unsuccessful interviews? Furthermore, can interviewees actually still get hired, even if they make tons of syntax errors?

In order to get at this question, we looked at the data. We restricted our dataset to interviews longer than 10 minutes with more than 5 unique instances of code being executed. This helped filter out interviews where interviewers didn’t actually want the interviewee to run code, or where the interview was cut short for some reason. We then measured the percent of code runs that resulted in errors.5 Of course, there are some limitations to this approach – for instance, candidates could execute code that does compile but gives a slightly incorrect answer. They could also get the right answer and write it to stderr! Nonetheless, this should give us a directional sense of whether or not there’s a difference.

The chart below gives a summary of this data. The x-axis shows the percentage of code executions that were error-free in a given interview. So an interview with 3 code executions and 1 error message would count towards the “30%-40%” bucket. The y-axis indicates the percentage of all interviews that fall in that bucket, for both successful and unsuccessful interviews. Just eyeballing the chart below, one gets the sense that on average, successful candidates run more code that goes off without an error. But is this difference statistically significant?


On average, successful candidates’ code ran successfully (didn’t result in errors) 64% of the time, whereas unsuccessful candidates’ attempts to compile code ran successfully 60% of the time, and this difference was indeed significant. Again, while we can’t make any causal claims, the main takeaway is that successful candidates do usually write code that runs better, despite what interviewers may tell you at the outset of an interview.

Should you wait and gather your thoughts before writing code?

“Never forget the power of silence, that massively disconcerting pause which goes on and on and may at last induce an opponent to babble and backtrack nervously.” -Lance Morrow

We were also curious whether or not successful interviewees tended to take their time in the interview. Interview questions are often complex! After being presented with a question, there might be some benefit to taking a step back and coming up with a plan, rather than jumping right into things. In order to get a sense of whether or not this was true, we measured how far into a given interview candidates first executed code. Below is a histogram showing how far into interviews both successful and unsuccessful interviewees first ran code. Looking quickly at the histogram, you can tell that successful candidates do in fact wait a bit longer to start running code, although the magnitude of the effect isn’t huge.


More specifically, on average, candidates with successful interviews first run code 27% of the way through the interview, whereas candidates with unsuccessful interviews first run code 23.9% of the way into the interview, and this difference is significant. Of course, there are alternate explanations for what’s happening here. For instance, perhaps successful candidates are better at taking the time to sweet-talk their interviewer. Furthermore, the usual caveat that we can’t make causal claims applies – if you just sit in an interview for an extra 5 minutes in complete silence, it won’t help your chances. Nonetheless, there does seem to be a difference between the two cohorts.


All in all, this post was our first attempt to understand what does and does not typically lead to an interviewer saying “you know what, I’d really like to hire this person.” Because all of our data are observational, its hard to make causal claims about what we see. While successful interviewees may exhibit certain behaviors, adopting those behaviors doesn’t guarantee success. Nonetheless, it does allow us to support (or call bullshit on) a lot of the advice you’ll read on the internet about how to be a successful interviewee.

That said, there is much still to be done. This was a first, quantitative pass over our data (which is, in many ways, a treasure trove of interview secrets), but we’re excited to do a deeper, qualitative dive and actually start to categorize different questions to see which carry the most signal as well as really get our head around 2nd order behaviors that you can’t measure easily by running a regex over a code sample or measuring how long an interview took. If you want to help us with this and are excited to listen to a bunch of technical interviews, drop me a line (at!

1All error bars in this post represent a 95% confidence interval.

2There were more languages than these on our platform, but the more obscure the language, the less data points we have. For instance, all interviews in Brainfuck were clearly successful. Kidding.

3The best engineers I’ve met have also been legendarily good at breaking down complex concepts and explaining them to laypeople. Why the infuriating myth of the socially awkward, incoherent tech nerd continues to exist, I have absolutely no idea.

4For every comparison of distributions in this post, we use both a Fisher-Pitman permutation test to compare the difference in the means of the distributions.

5We limit this analysis to interviews in Python because it lends itself particularly well to the identification of function definitions with a simple parsing script.

6We calculate this by looking at what percentage of the time the interviewee executed code that resulted in either an error or non-error output contained the term “error” or “traceback.”



LinkedIn endorsements are dumb. Here’s the data.

Posted on February 27th, 2017.

If you’re an engineer who’s been endorsed on LinkedIn for any number of languages/frameworks/skills, you’ve probably noticed that something isn’t quite right. Maybe they’re frameworks you’ve never touched or languages you haven’t used since freshman year of college. No matter the specifics, you’re probably at least a bit wary of the value of the LinkedIn endorsements feature. The internets, too, don’t disappoint in enumerating some absurd potential endorsements or in bemoaning the lack of relevance of said endorsements, even when they’re given in earnest.

Having a gut feeling for this is one thing, but we were curious about whether we could actually come up with some numbers that showed how useless endorsements can be, and we weren’t disappointed. If you want graphs and numbers, scroll down to the “Here’s the data” section below. Otherwise, humor me and read my completely speculative take on why endorsements exist in the first place.

LinkedIn endorsements are just noisy crowdsourced tagging

Pretend for a moment that you’re a recruiter who’s been tasked with filling an engineering role. You’re one of many people who pays LinkedIn ~$9K/year for a recruiter seat on their platform1. That hefty price tag broadens your search radius (which is otherwise artificially constrained) and lets you search the entire system. Let’s say you have to find a strong back-end engineer. How do you begin?

Unfortunately, LinkedIn’s faceted search (pictured below) doesn’t come with a “can code” filter2.

So, instead of searching for what you really want, you have to rely on proxies. Some obvious proxies, even though they’re not that great, might be where someone went to school or where they’ve worked before. However, if you need to look for engineering ability, you’re going to have to get more specific. If you’re like most recruiters, you’ll first look for the main programming language your company uses (despite knowledge of a specific language not being a good indicator of programming ability and despite most hiring managers not caring which languages their engineers know) and then go from there.

Now pretend you’re LinkedIn. You have no data about how good people are at coding, and though you do have a lot of resume/biographical data, that doesn’t tell the whole story. You can try relying on engineers filling in their own profiles with languages they know, but given that engineers tend to be pretty skittish about filling in their LinkedIn profile with a bunch of buzzwords, what do you do?

You build a crowdsourced tagger, of course! Then, all of a sudden, your users will do your work for you. Why do I think this is the case? Well, if LinkedIn cared about true endorsements rather than perpetuating the skills-based myth that keeps recruiters in their ecosystem, they could have written a weighted endorsement system by now, at the very least. That way, an endorsement from someone with expertise in some field might mean more than an endorsement from your mom (unless, of course, she’s an expert in the field).

But they don’t do that, or at least they don’t surface it in candidate search. It’s not worth it. Because the point of endorsements isn’t to get at the truth. It’s to keep recruiters feeling like they’re getting value out of the faceted search they’re paying almost $10K per seat for. In other words, improving the fidelity of endorsements would likely cannibalize LinkedIn’s revenue.

You could make the counterargument that despite the noise, LinkedIn endorsements still carry enough signal to be a useful first-pass filter and that having them is more useful than not having them. This is the question I was curious about, so I decided to cross-reference our users’ interview data with their LinkedIn endorsements.

The setup

So, what data do we have? First, for context, is a platform where people can practice technical interviewing anonymously with interviewers from top companies and, in the process, find jobs. Do well in practice, and you get guaranteed (and anonymous!) technical interviews at companies like Uber, Twitch, Lyft, and more. Over the course of our existence, we’ve amassed performance data from close to 5,000 real and practice interviews.

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question. Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role. Some examples of these interviews can be found on our public recordings page.

After every interview, interviewers rate interviewees on a few different dimensions, including technical ability. Technical ability gets rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!”. On our platform, a score of 3 or above has generally meant that the person was good enough to move forward. You can see what our feedback form looks like below:


As promised, I cross-referenced our data with our users’ LinkedIn profiles and found some interesting, albeit not that surprising, stuff.

Endorsements vs. what languages people actually program in

The first thing I looked at was whether the programming language people interviewed in most frequently had any relationship to the programming language for which they were most endorsed. It was nice that, across the board, people tended to prefer one language for their interviews, so we didn’t really have a lot of edge cases to contend with.

It turns out that people’s interview language of choice matched their most endorsed language on LinkedIn just under 50% of the time.

Of course, just because you’ve been endorsed a lot for a specific language doesn’t mean that you’re not good at the other languages you’ve been endorsed for. To dig deeper, I took a look at whether our users had been endorsed for their interview language of choice at all. It turns out that people were endorsed for their language of choice 72% of the time. This isn’t a particularly powerful statement, though, because most people on our platform have been endorsed for at least 5 programming languages.

That said, even when an engineer had been endorsed for their interview language of choice, that language appeared in their “featured skills” section only 31% of the time. This means that most of the time, recruiters would have to click “View more” (see below) to see the language that people prefer to code in, if it’s even listed in the first place.

So, how often were people endorsed for their language of choice? Quantifying endorsements3 is a bit fuzzy, but to answer this meaningfully, I looked at how often people were endorsed for that language relative to how often they were endorsed for their most-endorsed language, in the cases when the two languages weren’t the same (recall that this happened about half the time). Perhaps if these numbers were close to 1 most of the time, then endorsements might carry some signal. As you can see in the histogram below, this was not the case at all.

The x-axis above is how often people were endorsed for their interview language of choice relative to their most-endorsed language. The bars on the left are cases when someone was barely endorsed for their language of choice, and all the way to right are cases when people were endorsed for both languages equally as often. All told, the distribution is actually pretty uniform, making for more noise than signal.

Endorsements vs. interview performance

The next thing I looked at was whether there was any correspondence between how heavily endorsed someone was on LinkedIn and their interview performance. This time, to quantify the strength of someone’s endorsements4, I looked at how many times someone was endorsed for their most-endorsed language and correlated that to their average technical score in interviews on

Below, you can see a scatter plot of technical ability vs. LinkedIn endorsements, as well as my attempt to fit a line through it. As you can see, the R^2 is piss-poor, meaning that there isn’t a relationship between how heavily endorsed someone is and their technical ability to speak of.

Endorsements vs. no endorsements… and closing thoughts

Lastly, I took a look at whether having any endorsements in the first place mattered with respect to interview performance. If I’m honest, I was hoping there’d be a negative correlation, i.e. if you don’t have endorsements, you’re a better coder. After running some significance testing, though, it became clear that having any endorsements at all (or not) doesn’t matter.

So, where does this leave us? As long as there’s money to be made in peddling low-signal proxies, endorsements won’t go away and probably won’t get much better. It is my hope, though, that any recruiters reading this will take a second look at the candidates they’re sourcing and try to, where possible, look at each candidate as more than the sum of their buzzword parts.

Thanks to Liz Graves for her help with the data annotation for this post.

1Roughly 60% of LinkedIn’s revenue comes from recruiting, so you can see why this stuff matters.

2You know what comes with a can code filter? does! We know how people are doing rigorous, live technical interviews, which, in turn, lets us reliably predict how well they will do in future interviews. Roughly 60%3 of our candidates pass technical phone screens and make it onsite. Want to use us to hire?

3There are a lot of possible approaches to comparing endorsements, to each other and to other stuff. In this post, I decided to, as much as possible mimic how a recruiter might think about a candidate’s endorsements when looking at their profile. Recruiters are busy (I know; I used to be one) and get paid to make quick judgments. Therefore, given that LinkedIn doesn’t normalize endorsements for you, if a recruiter wanted to do it, they’d have to actually add up all of someone’s endorsements and then do a bunch of pairwise division. This isn’t sustainable, and it’s much easier and faster to look at the absolute numbers. For this exact reason, when comparing the endorsements for two languages, I chose to normalize the relative to each other rather than relative to all other endorsements. And when trying to quantify the strength of someone’s programming endorsements as a whole, I opted to just count the number of endorsements for someone’s most-endorsed language.

4See footnote 3 above; I used the same rationale.



Lessons from 3,000 technical interviews… or how what you do after graduation matters way more than where you went to school

Posted on December 28th, 2016.

The first blog post I published that got any real attention was called “Lessons from a year’s worth of hiring data“. It was my attempt to understand what attributes of someone’s resume actually mattered for getting a software engineering job. Surprisingly, as it turned out, where someone went to school didn’t matter at all, and by far and away, the strongest signal came from the number of typos and grammatical errors on their resume.

Since then, I’ve discovered (and written about) how useless resumes are, but ever since writing that first post, I’ve been itching to do something similar with’s data. For context, is a platform where people can practice technical interviewing anonymously and, in the process, find jobs — do well in practice, and you get guaranteed (and anonymous!) technical interviews at companies like Uber, Twitch, Lyft, and more. Over the course of our existence, we’ve amassed performance data from thousands of real and practice interviews. Data from these interviews sets us up nicely to look at what signals from an interviewee’s background might matter when it comes to performance.

As often happens, what we found was surprising, and some of it runs counter to things I’ve said and written on the subject. More on that in a bit.

The setup

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question (check out our recordings page to see this in action). Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from a mix of large companies like Google, Facebook, and Uber, as well as engineering-focused startups like Asana, Mattermark, KeepSafe, and more.

After every interview, interviewers rate interviewees on a few different dimensions, including technical ability. Technical ability gets rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!”. On our platform, a score of 3 or above has generally meant that the person was good enough to move forward. You can see what our feedback form looks like below:


The results

To run the analysis for this post, we cross-referenced interviewees’ average technical scores (circled in red in the feedback form above) with the attributes below to see which ones mattered most. Here’s the full attribute list1:

  • Attended a top computer science school
  • Worked at a top company
  • Took classes on Udacity/Coursera2
  • Founded a startup
  • Master’s degree
  • Years of experience

Of all of these, only 3 attributes emerged as statistically significant: top school, top company, and classes on Udacity/Coursera. Apparently, as the fine gentlemen of Metallica once said, nothing else matters. In the graph below, you can see the effect size of each of the significant attributes (attributes that didn’t achieve significance don’t have bars).

As I said at the outset, these results were quite surprising, and I’ll take a stab at explaining each of the outcomes below.

Top school & top company

Going into this, I expected top company to matter but not top school. The company thing makes sense — you’re selecting people who’ve successfully been through at least one interview gauntlet, so the odds of them succeeding at future ones should be higher.

Top school is a bit more muddy, and it was indeed the least impactful of the significant attributes. Why did schooling matter in this iteration of the data but didn’t matter when I was looking at resumes? I expect the answer lies in the disparity between performance in an isolated technical phone screen versus what happens when a candidate actually goes on site. With the right preparation, the technical phone interview is manageable, and top schools often have rigorous algorithms classes and a culture of preparing for technical phone screens (to see why this culture matters and how it might create an unfair advantage for those immersed in it, see my post about how we need to rethink the technical interview). Whether passing an algorithmic technical phone screen means you’re a great engineer is another matter entirely and hopefully the subject of a future post.


MOOC participation (Udacity and Coursera in particular, as those were the ones users gravitated to most) mattering as much as it did (and mattering way more than pedigree) was probably the most surprising finding here, and so it merited some additional digging.

In particular, I was curious about the interplay between MOOCs and top schools, so I partitioned MOOC participants into people who had attended top schools vs. people who hadn’t. When I did that, something startling emerged. For people who attended top schools, completing Udacity or Coursera courses didn’t appear to matter. However, for people who did not, the effect was huge, so huge, in fact, that it dominated the board. Moreover, interviewees who attended top schools performed significantly worse than interviewees who had not attended top schools but HAD taken a Udacity or Coursera course.

So, what does this mean? Of course (as you’re probably thinking to yourself while you read this), correlation doesn’t imply causation. As such, rather than MOOCs being a magic pill, I expect that people who gravitate toward online courses (and especially those who might have a chip on their shoulder about their undergrad pedigree and end up drinking from the MOOC firehose) already tend to be abnormally driven. But, even with that, I’d be hard pressed to say that completing great online CS classes isn’t going to help you become a better interviewee, especially if you didn’t have the benefit of a rigorous algorithms class up until then. Indeed, a lot of the courses we saw people take focused around algorithms, so it’s no surprise that supplementing your preparation with courses like this could be tremendously useful. Some of the most popular courses we saw were:

Design of Computer Programs
Intro to Algorithms
Computability, Complexity & Algorithms

Algorithms Specialization
Functional Programming Principles in Scala
Machine Learning
Algorithms on Graphs

Founder status

Having been a founder didn’t matter at all when it came to technical interview performance. This, too, isn’t that surprising. The things that make one a good founder are not necessarily the things that make one a good engineer, and if you just came out of running a startup and are looking to get back into an individual contributor role, odds are, your interview skills will be a bit rusty. This is, of course, true of folks who’ve been in industry but out of interviewing for some time, as you’ll see below.

Master’s degree & years of experience

No surprises here. I’ve ranted quite a bit about the disutility of master’s degrees, so I won’t belabor the point.

Years of experience, too, shouldn’t be that surprising. For context, our average user has about 5 years of experience, with most having between 2 and 10. I think we’ve all anecdotally observed that the time spent away from your schooling doesn’t do you any favors when it comes to interview prep. You can see a scatter plot of interview performance vs. years of experience below as well as my attempt to fit a line through it (as you can see, the R^2 is piss poor, meaning that there isn’t a relationship to speak of).

Closing thoughts

If you know me, or even if you’ve read some of my writing, you know that, in the past, I’ve been quite loudly opposed to the concept of pedigree as a useful hiring signal. With that in mind, I feel like I owe clearly acknowledge, up front, that we found this time runs counter to my stance. But that’s the whole point, isn’t it? You live, you get some data, you make some graphs, you learn, you make new graphs, and you adjust. Even with this new data, I’m excited to see that what mattered way more than pedigree was the actions people took to better themselves (in this case, rounding out their existing knowledge with MOOCs), regardless of their background.

Most importantly, these findings have done nothing to change’s core mission. We’re creating an efficient and meritocratic way for candidates and companies to find each other, and as long as you can code, we couldn’t care less about who you are or where you come from. In our ideal world, all these conversations about which proxies matter more than others would be moot non-starters because coding ability would stand for, well, coding ability. And that’s the world we’re building.

Thanks to Roman Rivilis for his help with data annotation for this post.

1For fun, we tried relating browser and operating system choice to interview performance, (smugly) expecting Chrome users to dominate. Not so. Browser choice didn’t matter, nor did what OS people used while interviewing.

2We got this data from looking at interviewees’ LinkedIn profiles.



You can’t fix diversity in tech without fixing the technical interview.

Posted on November 2nd, 2016.

In the last few months, several large players, including Google and Facebook, have released their latest and ultimately disappointing diversity numbers. Even with increased effort and resources poured into diversity hiring programs, Facebook’s headcount for women and people of color hasn’t really increased in the past 3 years. Google’s numbers have looked remarkably similar, and both players have yet to make significant impact in the space, despite a number of initiatives spanning everything from a points system rewarding recruiters for bringing in diverse candidates, to increased funding for tech education, to efforts to hire more diverse candidates in key leadership positions.

Why have gains in diversity hiring been so lackluster across the board?

Facebook justifies these disappointing numbers by citing the ubiquitous pipeline problem, namely that not enough people from underrepresented groups have access to the education and resources they need to be set up for success. And Google’s take appears to be similar, judging from what portion of their diversity-themed, forward-looking investments are focused on education.

In addition to blaming the pipeline, since Facebook’s and Google’s announcements, a growing flurry of conversations have loudly waxed causal about the real reason diversity hiring efforts haven’t worked. These have included everything from how diversity training isn’t sticky enough, to how work environments remain exclusionary and thereby unappealing to diverse candidates, to improper calibration of performance reviews to not accounting for how marginalized groups actually respond to diversity-themed messaging.

While we are excited that more resources are being allocated to education and inclusive workplaces, at, we posit another reason for why diversity hiring initiatives aren’t working. After drawing on data from thousands of technical interviews, it’s become clear to us that technical interviewing is a process whose results are nondeterministic and often arbitrary. We believe that technical interviewing is a broken process for everyone but that the flaws within the system hit underrepresented groups the hardest… because they haven’t had the chance to internalize just how much of technical interviewing is a numbers game. Getting a few interview invites here and there through increased diversity initiatives isn’t enough. It’s a beginning, but it’s not enough. It takes a lot of interviews to get used to the process and the format and to understand that the stuff you do in technical interviews isn’t actually the stuff you do at work every day. And it takes people in your social circle all going through the same experience, screwing up interviews here and there, and getting back on the horse to realize that poor performance in one interview isn’t predictive of whether you’ll be a good engineer.

A brief history of technical interviewing

A definitive work on the history of technical interviewing was surprisingly hard to find, but I was able to piece together a narrative by scouring books like How Would You Move Mount Fuji, Programming Interviews Exposed, and the bounty of the internets. The story goes something like this.

Technical interviewing has its roots as far back as 1950s Palo Alto, at Shockley Semiconductor Laboratories. Shockley’s interviewing methodology came out of a need to separate the innovative, rapidly moving, Cold War-fueled tech space from hiring approaches taken in more traditionally established, skills-based assembly-line based industry. And so, he relied on questions that could gauge analytical ability, intellect, and potential quickly. One canonical question in this category has to do with coins. You have 8 identical-looking coins, except one is lighter than the rest. Figure out which one it is with just two weighings on a pan balance.

The techniques that Shockley developed were adapted by Microsoft during the 90s, as the first dot-com boom spurred an explosion in tech hiring. As with the constraints imposed by both the volume and the high analytical/adaptability bar imposed by Shockley, Microsoft, too, needed to vet people quickly for potential — as software engineering became increasingly complex over the course of the dot-com boom, it was no longer possible to have a few centralized “master programmers” manage the design and then delegate away the minutiae. Even rank and file developers needed to be able to produce under a variety of rapidly evolving conditions, where just mastery of specific skills wasn’t enough.

The puzzle format, in particular, was easy to standardize because individual hiring managers didn’t have to come up with their own interview questions, and a company could quickly build up its own interchangeable question repository.

This mentality also applied to the interview process itself — rather than having individual teams run their own processes and pipelines, it made much more sense to standardize things. This way, in addition to questions, you could effectively plug and play the interviewers themselves — any interviewer within your org could be quickly trained up and assigned to speak with any candidate, independent of prospective team.

Puzzle questions were a good solution for this era for a different reason. Collaborative editing of documents didn’t become a thing until Google Docs’ launch in 2007. Without that capability, writing code in a phone interview was untenable — if you’ve ever tried to talk someone through how to code something up without at least a shared piece of paper in front of you, you know how painful it can be. In the absence of being able to write code in front of someone, the puzzle question was a decent proxy. Technology marched on, however, and its evolution made it possible to move from the proxy of puzzles to more concrete, coding-based interview questions. Around the same time, Google itself publicly overturned the efficacy of puzzle questions.

So where does this leave us? Technical interviews are moving in the direction of more concreteness, but they are still very much a proxy for the day-to-day work that a software engineer actually does. The hope was that the proxy would be decent enough, but it was always understood that that’s what they were and that the cost-benefit of relying on a proxy worked out in cases where problem solving trumped specific skills and where the need for scale trumped everything else.

As it happens, elevating problem-solving ability and the need for a scalable process are both eminently reasonable motivations. But here’s the unfortunate part: the second reason, namely the need for scalability, doesn’t apply in most cases. Very few companies are large enough to need plug and play interviewers. But coming up with interview questions and processes is really hard, so despite their differing needs, smaller companies often take their cues from the larger players, not realizing that companies like Google are successful at hiring because the work they do attracts an assembly line of smart, capable people… and that their success at hiring is often despite their hiring process and not because of it. So you end up with a de facto interviewing cargo cult, where smaller players blindly mimic the actions of their large counterparts and blindly hope for the same results.

The worst part is that these results may not even be repeatable… for anyone. To show you what I mean, I’ll talk a bit about some data we collected at

Technical interviewing is broken for everybody

Interview outcomes are kind of arbitrary is a platform where people can practice technical interviewing anonymously and, in the process, find jobs. Interviewers and interviewees meet in a collaborative coding environment and jump right into a technical interview question. After each interview, both sides rate one another, and interviewers rate interviewees on their technical ability. And the same interviewee can do multiple interviews, each of which is with a different interviewer and/or different company, and this opens the door for some interesting and somewhat controlled comparative analysis.

We were curious to see how consistent the same interviewee’s performance was from interview to interview, so we dug into our data. After looking at thousands of interviews on the platform, we’ve discovered something alarming: interviewee performance from interview to interview varied quite a bit, even for people with a high average performance. In the graph below, every represents the mean technical score for an individual interviewee who has done 2 or more interviews on The y-axis is standard deviation of performance, so the higher up you go, the more volatile interview performance becomes.

As you can see, roughly 25% of interviewees are consistent in their performance, but the rest are all over the place. And over a third of people with a high mean (>=3) technical performance bombed at least one interview.

Despite the noise, from the graph above, you can make some guesses about which people you’d want to interview. However, keep in mind that each person above represents a mean. Let’s pretend that, instead, you had to make a decision based on just one data point. That’s where things get dicey. Looking at this data, it’s not hard to see why technical interviewing is often perceived as a game. And, unfortunately, it’s a game where people often can’t tell how they’re doing.

No one can tell how they’re doing
I mentioned above that on, we collect post-interview feedback. In addition to asking interviewers how their candidates did, we also ask interviewees how they think they did. Comparing those numbers for each interview showed us something really surprising: people are terrible at gauging their own interview performance, and impostor syndrome is particularly prevalent. In fact, people underestimate their performance over twice as often as they overestimate it. Take a look at the graph below to see what I mean:

Note that, in our data, impostor syndrome knows no gender or pedigree — it hits engineers on our platform across the board, regardless of who they are or where they come from.

Now here’s the messed up part. During the feedback step that happens after each interview, we ask interviewees if they’d want to work with their interviewer. As it turns out, there’s a very strong relationship between whether people think they did well and whether they would indeed want to work with the interviewer — when people think they did poorly, even if they actually didn’t, they may be a lot less likely to want to work with you. And, by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did.

As a result, companies are losing candidates from all walks of life because of a fundamental flaw in the process.

Poor performances hit marginalized groups the hardest
Though impostor syndrome appears to hit engineers from all walks of life, we’ve found that women get hit the hardest in the face of an actually poor performance. As we learned above, poor performances in technical interviewing happen to most people, even people who are generally very strong. However, when we looked at our data, we discovered that after a poor performance, women are 7 times more likely to stop practicing than men:

A bevy of research appears to support confidence-based attrition as a very real cause for women departing from STEM fields, but I would expect that the implications of the attrition we witnessed extend beyond women to underrepresented groups, across the board.

What the real problem is

At the end of the day, because technical interviewing is indeed a game, like all games, it takes practice to improve. However, unless you’ve been socialized to expect and be prepared for the game-like aspect of the experience, it’s not something that you can necessarily intuit. And if you go into your interviews expecting them to be indicative of your aptitude at the job, which is, at the outset, not an unreasonable assumption, you will be crushed the first time you crash and burn. But the process isn’t a great or predictable indicator of your aptitude. And on top of that, you likely can’t tell how you’re doing even when you do well.

These are issues that everyone who’s gone through the technical interviewing gauntlet has grappled with. But not everyone has the wherewithal or social support to realize that the process is imperfect and to stick with it. And the less people like you are involved, whether it’s because they’re not the same color as you or the same gender or because not a lot of people at your school study computer science or because you’re a dropout or for any number of other reasons, the less support or insider knowledge or 10,000 foot view of the situation you’ll have. Full stop.

Inclusion and education isn’t enough

To help remedy the lack of diversity in its headcount, Facebook has committed to three actionable steps on varying time frames. The first step revolves around creating a more inclusive interview/work environment for existing candidates. The other two are focused on addressing the perceived pipeline problem in tech:

  • Short Term: Building a Diverse Slate of Candidates and an Inclusive Working Environment
  • Medium Term: Supporting Students with an Interest in Tech
  • Long Term: Creating Opportunity and Access

Indeed, efforts to promote inclusiveness and increased funding for education are extremely noble, especially in the face of potentially not being able to see results for years in the case of the latter. However, both take a narrow view of the problem and both continue to funnel candidates into a broken system.

Erica Baker really cuts to the heart of it in her blog post about Twitter hiring a head of D&I:

“What irks me the most about this is that no company, Twitter or otherwise, should have a VP of Diversity and Inclusion. When the VP of Engineering… is thinking about hiring goals for the year, they are not going to concern themselves with the goals of the VP of Diversity and Inclusion. They are going to say ‘hiring more engineers is my job, worrying about the diversity of who I hire is the job of the VP of Diversity and Inclusion.’ When the VP of Diversity and Inclusion says ‘your org is looking a little homogenous, do something about it,’ the VP of Engineering won’t prioritize that because the VP of Engineering doesn’t report to the VP of Diversity and Inclusion, so knows there usually isn’t shit the VP of Diversity and Inclusion can do if the Eng org doesn’t see some improvement in diversity.”

Indeed, this is sad, but true. When faced with a high-visibility conundrum like diversity hiring, a pragmatic and even reasonable reaction on any company’s part is to make a few high-profile hires and throw money at the problem. Then, it looks like you’re doing something, and spinning up a task force or a department or new set of titles is a lot easier than attempting to uproot the entire status quo.

As such, we end up with a newly minted, well-funded department pumping a ton of resources into feeding people who’ve not yet learned about the interviewing being a game into a broken, nondeterministic machine of a process made further worse by the fact that said process favors confidence and persistence over bona fide ability… and where the link between success in navigating said process and subsequent on-the-job performance is tenuous at best.

How to fix things

In the evolution of the technical interview, we saw a gradual reduction in the need for proxies as companies as the technology to write code together remotely emerged; with its advent, abstract, largely arbitrary puzzle questions could start to be phased out.

What’s the next step? Technology has the power to free us from relying on proxies, so that we can look at each individual as an indicative, unique bundle of performance-based data points. At, we make it possible to move away from proxies by looking at each interviewee as a collection of data points that tell a story, rather than one arbitrary glimpse of something they did once.

But that’s not enough either. Interviews themselves need to continue to evolve. The process itself needs to be repeatable, predictive of aptitude at the actual job, and not a system to be gamed, where a huge benefit is incurred by knowing the rules. And the larger organizations whose processes act as a template for everyone else need to lead the charge. Only then can we really be welcoming to a truly diverse group of candidates.



After a lot more data, technical interview performance really is kind of arbitrary.

Posted on October 13th, 2016. is a platform where people can practice technical interviewing anonymously, and if things go well, get jobs at top companies in the process. We started it because resumes suck and because we believe that anyone, regardless of how they look on paper, should have the opportunity to prove their mettle.

In February of 2016, we published a post about how people’s technical interview performance, from interview to interview, seemed quite volatile. At the time, we just had a few hundred interviews to draw on, so as you can imagine, we were quite eager to rerun the numbers with the advent of more data. After drawing on over a thousand interviews, the numbers hold up. In other words, technical interview outcomes do really seem to be kind of arbitrary.

The setup

When an interviewer and an interviewee match on, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question. After each interview, people leave one another feedback, and each party can see what the other person said about them once they both submit their reviews.

After every interview, interviewers rate interviewees on a few different dimensions, including technical ability. Technical ability gets rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!” (you can see the feedback form here). On our platform, a score of 3 or above has generally meant that the person was good enough to move forward.

At this point, you might say, that’s nice and all, but what’s the big deal? Lots of companies collect this kind of data in the context of their own pipelines. Here’s the thing that makes our data special: the same interviewee can do multiple interviews, each of which is with a different interviewer and/or different company, and this opens the door for some pretty interesting and somewhat controlled comparative analysis.

Performance from interview to interview really is arbitrary

If you’ve read our first post on this subject, you’ll recognize the visualization below. For the as yet uninitiated, every represents the mean technical score for an individual interviewee who has done 2 or more interviews on the platform. The y-axis is standard deviation of performance, so the higher up you go, the more volatile interview performance becomes. If you hover over each , you can drill down and see how that person did in each of their interviews. Anytime you see bolded text with a dotted underline, you can hover over it to see relevant data viz. Try it now to expand everyone’s performance. You can also hover over the labels along the x-axis to drill into the performance of people whose means fall into those buckets.

Standard Dev vs. Mean of Interviewee Performance
(1316 Interviews w/ 259 Interviewees)

As you can see, roughly 20% of interviewees are consistent in their performance (down from 25% the last time we did this analysis), and the rest are all over the place. If you look at the graph above, despite the noise, you can probably make some guesses about which people you’d want to interview. However, keep in mind that each represents a mean. Let’s pretend that, instead, you had to make a decision based on just one data point. That’s where things get dicey.1 For instance:

  • Many people who scored at least one 4 also scored at least one 2.
  • And as you saw above, a good amount of people who scored at least one 4 also scored at least one 1.
  • If we look at high performers (mean of 3.3 or higher), we still see a fair amount of variation.
  • Things get really murky when we consider “average” performers (mean between 2.6 and 3.3).

What do the most volatile interviewees have in common?

In the plot below, you can see interview performance over time for interviewees with the highest standard deviations on the platform (the cutoff we used was a standard dev of 1 or more, and this accounted for roughly 12% of our users). Note that the mix of dashed and dotted lines is purely visual — this way it’s easier to follow each person’s performance path.

So, what do the most highly volatile performers have in common? The answer appears to be, well, nothing. About half were working at top companies while interviewing, and half weren’t. Breakdown of top school was roughly 60/40. And years of experience didn’t have much to do with it either — a plurality of interviewees having between 2 and 6 years of experience, with the rest all over the board (varying between 1 and 20 years).

So, all in all, the factors that go into performance volatility are likely a lot more nuanced than the traditional cues we often use to make value judgments about candidates.

Why does volatility matter?

I discussed the implications of these findings for technical hiring at length in the last post, but briefly, a noisy, non-deterministic interview process does no favors to either candidates or companies. Both end up expending a lot more effort to get a lot less signal than they ought, and in a climate where software engineers are at such a premium, noisy interviews only serve to exacerbate the problem.

But beyond micro and macro inefficiencies, I suspect there’s something even more insidious and unfortunate going on here. Once you’ve done a few traditional technical interviews, the volatility and lack of determinism in the process is something you figure out anecdotally and kind of accept. And if you have the benefit of having friends who’ve also been through it, it only gets easier. What if you don’t, however?

In a previous post, we talked about how women quit interview practice 7 times more often than men after just one bad interview. It’s not too much of a leap to say that this is probably happening to any number of groups who are underrepresented/underserved by the current system. In other words, though it’s a broken process for everyone, the flaws within the system hit these groups the hardest… because they haven’t had the chance to internalize just how much of technical interviewing is a game. More on this subject in our next post!

What can we do about it?

So, yes, the state of technical hiring isn’t great right now, but here’s what we can say. If you’re looking for a job, the best piece of advice we can give you is to really internalize that interviewing is a numbers game. Between the kind of volatility we discussed in this post, impostor syndrome, poor evaluation techniques, and how hard it can be to get meaningful, realistic practice, it takes a lot of interviews to find a great job.

And if you’re hiring people, in the absence of a radical shift in how we vet technical ability, we’ve learned that drawing on aggregate performance is much more meaningful than a making such an important decision based on one single, arbitrary interview. Not only can aggregative performance help correct for an uncharacteristically poor performance, but it can also weed out people who eventually do well in an interview by chance or those who, over time, simply up and memorize Cracking the Coding Interview. At, even after just a handful of interviews, we have a much better picture of what someone is capable of and where they stack up than a single company would after a single interview, and aggregate data tells a much more compelling, repeatable story than one, arbitrary data point.

1At this point you might say that it’s erroneous and naive to compare raw technical scores to one another for any number of reasons, not the least of which is that one interviewer’s 4 is another interviewer’s 2. For a comprehensive justification of using raw scores comparatively, please check out the appendix to our previous post on this subject. Just to make sure the numbers hold up, I reran them, and this time, our R-squared is even higher than before (0.41 vs. 0.39 last time).

Huge thanks to Ian Johnson, creator of d3 Building Blocks, who made the graph entitled Standard Dev vs. Mean of Interviewee Performance (the one with the icons) as well as all the visualizations that go with it.



People are still bad at gauging their own interview performance. Here’s the data.

Posted on September 8th, 2016. is a platform where people can practice technical interviewing anonymously, and if things go well, get jobs at top companies in the process. We started it because resumes suck and because we believe that anyone, regardless of how they look on paper, should have the opportunity to prove their mettle.

At the end of 2015, we published a post about how people are terrible at gauging their own interview performance. At the time, we just had a few hundred interviews to draw on, so as you can imagine, we were quite eager to rerun the numbers with the advent of more data. After drawing on roughly one thousand interviews, we were surprised to find that the numbers have really held up, and that people continue to be terrible at gauging their own interview performance.

The setup

When an interviewer and an interviewee match on, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question (feel free to watch this process in action on our recordings page).  After each interview, people leave one another feedback, and each party can see what the other person said about them once they both submit their reviews.

If you’re curious, you can see what the feedback forms look like below — in addition to one direct yes/no question, we also ask about a few different aspects of interview performance using a 1-4 scale. We also ask interviewees some extra questions that we don’t share with their interviewers, and one of those questions is about how well they think they did. For context, a technical score of 3 or above seems to be the rough cut-off for hirability.

Feedback form for interviewers

Feedback form for interviewers

Feedback form for interviewees

Feedback form for interviewees

Perceived versus actual performance… revisited

Below are two heatmaps of perceived vs. actual performance per interview (for interviews where we had both pieces of data). In each heatmap, the darker areas represent higher interview concentration. For instance, the darkest square represents interviews where both perceived and actual performance was rated as a 3. You can hover over each square to see the exact interview count (denoted by “z”).

The first heatmap is our old data:

And the second heatmap is our data as of August 2016:

As you can see, even with the advent of a lot more interviews, the heatmaps look remarkably similar. The R-squared for a linear regression on the first data set is 0.24. And for the more recent data set, it’s dropped to 0.18. In both cases, even though some small positive relationship between actual and perceived performance does exist, it is not a strong, predictable correspondence.

You can also see there’s a non-trivial amount of impostor syndrome going on in the graph above, which probably comes as no surprise to anyone who’s been an engineer. Take a look at the graph below to see what I mean.

The x-axis is the difference between actual and perceived performance, i.e. actual minus perceived. In other words, a negative value means that you overestimated your performance, and a positive one means that you underestimated it. Therefore, every bar above 0 is impostor syndrome country, and every bar below zero belongs to its foulsome, overconfident cousin, the Dunning-Kruger effect.1

On (though I wouldn’t be surprised if this finding extrapolated to the qualified engineering population at large), impostor syndrome plagues interviewees roughly twice as often as Dunning-Kruger. Which, I guess, is better than the alternative.

Why people underestimate their performance

With all this data, I couldn’t resist digging into interviews where interviewees gave themselves 1’s and 2’s but where interviewers gave them 4’s to try to figure out if there were any common threads. And, indeed, a few trends emerged. The interviews that tended to yield the most interviewee impostor syndrome were ones where question complexity was layered. In other words, the interviewer would start with a fairly simple question and then, when the interviewee completed it successfully, they would change things up to make it harder. Lather, rinse, repeat. In some cases, an interviewer could get through up to 4 layered tiers in about an hour. Inevitably, even a good interviewee will hit a wall eventually, even if the place where it happens is way further out than the boundary for most people who attempt the same question.

Another trend I observed had to do with interviewees beating themselves up for issues that mattered a lot to them but fundamentally didn’t matter much to their interviewer: off-by-one errors, small syntax errors that made it impossible to compile their code (even though everything was semantically correct), getting big-O wrong the first time and then correcting themselves, and so on.

Interestingly enough, how far off people were in gauging their own performance was independent of how highly rated (overall) their interviewer was or how strict their interviewer was.

With that in mind, if I learned anything from watching these interviews, it was this. Interviewing is a flawed, human process. Both sides want to do a good job, but sometimes the things that matter to each side are vastly different. And sometimes the standards that both sides hold themselves to are vastly different as well.

Why this (still) matters for hiring, and what you can do to make it better

Techniques like layered questions are important to sussing out just how good a potential candidate is and can make for a really engaging positive experience, so removing them isn’t a good solution. And there probably isn’t that much you can do directly to stop an engineer from beating themselves up over a small syntax error (especially if it’s one the interviewer didn’t care about). However, all is not lost!

As you recall, during the feedback step that happens after each interview, we ask interviewees if they’d want to work with their interviewer. As it turns out, there’s a very statistically significant relationship between whether people think they did well and whether they’d want to work with the interviewer. This means that when people think they did poorly, they may be a lot less likely to want to work with you. And by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did.

How can one mitigate these losses? Give positive, actionable feedback immediately (or as soon as possible)! This way people don’t have time to go through the self-flagellation gauntlet that happens after a perceived poor performance, followed by the inevitable rationalization that they totally didn’t want to work there anyway.

1I’m always terrified of misspelling “Dunning-Kruger” and not double-checking it because of overconfidence in my own spelling abilities.



We built voice modulation to mask gender in technical interviews. Here’s what happened.

Posted on June 29th, 2016. is a platform where people can practice technical interviewing anonymously and, in the process, find jobs based on their interview performance rather than their resumes. Since we started, we’ve amassed data from thousands of technical interviews, and in this blog, we routinely share some of the surprising stuff we’ve learned. In this post, I’ll talk about what happened when we built real-time voice masking to investigate the magnitude of bias against women in technical interviews. In short, we made men sound like women and women sound like men and looked at how that affected their interview performance. We also looked at what happened when women did poorly in interviews, how drastically that differed from men’s behavior, and why that difference matters for the thorny issue of the gender gap in tech.

The setup

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question. Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from a mix of large companies like Google, Facebook, Twitch, and Yelp, as well as engineering-focused startups like Asana, Mattermark, and others. For more context, some examples of interviews done on the platform can be found on our public recordings page.

After every interview, interviewers rate interviewees on a few different dimensions.

Feedback form for interviewers

Feedback form for interviewers

As you can see, we ask the interviewer if they would advance their interviewee to the next round. We also ask about a few different aspects of interview performance using a 1-4 scale. On our platform, a score of 3 or above is generally considered good.

Women historically haven’t performed as well as men…

One of the big motivators to think about voice masking was the increasingly uncomfortable disparity in interview performance on the platform between men and women1. At that time, we had amassed over a thousand interviews with enough data to do some comparisons and were surprised to discover that women really were doing worse. Specifically, men were getting advanced to the next round 1.4 times more often than women. Interviewee technical score wasn’t faring that well either — men on the platform had an average technical score of 3 out of 4, as compared to a 2.5 out of 4 for women.

Despite these numbers, it was really difficult for me to believe that women were just somehow worse at computers, so when some of our customers asked us to build voice masking to see if that would make a difference in the conversion rates of female candidates, we didn’t need much convincing.

… so we built voice masking

Since we started working on, in order to achieve true interviewee anonymity, we knew that hiding gender would be something we’d have to deal with eventually but put it off for a while because it wasn’t technically trivial to build a real-time voice modulator. Some early ideas included sending female users a Bane mask.

Early voice masking prototype

Early voice masking prototype (drawing by Marcin Kanclerz)

When the Bane mask thing didn’t work out, we decided we ought to build something within the app, and if you play the videos below, you can get an idea of what voice masking on sounds like. In the first one, I’m talking in my normal voice.

And in the second one, I’m modulated to sound like a man.2

Armed with the ability to hide gender during technical interviews, we were eager to see what the hell was going on and get some insight into why women were consistently underperforming.

The experiment

The setup for our experiment was simple. Every Tuesday evening at 7 PM Pacific, hosts what we call practice rounds. In these practice rounds, anyone with an account can show up, get matched with an interviewer, and go to town. And during a few of these rounds, we decided to see what would happen to interviewees’ performance when we started messing with their perceived genders.

In the spirit of not giving away what we were doing and potentially compromising the experiment, we told both interviewees and interviewers that we were slowly rolling out our new voice masking feature and that they could opt in or out of helping us test it out. Most people opted in, and we informed interviewees that their voice might be masked during a given round and asked them to refrain from sharing their gender with their interviewers. For interviewers, we simply told them that interviewee voices might sound a bit processed.

We ended up with 234 total interviews (roughly 2/3 male and 1/3 female interviewees), which fell into one of three categories:

  • Completely unmodulated (useful as a baseline)
  • Modulated without pitch change
  • Modulated with pitch change

You might ask why we included the second condition, i.e. modulated interviews that didn’t change the interviewee’s pitch. As you probably noticed, if you played the videos above, the modulated one sounds fairly processed. The last thing we wanted was for interviewers to assume that any processed-sounding interviewee must summarily have been the opposite gender of what they sounded like. So we threw that condition in as a further control.

The results

After running the experiment, we ended up with some rather surprising results. Contrary to what we expected (and probably contrary to what you expected as well!), masking gender had no effect on interview performance with respect to any of the scoring criteria (would advance to next round, technical ability, problem solving ability). If anything, we started to notice some trends in the opposite direction of what we expected: for technical ability, it appeared that men who were modulated to sound like women did a bit better than unmodulated men and that women who were modulated to sound like men did a bit worse than unmodulated women. Though these trends weren’t statistically significant, I am mentioning them because they were unexpected and definitely something to watch for as we collect more data.

On the subject of sample size, we have no delusions that this is the be-all and end-all of pronouncements on the subject of gender and interview performance. We’ll continue to monitor the data as we collect more of it, and it’s very possible that as we do, everything we’ve found will be overturned. I will say, though, that had there been any staggering gender bias on the platform, with a few hundred data points, we would have gotten some kind of result. So that, at least, was encouraging.

So if there’s no systemic bias, why are women performing worse?

After the experiment was over, I was left scratching my head. If the issue wasn’t interviewer bias, what could it be? I went back and looked at the seniority levels of men vs. women on the platform as well as the kind of work they were doing in their current jobs, and neither of those factors seemed to differ significantly between groups. But there was one nagging thing in the back of my mind. I spend a lot of my time poring over interview data, and I had noticed something peculiar when observing the behavior of female interviewees. Anecdotally, it seemed like women were leaving the platform a lot more often than men. So I ran the numbers.

What I learned was pretty shocking. As it happens, women leave roughly 7 times as often as men after they do badly in an interview. And the numbers for two bad interviews aren’t much better. You can see the breakdown of attrition by gender below (the differences between men and women are indeed statistically significant with P < 0.00001).

Also note that as much as possible, I corrected for people leaving the platform because they found a job (practicing interviewing isn’t that fun after all, so you’re probably only going to do it if you’re still looking), were just trying out the platform out of curiosity, or they didn’t like something else about their experience.

A totally speculative thought experiment

So, if these are the kinds of behaviors that happen in the microcosm, how much is applicable to the broader world of software engineering? Please bear with me as I wax hypothetical and try to extrapolate what we’ve seen here to our industry at large. And also, please know that what follows is very speculative, based on not that much data, and could be totally wrong… but you gotta start somewhere.

If you consider the attrition data points above, you might want to do what any reasonable person would do in the face of an existential or moral quandary, i.e. fit the data to a curve. An exponential decay curve seemed reasonable for attrition behavior, and you can see what I came up with below. The x-axis is the number of what I like to call “attrition events”, namely things that might happen to you over the course of your computer science studies and subsequent career that might make you want to quit. The y-axis is what portion of people are left after each attrition event. The red curve denotes women, and the blue curve denotes men.

Now, as I said, this is pretty speculative, but it really got me thinking about what these curves might mean in the broader context of women in computer science. How many “attrition events” does one encounter between primary and secondary education and entering a collegiate program in CS and then starting to embark on a career? So, I don’t know, let’s say there are 8 of these events between getting into programming and looking around for a job. If that’s true, then we need 3 times as many women studying computer science than men to get to the same number in our pipelines. Note that that’s 3 times more than men, not 3 times more than there are now. If we think about how many there are now, which, depending on your source, is between 1/3 and a 1/4 of the number of men, to get to pipeline parity, we actually have to increase the number of women studying computer science by an entire order of magnitude.

Prior art, or why maybe this isn’t so nuts after all

Since gathering these findings and starting to talk about them a bit in the community, I began to realize that there was some supremely interesting academic work being done on gender differences around self-perception, confidence, and performance. Some of the work below found slightly different trends than we did, but it’s clear that anyone attempting to answer the question of the gender gap in tech would be remiss in not considering the effects of confidence and self-perception in addition to the more salient matter of bias.

In a study investigating the effects of perceived performance to likelihood of subsequent engagement, Dunning (of Dunning-Kruger fame) and Ehrlinger administered a scientific reasoning test to male and female undergrads and then asked them how they did. Not surprisingly, though there was no difference in performance between genders, women underrated their own performance more often than men. Afterwards, participants were asked whether they’d like to enter a Science Jeopardy contest on campus in which they could win cash prizes. Again, women were significantly less likely to participate, with participation likelihood being directly correlated with self-perception rather than actual performance.3

In a different study, sociologists followed a number of male and female STEM students over the course of their college careers via diary entries authored by the students. One prevailing trend that emerged immediately was the difference between how men and women handled the “discovery of their [place in the] pecking order of talent, an initiation that is typical of socialization across the professions.” For women, realizing that they may no longer be at the top of the class and that there were others who were performing better, “the experience [triggered] a more fundamental doubt about their abilities to master the technical constructs of engineering expertise [than men].”

And of course, what survey of gender difference research would be complete without an allusion to the wretched annals of dating? When I told the team about the disparity in attrition between genders, the resounding response was along the lines of, “Well, yeah. Just think about dating from a man’s perspective.” Indeed, a study published in the Archives of Sexual Behavior confirms that men treat rejection in dating very differently than women, even going so far as to say that men “reported they would experience a more positive than negative affective response after… being sexually rejected.”

Maybe tying coding to sex is a bit tenuous, but, as they say, programming is like sex — one mistake and you have to support it for the rest of your life.

Why I’m not depressed by our results and why you shouldn’t be either

Prior art aside, I would like to leave off on a high note. I mentioned earlier that men are doing a lot better on the platform than women, but here’s the startling thing. Once you factor out interview data from both men and women who quit after one or two bad interviews, the disparity goes away entirely. So while the attrition numbers aren’t great, I’m massively encouraged by the fact that at least in these findings, it’s not about systemic bias against women or women being bad at computers or whatever. Rather, it’s about women being bad at dusting themselves off after failing, which, despite everything, is probably a lot easier to fix.

1Roughly 15% of our users are female. We want way more, but it’s a start.

2If you want to hear more examples of voice modulation or are just generously down to indulge me in some shameless bragging, we got to demo it on NPR and in Fast Company.

3In addition to asking interviewers how interviewees did, we also ask interviewees to rate themselves. After reading the Dunning and Ehrlinger study, we went back and checked to see what role self-perception played in attrition. In our case, the answer is, I’m afraid, TBD, as we’re going to need more self-ratings to say anything conclusive.



A founder’s guide to making your first recruiting hire

Posted on April 26th, 2016.

Recently, a number of founder friends have asked me about how to approach their first recruiting hire, and I’ve found myself repeating the same stuff over and over again. Below are some of my most salient thoughts on the subject. Note that I’ll be talking a lot about engineering hiring because that’s what I know, but I expect a lot of this applies to other fields as well, especially ones where the demand for labor outstrips supply.

Don’t get caught up by flashy employment history; hustle trumps brands

At first glance, hiring someone who’s done recruiting for highly successful tech giants seems like a no-brainer. Google and Facebook are good at hiring great engineers, right? So why not bring in someone who’s had a hand in that success?

There are a couple of reasons why hiring straight out of the Googles and Facebooks of the world isn’t necessarily the best idea. First off, if you look at a typical recruiter’s employment history, you’re going to see a lot of short stints. Very likely this means that they were working as a contractor. While there’s nothing wrong with contract recruiting, per se, large companies often hire contract recruiters in batches, convert the best performers to full-time hires, and ditch the rest.1 That said, some of the best recruiters I know started out at Google. But I am inclined to believe they are exceptions.

The second and much more important reason not to blindly hire out of tech giants is the importance of scrappiness and hustle in this hire. If you work as a recruiter at Google, you’re basically plugged into the matrix. You have a readymade suite of tools that make it much easier to be successful. You have a database of candidates who have previously interviewed that spans a huge portion of the engineering population. Email discovery is easier. Reaching out to people is easier because you have templates that have been proven to work to rely on. And you can lean on the Google brand as a crutch. Who hasn’t been, at one point in their career, flattered by an email from a Google recruiter? As a result, if you’re sending these emails, you don’t have to go out of your way to write personal messages or to convince people that your company is cool and interesting and worth their time. You get that trust for free.

Contrast this setup with being the very first person in the recruiting org. You have no tools. You have no templates. You probably have no brand. You probably have, well, jack shit. You need someone who’s going to think critically about tooling and balance the need for tooling with a shoestring budget, especially in a space where most tooling has a price tag of at least $1K per month. You’re going to need someone whose methods are right for your particular situation rather than someone who does things because that’s just how they’ve always been done. You probably want someone who realizes that paying for a LinkedIn Recruiter seat is a huge fucking waste of money and that sourcing on LinkedIn, in general, is a black hole-level time suck. You want someone who is good at engaging with candidates independently of brand sparkle, which likely means someone who understands the value of personalization in their sourcing efforts. You want someone who compensates for your relatively unknown status with great candidate experience during your interview process. You want someone who won’t just blindly pay tens of thousands of dollars for career fair real estate because that’s just what you do, even though the only companies who get ROI on career fair attendance are ones with preexisting brands. And, apropos, you want someone who can start building a sparkly brand for you from day one because building a brand takes time. (More on brand-building in the last two sections on marketing chops and evangelism.)

Sales chops are hugely important, and you can test for those

People often ask me if having an engineering background is important for technical recruiters. My answer to that is always, “Yes, but.” Yes, it’s useful, but the main reason it’s useful is that it helps build credibility and rapport with candidates. A good salesperson can do that without all the trappings of engineering experience. To put it another way, at the end of the day, this is a sales job. Great engineers who are shitty salespeople will not do well at recruiting. Great salespeople with no engineering background will likely do well.

So, how can you test for sales aptitude? If the candidate is currently an in-house recruiter somewhere, I ask them to pitch me on the company’s product. If they’re an agency recruiter, I ask them to pitch me on one of their clients’ products. Most recruiters do a decent job of pitching the company as a good place to work, but unfortunately, many don’t have a very good understanding of what their company actually does. Given that they’re the first point of contact for candidates, it’s really important to be able to answer basic questions about product-market fit, challenges (both product and engineering), how the company makes money, how much traction there is, what the competition looks like, and so on. Moreover, a lack of interest in something this basic points to a lack of intellectual curiosity in general, and in a technical recruiter, this is a very poor sign because such a huge portion of the job is picking up new concepts and being able to talk about them intelligently to very smart people.

You want someone who can write

I was on the fence about whether to include this section because it sounds kind of obvious, but writing well is important in this role for two reasons. First off, your recruiter is likely going to be the first point of contact with candidates. And if you’re an early-ish company without much brand, correspondence with the recruiter will likely be the first time a candidate ever hears of you. So, you probably want that interaction to shine. And the other reason you want someone who cares about narrative, spelling, and grammar is that they will be the arbiter of these abilities in future recruiting hires. Enough said.

One exercise I like to have candidates for this role go through is writing mock sourcing emails to people at your company, as if they were still at their previous position. This portion of the interview process is probably the best lens into what it’s actually like to work with the candidate. In particular, because candidates are not likely to have a clear idea of what they’re pitching yet, I try to make this part of the process iterative and emphasize that I welcome any number of questions about anything, whether it’s the company’s offering, what companies my firm works with, what certain parts of the engineers’ profiles mean, or anything in between. What questions people ask, how they ask them, and how they deal with the ambiguity inherent in this assignment is part of the evaluation, as is the caliber of the research they did on each mock email recipient.

You want someone with marketing chops

I talked a bit earlier about how you probably have no brand to speak of at this point. I can’t stress enough how much easier having a brand makes hiring. Until you have one, especially in this climate, you’re going to be fighting so fucking hard for every one-off hire. If you can, you ought to put effort into branding such that you end up in the enviable position of smart people coming to you.

So why don’t early-ish companies do this across the board? Brand building is a pain in the ass, it takes time, and not all of your outbound efforts are going to be measurable, which can make it harder to get others in your org to buy in. If you can find someone who’s had even a little bit of marketing experience, they’ll be able to identify channels to get the word out, use their preexisting network to help with outsource-able tasks, and get the ball rolling on things like hosting events, which, if you’ve never done before, can be quite intimidating.

And because recruiting doesn’t live in a vacuum and needs help from other teams to send something high-signal and genuine into the world, someone with some marketing experience will likely have an easier time getting other teams to buy in and put time and resources into this endeavor, which brings me to my next point.

You want someone who can fearlessly evangelize the importance of recruiting… and get you to take an active role even when you don’t feel like it

The harsh reality is that the primary reason companies hire their first recruiter is so that hiring can be taken off the plate of the founders. It’s tempting to have the “set it and forget it” mentality in a founder’s shoes — recruiters aren’t cheap, so presumably if you pay them enough, they’ll just deal with this pesky hiring thing, and then you can get back to work. I get it. Hiring isn’t that fun, and as a founder, despite having been a recruiter myself, there are definitely days when I just want to pay someone to, for the love of god, take this bullshit off my hands so I can get back to talking to users and selling and figuring out what to build next and all sorts of other things.

But it doesn’t work that way. If you’re a founder, no one can sell your vision as well as you. And all that experience you’ve built up that makes you a subject matter expert probably also makes you pretty good at filtering candidates. You might take a lot of what’s in your head for granted, but transferring that over into someone else’s brain is going to take time and iteration. And you can never really dissociate from hiring entirely because the moment you do, the culture of “hiring is just the domain of recruiting” is going to trickle down into your culture, and over time, it will cost you the best people.

In my recruiting days, at a high level, I saw two types of hiring cultures. One had the hiring managers and teams taking an active role, participating in sourcing, tweaking interview questions to make them engaging and reflective of the work, and taking time to hang out with candidates, even if they weren’t interviewing yet. The other type had the recruiting org be largely disjoint from the teams it was hiring for. In this type of setup, team members would view recruiting as a hassle/necessary evil that took them away from their regular job, and most of the remaining trappings of the hiring process would be left in the hands of recruiters alone.

You can guess which type of company ends up with an enviable interview process, a sweet blog, cool, hiring-themed easter eggs in their code, and a wistful, pervading, nose-pressed-against-the-glass refrain of “I wish I could work there”. And you can, in turn, guess which company demands a CS degree and 10 years of [insert recent language name here] experience in their job descriptions.

Despite these realities, founders and hiring managers often forget how critical their role in hiring is because they have a ton of everyday tasks on their plates. This is why having your recruiter be a fearless evangelist is so important. This person needs to cheerfully yet insistently remind the team and especially founders (who are, after all, the ones who dictate culture) that time spent on hiring is part of their jobs. This person needs to be able to march into the CEO’s office and demand that they go and give a talk somewhere or consistently block off time on their calendar every week to send some sourcing emails. Or that they need to write some stuff somewhere on the internet such that people start to realize that their company is a thing. Marching into a CEO’s office and making demands is tough. You need a person who will do this without trepidation and who will be able to convince you, even when the sky is falling, that a few hours a week spent on hiring are a good use of your time.

In addition to these points, all the usual thought points about hiring someone who’s going to be growing a team apply here. Is this person already a strong leader? If not, can they grow into one? Are they going to be able to attract other talent to their team? Are they someone you want around, fighting alongside you in the dark, for a long time to come? And, though in an ideal world I’d choose someone with experience who also meets the criteria I’ve outlined in this guide, if ultimately faced with a choice between experience and someone green with hustle, charisma, writing ability, and smarts, I’ll choose the latter every time.

1As an aside, this process is an unfortunate side effect of employment law meant to protect contractors from being exploited. The thinking is that by capping the length of time that someone can work as a contractor, you can exert pressure on the company to turn them into full-time hires who have to be given benefits. But as with many well-intentioned, regulatory pieces of legislation, that’s not really what happens in practice. The practical takeaway, though, is that if someone is great at recruiting, they’re probably not going to have a bunch of short contracting stints.



Technical interview performance is kind of arbitrary. Here’s the data.

Posted on February 17th, 2016.

Note: Though I wrote most of the words in this post, there are a few people outside of whose work made it possible. Ian Johnson, creator of d3 Building Blocks, created the graph entitled Standard Dev vs. Mean of Interviewee Performance (the one with the icons) as well as all the interactive visualizations that go with it. Dave Holtz did all the stats work for computing the probability of people failing individual interviews. You can see more about his work on his blog. is a platform where people can practice technical interviewing anonymously and, in the process, find jobs. In the past few months, we’ve amassed data from hundreds of interviews, and when we looked at how the same people performed from interview to interview, we were really surprised to find quite a bit of volatility, which, in turn, made us question the reliability of single interview outcomes.

The setup

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice1, text chat, and a whiteboard and jump right into a technical question. Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from a mix of large companies like Google, Facebook, and Yelp, as well as engineering-focused startups like Asana, Mattermark, KeepSafe, and more. If you’d like to see an interview an action, head over to our public recordings page for a few examples.

After every interview, interviewers rate interviewees on a few different dimensions, including technical ability. Technical ability gets rated on a scale of 1 to 4, where 1 is “meh” and 4 is “amazing!” (you can see the feedback form here). On our platform, a score of 3 or above has generally meant that the person was good enough to move forward.

At this point, you might say, that’s nice and all, but what’s the big deal? Lots of companies collect this kind of data in the context of their own pipelines. Here’s the thing that makes our data special: the same interviewee can do multiple interviews, each of which is with a different interviewer and/or different company, and this opens the door for some pretty interesting and somewhat controlled comparative analysis.

Performance from interview to interview is pretty volatile

Let’s start with some visuals. In the graph below, every represents the mean technical score for an individual interviewee who has done 2 or more interviews on the platform2. The y-axis is standard deviation of performance, so the higher up you go, the more volatile interview performance becomes. If you hover over each , you can drill down and see how that person did in each of their interviews. Anytime you see bolded text with a dotted underline, you can hover over it to see relevant data viz. Try it now to expand everyone’s performance. You can also hover over the labels along the x-axis to drill into the performance of people whose means fall into those buckets.

Standard Dev vs. Mean of Interviewee Performance
(299 Interviews w/ 67 Interviewees)

As you can see, roughly 25% of interviewees are consistent in their performance, and the rest are all over the place3. If you look at the graph above, despite the noise, you can probably make some guesses about which people you’d want to interview. However, keep in mind that each represents a mean. Let’s pretend that, instead, you had to make a decision based on just one data point. That’s where things get dicey. For instance:

  • Many people who scored at least one 4 also scored at least one 2.
  • If we look at high performers (mean of 3.3 or higher), we still see a fair amount of variation.
  • Things get really murky when we consider “average” performers (mean between 2.6 and 3.3).

To me, looking at this data and then pretending that I had to make a hiring decision based on one interview outcome felt a lot like peering into some beautiful, lavishly appointed parlor through a keyhole. Sometimes you see a piece of art on the wall, sometimes you see the liquor selection, and sometimes you just see the back of the couch.

At this point you might say that it’s erroneous and naive to compare raw technical scores to one another for any number of reasons, not the least of which is that one interviewer’s 4 is another interviewer’s 2. We definitely share this concern and address it in the appendix of this post. It does bear mentioning, though, that most of our interviewers are coming from companies with strong engineering brands and that correcting for brand strength didn’t change interviewee performance volatility, nor did correcting for interviewer rating.

So, in a real life situation, when you’re trying to decide whether to advance someone to onsite, you’re probably trying to avoid two things — false positives (bringing in people below your bar by mistake) and false negatives (rejecting people who should have made it in). Most top companies’ interviewing paradigm is that false negatives are less bad than false positives. This makes sense right? With a big enough pipeline and enough resources, even with a high false negative rate, you’ll still get the people you want. With a high false positive rate, you might get cheaper hiring, but you do potentially irreversible damage to your product, culture, and future hiring standards in the process. And of course, the companies setting the hiring standards and practices for an entire industry ARE the ones with the big pipelines and seemingly inexhaustible resources.

The dark side of optimizing for high false negative rates, though, rears its head in the form of our current engineering hiring crisis. Do single interview instances, in their current incarnation, give enough signal? Or amidst so much demand for talent, are we turning away qualified people because we’re all looking at a large, volatile graph through a tiny keyhole?

So, hyperbolic moralizing aside, given how volatile interview performance is, what are the odds that a good candidate will fail an individual phone screen?

Odds of failing a single interview based on past performance

Below, you can see the distribution of mean performance throughout our population of interviewees.

In order to figure out the probability that a candidate with a given mean score would fail an interview, we had to do some stats work. First, we broke interviewees up into cohorts based on their mean scores (rounded to the nearest 0.25). Then, for each cohort, we calculated the probability of failing, i.e. of getting a score of 2 or less. Finally, to work around our starting data set not being huge, we resampled our data. In our resampling procedure, we treated an interview outcome as a multinomial distribution, or in other words, pretended that each interview was a roll of a weighted, 4-sided die corresponding to that candidate’s cohort. We then re-rolled the dice a bunch of times to create a new, “simulated” dataset for each cohort and calculated new probabilities of failure for each cohort using these data sets. Below, you can see the results of repeating this process 10,000 times.

As you can see, a lot of the distributions above overlap with one another. This is important because these overlaps tell us that there may not be statistically significant differences between those groups (e.g. between 2.75 and 3). Certainly, with the advent of LOT more data, the delineations between cohorts may become clearer. On the other hand, if we do need a huge amount of data to detect differences in failure rate, it might suggest that people are intrinsically highly variable in their performance. At the end of the day, while we can confidently say that there is a significant difference between the bottom end of the spectrum (2.25) versus the top end (3.75), for people in the middle, things are murky.

Nevertheless, using these distributions, we did attempt to compute the probability that a candidate with a certain mean score would fail a single interview (see below — the shaded areas encapsulate a 95% confidence interval). The fact that people who are overall pretty strong (e.g. mean ~= 3) can mess up technical interviews as much as 22% of the time shows that there’s definitely room for improvement in the process, and this is further exacerbated by the general murkiness in the middle of the spectrum.

Is interviewing doomed?

Generally, when we think of interviewing, we think of something that ought to have repeatable results and carry a strong signal. However, the data we’ve collected, meager though it might be, tells a different story. And it resonates with both my anecdotal experience as a recruiter and with the sentiments we’ve seen echoed in the community. Zach Holman’s Startup Interviewing is Fucked hits on the disconnect between interview process and the job it’s meant to fill, the fine gentlemen of TripleByte reached similar conclusions by looking at their own data, and one of the more poignant expressions of inconsistent interviewing results recently came from

You can bet that many people who are rejected after a phone screen by Company A but do better during a different phone screen and ultimately end up somewhere traditionally reputable are getting hit up by Company A’s recruiters 6 months later. And despite everyone’s best efforts, the murky, volatile, and ultimately stochastic circle jerk of a recruitment process marches on.

So yes, it’s certainly one possible conclusion is that technical interviewing itself is indeed fucked and doesn’t provide a reliable, deterministic signal for one interview instance. Algorithmic interviews are a hotly debated topic and one we’re deeply interested in teasing apart. One thing in particular we’re very excited about is tracking interview performance as a function of interview type, as we get more and more different interviewing types/approaches happening on the platform. Indeed, one of our long-term goals is to really dig into our data, look at the landscape of different interview styles, and make some serious data-driven statements about what types of technical interviews lead to the highest signal.

In the meantime, however, I am leaning toward the idea that drawing on aggregate performance is much more meaningful than a making such an important decision based on one single, arbitrary interview. Not only can aggregative performance help correct for an uncharacteristically poor performance, but it can also weed out people who eventually do well in an interview by chance or those who, over time, submit to the beast and memorize Cracking the Coding Interview. I know it’s not always practical or possible to gather aggregate performance data in the wild, but at the very least, in cases where a candidate’s performance is borderline or where their performance differs wildly from what you’d expect, it might make sense to interview them one more time, perhaps focusing on slightly different material, before making the final decision.

Appendix: The part where we tentatively justify using raw scores for comparative performance analysis

For the skeptical, inquiring minds among you who realize that using raw coding scores to evaluate an interviewee has some pretty obvious problems, we’ve included this section. The issue is that even though our interviewers tend to come from companies with high engineering bars, raw scores are still comprised of just one piece of feedback, they don’t adjust for interviewer strictness (e.g. one interviewer’s 4 could be another interviewer’s 2), and they don’t adjust well to changes in skill over time. Internally, we actually use a more complex and comprehensive rating system when determining skill, and if we can show that raw scores align with the ratings we calculate, then we don’t feel so bad about using raw scores comparatively.

Our rating system works something like this:

  1. We create a single score for each interview based on a weighted average of each feedback item.
  2. For each interviewer, we pit all the interviewees they’ve interviewed against one another using this score.
  3. We use a Bayesian ranking system (a modified version of Glicko-2) to generate a rating for each interviewee based on the outcome of these competitions.

As a result, each person is only rated based on their score as it compares to other people who were interviewed by the same interviewer. That means one interviewer’s score is never directly compared to another’s, and so we can correct for the hairy issue of inconsistent interviewer strictness.

So, why am I bringing this up at all? You’re all smart people, and you can tell when someone is waving their hands around and pretending to do math. Before we did all this analysis, we wanted to make sure that we believed our own data. We’ve done a lot of work to build a ratings system we believe in, so we correlated that with raw coding scores to see how strong they are at determining actual skill.

These results are pretty strong. Not strong enough for us to rely on raw scores exclusively but strong enough to believe that raw scores are useful for determining approximate candidate strength.

1While listening to interviews day in and day out, I came up with a drinking game. Every time someone thinks the answer is hash table, take a drink. And every time the answer actually is hash table, take two drinks.4

2This is data as of January 2016, and there are only 299 interviews because not all interviews have enough feedback data and because we threw out everyone with less than 2 interviews. Moreover, one thing we don’t show in this graph is the passage of time, so you can see people’s performance over time — it’s kind of a hot mess.

3We were curious to see if volatility varied at all with people’s mean scores. In other words, were weaker players more volatile than strong ones? The answer is no — when we ran a regression on standard deviation vs. mean, we couldn’t come up with any meaningful relationship (R-squared ~= 0.03), which means that people are all over the place regardless of how strong they are on average.

4I almost died.

Thanks to Andrew Marsh for co-authoring the appendix, to Plotly for making a terrific graphing product, and to everyone who read drafts of this behemoth.



Engineers can’t gauge their own interview performance. And that makes them harder to hire.

Posted on December 15th, 2015. is an anonymous technical interviewing platform. We started it because resumes suck and because we believe that anyone, regardless of how they look on paper, should have the opportunity to prove their mettle. In the past few months, we’ve amassed over 600 technical interviews along with their associated data and metadata. Interview questions tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role at a top company, and interviewers typically come from a mix of larger companies like Google, Facebook, and Twitter, as well as engineering-focused startups like Asana, Mattermark, KeepSafe, and more.

Over the course of the next few posts, we’ll be sharing some { unexpected, horrifying, amusing, ultimately encouraging } things we’ve learned. In this blog’s heroic maiden voyage, we’ll be tackling people’s surprising inability to gauge their own interview performance and the very real implications this finding has for hiring.

First, a bit about setup

When an interviewer and an interviewee match on our platform, they meet in a collaborative coding environment with voice, text chat, and a whiteboard and jump right into a technical question. After each interview, people leave one another feedback, and each party can see what the other person said about them once they both submit their reviews. If both people find each other competent and pleasant, they have the option to unmask. Overall, interviewees tend to do quite well on the platform, with just under half of interviews resulting in a “yes” from the interviewer.

If you’re curious, we have a few public recordings of interviews done on the platform, so you can watch and see what an interview is really like. In addition to these, our feedback forms are attached below. There is one direct yes/no question, and we also ask about a few different aspects of interview performance using a 1-4 scale. We also ask interviewees some extra questions that we don’t share with their interviewers, and one of those questions is about how well they think they did. In this post, we’ll be focusing on the technical score an interviewer gives an interviewee and the interviewee’s self-assessment (both are circled below). For context, a technical score of 3 or above seems to be the rough cut-off for hirability.

Feedback form for interviewers

Feedback form for interviewers

Feedback form for interviewees

Feedback form for interviewees

Perceived versus actual performance

Below, you can see the distribution of people’s actual technical performance (as rated by their interviewers) and the distribution of their perceived performance (how they rated themselves) for the same set of interviews.1

You might notice right away that there is a little bit of disparity, but things get interesting when you plot perceived vs. actual performance for each interview. Below, is a heatmap of the data where the darker areas represent higher interview concentration. For instance, the darkest square represents interviews where both perceived and actual performance was rated as a 3. You can hover over each square to see the exact interview count (denoted by “z”).

If you run a regression on this data2, you get an R-squared of only 0.24, and once you take away the worst interviews, it drops down even further to a 0.16. For context, R-squared is a measurement of how well you can fit empirical data to some mathematical model. It’s on a scale from 0 to 1 with 0 meaning that everything is noise and 1 meaning that everything fits perfectly. In other words, even though some small positive relationship between actual and perceived performance does exist, it is not a strong, predictable correspondence.

You can also see there’s a non-trivial amount of impostor syndrome going on in the graph above, which probably comes as no surprise to anyone who’s been an engineer.

Gayle Laakmann McDowell of Cracking the Coding Interview fame has written quite a bit about how bad people are at gauging their own interview performance, and it’s something that I had noticed anecdotally when I was doing recruiting, so it was nice to see some empirical data on that front. In her writing, Gayle mentions that it’s the job of a good interviewer to make you feel like you did OK even if you bombed. I was curious about whether that’s what was going on here, but when I ran the numbers, there wasn’t any relationship between how highly an interviewer was rated overall and how off their interviewees’ self-assessments were, in one direction or the other.

Ultimately, this isn’t a big data set, and we will continue to monitor the relationship between perceived and actual performance as we host more interviews, but we did find that this relationship emerged very early on and has continued to persist with more and more interviews — R-squared has never exceeded 0.26 to date.

Why this matters for hiring

Now here’s the actionable and kind of messed up part. As you recall, during the feedback step that happens after each interview, we ask interviewees if they’d want to work with their interviewer. As it turns out, there’s a very statistically significant relationship (p < 0.0008) between whether people think they did well and whether they’d want to work with the interviewer. This means that when people think they did poorly, they may be a lot less likely to want to work with you3. And by extension, it means that in every interview cycle, some portion of interviewees are losing interest in joining your company just because they didn’t think they did well, despite the fact that they actually did.

How can one mitigate these losses? Give positive, actionable feedback immediately (or as soon as possible)! This way people don’t have time to go through the self-flagellation gauntlet that happens after a perceived poor performance, followed by the inevitable rationalization that they totally didn’t want to work there anyway.

Lastly, a quick shout-out to Statwing and Plotly for making terrific data analysis and graphing tools respectively.

1There are only 254 interviews represented here because not all interviews in our data set had comprehensive, mutual feedback. Moreover, we realize that raw scores don’t tell the whole story and will be focusing on standardization of these scores and the resulting rat’s nest in our next post. That said, though interviewer strictness does vary, we gate interviewers pretty heavily based on their background and experience, so the overall bar is high and comparable to what you’d find at a good company in the wild.

2Here we are referring to linear regression, and though we tried fitting a number of different curves to the data, they all sucked.

3In our data, people were 3 times less likely to want to work with their interviewers when they thought they did poorly.



Resumes suck. Here’s the data.

Posted on November 10th, 2014.

Note: This post is syndicated from Aline Lerner’s personal blog. Aline is the CEO and co-founder of, and results like these are what inspired her to start this company.

About a year ago, after looking at the resumes of engineers we had interviewed at TrialPay in 2012, I learned that the strongest signal for whether someone would get an offer was the number of typos and grammatical errors on their resume. On the other hand, where people went to school, their GPA, and highest degree earned didn’t matter at all. These results were pretty unexpected, ran counter to how resumes were normally filtered, and left me scratching my head about how good people are at making value judgments based on resumes, period. So, I decided to run an experiment.

In this experiment, I wanted to see how good engineers and recruiters were at resume-based candidate filtering. Going into it, I was pretty sure that engineers would do a much better job than recruiters. (They are technical! They don’t need to rely on proxies as much!) However, that’s not what happened at all. As it turned out, people were pretty bad at filtering resumes across the board, and after running the numbers, it began to look like resumes might not be a particularly effective filtering tool in the first place.


The setup was simple. I would:

  1. Take resumes from my collection.
  2. Remove all personally identifying info (name, contact info, dates, etc.).
  3. Show them to a bunch of recruiters and engineers.
  4. For each resume, ask just one question: Would you interview this candidate?

Essentially, each participant saw something like this:

If the participant didn’t want to interview the candidate, they’d have to write a few words about why. If they did want to interview, they also had the option of substantiating their decision, but, in the interest of not fatiguing participants, I didn’t require it.

To make judging easier, I told participants to pretend that they were hiring for a full-stack or back-end web dev role, as appropriate. I also told participants not to worry too much about the candidate’s seniority when making judgments and to assume that the seniority of the role matched the seniority of the candidate.

For each resume, I had a pretty good idea of how strong the engineer in question was, and I split resumes into two strength-based groups. To make this judgment call, I drew on my personal experience — most of the resumes came from candidates I placed (or tried to place) at top-tier startups. In these cases, I knew exactly how the engineer had done in technical interviews, and, more often than not, I had visibility into how they performed on the job afterwards. The remainder of resumes came from engineers I had worked with directly. The question was whether the participants in this experiment could figure out who was who just from the resume.

At this juncture, a disclaimer is in order. Certainly, someone’s subjective hirability based on the experience of one recruiter is not an oracle of engineering ability — with the advent of more data and more rigorous analysis, perhaps these results will be proven untrue. But, you gotta start somewhere. That said, here’s the experiment by the numbers.

  • I used a total of 51 resumes in this study. 64% belonged to strong candidates.
  • A total of 152 people participated in the experiment.
  • Each participant made judgments on 6 randomly selected resumes from the original set of 51, for a total of 716 data points1.

If you want to take the experiment for a whirl yourself, you can do so here.

Participants were broken up into engineers (both engineers involved in hiring and hiring managers themselves) and recruiters (both in-house and agency). There were 46 recruiters (22 in-house and 24 agency) and 106 engineers (20 hiring managers and 86 non-manager engineers who were still involved in hiring).


So, what ended up happening? Below, you can see a comparison of resume scores for both groups of candidates. A resume score is the average of all the votes each resume got, where a ‘no’ counted as 0 and a ‘yes’ vote counted as 1. The dotted line in each box is the mean for each resume group — you can see they’re pretty much the same. The solid line is the median, and the boxes contain the 2nd and 3rd quartiles on either side of it. As you can see, people weren’t very good at this task — what’s pretty alarming is that scores are all over the place, for both strong and less strong candidates.

Another way to look at the data is to look at the distribution of accuracy scores. Accuracy in this context refers to how many resumes people were able to tag correctly out of the subset of 6 that they saw. As you can see, results were all over the board.

On average, participants guessed correctly 53% of the time. This was pretty surprising, and at the risk of being glib, according to these results, when a good chunk of people involved in hiring make resume judgments, they might as well be flipping a coin.

What about performance broken down by participant group? Here’s the breakdown:

  • Agency recruiters – 56%
  • Engineers – 54%
  • In-house recruiters – 52%
  • Eng hiring managers – 48%

None of the differences between participant groups were statistically significant. In other words, all groups did equally poorly. For each group, you can see how well people did below.

To try to understand whether people really were this bad at the task or whether perhaps the task itself was flawed, I ran some more stats. One thing I wanted to understand, in particular, was whether inter-rater agreement was high. In other words, when rating resumes, were participants disagreeing with each other more often than you’d expect to happen by chance? If so, then even if my criteria for whether each resume belonged to a strong candidate wasn’t perfect, the results would still be compelling — no matter how you slice it, if people involved in hiring consistently can’t come to a consensus, then something about the task at hand is too ambiguous.

The test I used to gauge inter-rater agreement is called Fleiss’ kappa. The result is on the following scale of -1 to 1:

  • -1 perfect disagreement; no rater agrees with any other
  • 0 random; the raters might as well have been flipping a coin
  • 1 perfect agreement; the raters all agree with one another

Fleiss’ kappa for this data set was 0.13. 0.13 is close to zero, implying just mildly better than coin flip. In other words, the task of making value judgments based on these resumes was likely too ambiguous for humans to do well on with the given information alone.

TL;DR Resumes might actually suck.

Some interesting patterns

In addition to the finding out that people aren’t good at judging resumes, I was able to uncover a few interesting patterns.

Times didn’t matter
We’ve all heard of and were probably a bit incredulous about the study that showed recruiters spend less than 10 seconds on a resume on average. In this experiment, people took a lot longer to make value judgments. People took a median of 1 minute and 40 seconds per resume. In-house recruiters were fastest, and agency recruiters were slowest. However, how long someone spent looking at a resume appeared to have no bearing, overall, on whether they’d guess correctly.

Different things mattered to engineers and recruiters
Whenever a participant deemed a candidate not worth interviewing, they had to substantiate their decision. Though these criteria are clearly not the be-all and end-all of resume filtering — if they were, people would have done better — it was interesting to see that engineers and recruiters were looking for different things.2

recruiter rejection reasons
engineer rejection reasons copy

Incidentally, lack of relevant experience didn’t refer to lack of experience with a specific stack. Verbatim rejection reasons under this category tended to say stuff like “projects not extensive enough”, “lack of core computer science”, or “a lot of academic projects around EE, not a lot on the resume about programming or web development”. Culture fit in the engineering graph denotes concerns about engineering culture fit, rather than culture fit overall. This could be anything from concern that someone used to working with Microsoft technologies might not be at home in a RoR shop to worrying that the candidate is too much of a hacker to write clean, maintainable code.

Different groups did better on different kinds of resumes
First of all, and not surprisingly, engineers tended to do slightly better on resumes that had projects. Engineers also tended to do better on resumes that included detailed and clear explanations of what the candidate worked on. To get an idea of what I mean by detailed and clear explanations, take a look at the two versions below (source: Lessons from a year’s worth of hiring data). The first description can apply to pretty much any software engineering project, whereas after reading the second, you have a pretty good idea of what the candidate worked on.


Recruiters, on the other hand, tended to do better with candidates from top companies. This also makes sense. Agency recruiters deal with a huge, disparate candidate set while also dealing with a large number of companies in parallel. They’re going to have a lot of good breadth-first insight including which companies have the highest engineering bar, which companies recently had layoffs, which teams within a specific company are the strongest, and so on.

Resumes just aren’t that useful

So, why are people pretty bad at this task? As we saw above, it may not be a matter of being good or bad at judging resumes but rather a matter of the task itself being flawed — at the end of the day, the resume is a low-signal document.

If we’re honest, no one really knows how to write resumes particularly well. Many people get their first resume writing tips from their university’s career services department, which is staffed with people who’ve never held a job in the field they’re advising for. Shit, some of the most fervent resume advice I ever got was from a technical recruiter, who insisted that I list every technology I’d ever worked with on every single undergrad research project I’d ever done. I left his office in a cold sweaty panic, desperately trying to remember what version of Apache MIT had been running at the time.

Very smart people, who are otherwise fantastic writers, seem to check every ounce of intuition and personality at the door and churn out soulless documents expounding their experience with the software development life cycle or whatever… because they’re scared that sounding like a human being on their resume or not peppering it with enough keywords will eliminate them from the applicant pool before an engineer even has the chance to look at it.

Writing aside, reading resumes is a shitty and largely thankless task. If it’s not your job, it’s a distraction that you want to get over with so you can go back to writing code. And if it is your job, you probably have a huge stack to get through, so it’s going to be hard to do deep dives into people’s work and projects, even if you’re technical enough to understand them, provided they even include links to their work in the first place. On top of that, spending more time on a given resume may not even yield a more accurate result, at least according to what I observed in this study.

How to fix top-of-the-funnel filtering

Assuming that my results are reproducible and people, across the board, are really quite bad at filtering resumes, there are a few things we can do to make top-of-the-funnel filtering better. In the short term, improving collaboration across different teams involved in hiring is a good start. As we saw, engineers are better at judging certain kinds of resumes, and recruiters are better at others. If a resume has projects or a GitHub account with content listed, passing it over to an engineer to get a second opinion is probably a good idea. And if a candidate is coming from a company with a strong brand, but one that you’re not too familiar with, getting some insider info from a recruiter might not be the worst thing.

Longer-term, how engineers are filtered fundamentally needs to change. In my TrialPay study, I found that, in addition to grammatical errors, one of the things that mattered most was how clearly people described their work. In this study, I found that engineers were better at making judgments on resumes that included these kinds of descriptions. Given these findings, relying more heavily on a writing sample during the filtering process might be in order. For the writing sample, I am imagining something that isn’t a cover letter — people tend to make those pretty formulaic and don’t talk about anything too personal or interesting. Rather, it should be a concise description of something you worked on recently that you are excited to talk about, as explained to a non-technical audience. I think the non-technical audience aspect is critical because if you can break down complex concepts for a layman to understand, you’re probably a good communicator and actually understand what you worked on. Moreover, recruiters could actually read this description and make valuable judgments about whether the writing is good and whether they understand what the person did.

Honestly, I really hope that the resume dies a grisly death. One of the coolest things about coding is that it doesn’t take much time/effort to determine if someone can perform above some minimum threshold — all you need is the internets and a code editor. Of course, figuring out if someone is great is tough and takes more time, but figuring out if someone meets a minimum standard, mind you the same kind of minimum standard we’re trying to meet when we go through a pile of resumes, is pretty damn fast. And in light of this, relying on low-signal proxies doesn’t make sense at all.


A huge thank you to:

  • All the engineers who let me use their resumes for this experiment
  • Everyone who participated and took the time to judge resumes
  • The fine people at Statwing and Plotly
  • Stan Le for doing all the behind-the-scenes work that made running this experiment possible
  • All the smart people who were kind enough to proofread this behemoth
1This number is less than 152*6=912 because not everyone who participated evaluated all 6 resumes.
2I created the categories below from participants’ full-text rejection reasons, after the fact.