Hey, Aline (founder of interviewing.io) here. We’re trying something new. Up till now, all posts on this blog have been written by interviewing.io employees or contractors. Why? Frankly, it’s hard to find great content in the recruiting space. There’s so much fluff and bad advice out there, and we didn’t want any part of that.
The other day though, I was reading Hacker News and saw an article by Uduak Obong-Eren about how he did over 60 technical interviews in 30 days and what he learned from that gauntlet of an experience. I thought it was honest, vulnerable, well-written, and brimming with actionable advice. So, I reached out to him to see if he’d want to write something else. Fortunately, he did, and the article below is the inaugural post in what I hope will become our Guest Author series. You can read more about Uduak in the bio below.
A quick note because this is the first time we’re doing this. One of the things I’m most excited about with this new Guest Author series is the diversity of opinions it will bring to our blog. Technical interviewing and hiring is fraught with controversy, and not everything these posts contain will be in line with my opinions or the official opinions of interviewing.io. But that’s what’s great about it. After over a decade in this business, I *still* don’t think there’s a right way to conduct interviews, and I think hiring is always going to be a bit of a mess because it’s a fundamentally human process. Even if we don’t always agree, I do promise that the content we put forth will be curated, high quality, and written by smart people who are passionate about this space.
Now, off we go!
Uduak Obong-Eren is a Software Engineer based in the San Francisco Bay Area who is passionate about architecting and building scalable software systems. He has about five years of industry experience and holds a Masters in Software Engineering from Carnegie Mellon University. He is also an open source enthusiast and writes technical articles – you can view some of his writings at https://meekg33k.dev. He especially enjoys conducting free mock technical interviews to help folks get better at technical interviewing. You can follow him on Twitter @meekg33k.
What is the one thing you would look out for if you had to join a company?
Sometime between January and February 2020, I wanted to change jobs and was looking to join a new company. This, among other reasons, led me to embark on a marathon of technical interviews – 60+ technical interviews in 30 days.
Doing that many number of interviews in such a short time meant I had an interesting mix of experiences from the various companies I interviewed with, each with their unique culture and values that often reflected in the way their interviews were conducted, intentionally or not.
In this article, I will be sharing some of the red flags I observed while I was on this marathon of technical interviews. I will not be mentioning names of any companies because that’s not the intent behind this article.
The goal of this article is also not to make you paranoid and be on the hunt for red flags in your next interview, far from it. Rather the goal is to equip you with knowledge to help you immediately identify exactly the same or similar red flags in your next interview, and hopefully identifying them will set you up to better handle them.
Even though the stories I’ll be sharing come from my marathon of technical interviews, these red flags do not apply only to technical interviews. They apply to all kinds of interviews and so there’s a lot to learn here for everyone.
The Red FlagsYour interviewer is only open to solving the problem ONE way
In the world of computing and in life generally, for any given problem, there is typically more than one way to solve that problem. For example, given a sorting problem, you could solve it using a merge-sort algorithm or a heap sort algorithm.
Having this rich number of techniques to solve a problem makes it even more interesting and the general expectation in technical interviews is that you should have the flexibility to solve a problem using your preferred technique.
I had an interview where the interviewer asked me to solve an algorithmic problem. I had started solving the problem using a specific technique when the interviewer stopped me in my tracks and asked that I use another technique.
When I probed a bit further to know why, it appeared that the reason he asked that I used the second wasn’t to test my knowledge of that second technique; it was because he was more ‘comfortable’ with that approach.
It is different if the interviewer wants to test your knowledge of something very specific. For example, given a problem that can be solved using iteration and recursion, the interviewer may want to test your knowledge of recursion and can ask you to solve the problem recursively. That wasn’t the case here.
I ended up using both techniques and discussed the trade-offs but frankly that experience left a bad taste in my mouth especially because that interview was with the hiring manager — my would-be manager, who is someone that can significantly influence your career growth and trajectory.
Undue pressure to accept an offer letter
It’s very exciting and fulfilling when you go through all preliminary stages of a technical interview, through to the onsite interview (remote or in-person) and then you receive that “Congratulations <insert name here>, we are pleased to offer you…” email.
However, that excitement often becomes short-lived when there is some form of pressure from your soon-to-be employer to accept the offer. It’s a bit more manageable when the pressure comes from someone in HR or the recruiter, but when it’s from the hiring manager, that can be harder to manage.
That was the case for me when I interviewed with a startup based in Palo Alto. They were a small company in terms of staff strength. My onsite interview with them had gone quite well. I had a good conversation with the hiring manager and an even better conversation with the VP of Engineering, so much so that I could tell that I was going to be extended an offer. I asked to know how long I had to accept the offer letter and I was told seventy-two hours.
The offer letter arrived later that evening, and it looked great — a six-figure offer definitely didn’t seem like a bad start. I was also at the final stage of the interview process with other companies too and thankfully, I had enough time to negotiate and accept the offer, or so I thought.
Then the pressure started, incessant calls from the hiring manager and the VP of Engineering, back-to-back emails, all within the allotted time. So much was the pressure that it got to a point where I wasn’t sure I wanted to negotiate the offer anymore. I turned down the offer.
I turned down the offer because the experience got me thinking about the company’s work culture. Were the methods employed by the company to get me to accept the offer indicative of their work culture? If they needed to get something done, how far would they go?
Now don’t get me wrong, yes the company wants to employ you, yes the recruiting team wants to ‘close the deal’ however, it’s very important to pay attention to how the company does this. Do they remain professional about it?
A company’s values go beyond what they say, it shows in what they do and how they do it.
Not enough clarity about your role
Among the many reasons why you would join a company is your desire to be involved in valuable work. I had the opportunity to join a US company based in Boulder, Colorado. They had contracted a recruiting agency to help them find someone to fill a Software Engineer position in their firm.
The hiring process started with an exploratory interview with the recruiting agency closely followed by a second interview with a recruiter from the company. In both interviews, I couldn’t get a clear sense of the details of my specific role was — what team I would be on, what kinds of projects I’d be working on, what the career growth pathway was, etc.
I understand that sometimes companies can be going through restructuring, but that didn’t seem to be the case here. It seemed more like the company was focused on completing their headcount. Even though there’s nothing wrong with completing a headcount, I think there is everything wrong with not having a clear purpose for a role for a couple of reasons:
It means the role may not be critical to the company’s core business.
If the role isn’t that important, it may mean when a layoff comes, your position may be impacted.
On a more personal note, I don’t want to be just a number. I want to work at a place where I have the opportunity to contribute in an impactful way, and I like to believe you would too. So it’s important to get clarity about your role, for where you are today and for future career growth.
Consistent lack of interest or low morale from interviewers
When looking to join a company, one of the things you simply must care about is the team you will end up working on. At least 25% of your waking hours will be spent interacting with that team whether in-person or virtually.
Interviews offer you an opportunity to experience firsthand what it will look like to work with your prospective teammates, especially since, unless you’re interviewing at a huge company, your interviewers are likely to become your teammates.
If through all the different stages of the interview process, you experience a consistent lack of interest or low morale from your interviewers, you might want to pay attention .
When I experienced that during one of my interviews, I couldn’t exactly tell what the cause was, but I knew something just wasn’t right. After some internal tussle, I decided to trust my gut feelings and ended the interview process with the company.
Fast forward to two months after, two of my interviewers (would-be teammates) had left the company and joined another company (no I wasn’t stalking, I just checked on LinkedIn).
Now I’m not saying that during the interview process, there won’t be one or two people, who because of their busy schedules, would have preferred to be doing something else rather than interviewing. Yet, when all of the interviewers don’t want to be there, you certainly want to pay attention to that.
A lack of interest or low morale could be pointers to a combination of any of the following:
Your prospective team-mates may be experiencing burnout.
Some internal dissatisfaction with company — culture, policies, something, anything.
The team isn’t that interested in you (hard pill to swallow?), maybe they don’t see you as a long-term hire.
Or it could be for reasons that I have not included here, but I implore you to not ignore this red flag if you see it in your next interview.
Your interviewers aren’t prepared for the interview
Have you been in an interview before where the interviewer doesn’t seem to have any questions to ask you? Trust me it can get really awkward.
That was my experience during a technical phone-screen interview with an educational technology company based in California. The interviewer wasn’t prepared for our interview and didn’t have any questions at hand. He wasn’t even sure of who he was interviewing and what role I was interviewing for. It wasn’t a pleasant experience.
I understand that there are a myriad of reasons why interviewers may not be prepared for an interview. Some of which include:
Lack of proper planning by the HR/recruiting team.
Last-minute changes on the interviewee.
Busy schedules for the interviewer.
The interviewer just wasn’t prepared.
I typically won’t act on this red flag in isolation. I will be looking for other red flags in a bid to form a cluster of patterns before making any decision.
Lack of a clear direction on where the company is headed
It’s fulfilling to be a part of a company that is involved in meaningful work that creates value for its users. Joining such a company would mean you have the desire to contribute in helping the organization meet its goals. This invariably means the organization must have some goals right?
I was contacted by a startup based in San Francisco via AngelList. I had a first introductory call with a recruiter from the company, closely followed by a phone screen technical interview.
In both interviews, even though the interviewers shared some details about the company, there was a lot of vagueness and about the company’s direction and where the company was headed.
I particularly remember that one question I asked at the time, was about how the company would deal with its growing competition. Sadly, the answers I got didn’t seem convincing and the company later got acquired by the competition.
When you are interviewing to join a company, you are selling more than just your skills, but also yourself — your unique experience. While it’s important to do that, I think it’s equally important that the company should be able to sell you on its vision and what it hopes to achieve.
When I think of joining a company, I picture myself in that company for the next 2–5 years. If my vision for where I want to be in my career doesn’t align with the company’s vision, that is a mismatch that shouldn’t be ignored.
Conclusion
We sometimes focus more on securing the job and even though that is very important, even more important than getting the job is staying fulfilled on the job. For me, fulfillment meant joining a company that had a clear vision of where they were headed, working in a role that was critical to the company’s business while being equipped with a lot of growth opportunities.
Hopefully, these red flags I have shared will equip you to make better decisions on what companies you choose to grow your career with. I would generally not advise making a decision based on one or two red flags, but if you see a cluster of red flags, you shouldn’t ignore them. I wish you the best in your career journey.
If you ever need someone to do a mock interview with you, feel free to schedule one here or you can reach out directly to me on Twitter @meekg33k.
And if you’d like a list of things to ask companies while you’re interviewing that may help you identify these red flags (and others!) sooner, take a look at this one.
If you have something to say about your adventures in interviewing or hiring, write a guest post on our blog! Please email me at aline@interviewing.io to get started.
I started interviewing.io because I was frustrated with how inefficient and unfair hiring was and how much emphasis employers placed on resumes.
But the problem is bigger than resumes. We’ve come to learn that interview practice matters just as much. The resume gets you in the door, and your interview performance is what gets you the offer. But, even though technical interviews are hard and scary for everyone — many of our users are senior engineers from FAANG who are terrified of getting back out there and code up the kinds of problems they don’t usually see at work while someone breathes down their neck — interview prep isn’t equitably distributed.
As you may have read, if you follow interviewing.io news, COVID-19 turned our world upside down. In its wake, the pandemic left a deluge of hiring slowdowns and freezes. For a recruiting marketplace, this was an existential worst nightmare — in a matter of weeks, we found ourselves down from 7-figure revenue to literally nothing. Companies didn’t really want or need to pay for hiring anymore, and we were screwed.
Then, we pivoted and started charging our users, who had previously been able to practice on our platform completely for free (albeit with some strings, more on that in a moment). While this pivot was the right thing to do — without it, we would have had to shut down the company, unable to provide any practice at all — charging people, especially those from underrepresented backgrounds, didn’t sit right with us, and in our last post announcing our model, we made the following promises:
We’d ALWAYS have a free tier
We’d immediately start working on a fellowship for engineers from underrepresented backgrounds or in a visa crisis/experiencing financial hardship ← That’s what this post is about!
We’d find a way to let people defer their payments
We launched with a free tier, and it’s still there and going strong. We’re still working on deferred payments and are in the thick of user research and price modeling.
But, the rest of this post is about the 2nd promise. To wit, I’m so proud to tell you that we’ve officially launched the first (pilot) cohort of the interviewing.io Technical Interview Practice Fellowship. This cohort will be focused on engineers from backgrounds that are underrepresented in tech. We are acutely aware, of course, that our first cohort couldn’t capture everyone who’s underrepresented, that gender and race isn’t enough, and that we need to do more for our users who can’t afford our price tags, regardless of who they are or where they come from.
Our hope is to expand this Fellowship to anyone who needs it.
We’re also working on the much harder problem of how to navigate the visa situation we’re in right now (different than when we wrote the first post, sadly… but especially important to me, given that I’m an immigrant myself).
What is the Fellowship, and why does it exist?
Before we tell you a little bit about the Fellows in our inaugural cohort and what the Fellowship entails, a quick word about why this matters.
In order to get a job as a software engineer, it’s not enough to have a degree in the field from a top school. However you learned your coding skills, you also have to pass a series of rigorous technical interviews, focusing on analytical problem solving, algorithms, and data structures.
To account for both of these limitations, the best strategy to maximize your chances of success is to practice a lot so you can 1) get better and 2) accept that the results of a single interview are not the be-all and end-all of your future aptitude as a software engineer and that it’s ok to keep trying.
The main problem created by modern interview techniques is that, despite interview practice being such a critical prerequisite to success in this field, access to practice isn’t equitably distributed. We want to fix this, and we’re well equipped to do so. Based on our data, engineers are twice as likely to pass a real interview after they’ve done 3-5 practice sessions on our platform.
Our Fellows will get these practice sessions completely for free. These will be 1:1 hour-long sessions with senior engineers from a top company who have graciously volunteered their time and expertise. Huge thank you and a big shout-out to them all.
After each session, Fellows will get actionable feedback that will help them in their upcoming job search, and we will be helping Fellows connect with top companies as well.
Note: We’d like to be able to offer even more support – and are actively seeking more partners to do so. Please see the How you can help section below if you or your organization would like to get involved!
Why now?
The world seems to be in a place, now more than ever, to have the conversation about race, gender, socioeconomic, and other kinds of equity, in hiring. This is our small part of that conversation.
Who are the Fellows?
After opening up our application process, we close to 1,000 submissions in a week, and (though it was really, really hard) we culled those down to 56 Fellows.
Our first cohort is:
82% Black, Latinx, and/or Indigenous
53% women
55% senior (4+ years of experience) & 45% junior (0-3 years of experience)
Here are some of their (anonymized) stories. There were a lot of stories like these.
My goal is to keep pressing as well as to share and give to underrepresented communities because the journey in tech can be isolating. Often I am the only one. It is critical that there are more people that look like me that are engineers *and* ascend the leadership ladder.
My parents immigrated from [redacted] to The Bronx without a formal education. I’m the first individual in my household to graduate from college and I’m the only Software Engineer in my family. I grew up in a poor neighborhood where many individuals had limited economic and educational opportunity. I aim to make the path to become a Software Engineer easier for those who were in my situation.
My journey to becoming a software engineer almost never happened. Throughout my undergraduate studies I was faced with having to drop out multiple times, due to the immigration status of my parents…. I was tasked with assisting in my family’s living situation and paying for school. I worked full time and started my own construction company in order to take care of my family and studies. It was always tough having to work 8-10 hours a day and then going to class or doing homework… Becoming a software engineer was always a goal of mine, and realizing that goal was well worth the struggle, given the struggle my parents went through to bring us here in the first place.
I spent 5 years in public education working directly with marginalized communities in the struggle for equity. My journey through software engineering is a continuation of this spirit of advocacy and changemaking. Software engineering is a tool to be put at the service of advocacy.
What can I do to help?
There are a number of ways you can help and get involved!
Help sponsor future Fellowship cohorts & create scholarships for underrepresented engineers!
Every Fellow in this first cohort represents at least 100X who are not. We have the tech to scale the hell out of this program, and all we need is backing and resources from people or organizations who recognize there’s a need (donations are tax-deductible). Please email fellowship-sponsors@interviewing.io if you’d like to get involved or want more information.
Hire through us!
Despite mounting evidence that resumes are poor predictors of aptitude, companies were obsessed with where people had gone to school and worked previously. On interviewing.io, software engineers, no matter where they come from or where they’re starting, can book anonymous mock interviews with senior interviewers from top companies. We use data from these interviews to identify top performers much more reliably than a resume, and fast-track them to real job interviews with employers on our platform through the same anonymous, fair process. Because we use data, not resumes, our candidates end up getting hired consistently by companies like Facebook, Uber, Twitch, Lyft, Dropbox, and many others, and 40% of the hires we’ve made to date have been candidates from non-traditional backgrounds. Many of our candidates have literally been rejected based on their resumes by the same employer who later hired them when they came through our anonymous platform (one notable candidate was rejected 3 times from a top-tier public company based on his resume before he got hired at that same company through our anonymous interview format).
Buy an individual practice session for someone who can’t afford it
If you know individual engineers who need interview practice but can’t afford it, use our handy interview gifting feature. Interviews are $100 each. They’re not cheap, but we have to price them that way to pay for interviewer time (interviewers are senior FAANG engineers) and cover our costs. Sadly that means practice interviews are not affordable to everyone. Even if you can’t get involved to help us fund interviews at scale, if you know someone who needs practice but can’t afford it, you can buy them an anonymous mock interview or two individually. It’s the best gift you can give to an engineer who’s starting their job search.
I started interviewing.io 5 years ago. After working as both an engineer and a recruiter, my frustration with how inefficient and unfair hiring had reached a boiling point. What made me especially angry was that despite mounting evidence that resumes are poor predictors of aptitude, employers were obsessed with where people had gone to school and worked previously. In my mind, any great engineer, regardless of how they look on paper, should have the opportunity to get their foot in the door wherever they choose.
So, we set out to build a better system. On interviewing.io, software engineers can book anonymous mock interviews with senior engineers from companies like Facebook, Google, and others, and if they do well in practice, get fast-tracked with top employers regardless of how they look on paper. Fast-tracking means that you bypass resume screens, scheduling emails, and recruiter calls, and go straight to the technical interview (which, by the way, is still anonymous1) at companies of your choice. Because we use interview data, not resumes, our candidates end up getting hired consistently by companies like Facebook, Uber, Twitch, Lyft, Dropbox, and many others, and 40% of the hires we’ve made to date have been candidates from non-traditional backgrounds. What’s nuts is that many of our candidates have literally been rejected based on their resumes by the same employer who later hired them when they came through interviewing.io. One notable candidate was rejected three times from a top-tier public company based on his resume before he got hired at that same company through us.
Over the past 5 years, we’ve hosted over 50,000 technical interviews (both practice and real) on our platform. Our YouTube channel, where you can watch other people interview, has gotten over 3.5M views, and, most importantly, we have helped thousands of engineers get great jobs.
All practice interviews are completely anonymous and include actionable, high-fidelity feedback
Despite that for our entire, multi-year existence, we’ve been in beta. Over the past year or so, this increasingly inaccurately named “beta” became kind of a smoke screen. Our product was stable, we had plenty of interviewers, but sadly, we couldn’t serve many of the people who needed us most. Because we made money by charging companies for hires, despite a growing waitlist of >180,000 engineers, we could only serve the ones whom we had a shot at placing, i.e. engineers who 1) were located in a city where we had customers and 2) had 4 or more years of experience—sadly, despite our best efforts, employers across the board were not willing to pay for junior hires.
Then, COVID-19 happened and with it, a deluge of hiring slowdowns and freezes. In a matter of weeks, we found ourselves down from 7-figure revenue to literally nothing. Companies didn’t really want or need to pay for hiring anymore.
But these hiring slowdowns freezes weren’t just affecting us. In parallel, we saw a growing sea of layoffs, and we realized that, soon, more and more candidates would be vying for a shrinking number of jobs. On top of that, because a disproportionate amount of layoffs targeted recruiters, an overworked in-house skeleton recruiting team would go back to relying on resumes and other old-school proxies, unintentionally marginalizing non-traditional candidates once again. We also realized that many of the folks getting laid off would be here on visas, which meant that they’d have a paltry 60 days to find their next job or risk deportation.
So, we made a hard call. You may know that historically, we’ve offered completely free mock interviews. What you may not know, is that we pay our professional interviewers, as it’s the only way to ensure that we have seasoned, experienced engineers delivering a consistent, realistic, and high-quality experience. This is often our largest expense.
Since we previously funded practice by charging companies for hires, we had to find another revenue stream to continue. In the face of either shutting down the company, unable to provide any practice at all, or to begin to charge for premium practice interviews, we made the choice to launch a paid tier. But, because charging for practice felt anathema to our mission, we knew we needed some ground rules in place. I started this company to fix hiring, after all, and that’s why the people who work at interviewing.io are here, too.
After 50,000 interviews, our data shows that where someone goes to school has no correspondence to their interview performance. Despite that, we are all too aware that just because aptitude is uniformly distributed among populations, resources are not. We understand that paying for interviews will be prohibitive to many of the people who need help most, so our ground rules and goals for this pivot were as follows:
We’d ALWAYS have a free tier
We’d immediately start working on a fellowship for engineers from underrepresented backgrounds or in a visa crisis experiencing financial hardship
We’d find a way to let people defer their payments
There is an upside to our new model though. Now that we’re no longer strictly beholden to employers, we’re able to open up interviewing.io to engineers of all seniority levels and many more locations. And because our revenue isn’t coming from placements but directly from practice, we don’t have to constrain our users to a limited number of practice interviews.
So, as of today, interviewing.io is open to engineers of all experience levels in North America and the UK. Engineers who sign up can book mock interviews 24 hours out, and top performers will still get fast-tracked for great jobs.2
You can book either free (peer practice with other users) or premium interviews with experienced interviewers from FAANG, as early as 24 hours out
Here’s how it works:
If you’re able to pay, you can now book unlimited interviews with professional interviewers from FAANG and other top companies. We’re no longer constraining you to 2 or 3, and we have the interviewer supply to make this work. Interviews cost between $100 and $225.
If you can’t pay, there’s a free tier where you can do peer practice with other users.
What about the goals and ground rules above? We’ve already made some headway against these. The free tier was there from day one. Moreover, a number of our professional interviewers have stepped up and volunteered their time to help the people who need it most prepare for free (if you’d like to volunteer your time to help engineers from underrepresented groups practice interviewing, please email interviewers@interviewing.io with the subject line ‘Volunteer’), and a formal fellowship is in the works. Lastly, we’re working on a deferred payment plan, where users who buy premium interviews will not have to pay us for them until they find their next job.
COVID-19 has changed a lot of things for our team (that’s us above doing the remote work thing). But not how much we want to fix hiring.
Look, whether you got laid off, are a student who’s reeling from your internship being withdrawn, lost your visa, whether you’re fortunate enough to still be employed but worry about your job stability, or if you just want to help making hiring fairer, we hope you’ll take us for a spin. Technical interviews are hard, and hiring is broken. And whether you’re new to interviewing or are just rusty after being off the market, and whether you can pay or not, we have your back.
The coming months are going to be hard. We know that in the current climate more people than ever are feeling helpless, and the world feels like it’s burning. And it might not even seem like technical interviews matter that much. But they do to us… because this is our way of creating a world where access to opportunity isn’t determined by who you are or where you come from but by what you can do.
P.S. If you don’t need practice for yourself but you or your organization want to help others get awesome at technical interviews, check out our new gifting feature.
Recursive Cactus has been working as a full-stack engineer at a well-known tech company for the past 5 years, but he’s now considering a career move.
Over the past 6 months, Recursive Cactus (that’s his anonymous handle on interviewing.io) has been preparing himself to succeed in future interviews, dedicating as much as 20-30 hours/week plowing through LeetCode exercises, digesting algorithms textbooks, and of course, practicing interviews on our platform to benchmark his progress.
Recursive Cactus’s typical weekday schedule
Time
Activity
6:30am – 7:00am
Wake up
7:00am – 7:30am
Meditate
7:30am – 9:30am
Practice algorithmic questions
9:30am – 10:00am
Commute to work
10:00am – 6:30pm
Work
6:30pm – 7:00pm
Commute from work
7:00pm – 7:30pm
Hang out with wife
7:30pm – 8:00pm
Meditate
8:00pm – 10:00pm
Practice algoirthmic questions
Recursive Cactus’s typical weekend schedule
Time
Activity
8:00am – 10:00am
Practice algorithmic questions
10:00am – 12:00pm
Gym
12:00pm – 2:00pm
Free time
2:00pm – 4:00pm
Practice algorithmic questions
4:00pm – 7:00pm
Dinner with wife & friends
7:00pm – 9:00pm
Practice algorithmic questions
But this dedication to interview prep has been taking an emotional toll on him, his friends, and his family. Study time crowds out personal time, to the point where he basically has no life beyond work and interview prep.
“It keeps me up at night: what if I get zero offers? What if I spent all this time, and it was all for naught?”
We’ve all been through the job search, and many of us have found ourselves in a similar emotional state. But why is Recursive Cactus investing so much time, and what’s the source of this frustration?
He feels he can’t meet the engineer hiring bar (aka “The Bar”), that generally accepted minimum level of competency that all engineers must exhibit to get hired.
To meet “The Bar,” he’s chosen a specific tactic: to look like the engineer that people want, rather than just be the engineer that he is.
It seems silly to purposefully pretend to be someone you’re not. But if we want to understand why Recursive Cactus acts the way he does, it helps to know what “The Bar” actually is. And when you think a little more about it, you realize there’s not such a clear definition.
Defining “The Bar”
What if we look at how the FAANG companies (Facebook, Amazon, Apple, Netflix, Google) define “The Bar?” After all, it seems those companies receive the most attention from pretty much everybody, job seekers included.
Few of them disclose specific details about their hiring process. Apple doesn’t publicly share any information. Facebook describes the stages of their interview process, but not their assessment criteria. Netflix and Amazon both say they hire candidates that fit their work culture/leadership principles. Neither Netflix nor Amazon describes exactly how they assess against their respective principles. However, Amazon does share how interviews get conducted as well as software topics that could be discussed for software developer positions.
The most transparent of all FAANGs, Google publicly discloses its interview process with the most detail, with Laszlo Bock’s book Work Rules! adding even more insider color about how their interview process came to be.
And speaking of tech titans and the recent past, Aline (our founder) mentioned the 2003 book How Would You Move Mount Fuji? in a prior blog post, which recounted Microsoft’s interview process when they were the pre-eminent tech behemoth of the time.
In order to get a few more data points about how companies assess candidates, I also looked at Gayle Laakmann McDowell’s “Cracking the Coding Interview”, which is effectively
the Bible of interviewing for prospective candidates, as well as Joel Spolsky’s Guerilla Guide to Interviewing 3.0, written by an influential and well-known figure within tech circles over the past 20-30 years.
– “The goal of Microsoft’s interviews is to assess a general problem-solving ability rather than a specific competency.” – “Bandwidth, inventiveness, creative problem-solving ability, outside-the-box thinking” – “Hire for what people can do rather than what they’ve done” – Motivation
Defining “Intelligence”
It’s not surprising that coding and technical knowledge would be part of any company’s software developer criteria. After all, that is the job.
But beyond that, the most common criteria shared among all these entities is a concept of intelligence. Though they use different words and define the terms slightly differently, all point to some notion of what psychologies call “cognitive ability.”
Source
Definition of cognitive ability
Google
“General Cognitive Ability. Not surprisingly, we want smart people who can learn and adapt to new situations. Remember that this is about understanding how candidates have solved hard problems in real life and how they learn, not checking GPAs and SATs.”
Microsoft (circa 2003)
“The goal of Microsoft’s interviews is to assess a general problem-solving ability rather than a specific competency… It is rarely dear what type of reasoning is required or what the precise limits of the problem are. The solver must nonetheless persist until it is possible to bring the analysis to a timely and successful conclusion.”
Joel Spolsky
“For some reason most people seem to be born without the part of the brain that understands pointers.”
Gayle Laakmann McDowell
“If you’re able to work through several hard problems (with some help, perhaps), you’re probably pretty good at developing optimal algorithms. You’re smart.”
All these definitions of intelligence resemble early 19th-century psychologist Charles Spearman’s theory of intelligence, the most widely acknowledged framework for intelligence.After performing a series of cognitive tests on school children, Spearman found that those who did well in one type of test tended to also perform well in other tests. This insight led Spearman to theorize that there exists a single underlying general ability factor (called “g” or “g factor”) influencing all performance, separate from specific, task-specific abilities (named “s”).
If you believe in the existence of “g” (many do, some do not… there exist different theories of intelligence), finding candidates with high measures of “g” aligns neatly with the intelligence criteria companies look for.
While criteria like leadership and culture fit matter to companies, “The Bar” is not usually defined in those terms. “The Bar” is defined as having technical skills but also (and perhaps more so) having general intelligence. After all, candidates aren’t typically coming to interviewing.io to specifically practice leadership and culture fit.
The question then becomes how you measure these two things. Measuring technical skills seems tough but doable, but how do you measure “g?”
Measuring general intelligence
Mentioned in Bock’s book, Frank Schmidt’s and John Hunter’s 1998 paper “The Validity and Utility of Selection Methods in Personnel Psychology” attempted to answer this question by analyzing a diverse set of 19 candidate selection criteria to see which predicted future job performance the best. The authors concluded general mental ability (GMA) best predicted job performance based on a statistic called “predictive validity.”
In this study, a GMA test referred to an IQ test. But for Microsoft circa 2003, puzzle questions like “How many piano tuners are there in the world?” appear to have taken the place of IQ tests for measuring “g”. Their reasoning:
“At Microsoft, and now at many other companies, it is believed that there are parallels between the reasoning used to solve puzzles and the thought processes involved in solving the real problems of innovation and a changing marketplace. Both the solver of a puzzle and a technical innovator must be able to identify essential elements in a situation that is initially ill-defined.”
– “How Would You Move Mount Fuji?” – page 20
Fast forward to today, Google denounces this practice, concluding that “performance on these kinds of questions is at best a discrete skill that can be improved through practice, eliminating their utility for assessing candidates.”
So here we have two companies who test for general intelligence, but who also fundamentally disagree on how to measure it.
Are we measuring specific or general intelligence?
But maybe as Spolsky and McDowell have argued, the traditional algorithmic and computer science-based interview questions are themselves effective tests for general intelligence. Hunter & Schmidt’s study contains some data points that could support this line of reasoning. Among all single-criteria assessment tools, work sample tests possessed the highest predictive validity. Additionally, when observing the highest validity regression result of two-criteria assessment tool (GMA test plus work sample test), the standardized effect size on the work sample rating was larger than that of the GMA rating, suggesting a stronger relationship with future job performance.
If you believe algorithmic exercises function as work sample tests in interviews, then the study suggests traditional algorithm-based interviews could predict future job performance, maybe even more than a GMA/IQ test.
Recursive Cactus doesn’t believe there’s a connection.
There’s little overlap between the knowledge acquired on the job and knowledge about solving algorithmic questions. Most engineers rarely work with graph algorithms or dynamic programming. In application programming, lists and dictionary-like objects are the most common data structures. However, interview questions involving those are often seen as trivial, hence the focus on other categories of problems.
– Recursive Cactus
In his view, algorithms questions are similar to Microsoft’s puzzle questions: you learn how to get good at solving interview problems, which to him don’t ever show up in actual day-to-day work, which, if true, wouldn’t actually fit with the Schmidt & Hunter study.
Despite Recursive Cactus’s personal beliefs, interviewers like Spolsky still believe these skills are vital to being a productive programmer.
A lot of programmers that you might interview these days are apt to consider recursion, pointers, and even data structures to be a silly implementation detail which has been abstracted away by today’s many happy programming languages. “When was the last time you had to write a sorting algorithm?” they snicker.
Still, I don’t really care. I want my ER doctor to understand anatomy, even if all she has to do is put the computerized defibrillator nodes on my chest and push the big red button, and I want programmers to know programming down to the CPU level, even if Ruby on Rails does read your mind and build a complete Web 2.0 social collaborative networking site for you with three clicks of the mouse.
– Joel Spolsky
Spolsky seems to concede that traditional tech interview questions might not mimic actual work problems, and therefore wouldn’t act as work samples. Rather, it seems he’s testing for general computer science aptitude, which is general in a way, but specific in other ways. General intelligence within a specific domain, one might say.
That is, unless you believe intelligence in computer science is general intelligence. McDowell suggests this:
There’s another reason why data structure and algorithm knowledge comes up: because it’s hard to ask problem-solving questions that don’t involve them. It turns out that the vast majority of problem-solving questions involve some of these basics.
– Gayle Laakmann McDowell
This could be true assuming you view the world primarily through computer science lenses. Still, it seems pretty restrictive to suggest people who don’t speak the language of computer science would have more difficulty solving problems.
At this point, we’re not really talking about measuring general intelligence as Spearman originally defined it. Rather, it seems we’re talking about specific intelligence, defined or propagated by those grown or involved in traditional computer science programs, and conflating that with general intelligence (Spolsky, McDowell, Microsoft’s Bill Gates, and 4 of 5 FAANG founders studied computer science at either some Ivy League university or Stanford).
Maybe when we’re talking about “The Bar,” we’re really talking about something subjective, based on whoever is doing the measuring, and whose definition might not be consistent from person-to-person.
Looking at candidate assessment behavior from interviewers on the interviewing.io platform, you can find some evidence that supports this hypothesis.
“The Bar” is subjective
On the interviewing.io platform, people can practice technical interviews online and anonymously, with interviewers from top companies on the other side. Interview questions on the platform tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role, and interviewers typically come from companies like Google, Facebook, Dropbox, Airbnb, and more. Check out our interview showcase to see how this all looks and to watch people get interviewed. After every interview, interviewers rate interviewees on a few different dimensions: technical skills, communication skills, and problem-solving skills. Each dimension gets rated on a scale of 1 to 4, where 1 is “poor” and 4 is “amazing!”. You can see what our feedback form looks like below:
If you do well in practice, you can bypass applying online/getting referrals/talking to recruiters and instead immediately book real technical interviews directly with our partner companies (more on that in a moment).
When observing our most frequent practice interviewers, we noticed differences across interviewers in the percent of candidates that person would hire, which we call the passthrough rate. Passthrough rates ranged anywhere between 30% and 60%. At first glance, certain interviewers seemed to be a lot stricter than others.
Because interviewees and interviewers are anonymized and matched randomly[1], we wouldn’t expect the quality of candidates to vary much across interviewers, and as a result, wouldn’t expect interviewee quality to explain the difference. Yet even after accounting for candidate attributes like experience level, differences in passthrough rates persist[2].
Maybe some interviewers choose to be strict on purpose because their bar for quality is higher. While it’s true that candidates who practiced with stricter interviewers tended to receive lower ratings, they also tended to perform better on their next practice.
This result could be interpreted in a couple of ways:
Candidates get so beat up by strict interviewers that they tended to improve more between practices, striving to meet their original interviewer’s higher bar
If the latter were true, you would expect that candidates who practiced with stricter interviewers would perform better in real company interviews. However, we did not find a correlation between interviewer strictness and future company interview passthrough rate, based on real company interviews conducted on our platform[3].
Interviewers on our platform represent the kinds of people a candidate would encounter in a real company interview, since those same people also conduct phone screens and onsites at the tech companies you’re all applying to today. And because we don’t dictate how interviewers conduct their interviews, these graphs could be describing the distribution of opinions about your interview performance once you hang up the phone or leave the building.
This suggests that, independent of your actual performance, whom you interview with could affect your chance of getting hired. In other words, “The Bar” is subjective.
This variability across interviewers led us to reconsider our own internal definition of “The Bar,” which determined which candidates were allowed to interview with our partner companies. Our definition strongly resembled Spolsky’s binary criteria (“be smart”), heavily weighing an interviewer’s Would Hire opinion way more than our other 3 criteria, leading to the bimodal, camel-humped distribution below.
While our existing scoring system correlated decently with future interview performance, we found that an interviewer’s Would Hire rating wasn’t as strongly associated with future performance as our other criteria were. We lessened the weight on the Would Hire rating, which in turn improved our predictive accuracy[4]. Just like in “Talledega Nights” when Ricky Bobby learned there existed places other than first place and last place in a race, we learned that it was more beneficial to think beyond the binary construct of “hire” vs. “not hire,” or if you prefer, “smart” vs. “not smart.”
Of course, we didn’t get rid of all the subjectivity, since those other criteria were also chosen by the interviewer. And this is what makes assessment hard: an interviewer’s assessment is itself the measure of candidate ability.
If that measurement isn’t anchored to a standard definition (like we hope general intelligence would be), then the accuracy of any given measurement becomes less certain. It’s as if interviewers used measuring sticks of differing lengths, but all believed their own stick represented the same length, say 1 meter.
When we talked to our interviewers to understand how they assessed candidates, it became even more believable that different people might be using measuring sticks of differing lengths. Here are some example methods of how interviewers rated candidates:
Ask 2 questions. Pass if answer both
Ask questions of varying difficulty (easy, medium, hard). Pass if answers a medium
Speed of execution matters a lot, pass if answers “fast” (“fast” not clearly defined)
Speed doesn’t matter much, pass if have a working solution
Candidates start with full points. When candidates make mistakes, start docking points
Having different assessment criteria isn’t necessarily a bad thing (and actually seems totally normal). It just introduces more variance to our measurements, meaning our candidates’ assessments might not be totally accurate.
The problem is, when people talk about “The Bar,” that uncertainty around measurement usually gets ignored.
You’ll commonly see people advising you only to hire the highest quality people.
A good rule of thumb is to hire only people who are better than you. Do not compromise. Ever.
– Laszlo Bock
Don’t lower your standards no matter how hard it seems to find those great candidates.
– Joel Spolsky
In the Macintosh Division, we had a saying, “A player hire A players; B players hire C players”–meaning that great people hire great people.
– Guy Kawasaki
Every person hired should be better than 50 percent of those currently in similar roles – that’s raising the bar.
– Amazon Bar Raiser blog post
All of this is good advice, assuming “quality” could be measured reliably, which as we’ve seen so far, isn’t necessarily the case.
Even when uncertainty does get mentioned, that variance gets attributed to the candidate’s ability, rather than the measurement process or the person doing the measuring.
[I]n the middle, you have a large number of “maybes” who seem like they might just be able to contribute something. The trick is telling the difference between the superstars and the maybes, because the secret is that you don’t want to hire any of the maybes. Ever.
…
If you’re having trouble deciding, there’s a very simple solution. NO HIRE. Just don’t hire people that you aren’t sure about.
– Joel Spolsky
Assessing candidates isn’t a fully deterministic process, yet we talk about it like it is.
Why “The Bar” is so high
“Compromising on quality” isn’t really about compromise, it’s actually about decision-making in the face of uncertainty. And as you see from the quotes above, the conventional strategy is to only hire when certain.
No matter what kind of measuring stick you use, this leads to “The Bar” being set really high. Being really certain about a candidate means minimizing the possibility of making a bad hire (aka “false positives”). And companies will do whatever they can to avoid that.
A bad candidate will cost a lot of money and effort and waste other people’s time fixing all their bugs. Firing someone you hired by mistake can take months and be nightmarishly difficult, especially if they decide to be litigious about it.
– Joel Spolsky
Hunter and Schmidt quantified the cost of a bad hire: “The standard deviation… has been found to be at minimum 40% of the mean salary,” which in today’s terms would translate to $40,000 assuming a mean engineer salary of $100,000/year.
But if you set “The Bar” too high, chances are you’ll also miss out on some good candidates (aka “false negatives”). McDowell explains why companies don’t really mind a lot of false negatives:
“From the company’s perspective, it’s actually acceptable that some good candidates are rejected… They can accept that they miss out on some good people. They’d prefer not to, of course, as it raises their recruiting costs. It is an acceptable tradeoff, though, provided they can still hire enough good people.”
In other words, it’s worth holding out for a better candidate if the difference in their expected output is large, relative to the recruiting costs from continued searching. Additionally, the costs of HR or legal issues downstream from potentially problematic employees also tilt the calculation toward keeping “The Bar” high.
This is a very rational cost-benefit calculation. But has anyone ever done this calculation before? If you have done it, we’d love to hear from you. Otherwise, it seems difficult to do.
Given that nearly everyone is using hand-wavy math, if we do the same, maybe we can convince ourselves that “The Bar” doesn’t have to be set quite so high.
As mentioned before, the distribution of candidate ability might not be so binary, so Spolsky’s nightmare bad hire scenario wouldn’t necessarily happen with all “bad” hires, meaning the expected difference in output between “good” and “bad” employees might be lower than perceived.
Recruiting costs might be higher than perceived because finding and employing +1 standard deviation employees gets increasingly difficult. By definition, fewer of those people exist as your bar rises. Schmidt and Hunter’s “bad hire” calculation only compares candidates within an applicant pool. The study does not consider the relative cost of getting high-quality candidates into the applicant pool to begin with, which tends to be the more significant concern for many of today’s tech recruiting teams. And when you consider that other tech companies might be employing the same hiring strategy, competition would increase the average probability that offers get rejected, extending the time to fill a job opening.
Estimating the expected cost of HR involvement is also difficult. No one wants to find themselves interacting with HR. But then again, not all HR teams are as useless as Toby Flenderson.
Taken together, if the expected output between “good” and “bad” candidates were less than expected, and the recruiting costs were higher than perceived, it would make less sense to wait for a no-brainer hire, meaning “The Bar” might not have to be set so high.
Even if one does hire an underperformer, companies could adopt the tools of training and employee management to mitigate the negative effects from some disappointing hires. After all, people can and do become more productive over time as they acquire new skills and knowledge.
Employee development seems to rarely get mentioned in conjunction with hiring (Laszlo Bock makes a few connections here and there, but the topics are mostly discussed separated). But when you add employee development into the equation above, you start to see the relationship between hiring employees and developing employees. You can think of it as different methods for acquiring more company output from different kinds of people: paying to train existing employees versus paying to recruit new employees.
You can even think of it as a trade-off. Instead of developing employees in-house, why not outsource that development? Let others figure out how to develop the raw talent, and later pay recruiters to find them when they get good. Why shop the produce aisle at Whole Foods and cook at home when you can just pay Caviar to deliver pad thai to your doorstep? Why spend time managing and mentoring others when you can spend that time doing “real work” (i.e. engineering tasks)?
Perhaps “The Bar” is set high because companies don’t develop employees effectively, which puts more pressure on the hiring side of the company to yield productive employees.
Therefore, companies can lower their risk by shifting the burden of career development onto the candidates themselves. In response, candidates like Recursive Cactus have little choice but to train themselves.
Initially, I thought Recursive Cactus was a crazy outlier in terms of interview preparation. But apparently, he’s not alone.
Candidates are training themselves
Last year we surveyed our candidates about how many hours they spent preparing for interviews. Nearly half of the respondents reported spending 100 hours or more on interview preparation[5].
We wondered whether hiring managers and recruiters had the same expectations for candidates they encounter, Aline asked a similar question on Twitter, and results suggest they vastly underestimate the work and effort candidates endure prior to meeting with a company.
Are you a hiring manager or recruiter? If so, please vote on my poll! Results will be shared in an upcoming, titillating blog post.
How many dedicated hours of interview prep do you believe candidates should invest in their job search?
Decision makers clearly underestimate the amount of work candidates put into job hunt preparation. The discrepancy seems to reinforce the underlying and unstated message pervading all these choices around how we hire: If you’re not one of the smart ones (whatever that means), it’s not our problem. You’re on your own.
“The Bar” revisited
So this is what “The Bar” is. “The Bar” is a high standard set by companies in order to avoid false positives. It’s not clear whether companies have actually done the appropriate cost-benefit analysis when setting it, and it’s possible it can be explained by an aversion to investing in employee development.
“The Bar” is in large part meant to measure your general intelligence, but the actual instruments of measurement don’t necessarily follow the academic literature that underlies it. You can even quibble about the academic literature[6]. “The Bar” does measure specific intelligence in computer science, but that measurement might vary depending on who conducts your interview.
Despite the variance that exists across many aspects of the hiring process, we talk about “The Bar” as if it were deterministic. This allows hiring managers to make clear binary choices but discourages them to think critically about whether their team’s definition of “The Bar” could be improved.
And that helps us understand why Recursive Cactus spends so much time practicing. He’s training himself partially because his current company isn’t developing his skills. He’s preparing for the universe of possible questions and interviewers he might encounter because hiring criteria varies a lot, which cover topics that won’t necessarily be used in his day-to-day work, all so he can resemble someone that’s part of the “smart” crowd.
That’s the system he’s working within. Because the system is the way it is, it’s had significant impact on his personal life.
My wife’s said on more than one occasion that she misses me. I’ve got a rich happy life, but I don’t feel I can be competitive unless I put everything else on hold for months. No single mom can be doing what I’m doing right now.
– Recursive Cactus
This impacts his current co-workers too, whom he cares about a lot.
This process is sufficiently demanding that I’m no longer operating at 100% at work. I want to do the best job at work, but I don’t feel I can do the right thing for my future by practicing algorithms 4 hours a day and do my job well.
I don’t feel comfortable being bad at my job. I like my teammates. I feel a sense of responsibility. I know I won’t get fired if I mail it in, but I know that it’s them that pick up the slack.
– Recursive Cactus
It’s helpful to remember that all the micro decisions made around false positives, interview structure, brain teasers, hiring criteria, and employee development add up to define a system that, at the end of the day, impacts people’s personal lives. Not just the lives of the job hunters themselves, but also all the people that surround them.
Hiring is nowhere near a solved problem. Even if we do solve it somehow, it’s not clear we would ever eliminate all that uncertainty. After all, projecting a person’s future work output after spending an hour or two with them in an artificial work setting seems kinda hard. While we should definitely try to minimize uncertainty, it might be helpful to accept it as a natural part of the process.
This system can be improved. Doing so requires not only coming up with new ideas, but also revisiting decades-old ideas and assumptions, and expanding upon that prior work rather than anchoring ourselves to it.
We’re confident that all of you people in the tech industry will help make tech hiring better. We know you can do it, because after all, you’re smart.
[1]There does exist some potential for selection bias, particularly around the time frames when people choose to practice. Cursory analysis suggests there’s not much of a relationship, but we’re currently digging in deeper (hinthint: could be a future blog post). You can also choose between the traditional algorithmic interview vs. a systems design interview, but the vast majority opt for the traditional interview. The passthrough rates shown are for the traditional interview.↩ [2]You might be wondering about the relative quality of candidates on interviewing.io. While it’s hard to pin down the true distribution of quality (which is the underlying question of this blog post), on average our practice interviewers have communicated to us that the quality of candidates on interviewing.io tends to be similar to the quality of candidates they encounter during their own company’s interview process, particularly during the phone screen process.↩ [3]This only includes candidates who have met our internal hiring bar and attended a company interview on our site. This does not represent the entire population of candidates who have interviewed with an interviewer.↩ [4]For those of you that have used interviewing.io before, you may remembere that we had an algorithm that adjusted for interviewer strictness. Upon further inspection, we found this algorithm also introduced variance to candidate scores in really unexpected ways. Because of this, we don’t rely on this algorithm as heavily.↩ [5]Spikes at 100 and 200 hours occurred because of an error in the labeling and max value of the survey question. The 3 survey questions asked were the following: 1) In your most recent job search, how many hours did you spend preparing for interviews? 2) How many hours did you spend on interview preparation before signing up for interviewing.io? 3) How many hours did you spend on interview preparation after signing up for interviewing.io (not including time using interviewing.io itself)? Each question had a max value of 100 hours, but many respondents responded to questions 2) and 3) where their sum exceeded 100. The distribution here shows the sum of 2) and 3). The median from question 1) responses was 94, nearly identical to the median of the sum of 2) and 3), so we used the sum to observe the shape of the distribution beyond 100 hours. Key lessons: assume a larger max value than you’d expect, and double-check your survey.↩ [6]I found the study a little hard to reason about, mainly because I’m not a psychologist, so techniques like meta analysis were a little foreign to me even if the underlying statistical tools were familiar. It’s not a question of whether the tools are valid, it’s that reasoning about the study’s underlying data was difficult. Similar to spaghetti code, the validation of the underlying datasets is spread across decades of prior academic papers, which makes it difficult to follow. It’s likely this is the nature of psychology, where useful data is harder to acquire, at least compared to the kinds of data we deal with in tech. Beyond that, I also had other questions about their methodology, which this article asks in far greater detail than I could. ↩
One of the things that sucks most about technical interviews is that they’re a black box—candidates (usually) get told whether they made it to the next round, but they’re rarely told why they got the outcome that they did. Lack of feedback, or feedback that doesn’t come right away, isn’t just frustrating to candidates. It’s bad for business. We did a whole study on this. It turns out that candidates chronically underrate and overrate their technical interview performance, like so:
Where this finding starts to get actionable is that there’s a statistically significant relationship between whether people think they did well in an interview and whether they’d want to work with you. In other words, in every interview cycle, some portion of interviewees are losing interest in joining your company just because they don’t think they did well, even when they actually did. It makes sense… when you suspect you might not have done well, you’re prone to embark on a painful bout of self-flagellation, and to make it stop, you’ll rationalize away the job by telling yourself that you totally didn’t want to work there anyway.
Practically speaking, giving instant feedback to successful candidates can do wonders for increasing your close rate.
In addition to making candidates you want today more likely to join your team, feedback is crucial for the candidates you might want down the road. Technical interview outcomes are highly non-deterministic. According to our data, only about 25% of candidates perform consistently from interview to interview. Why does this matter? If interview outcomes are erratic, it means that the same candidate you reject this time might be someone you want to hire in 6 months. It’s in your interest to forge a good relationship with them now and be cognizant of and humble about the flaws in your hiring process.
I thought this tweet captured my sentiments particularly well.
So, despite the benefits, why do most companies persist in giving slow feedback or none at all? I surveyed founders, hiring managers, recruiters and labor lawyers (and also put out some questions to the Twitterverse) to understand why anyone who’s ever gone through interviewer training has been told in no uncertain terms to not give feedback.
As it turns out, feedback is discouraged primarily because companies are scared of getting sued… and because interviewers fear defensive candidate backlash. In some cases, giving feedback is avoided just because companies view it as a no-upside hassle.
The sad truth is that hiring practices have not caught up with market realities. Many of the hiring practices we take for granted today originated in a world where there was a surplus of candidates and a shortage of jobs. This extends to everything from painfully long take-home assignments to poorly written job descriptions. And post-interview feedback is no exception. As Gayle Laakmann McDowell, author of Cracking the Coding Interview, explains on Quora:
“Companies are not trying to create the most perfect process for you. They are trying to hire—ideally efficiently, cheaper, and effectively. This is about their goals, not yours. Maybe when it’s easy they’ll help you too, but really this whole process is about them… Companies do not believe it helps them to give candidates feedback. Frankly, all they see is downside.”
Look, I’m guilty of this, too. Here’s a rejection email I wrote when I was head of technical recruiting at TrialPay. This email makes me want to go back in time and punch myself in the face and then wish myself the best in my future endeavors to not get punched in the face.
These kinds of form letter rejections (which I guess is better than just leaving the person hanging) make a lot of sense when you have a revolving door of disposable candidates. They are completely irrational in this brave new world where candidates have more leverage than companies. But, because HR is fundamentally a cost center tasked with risk mitigation (rather than a profit center tasked with, you know, making stuff better), and because engineers on the ground only have so many cognitive cycles to tackle hard stuff outside their job descriptions, we continue to march forward on autopilot, perpetuating outdated and harmful practices like this one.
In this hiring climate, companies should move toward practices that give candidates a better interview experience. Is fear of litigation and discomfort legit enough to keep companies from giving feedback? Does optimizing for fear and a few bad actors in lieu of candidate experience make sense in the midst of a severe engineering shortage? Let’s break it down.
Does the fear of getting sued even make sense?
While researching this piece, I spoke to a few labor lawyers and ran some Lexis Nexis searches to see just how often a company’s constructive feedback (i.e. not “durrrr we didn’t hire you because you’re a woman”) to a rejected eng candidate has resulted in litigation.
Hey, guess what? IT’S ZERO! THIS HAS NEVER HAPPENED. EVER.1
As some of my lawyer contacts pointed out, a lot of cases get settled out of court, and that data is much harder to get. But in this market, creating poor candidate experience to hedge against something that is highly unlikely seems… irrational at best and destructive at worst.
What about candidates getting defensive?
At some point, I stopped writing trite rejection emails like the one above, but I was still beholden to my employer’s rules about written feedback.2 As an experiment, I tried giving candidates verbal feedback over the phone.
For context, I had a unique, hybrid role at TrialPay. Though my title was Head of Technical Recruiting, which meant I was accountable for normal recruiter stuff like branding and sourcing and interview process logistics, my role had one unique component. Because I had previously been a software engineer, to take the heat off the long-suffering eng team, I was often the first line of defense for technical interviews and conducted something like 500 of them that year.
After doing a lot of interviews day in and day out, I became less shy about ending them early when it was clear that a candidate wasn’t qualified (e.g. they couldn’t get through the brute force solution to the problem, let alone optimize). Did ending interviews early cause candidates to fly off the handle or feel particularly awkward, as many people suspect?
In my experience, cutting things off and saying nothing about why is a lot more awkward and leads to more defensiveness than letting candidates know what the deal is. Some candidates will get defensive (at which point you can politely end the call), but if you offer constructive feedback—let them know what went wrong, make some recommendations about books to read, point them to problem repositories like Leetcode3, etc.—most will be grateful. My personal experience with giving feedback has been overwhelmingly positive. I used to love mailing books to candidates, and I formed lasting relationships with many. Some became early interviewing.io users a few years later.
Anyway, the way to avoid negative reactions and defensiveness from candidates is to practice giving feedback in a way that’s constructive. We’ll cover this next.
So if giving feedback isn’t actually risky and has real upsides, how does one do it?
When I started interviewing.io, it was the culmination of what I had started experimenting with at TrialPay. It was clear to me that feedback is a Good Thing and that candidates liked it… which in this market means it’s also good for companies. But, we still had to grapple with prospective customers’ (pretty irrational) fears about the market being flooded with defensive candidates with a lawyer on speed dial.
For context, interviewing.io is a hiring marketplace. Before talking to companies, engineers can practice technical interviewing anonymously, and if things go well, unlock our jobs portal, where they can bypass the usual top-of-funnel cruft (applying online, talking to recruiters or “talent managers,” finding friends who can refer them) and book real technical interviews with companies like Microsoft, Twitter, Coinbase, Twitch, and many others… often as early as the next day.
The cool thing is that both practice and real interviews with companies take place within the interviewing.io ecosystem, and wrt feedback, you’ll see why this matters in a moment.
Before we started working with employers, we spent some time building out our practice platform and getting the mechanics right. For practice interviews, our post-interview feedback forms looked like this:
The feedback form that an interviewer fills out
After each practice interview, interviewers fill out the form above. Candidates fill out a similar form rating their interviewer. When both parties fill out their forms, they can see each other’s responses.
If you’re curious, you can watch people practicing and read real feedback that they got in our public showcase. Here’s a snapshot:
When we started letting employers hire on our platform, we just recycled this post-interview feedback format, told them they should leave feedback to help us calibrate and because it’s good for candidate experience, and fervently hoped that they wouldn’t have an issue with it.
To our surprise and delight, employers were eminently willing to leave feedback. On our platform, candidates were able to see whether they passed or not and exactly how they did, just a few minutes after the interview was over, stopping the rising tide of post-interview anxiety and self-flagellation in its tracks, and, as we’ve said, increasing the likelihood that a great candidate will accept an offer.
A real, successful company interview on interviewing.io
And if a candidate failed an interview, they got to see exactly why they failed and what they needed to work on, probably for the first time ever in interview history.
A real, failed company interview on interviewing.io
Anonymity makes it easier to give feedback
On interviewing.io, interviews are anonymous: an employer knows literally nothing about the candidate before and during the interview (employers can even enable our real-time voice masking feature). Candidates’ identities are only revealed after a successful interview, i.e. after the employer has already submitted feedback.
We insist on anonymity because about 40% of our top-performing candidates are non-traditional, and we don’t want lack of pedigree or an unusual background to open them up to bias. Because interviews are anonymous, it’s often impossible to discriminate on the basis of age or gender or background. Therefore, feedback has to be constructive, by design, because the only info the interviewer has to go on is how well the candidate is performing in the interview. In addition to helping candidates get a fair shot, this anonymity provides something of a safety net for employers—it’s harder to build a discrimination case out of the feedback when the employer doesn’t know the identity of the candidate.
In many other contexts, anonymity can be destructive because of reduced accountability. But in the interview process, we’ve discovered over and over that anonymity sets free the better angels of our nature and creates a kinder, more inclusive interview experience for candidates and employers alike.
Building post-interview feedback into your eng hiring process
So, how can you fold these learnings into your process? Of course, the easiest way to get your feet wet is to start using interviewing.io. We’ll get you candidates you wouldn’t be sourcing without us, and we’ll empower you to give them the best interview experience you can.
But even if you don’t use us, based on how unlikely it is that you’re going to get sued or deal with angry candidates, we strongly recommend having your interviewers provide constructive feedback (like in the examples above) after every interview for all candidates, whether they pass or not, over email.
Here are a few strategies for delivering constructive feedback:
Be clear that it’s a no-go. Ambiguity is psychologically difficult in a stressful situation. For instance: Thank you for interviewing with us. Unfortunately, you didn’t pass the interview.
After you make it clear that it’s a no-go, tell them something nice. Find something about their performance—an answer they gave, or the way they thought through a problem, or how they asked the right questions—and share it with them. They’ll be more receptive to the rest of your feedback once they know that you’re on their side. For instance: Despite the fact that it didn’t work out this time, you did {x,y,z} really well, and I think that you can do much better in the future. Here are some suggestions for what you can work on.
When you give suggestions, be specific and constructive. Don’t tell them that they royally screwed the whole operation and need to rethink their line of work. Instead, focus on specific things they can work on. Or, to put it another way, “Hey, familiarize yourself with big O notation. It’s not as scary as it sounds bc it comes up a lot in these kinds of interviews.”4 doesn’t say “you’re dumb and your work experience is dumb and you should feel bad” or “you seem like an asshole.” It says you should familiarize yourself with big O notation.
Make recommendations. Is there a book they could read? If they’re promising but just lack knowledge, it’s a really nice gesture to ship said book to them.
If you think the candidate is on their way to becoming a great engineer (especially if they take your recommendations and advice!), let them know that they can contact you again in a few months. You’ll build goodwill with someone who, even if they don’t work with you in the future, will talk about you to others. And when they do improve, you’ll be in a better position to bring them onto your team.
1If you know of such a case, please tell me, and I’ll update this post/associated content accordingly ASAP.↩ 2This is a rule pretty much every company has, ever. It’s not just TrialPay, which was a great place to work and whose defensive HR policies, like every other company’s, were in no way indicative of their workplace culture.↩ 3Whether algorithmic problems of the type you’d find on Leetcode are the best way to interview is a question worth asking… and I’ve since come to feel pretty strongly that many of them are not. But that’s out of scope for this piece.↩ 4I recently discovered an amazing online book that makes big O approachable, practical, and not scary (all without talking down to the reader): Grokking Algorithms by Aditya Bhargava.↩
Guest Author
on September 15th, 2020.
Uduak Obong-Eren is a Software Engineer based in the San Francisco Bay Area who is passionate about architecting and building scalable software systems. He has about five years of industry experience and holds a Masters in Software Engineering from Carnegie Mellon University. He is also an open source enthusiast and writes technical articles – you can view some of his writings at https://meekg33k.dev. He especially enjoys conducting free mock technical interviews to help folks get better at technical interviewing. You can follow him on Twitter @meekg33k.
Aline Lerner 
Ken Pascual 
