Refuting Bloomberg's analysis: ChatGPT isn't racist. But it is bad at recruiting.

Bloomberg recently published an article saying ChatGPT has racial bias when evaluating resumes. We re-ran their numbers and saw that they didn't do any statistical significance testing. In other words, there's no evidence of racial bias, at least not in this data set. However, ChatGPT has another kind of bias -- it drastically overestimates the value of top companies and top schools and does non-traditional candidates a disservice as a result.

Want to work from home? That’ll be $46,282.

Software engineering jobs come with a lot of perks. But that doesn’t mean our jobs are perfect. Work requires us to commute, reduces the time we can spend with family, increases our stress levels, and forces us to deal with teammates we don't gel with. And sometimes we work for companies with questionable morals and use technologies we don’t enjoy. 

For some of us, it’s worth trading cash for a job that fits into our lives better. But how much cash?

To figure this out, we surveyed our users about times they took jobs with lower compensation, why they did it, and how much money they left on the table. We have the numbers!

How hard is it to cheat in technical interviews with ChatGPT? We ran an experiment.

Does ChatGPT make it easy to cheat in technical interviews? To find out, we ran an experiment where we instructed interviewees on our platform to use ChatGPT in their interviews, unbeknownst to their interviewers. The results were surprising, but as a preview, know this: companies need to change the types of interview questions they are asking—immediately.

When is hiring coming back? Our predictions for 2024.

Predictions are hard, and, inevitably, most of them turn out wrong. But we’d like to brave the scathing mockery of the internets and try anyway! Our courage is bolstered by some useful data we have (both proprietary and gathered from the internet), which we’ll use to guess what will happen in 2024 and to answer the question foremost in many of our minds: When is hiring coming back?

Why AI can’t do hiring

The recent exciting and somewhat horrifying inflection point in AI capability tipped me into writing this blog post.

I simply don't believe that AI can do hiring. My argument isn't about bias (though bias is a real problem) or that it's technologically impossible. It's just that the training data simply isn't available.

Most people believe that if you can somehow combine what's available on LinkedIn, GitHub, and the social graph (who follows whom on Twitter etc.), you'll be able to find the good engineers who are actively looking and also figure out what they want. This is wrong. None of those 3 sources are actually useful. At the end of the day, you can’t use AI for hiring if you don’t have the data. And if you have the data, then you don’t strictly need AI.

Does posting Open To Work on LinkedIn help or hurt? A tale of two labor markets.

If you’re a software engineer who’s on the market, should you list yourself as OpenToWork? Does doing so carry a negative signal? And with the recent deluge of layoffs at tech companies, has the meaning of OpenToWork changed?

TL;DR It used to be bad. Now, it's not. Moreover, it's clear to us that the people who were laid off in the 2nd half of 2022 and 2023 so far are great... and that, by and large, these layoffs were indeed NOT based on performance.

Why you shouldn’t list certifications on LinkedIn

People often suggest that interviewing.io should create a certification that our users can post on their LinkedIn profile, e.g., something like “Top 10% performer on interviewing.io”. Presumably, these certifications would signal to recruiters that this person is a good engineer and worth reaching out to and should carry more signal than where said person went to school or worked previously.

I've always thought certifications were a terrible idea, and I’ve resisted building them. Now, we've finally dug into the data to see if my hatred of them holds water. TL;DR it does.

Our business depends on having the best interviewers, so we built an interviewer rating system. And you can too.

interviewing.io is an anonymous mock interview platform and eng hiring marketplace. Engineers use us for mock interviews, and we use the data from those interviews to surface top performers, in a much fairer and more predictive way than a resume. If you’re a top performer on interviewing.io, we fast-track you at the world’s best companies.

We make money in two ways: engineers pay us for mock interviews, and employers pay us for access to the best performers. To keep our engineer customers happy, we have to make sure that our interviewers deliver value to them by conducting realistic mock interviews and giving useful, actionable feedback afterwards. To keep our employer customers happy, we have to make sure that the engineers we send them are way better than the ones they’re getting without us. Otherwise, it’s just not worth it for them.

This means that we live and die by the quality of our interviewers, in a way that no single employer does, no matter how much they say they care about people analytics or interviewer metrics or training. If we don’t have really well-calibrated interviewers, who also create great candidate experience, we don’t get paid.

In this post, we’ll explain exactly how we compute and use these metrics to get the best work out of our interviewers.

The definitive list of companies who are hiring engineers right now

interviewing.io is both a mock interview platform and an eng hiring marketplace (engineers use us for technical interview practice, and top performers get fast-tracked at companies), so we have some unique insights into how recent hiring freezes have affected engineers’ behavior. We also have unique insight into which companies are actually hiring. As such, in the spirit of being useful during a hard and uncertain time, we thought it’d be interesting to survey our users to see what’s actually going on in the market. TL;DR There are lots of engineers actively looking. There are also lots of companies who are actively hiring. Read the actual post to see the full list of 447 U.S. companies who are hiring software engineers right now.

Does communication matter in technical interviewing? We looked at 100K interviews to find out.

The interviewing.io platform has hosted and collected feedback from over 100K technical interviews, split between mock interviews and real ones. It’s generally accepted that to pass a technical interview, you have to not only come up with a solution to the problem (or at least make good headway), but you also have to do a good job of articulating your thoughts, explaining to your interviewer what you’re doing as you’re doing it, and coherently discussing tradeoffs and concepts like time and space complexity. But how important is communication in technical interviews, really? We looked at the data, and it turns out that talk is cheap. Read on to find out how and why.

We analyzed 100K technical interviews to see where the best performers work. Here are the results.

At interviewing.io, we’ve hosted over 100K technical interviews, split between mock interviews and real ones. As it happens, we know where our users currently work – they tell us that when they sign up. Given that we have this data AND given that we know how well people do in their interviews, we thought it would be interesting to see which companies’ engineers are especially good at technical interviews. Our resulting top ten lists are in this post!

The Eng Hiring Bar: What the hell is it?

Recursive Cactus has been working as a full-stack engineer at a well-known tech company for the past 5 years, but he’s now considering a career move. Over the past 6 months, Recursive Cactus (that’s his anonymous handle on interviewing.io) has been preparing himself to succeed in future interviews, dedicating as much as 20-30 hours/week plowing through LeetCode exercises, digesting algorithms textbooks, and of course, practicing interviews on our platform to benchmark his progress.

But this dedication to interview prep has been taking an emotional toll on him, ...

No engineer has ever sued a company because of constructive post-interview feedback. So why don’t employers do it?

One of the things that sucks most about technical interviews is that they’re a black box—candidates (usually) get told whether they made it to the next round, but they’re rarely told why they got the outcome that they did. Lack of feedback, or feedback that doesn’t come right away, isn’t just frustrating to candidates. It’s bad for business. We did a whole study on this. It turns out that candidates chronically underrate and overrate their technical interview performance, like so: Where this finding starts to get actionable is that there’s a statistically significant relationship between whether people think they did well in an interview and whether they’d want to work with you. In other words, …

We ran the numbers, and there really is a pipeline problem in eng hiring.

If you say the words “there’s a pipeline problem” to explain why we’ve failed to make meaningful progress toward gender parity in software engineering, you probably won’t make many friends (or many hires). The pipeline problem argument goes something like this: “There aren’t enough qualified women out there, so it’s not our fault if we don’t hire them.” Many people don’t like this reductive line of thinking because it ignores the growing body of research that points to unwelcoming environments that drive underrepresented talent out of tech: STEM in early education being unfriendly to children from underrepresented backgrounds, lack of a level playing field and unequal access to quality STEM education (see this study on …

You probably don’t factor in engineering time when calculating cost per hire. Here’s why you really should.

Whether you’re a recruiter yourself or an engineer who’s involved in hiring, you’ve probably heard of the following two recruiting-related metrics: time to hire and cost per hire. Indeed, these are THE two metrics that any self-respecting recruiting team will track. Time to hire is important because it lets you plan — if a given role has historically taken 3 months to fill, you’re going to act differently when you need to fill it again than if it takes 2 weeks. And, traditionally, cost per hire has been a planning tool as well — if you’re setting recruiting budgets for next year and have a headcount in mind, seeing what recruiting spent last year is …

Can fake names create bias? An exploration into interviewing.io’s pseudonym generator

Hello everyone, my name is Atomic Artichoke, and I’m the newest employee of the interviewing.io team, having joined a couple months ago as a Data Scientist. Atomic Artichoke isn’t my real name, of course. That’s the pseudonym the interviewing.io platform gave me, right before I took my final interview with the company. If you’ve never used interviewing.io before (and hey, if you haven’t already, why not sign up now?), it’s a platform where you can practice technical interviewing anonymously with experienced engineers (and do real job interviews anonymously too). When it’s time to interview, you and your partner meet in a collaborative coding environment with voice, text chat, and a whiteboard (check out recordings of …

There is a real connection between technical interview performance and salary. Here’s the data.

At the end of the day, money is a huge driver for the decisions we make about what jobs to go after. In the past, we’ve written about how to negotiate your salary, and there are a lot of labor statistics and reports out there looking at salaries in the tech industry as a whole. But as with many things in eng hiring, there’s very little concrete data on whether technical interview performance plays a role in compensation offers. So we set out to gather the data and asked our users who had gone on to successfully get jobs after using our platform to share their salary info. With our unique dataset of real coding …

Impostor syndrome strikes men just as hard as women... and other findings from thousands of technical interviews

The modern technical interview is a rite of passage for software engineers and (hopefully!) the precursor to a great job. But it’s also a huge source of stress and endless questions for new candidates. Just searching “how do I prepare for a technical interview” turns up millions of Medium posts, coding bootcamp blogs, Quora discussions, and entire books. Despite all this conversation, people struggle to know how they’re even doing in interviews. In a previous post, we found that a surprisingly large number of interviewing.io’s users consistently underestimate their performance, making them more likely to drop out of the process and ultimately harder to hire. Now, and with considerably more data (over 10k interviews led …

We looked at how a thousand college students performed in technical interviews to see if where they went to school mattered. It didn't.

interviewing.io is a platform where engineers practice technical interviewing anonymously. If things go well, they can unlock the ability to participate in real, still anonymous, interviews with top companies like Twitch, Lyft and more. Earlier this year, we launched an offering specifically for university students, with the intent of helping level the playing field right at the start of people’s careers. The sad truth is that with the state of college recruiting today, if you don’t attend one of very few top schools, your chances of interacting with companies on campus are slim. It’s not fair, and it sucks, but university recruiting is still dominated by career fairs. Companies pragmatically choose to visit the same …

What do the best interviewers have in common? We looked at thousands of real interviews to find out.

At interviewing.io, we’ve analyzed and written at some depth about what makes for a good interview from the perspective of an interviewee. However, despite the inherent power imbalance, interviewing is a two-way street. I wrote a while ago about how, in this market, recruiting isn’t about vetting as much as it is about selling, and not engaging candidates in the course of talking to them for an hour is a woefully missed opportunity. But, just like solving interview questions is a learned skill that takes time and practice, so, too, is the other side of the table. Being a good interviewer takes time and effort and a fundamental willingness to get out of autopilot and …

We analyzed thousands of technical interviews on everything from language to code style. Here's what we found.

If you’re reading this post, there’s a decent chance that you’re about to re-enter the crazy and scary world of technical interviewing. Maybe you’re a college student or fresh grad who is going through the interviewing process for the first time. Maybe you’re an experienced software engineer who hasn’t even thought about interviews for a few years. Either way, the first step in the interviewing process ...

LinkedIn endorsements are dumb. Here’s the data.

If you’re an engineer who’s been endorsed on LinkedIn for any number of languages/frameworks/skills, you’ve probably noticed that something isn’t quite right. Maybe they’re frameworks you’ve never touched or languages you haven’t used since freshman year of college. No matter the specifics, you’re probably at least a bit wary of the value of the LinkedIn endorsements feature. The internets, too, don’t disappoint in enumerating some absurd potential endorsements or in bemoaning the lack of relevance of said endorsements, even when they’re given in earnest. Having a gut feeling for this is one thing, but we were curious about whether we could actually come up with some numbers that showed how useless endorsements can be, and …

Lessons from 3,000 technical interviews… or how what you do after graduation matters way more than where you went to school

The first blog post I published that got any real attention was called “Lessons from a year’s worth of hiring data“. It was my attempt to understand what attributes of someone’s resume actually mattered for getting a software engineering job. Surprisingly, as it turned out, where someone went to school didn’t matter at all, and by far and away, the strongest signal came from the number of typos and grammatical errors on their resume. Since then, I’ve discovered (and written about) how useless resumes are, but ever since writing that first post, I’ve been itching to do something similar with interviewing.io’s data. For context, interviewing.io is a platform where people can practice technical interviewing anonymously …

After a lot more data, technical interview performance really is kind of arbitrary.

interviewing.io is a platform where people can practice technical interviewing anonymously, and if things go well, get jobs at top companies in the process. We started it because resumes suck and because we believe that anyone, regardless of how they look on paper, should have the opportunity to prove their mettle. In February of 2016, we published a post about how people’s technical interview performance, from interview to interview, seemed quite volatile. At the time, we just had a few hundred interviews to draw on, so as you can imagine, we were quite eager to rerun the numbers with the advent of more data. After drawing on over a thousand interviews, the numbers hold up. In …

People are still bad at gauging their own interview performance. Here’s the data.

interviewing.io is an anonymous technical interviewing platform. We started it because resumes suck and because we believe that anyone, regardless of how they look on paper, should have the opportunity to prove their mettle. In the past few months, we’ve amassed over 600 technical interviews along with their associated data and metadata. Interview questions tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role at a top company, and interviewers typically come from a mix of larger companies like Google, Facebook, and Twitter, as well as engineering-focused startups like Asana, Mattermark, KeepSafe, and more. Over the course of the next few posts, we’ll be sharing some …

We built voice modulation to mask gender in technical interviews. Here’s what happened.

interviewing.io is a platform where people can practice technical interviewing anonymously and in the process, find jobs based on their interview performance rather than their resumes. Since we started, we’ve amassed data from thousands of technical interviews, and in this blog, we routinely share some of the surprising stuff we’ve learned. In this post, I’ll talk about what happened when we built real-time voice masking to investigate the magnitude of bias against women in technical interviews. **In short, we made men sound like women and...

Technical interview performance is kind of arbitrary. Here’s the data.

interviewing.io is a platform where people can practice technical interviewing anonymously and, in the process, find jobs. In the past few months, we’ve amassed data from hundreds of interviews, and when we looked at how the same people performed from interview to interview, we were really surprised to find quite a bit of volatility, which, in turn, made us question the reliability of single interview outcomes.

Engineers can't gauge their own interview performance. And that makes them harder to hire.

interviewing.io is an anonymous technical interviewing platform. We started it because resumes suck and because we believe that anyone regardless of how they look on paper, should have the opportunity to prove their mettle. In the past few months, we’ve amassed over 600 technical interviews along with their associated data and metadata. Interview questions tend to fall into the category of what you’d encounter at a phone screen for a back-end software engineering role at a top company, and interviewers typically come from a mix...

Resumes suck. Here's the data.

About a year ago, after looking at the resumes of engineers we had interviewed at TrialPay in 2012, I learned that the strongest signal for whether someone would get an offer was the number of typos and grammatical errors on their resume. On the other hand, where people went to school, their GPA, and highest degree earned didn’t matter at all. These results were pretty unexpected, ran counter to how resumes were normally filtered, and left me scratching my head ...

Lessons from a year’s worth of hiring data

I ran technical recruiting at TrialPay for a year before going off to start my own agency. Because I used to be an engineer, one part of my job was conducting first-round technical interviews, and between January 2012 and January 2013, I interviewed roughly 300 people for our back-end/full-stack engineer position.

TrialPay was awesome and gave me ...