You now need to do 15% better in technical interviews than you did at the start of 2022 (and the bar will keep rising).

interviewing.io is a technical mock interview platform and technical recruiting marketplace, so we have a ton of useful data around technical interviewing and hiring. One of the most useful pieces of data in the current climate is the ever-changing technical interview bar – throughout 2022, it’s gotten progressively harder to pass technical interviews, and it’s only going to keep getting harder… because employers are gaining more leverage in the market.

One might say that because we’re a mock interview platform, publishing this post is self-serving. The reality is that we take no joy in this content. Ever since hiring freezes and layoffs have started dominating the news cycle, we’ve lost about half of our interview volume, many of the prominent employers who were hiring through us as recently as Q2 of this year have paused. So, trust us, we’d much rather be writing about something else while living in a hiring boom. And look, whether we want to see it or not or whether we have vested interest in publishing it or not, the data is the data, and the data doesn’t care. Realistically, neither will the employers who are still interviewing and hiring.

We hope that what we’re about to share will help you enter an increasingly unforgiving labor market, the likes of which engineers haven’t seen since the early 2000s, with your eyes wide open.

The state of the world

Despite not wanting to believe it (as Upton Sinclair said, “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”), I’m coming around to the idea that engineers are going to lose significant leverage in the market.

In the graph above, the blue line is the number of open tech jobs, according to TrueUp. TrueUp is a tech job index and layoff tracker, started by Amit Taylor. One of its limitations is that it tracks jobs at somewhat arbitrarily defined “tech” companies and excludes jobs (even if they’re technical) at non-tech companies (e.g. Walmart… their example, not ours). However, it’s the best aggregation of open jobs over time that I’ve been able to find, and I think it’s as good a proxy as any.

The red line comes from our own analysis of tech layoffs (it’s on its own axis because the blue line represents the number of open tech jobs across all disciplines, whereas the red line is engineers only). We recently looked at layoff lists on layoffs.fyi, tagged them by function, and did some corrections for how likely people in different functions were to opt in to those lists. We ultimately learned that engineers comprised about 5% of total layoffs overall (though eng departments were hit harder than we expected, and at companies that did layoffs, about 12% of the eng team was let go). As such, we counted up total layoffs in the US by month and took 5% of that.

TL;DR The number of open jobs is shrinking, and the number of unemployed engineers is growing.

How much will this actually affect hiring practices and how high the bar will actually get?

It’s hard to say, and I’m not an economist, but I do run an eng hiring marketplace, and we have some proprietary data I’d like to share with you all.

The rising eng bar

interviewing.io is both a mock interview platform and an eng hiring marketplace – engineers use us for technical interview practice, and top performers get fast-tracked at companies.

Companies can actually interview our top performers anonymously, right on our platform, and leave feedback after each interview. If the candidate passes, they unmask and move to the next step (typically an onsite). Feedback is both quantitative and qualitative, and in addition to telling us if the candidate passed, companies also rate them on technical ability, communication ability, and problem solving ability. Technical ability is the most predictive and is therefore weighed the most heavily in our scoring system.

While we’re not contractually allowed to list publicly which companies hire through us, it’s historically been a mix of FAANGs, FAANG-adjacent top tier companies (e.g. ride sharing, file sharing), and up and coming startups. That is to say that we’re confident that our data about the eng bar is indeed representative of what’s happening at the top end of the market.

To figure out where the bar is and how it’s changed, we averaged the technical scores for successful interviews among senior engineers¹ over the last few quarters and graphed it against trueup.io’s number of open tech jobs (same as the blue line in the graph above). Below, you can see the results. The shape of the 2 curves surprised us in its uncanniness. Since January of 2022, after a brief rise in Q2, tech jobs have contracted by 40%. At the same time, after a brief dip in Q2, the bar for a successful technical interview has increased by 10 percentile points (or by about 15%).²

The interviewing.io technical interview bar index

Here’s our quantification of where the “eng bar” is. We’ll call it the interviewing.io technical interview bar index (ITIB index). We used historical data to measure the link between the number of open tech jobs and the typical performance of people who cleared a real technical interview on our platform. As the plot above shows, it’s a negative relationship: when the job pool shrinks, candidates have to perform better in technical interviews.

Our regression estimates show that for every 50K drop in tech jobs, candidates need to perform about 3 percentiles higher in interviews. Given the almost halving of jobs that has happened so far in 2022, candidates went from needing to beat about two thirds of candidates who had also gotten to the phone screen stage to needing to beat more than three quarters.

One caveat to this approach is that interviewing.io probably doesn’t represent the whole eng market for 2 reasons: selection bias among employers and selection bias among candidates. As I mentioned earlier, the employers who hire through us tend to have a high bar and conduct a specific kind of interview (typically algorithmic in the phone screen). Moreover, only people who do really well in mock interviews get to the point where they can talk to real companies, so these percentiles are comparing people who are already very strong performers to one another rather than to everyone in the pool. However, because they’re relative rather than absolute, we feel reasonably confident that these numbers can be generalized to the overall hiring market.

The ITIB index captures this in a single number, measuring the current technical bar in percentile terms. If it’s at 40, you need to be better than 40% of engineers to get a Yes. Right now, it’s at 78, up from 68 in Q1 of 2022.

We’ll be tracking this index on a monthly basis on Twitter. Follow us at @interviewingio.

Footnotes