interviewing.io logo interviewing.io blog
better interviewing through data

We ran the numbers, and there really is a pipeline problem in eng hiring.

Introduction

Aline Lerner

Aline Lerner

Aline is the CEO and founder of interviewing.io.


LATEST POSTS

3 exercises to craft the kind of employer brand that actually makes engineers want to work for you 14th May, 2019

You probably don’t factor in engineering time when calculating cost per hire. Here’s why you really should. 24th April, 2019

Navigation

Uncategorized

We ran the numbers, and there really is a pipeline problem in eng hiring.

Posted on .

If you say the words “there’s a pipeline problem” to explain why we’ve failed to make meaningful progress toward gender parity in software engineering, you probably won’t make many friends (or many hires). The pipeline problem argument goes something like this: “There aren’t enough qualified women out there, so it’s not our fault if we don’t hire them.”

Many people don’t like this reductive line of thinking because it ignores the growing body of research that points to unwelcoming environments that drive underrepresented talent out of tech: STEM in early education being unfriendly to children from underrepresented backgrounds, lack of a level playing field and unequal access to quality STEM education (see this study on how few of California’s schools offer AP Computer Science for instance), hostile university culture in male-dominated CS programs, biased hiring practices, and ultimately non-inclusive work environments that force women and people of color to leave tech at disproportionately high rates.1

However, because systemic issues can be hard to fix (they can take years, concerted efforts across many big organizations, and even huge socioeconomic shifts), the argument against the pipeline problem tends to get reduced to “No, the candidates are there. We just need to fix the bias in our process.”

This kind of reductive thinking is also not great. For years, companies have been pumping money and resources into things like unconscious bias training (which has been shown not to work), anonymizing resumes, and all sorts of other initiatives, and the numbers have barely moved. It’s no wonder tech eventually succumbs to a “diversity fatigue” that comes from trying to make changes and not seeing results.

We ran the numbers and learned that there really IS a pipeline problem in hiring — there really aren’t enough women to meet demand… if we keep hiring the way we’re hiring. Namely, if we keep over-indexing on CS degrees from top schools, and even if we remove unconscious bias from the process entirely, we will not get to gender parity. And yes, there is a way to surface strong candidates without relying on proxies like a college degree. We’ll talk about that toward the end.

Our findings ARE NOT meant to diminish the systemic issues that make engineering feel unwelcome to underrepresented talent, nor to diminish our ability to work together as an industry to effect change — to enact policy changes like access to CS in public schools, for instance. Our findings ARE meant to empower those individuals already working very hard to make hiring better who find themselves frustrated because, despite their efforts, the numbers aren’t moving. To those people, we say, please don’t lose sight of the systemic problems, but in the short term, there are things you can do that will yield results. We hope that, over time, by addressing both systemic pipeline issues and biases, we will get to critical mass of women from all backgrounds in engineering positions, and that these women, in turn, will do a lot to change the pipeline problem by providing role models and advocates and by changing the culture within companies.

Lastly, an important disclaimer before we proceed. In this post, we chose to focus on gender (and not on race). This decision was mostly due to the dearth of publicly available data around race and intersectionality in CS programs/bootcamps/MOOCs.2 While this analysis does not examine race and intersectionality, it is important to note that we recognize: 1) Not all women have the same experience in their engineering journey, and 2) tech’s disparities by gender is no more important than the lack of representation of people of color in engineering. We will revisit these subjects in a future post.

The percentage of women engineers is low and likely worse than reported

It’s very hard to have a cogent public conversation about diversity when there is no standardization of what statistic means what. As this post is about engineering specifically, we needed to find a way to get at how many women engineers are actually in the market and work around two big limitations in how FAAMNG (Facebook, Amazon, Apple, Microsoft, Google, and Netflix) report their diversity numbers.

The first limitation is that FAAMNG’s numbers are global. Why does this matter? It turns out that other countries, especially those where big companies have their offshore development offices, tend to have a higher percentage of female developers.3 In India, for instance, about 35% of developers are women; in the U.S., it’s 16%. Why are these numbers reported globally? The cynic in me says that it’s likely because the U.S. numbers, on their own, are pretty dismal, and these companies know it.4 To account for this limitation and get at the U.S. estimate, we did some napkin math and conservatively cut each company’s female headcount by 20%.

The second limitation is that reported numbers are based on “technical roles,” which Facebook at least defines very broadly: “A position that requires specialization and knowledge needed to accomplish mathematical, engineering, or scientific related duties.” I expect the other giants use something similar. What are the implications of this fairly broad definition? Given that product management and UX design count as technical roles, we did some more napkin math and removed ~20% to correct for PMs and designers.4

With these limitations in mind, below is a graph comparing the makeup of the U.S. population to its representation in tech at FAAMNG companies where said data was available, as well as an estimate of women in engineering specifically.

There's still a lot of work to do if we want to reach gender parity in engineering

 

If we want to reach gender parity in engineering, especially when we correct for women in the U.S. (and whether they’re actually software engineers), you can see that we have a long way to go.

Is it a pipeline problem?

So, are there just not enough qualified women in the hiring pool? It turns out that we’re actually hiring women at pretty much the same rate that women are graduating with CS degrees from four-year universities — out of the 71,420 students who graduated with a CS degree in 2017, 13,654, or ~20%, were women.5 So maybe we just need more women to get CS degrees?

Top tech companies and their non-profit arms have been using their diversity and inclusion budgets to bolster education initiatives, in the hopes that this will help them improve gender diversity in hiring. Diversity initiatives started taking off in earnest in 2014, and in 4 years, enrollment in CS programs grew by about 60%. It’s not anywhere near enough to get to gender parity.

And even if we could meaningfully increase the number of women enrolling in CS programs overall, top companies have historically tended to favor candidates from elite universities (based on some targeted LinkedIn Recruiter searches, 60% of software engineers at FAAMNG hold a degree from a top 20 school). You can see enrollment rates of women in 3 representative top computer science programs below. Note that while the numbers are mostly going up, growth is linear and not very fast.

Growth in women's undergraduate computer science enrollment at top schools

Sources: UC-Berkeley, MIT (1 and 2), and Stanford enrollment data

To see if it’s possible to reach gender parity if we remove unconscious bias but keep hiring primarily from top schools, let’s build a model. For the purposes of this model let’s focus solely on new jobs — if companies want to meet their diversity goals, at a minimum they need to achieve parity on any new jobs they’ve created. Based on the US BLS’s projections, the number of software engineering jobs is estimated to increase by 20% by 2028 (or about 1.8% annually). Today, the BLS estimates there are about 4 million computer-related jobs. This projects to about 70,000 new jobs created this year, increasing to 85,000 new jobs created in 2028.

If the goal is to hit gender parity in the workforce, our goal should be to have 50% of these new seats filled by women.

To see if this is possible, let’s project the growth of the incoming job pool over the same timeframe. Based on NCES’ 2017 survey, computer science graduates have grown annually anywhere between 7% and 11% this decade. Let’s optimistically assume this annual growth rate persists at 10%. Let’s also assume that the
percentage of graduates who are women remains at 20%, which has been true for the last 15 years. But, there are some gotchas.

First, there’s no guarantee that the seats earmarked for women actually get filled by women, particularly in a world where male CS graduates will continue to outnumber females 4-to-1. Not all of these jobs will be entry-level, so some portion of these jobs will be pulling from an already female-constrained pool of senior candidates. Finally, there’s no guarantee that traditional 4-year colleges will be able to support the projected influx of computer science candidates, particularly from the top-tier universities that companies usually prefer. Below, we graph the net new seats we’d need to fill if women held half of software engineering jobs (blue line) vs. how many women are actually available to hire if we keep focusing largely on educational pedigree in our recruiting efforts (red line). As you can see, it’s not possible to hit our goals, whether or not we’re biased against women at any point in the hiring process.6

 

So if the pipeline is at least partially to blame, what can we do?

You saw above that enrollment in undergraduate computer science programs among women is growing linearly. Rising tuition costs coupled with 4-year universities’ inability to keep up with demand for computer science education have forced growing numbers of people to go outside the system to learn to code.

Below is a graph of the portion of developers who have a bachelor’s degree in computer science and the portion of developers who are at least partially self-taught, according to the Stack Overflow Developer Surveys from 2015 to 2019. As you can see, in 2015, the numbers were pretty close, and then, with the emergence of MOOCs, there was a serious spurt, with more likely to come.

Education sources for software engineers

Sources: Stack Overflow Developer Surveys for 2015, 2016, 2018, and 2019

The rate of change in alternative, more affordable education is rapidly outpacing university enrollment. Unlike enrollment in traditional four-year schools, enrollment in MOOCs and bootcamps is growing exponentially.

In 2015 alone, over 35 million people have signed up for at least one MOOC course, and in 2018 MOOCs collectively had over 100M students. Of course, many people treat MOOCs as a supplement to their existing educational efforts or career rather than relying on MOOCs entirely to learn to code. This is something we factored into our model.

Despite their price tag (most charge on the order of $10-20K), bootcamps seem like a rational choice when compared to the price of top colleges. Since 2013, bootcamp enrollment has grown 9X, with a total of 20,316 grads in 2018. Though these numbers represent enrollment across all genders7 and the raw number of grads lags behind CS programs (for now), below you can see that the portion of women graduating from bootcamps is also on the rise and that graduation from online programs has actually reached gender parity (as compared to 20% in traditional CS programs).

Source: Course Report’s Data Dive: Gender in Coding Bootcamps

Source: Course Report’s Data Dive: Gender in Coding Bootcamps

Of course, one may rightfully question the quality of grads from alternative education programs. We factored in bootcamp placement rates in building our updated model below.

Outside of alternative education programs, the most obvious thing we can do to increase the supply of qualified women engineers is to expand our pipeline to include strong engineers who don’t hail from top schools or top companies.

In previous posts, we looked at the relationship between interview performance and traditional credentialing and found that participation in MOOCs mattered almost twice as much for interview performance than whether the candidate had worked at a top company. And top school was least predictive of performance and sometimes not at all. And some of my earlier research indicates that the most predictive attribute of a resume is the number of typos and grammatical errors (more is bad), rather than top school or top company. In this particular study, experience at a top company mattered a little, and a degree from a top school didn’t matter at all.

But, even if lower-tier schools and alternative programs have their fair share of talent, how do we surface the most qualified candidates? After all, employers have historically leaned so hard on 4-year degrees from top schools because they’re a decent-seeming proxy. Is there a better way?

But culling non-traditional talent is hard… that’s why we rely on pedigree and can’t change how we hire!

In this brave new world, where we have the technology to write code together remotely, and where we can collect data and reason about it, technology has the power to free us from relying on proxies, so that we can look at each individual as an indicative, unique bundle of performance-based data points. At interviewing.io, we make it possible to move away from proxies by looking at each interviewee as a collection of data points that tell a story, rather than a largely signal-less document a recruiter looks at for 10 seconds and then makes a largely arbitrary decision before moving on to the next candidate.

Of course, this post lives on our blog, so I’ll take a moment to plug what we do. In a world where there’s a growing credentialing gap and where it’s really hard to figure out how to separate a mediocre non-traditional candidate from a stellar one, we can help. interviewing.io helps companies find and hire engineers based on ability, not pedigree. We give out free mock interviews to engineers, and we use the data from these interviews to identify top performers, independently of how they look on paper. Those top performers then get to interview anonymously with employers on our platform (we’ve hired for Lyft, Uber, Dropbox, Quora, and many other great, high-bar companies). And this system works. Not only are our candidates’ conversion rates 3X the industry standard (about 70% of our candidates ace their phone screens, as compared to 20-25% in a typical, high-performing funnel), about 40% of the hires made by top companies on our platform have come from non-traditional backgrounds. Because of our completely anonymous, skills-first approach, we’ve seen an interesting phenomenon happen time and time again: when an engineer unmasks at the end of a successful interview, the company in question realizes that the student who just aced their phone screen was one whose resume was sitting at the bottom of the pile all along (we recently had someone get hired after having been rejected by that same company 3 times based on his resume!).

Frankly, think of how much time and money you’re wasting competing for only a small pool of superficially qualified candidates when you could be hiring overlooked talent that’s actually qualified. Your CFO will be happier, and so will your engineers. Look, whether you use us or something else, there’s a slew of tech-enabled solutions that are redefining credentialing in engineering, from asynchronous coding assessments like CodeSignal or HackerRank to solutions that vet candidates before sending them to you, like Triplebyte, to solutions that help you vet your inbound candidate pool, like Karat.

And using these new tools isn’t just paying lip service to a long-suffering D&I initiative. It gets you the candidates that everyone in the world isn’t chasing without compromising on quality, helps you make more hires faster, and just makes hiring fairer across the board. And, yes, it will also help you meet your diversity goals. Here’s another model.

How does changing your hiring practices improve the pipeline?

Above, you saw our take on status quo supply and demand of women engineers — basically how many engineers are available to hire using today’s practices versus how many we’d need to actually reach gender parity. Now, let’s see what it looks like when we include candidates without a traditional pedigree (yellow line).

 

As you can see, broadening your pipeline isn’t a magic pill, and as long as demand for software engineers continues to grow, it’s still going to be really hard, systemic changes to our society notwithstanding. If we do make these changes, however, the tech industry as a whole can accelerate its path toward gender parity and potentially get there within a decade.

What about specific companies? An interactive visualization.

So far we’ve talked about trends in the industry as a whole. But, how do these insights affect individual employers? Below is an interactive model where you visualize when Google, Facebook, or your company (where you can plug in your hiring numbers) will be able to hit their goals based on current hiring practices versus the more inclusive ones we advocate in this post. Unlike the industry as a whole, built into this visualization is the idea of candidate reach, as well as hire rates — one company can’t source and hire ALL the women (as much as they might want to). Of course, the stronger your brand, the higher your response rates will be.

We made some assumptions about response rates to sourcing outreach for both Google and Facebook. Specifically, we guessed a 60%-70% response rate for these giants based on the strength of their brand and their army of sourcers — when those companies reach out and tenaciously follow up, you’ll probably respond eventually.8 We also made some assumptions about their hire rates (5-10% of interviewed candidates). You can see both sets of assumptions below. And you can see that even with all the changes we propose, in our model, Google and Facebook will still not get to gender parity!

We also included a tab called “Your company” where you can play around with the sliders and see how long it would take your company to get to gender parity/whether it’s possible. There, we made much more conservative assumptions about response rates!

As you can see, for the giants, getting to gender parity is a tall order even with broadening your pipeline to include non-traditional candidates. And while it may be easier for smaller companies to get there without making drastic changes, when you’re small is exactly the right time to get fairer hiring into your DNA. It’s much harder to turn the ship around later on.

Conclusion

Regardless of whether you’re a giant or a small company, as long as hiring practices largely limits itself to top schools, the status quo will continue to be fundamentally inefficient, unmeritocratic, and elitist, and any hope of reaching gender parity will be impossible. Look, there are no easy fixes or band-aids when it comes to diversifying your workforce. Rather than continuing to look for hidden sources of women engineers (I promise, we’re not all hiding somewhere, just slightly out of reach) or trying to hire all the women from top schools, the data clearly shows that the only path forward is to improve hiring for everyone by going beyond top schools and hiring the best people for the job based on their ability, not how they look on paper.

I was recently in a pitch meeting where I got asked what interviewing.io’s mission is. I said that it’s to make hiring more efficient. The investors in the room were a bit surprised by this and asked, given that I care about hiring being fair, why that’s not the mission. First off, “fair” is hard to define and open to all manners of interpretation, whereas in an efficient hiring market, a qualified candidate, by definition, gets the job, with the least amount of pain and missteps. In other words, meritocracy is a logical consequence of efficiency. Secondly, and even more importantly, while I firmly believe that most people at companies want to do “the right thing”, it’s much easier to actually do the right thing in a big organization when it’s also cheaper, better, and faster.

All that’s to say that there are no shortcuts, and the most honorable (and most viable) path forward is to make hiring better for everyone and then hit your diversity goals in the process (or at least get closer to them). Software engineering is supposed to be this microcosm of the American dream — anyone can put in the work, learn to code, and level up, right? Until we own our very conscious biases about pedigree and change how we hire, that dream is a hollow pipe.

Appendix: Model Description and assumptions

To assess whether there exists a pipeline problem, we need to estimate the number of job openings that exist, as well as the number of recent female job market entrants that could feasibly fill those roles. If a pipeline problem does exist, the number of job openings would be greater than the number of female entrants.

For this analysis, we focused on new jobs created over the next 10 years and ignored openings from existing jobs due to attrition. Unfortunately, engineering does have a significantly higher attrition rate for women than other industries, so likely the numbers are worse than they appear in our models.9

That said, if a company wants to meet its diversity goals, it seems reasonable to expect them to do so with jobs that don’t yet exist, rather than on existing jobs whose pool of candidates we know are dominated by men.

Demand: Projected new jobs created

Tech industry net new jobs created = (# tech industry jobs prior year) x (annual growth rate)

Assumptions:

Supply: Projected new women in job pool from top tier universities

New women in job pool = (# CS graduates prior year) x (annual CS graduate growth rate) x (% CS graduates that are women) * (% CS graduates from top tier schools)

Assumptions:

Supply: Projected new women in job pool beyond top tier universities

This represents female bootcamp graduates plus female CS graduates not from top schools.

New women in job pool from beyond top schools =
(# bootcamp graduates prior year) x (%  bootcamp graduates that are women) x (% annual bootcamp graduate growth)
+ (# CS graduates prior year) x (annual CS graduate growth rate) x (% women in CS grads) x (% CS graduates not from top tier schools)

Bootcamp assumptions:

  • Current # bootcamp graduates: 20,000 (Course Report)
  • % bootcamp graduates that are women: 40% (Switchup)
  • Bootcamp graduates growth rate: 10% (Course Report)
  • Placement rate: 50% (Switchup)
  • % CS graduates not from top tier schools: 75% (see assumption from “Supply: Projected new women in job pool from top tier universities”)

Assumptions for CS graduates beyond top tier universities are the same as those found under “Supply: Projected new women in job pool from top tier universities”, but taking the remaining 75% of CS graduates excluded there.

Company-specific Demand: Projected number of candidates needed to source for job openings

Number of women to source = (# Engineers employed prior year) x (% annual growth rate) x (% diversity goal) x (1 / hire rate) x (1 / sourcing response rate)

In practice, companies typically need to contact many people for any single job opening, since there is plenty of inherent variability in the sourcing and interview process. This line describes how many people your company would have to reach to fill all new job openings created, based on assumptions about your company’s hiring practices.

1The Kapor Center has some great, data-rich reports on attrition in tech as well as systemic factors that contribute to the leaky pipeline. For a detailed study of attrition in tech, please see the Tech Leavers Study. For a comprehensive look at systemic factors that contribute to the leaky tech pipeline (and a series of long-term solutions), please see this comprehensive report. For a survey of the deplorable state of computer science education in California schools, please see this report.
2For instance, MIT keeps their race stats behind a login wall.
3According to a HackerRank study, “India, the United Arab Emirates, Romania, China, Sri Lanka, and Italy are the six countries with the highest percentage of women developers, [whereas the] U.S. came in at number 11.”.
4We assumed a 1:7 ratio of PMs to engineers and a 1:7 ratio of designers to engineers on product teams. Removing PMs and designers from our numbers does not mean to imply that their representation in tech doesn’t matter but rather to scope this post specifically to software engineers.).
5The National Center for Education Statistics doesn’t yet list graduation rates beyond 2017… the new numbers might be a bit higher, as you’ll see when you look at enrollment numbers for CS a bit further down in the post..
6This is not independent of the idea that deep, systemic issues within the fabric of our society (such as the ones we mention at the beginning of the post) are keeping women from entering engineering in droves. But, as I mentioned in the intro, laying the blame at the feet of these systemic issues entirely paralyzes us and prevents us from fixing the things we can fix.
7We couldn’t find a public record of women’s numbers by year and so are relying on graduation rates from Course Report as a proxy.
8These rates might seem high to recruiters reading this post. They might be high. We did try to correct for 2 things, both of which made our estimates higher: 1) this includes university and junior candidates, who tend to be way more responsive, and 2) this isn’t per sourcing attempt but over the lifetme of a candidate, so it includes followups, outreach to candidates who had previously been declined, and so on. However, if this still seems off, please write in and tell us!
9There are two good sources that look at attrition among women in tech. One is Women in Tech: The Facts, published by the National Center for Women&IT. The other is the excellent Tech Leavers Study by the Kapor Center.

Thank you to the interviewing.io data & eng team for all the data modeling, projections, and visualizations, as well as everyone who proofread the myriad long drafts.

Aline Lerner

Aline Lerner

Aline is the CEO and founder of interviewing.io.

Comments
  • user

    AUTHOR Charles Pritchett

    Posted on 6:31 pm December 3, 2019.
    Reply

    James Damore very succinctly provided the answer for why there’s a pipeline difference: men and women just flat out have different interests.
    The real question is why do really smart people who see the differences between men and women (women like going into nursing more than men, men like going into tech more than women, etc) refuse to acknowledge this?
    Is India, where random women get raped on the street, really some enlightened society? Or are women going into technology there because it’s one of the few routes to the middle class for a lot of them?
    Really makes you think…

  • user

    AUTHOR James

    Posted on 4:24 pm December 3, 2019.
    Reply

    In India everybody can become an engineer for companies like Infosys, Wipro, etc. Even if you don’t know how to write hello world program, they will still hire you. If you can filter the male female ratio only for tech companies, then it will be a better comparison.

  • user

    AUTHOR MarcusG

    Posted on 12:55 pm December 3, 2019.
    Reply

    Despite extensive effort towards diversity my company in Oil and Gas industry fails to attract female software engineers lingering at merely 16% women in tech and managerial roles. There is definitely a pipeline problem. We had a case where one man had a problem working under a female manager and he was fired next day. I spent hours on workplace professionalism, unconscious bias, inclusion training. The effort is great, the results are none.

  • user

    AUTHOR Ben

    Posted on 12:17 pm December 3, 2019.
    Reply

    Typo in the Appendix, paragraph 2:

    Unfortunately, engineering does have a significantly higher retention rate for women than other industries, so likely the numbers are worse than they appear in our models.

    retention -> attrition

    • user

      AUTHOR Aline Lerner

      Posted on 12:22 pm December 3, 2019.
      Reply

      thank you, fixing now!

  • user

    AUTHOR Radhika

    Posted on 9:58 am December 3, 2019.
    Reply

    Doesn’t this prove that there _isn’t_ a pipeline problem in most engineering organizations? Considering that companies at a lower tier than FAANMG (and other well-known companies like that) are sourcing folks from all genders from lower-tier schools and bootcamps, what’s the excuse there?

    I see this post as an argument for elite companies to stop focusing so much on elite credentials, but what does this say about the startup with 50 engineers that are all men? What does this say about the IT company in Nevada? What does this say about IBM?

  • View Comments (10) ...