Uncategorized

Technical phone screen superforecasters

Posted by Guest Author on November 25th, 2020.

Hey, Aline (founder of interviewing.io) here. This is the third post in our Guest Author series.

In this post, our latest Guest Author looks at interviews from the company’s perspective. So much engineering time goes into interviewing… we know this firsthand, but what can be done about it? Some companies solve this problem by introducing homework. In this post, our Author digs into some historical data to unearth a really clever, elegant way to save eng time that’s also better for candidate experience!

If you have strong opinions about interviewing or hiring that you’ve been itching to write about, we’d love to hear from you. Please email me at aline@interviewing.io to get started.

Alexey Komissarouk is a growth engineering leader. He’s currently working on growth at MasterClass. Before that, he spent 2016-2020 at Opendoor, first as an early engineer, then as an Engineering Manager. Between 2013 and 2016, he built out a product engineering consulting company, helping clients such as Dropbox, Pebble, Boomerang, and Binti grow and expand lines of business through a combination of product management and engineering. In his other lives, Alexey co-founded a boutique work+travel company, Hacker Paradise. Since the company’s inception in 2015, they’ve run trips to over a dozen locations and been joined by more than 800 alumni.

“The new VP wants us to double engineering’s headcount in the next six months. If we have a chance in hell to hit the hiring target, you seriously need to reconsider how fussy you’ve become.”

It’s never good to have a recruiter ask engineers to lower their hiring bar, but he had a point. It can take upwards of 100 engineering hours to hire a single candidate, and we had over 50 engineers to hire.  Even with the majority of the team chipping in, engineers would often spend multiple hours a week in interviews. Folks began to complain about interview burnout. Also, fewer people were actually getting offers; the onsite pass rate had fallen by almost a third, from ~40% to under 30%. This meant we needed even more interviews for every hire.

Visnu and I were early engineers bothered most by the state of our hiring process. We dug in. Within a few months, the onsite pass rate went back up, and interviewing burnout receded.  We didn’t lower the hiring bar, though. There was a better way.

We took the company’s best technical interviewers and organized them into a dedicated Phone Screen Team. No longer would engineers be assigned between onsite interviews and preliminary phone screens at recruiting coordinators’ whims. The Phone Screen Team specialized in phone screens; everybody else did onsites.

Honestly, all I wanted at the start was to see if I was a higher-signal interviewer than my buddy Joe. So I graphed people’s phone screen pass rate against how those candidates performed in their onsite pass rate.

Joe turned out to be the better interviewer. More importantly, I stumbled into the fact that a number of engineers doing phone screens performed consistently better across the board. They both had more candidates pass their phone screens and then those candidates would get offers at a higher rate.

These numbers were consistent, quarter over quarter. As we compared the top quartile of phone screeners to everybody else, the difference was stark. Each group included a mix of strict and lenient phone screeners; on average, both groups had a phone screen pass rate of 40%.

The similarities ended there: the top quartile’s invitees were twice as likely to get an offer after the onsite (50% vs 25%). These results also were consistent across quarters.[1]

Armed with newfound knowledge of phone screen superforecasters, the obvious move was to have them do all the interviews. In retrospect, it made a ton of sense that some interviewers were “just better” than others.

A quarter after implementing the new process, the “phone screen to onsite” rate stayed constant, but the “onsite pass rate” climbed from ~30% to ~40%, shaving more than 10 hours-per-hire[2]. Opendoor was still running this process when I left several years later.

You should too[3], [4].

Get your Lever or Greenhouse (or ATS of choice) into an analyzable place somewhere, and then quantify how well interviewers perform[5]. There’s lots of ways to analyze performance; here’s a simple approach which favors folks who generated lots of offers from as few as possible onsites and phone screens.

You can adjust the constants to where zero would match a median interviewer. A score of zero, then, is good. Your query will look something like this:

Ideally, hires would also be included in the funnel, since a great phone screen experience would make a candidate more likely to join. I tried including them; unfortunately, the numbers get too small and we start running out of statistical predictive power.

Phone Screen interviewers no longer do onsite interviews (except as emergency backfills). The questions they ask are now retired from the onsite interview pool to avoid collisions.

Ask the engineers to identify and block off 4 hour-long weekly slots to make available to recruiting (recruiting coordinators will love you). Use a tool like youcanbook.me or calendly to create a unified availability calendar. Aim to have no more than ~2.5 interviews per interviewer per week. To minimize burnout, one thing we tried was to take 2 weeks off interviewing every 6 weeks. 

To avoid conflict, ensure that interviewers’ managers are bought in to the time commitment and incorporate their participation during performance reviews.

When new engineers join the company and start interviewing, they will initially conduct on-site interviews only. If they perform well, consider inviting them into the phone screen team as slots open up. Encourage new members to keep the same question they were already calibrated on, but adapt it to the phone format as needed. In general, it helps to make the question easier and shorter than if you were conducting the interview in person.

When onboarding a new engineer onto the team, have them shadow a current member twice, then be reverse-shadowed by that member twice. Discuss and offer feedback after each shadowing.

Interviewing can get repetitive and lonely. Fight this head-on by having recruiting coordinators add a second interviewer (not necessarily from the team) to join 10% or so of interviews and discuss afterwords.

Hold a monthly retrospective with the team and recruiting, with three items on the agenda:

• discuss potential process improvements to the interviewing process
• review borderline interviews with the group to review together, if your interviewing tool supports recording and playback
• have interviewers read through feedback their candidates got from onsite interviewers and look for consistent patterns

Eventually, interviewers may get burnt out and say things like “I’m interviewing way more people than others on my actual team – why? I could just go do onsite interviews.” This probably means it’s time to rotate them out. Six months feels about right for a typical “phone screen team” tour of duty, to give people a rest. Some folks may not mind and stay on the team for longer.

Buy exclusive swag for team members. Swag are cheap and these people are doing incredibly valuable work. Leaderboards (“Sarah interviewed 10 of the new hires this year”) help raise awareness. Appreciation goes a long way.

Also, people want to be on teams with cool names. Come up with a cooler name than “Phone Screen Team.” My best idea so far is “Ambassadors.”

There’s something very Dunder Mifflin about companies that create Growth Engineering organizations to micro-optimize conversion, only to have those very growth engineers struggle to focus due to interview thrash from an inefficient hiring process. These companies invest millions into hiring, coaching and retaining the very best sales people. Then they leave recruiting – selling the idea of working at the company – in the hands of an engineer that hasn’t gotten a lick of feedback on their interviewing since joining two years ago, with a tight project deadline on the back of her mind.

If you accept the simple truth that not all interviewers are created equal, that the same rigorous quantitative process with which you improve the business should also be used to improve your internal operations, and if you’re trying to hire quickly, you should consider creating a Technical Phone Screen Team.

1. Was this statistically significant, or are you conducting pseudoscience? Definitely pseudoscience. Folks in the sample were conducting about 10 interviews a month, ~25 per quarter. Perhaps not yet ready to publish in Nature but meaningful enough to infer from, especially considering the relatively low cost of being wrong. 

2. Why didn’t the on-site pass rate double, as predicted? First, not all of the top folks ended up joining the team. Second, the best performers did well because of a combination of skill (great interviewers, friendly, high signal) and luck (got better candidates). Luck is fleeting, resulting in a regression to the mean.

3. What size does this start to make sense at? Early on, you should just identify who you believe your best interviewers are and have them (or yourself) do all the phone screens. Then, once you start hiring rapidly enough that you are doing about 5-10 phone screens a week, run the numbers and invite your best 2-3 onsite interviewers to join and create the team.

4. What did you do for specialized engineering roles? They had their own dedicated processes. Data Science ran a take home, Front-End engineers had their own Phone Screen sub-team, and Data and ML Engineers went through the general full-stack engineer phone screen.

5. Didn’t shrinking your Phone Screener pool hurt your diversity? In fact, the opposite happened. First, the phone screener pool had a higher percentage of women than the engineering organization at the time; second, a common interviewing anti-pattern is “hazing” – asking difficult questions and then rejecting somebody for “not even remembering about Kahn’s algorithm, lolz.” The best phone screeners don’t haze, bringing a more diverse group onsite.

Uncategorized

I’ve conducted over 600 technical interviews on interviewing.io. Here are 5 common problem areas I’ve seen.

Posted by Guest Author on October 15th, 2020.

Hey, Aline (founder of interviewing.io) here. This is the second post in our Guest Author series The first post talked about red flags you might encounter while interviewing with companies. Complementarily, this post, authored by one of our prolific, long-time interviewers, explores common missteps that interviewees make.

One of the things I’m most excited about with the Guest Author series is the diversity of opinions it’s bringing to our blog. Technical interviewing and hiring is fraught with controversy, and not everything these posts contain will be in line with my opinions or the official opinions of interviewing.io. But that’s what’s great about it. After over a decade in this business, I still don’t think there’s a right way to conduct interviews, and I think hiring is always going to be a bit of a mess because it’s a fundamentally human process. Even if we don’t always agree, I do promise that the content we put forth will be curated, high quality, and written by smart people who are passionate about this space.

If you have strong opinions about interviewing or hiring that you’ve been itching to write about, we’d love to hear from you. Please email me at aline@interviewing.io to get started.

William Ian Douglas goes by “Ian”, and uses he/him pronouns. He lives in the Denver, Colorado region and graduated from a Computer Engineering program in 1996. His career spans back-end systems, API architecture, DevOps/DBA duties and security, and has been a team lead managing small teams, and Director of Engineering. Ian branched out into professional technical interview coaching in 2014, and in 2017 pivoted his entire career to teaching software development for the Turing School of Software & Design in the Denver area. He joined interviewing.io as a contract interviewer in the summer of 2017 and is a big fan of the data analytics blog posts that IIO produces to help expose and eliminate bias in our tech industry interviews. Ian writes technical coaching information at https://techinterview.guide and you can reach him on Twitter, LinkedIn and GitHub.

I recently conducted my 600th interview on interviewing.io (IIO). I’d like to share lessons learned, why I approach interviews the way that I do, and shed some light on common problem areas I see happen in technical interviews. Every interviewer on the platform is different, and so your results may vary. We have some excellent folks helping out on the platform, and have a wonderful community working to better ourselves.

During our interviews on IIO, we rate people on three 4-point scales. A score of 1 means they did extremely poorly, and a 4 means they did extremely well in that category. I typically start my interview where everyone gets 3 out of 4 points right away, and then earn/lose points as the interview goes on.

Every interviewer on the platform will have some aspect that they favor over others. My own bias as an interviewer tends to be around communication and problem solving, which I’ll point out below.

In this category, I grade a candidate on how proficient they seem in their language of choice, whether they had significant problems coding an algorithm of a particular style, if I needed to give a lot of hints during coding.

Here, I grade a candidate on how well they break the problem into smaller pieces, come up with a strategy for solving the smaller problems, and also debugging issues along the way. The ability to think through problems while debugging is just as important as writing the code in the first place. Are they stumped when a problem happens, or are they able to find the root cause on their own?

Interviewers really want to hear your decision-making process. This is also very important when debugging code. I tended to hire folks who would fit in well on smaller teams or clusters of developers. With that in mind, collaboration and easy communication are a good way to win me over.

Here are the top problem areas I see in interviews, not just on IIO, but in general. I hope you find this advice helpful.

I see this in developers of all types and levels, but mostly in the “intermediate” level of 2-5 years of experience. They hear a problem, talk about a high-level design for 30 seconds or less, and are eager to get coding. They feel like they’re on a timer. They want to rush to get things finished. It’s a race to the finish line. First one across the finish line is the winner.

Right?

Nope.

Please, slow down. Plan your work. And share your thought process along the way.

People who take time to think out a mid-level design, whether that’s pseudocode or just writing out notes of their approach, tend to spend less time debugging their code later. Folks who jump right into coding fall into what I call “design-as-you-go” problems, where you’re spending lots of time refactoring your code because you need to change a parameter passed or a return value, or wait, that loop is in the wrong place, etc.. This is very easy to spot as an interviewer.

Spending some time on mid-level design doesn’t guarantee your success, but it might save you time in the long run by thinking through your plan a little deeper, and that extra time you bought could be used to fix problems later.

Also, as an interviewer, I want to see you succeed. Especially if you’re “on-site” (in person, or remote nowadays) because you’re costing our company a lot more money in an on-site interview process. While I need to be fair to all candidates in the amount of help I can give, if I can see your design ahead of time, and spot a flaw in that design, I can ask leading questions to guide you to the problem and correct your approach earlier.

If you jump straight into code, I have no idea if your implementation is even going to work, and that’s not a great place to put your interviewer. It’s much harder for me to correct a design when you have 100 lines of Java code written before I really understand what’s going on in your code.

I saw this lack of planning backfire in a horrible way in a real interview in 2012. The candidate was brought to my interview room by someone in HR, asked if they would like a bottle of water, and promised to return. We introduced ourselves and got down to the technical challenge. The candidate shared no details, no design, barely talked about a high-level approach, wrote nothing down, and started writing code on a whiteboard. (This would be my second-to-last whiteboard interview I ever conducted, I hate whiteboard interviews!) HR showed up a few minutes later, knocking loudly on the door, offering the bottle of water and leaving. The candidate, grateful for a drink, uncapped the bottle and started to take a sip when this awful, draining look came over their face. The distraction of delivering a bottle of water made them completely lose their train of thought, and I couldn’t help them recover because they hadn’t shared any details with me about their approach. They spent several minutes re-thinking the problem and starting over.

On the “other side” of this coin, however, you can spend “too long” on the design stage and run out of time to implement your genius plan. I’ve seen candidates talk through a mid-level design, then write notes, then manually walk through an example with those notes to really make sure their plan is a good one, and now they only have a few minutes left to actually implement the work. Extra points on communication, maybe, but we need to see some working code, too.

So what’s the best approach here?

I typically recommend practicing until you spend about 5 minutes thinking through high-level design choices, 5 minutes to plan and prove the mid-level design, and then get to work on code. The good news here is that “practice makes better” — the more you practice this design break-down and problem solving, the better you’ll get. More on this later.

This is a term I’ve coined over the years, where you start to say a thought out loud, finish the thought in your head, and then change something about your code. It usually sounds something like this:

“Hmm, I wonder if I could … … … no, never mind, I’ll just do this instead.”

Back to my bias for communication.

Interviewers want to know what’s going on in your thought process. It’s important that they know how you’re making decisions. How are you qualifying or disqualifying ideas? Why are you choosing to implement something in a particular way? Did you spot a potential problem in your code? What was it?

This missing information is a hidden treasure for your interviewer. It takes mere seconds to change your communication to something more like this:

“I wonder if … hmm … well, I was thinking about implementing this as a depth-first-search, but given a constraint around ___ I think a better approach might be ___, what do you think?”

That took maybe 2 or 3 extra seconds, and you’ve asked for my opinion or buy-in, we can consider possibilities together, and now we’re collaborating on the process. You already feel like my future coworker!

An interview challenge I often ask as a warm-up question goes something like this: 

You have a grouping of integer numbers. Write a method that finds two numbers that add up to a given target value, stop immediately, and report those numbers. Return two ‘null’ values if nothing is found.

This is a great question that shows me how you think about algorithms and the kinds of assumptions you make when you hear a problem.

I’ve been coding for a pretty long time. Since 1982, actually. There’s no data structure called “a grouping” in any language I’ve ever used. So what assumptions are you going to make about the problem?

Most candidates immediately assume the “grouping” of numbers is in an array. You can successfully solve this problem by using an array to store your numbers. Your algorithm will likely be an O(n^2) (n-squared) algorithm because you’ll be iterating over the data in an exponential way: for each value, iterate through the rest of the values. There’s a more efficient way to solve this in O(n) time by choosing a different data structure.

Go ahead and ask your interviewer questions about the problem. If they tell you to make your own assumptions that’s different, but ask if they’re good assumptions. Ask if there are alternate data sets that you’ll be using as test cases which could impact your algorithm.

Wait, what?

Yeah, you read me right.

Yes, you’re there for the interview, but you’re there to show them how you’ll work on the team, and teams work best when there is clear, open communication and a sense of collaboration. Spend the first few minutes of the interview setting expectations, especially around communication and work process.

There’s nothing wrong with having this kind of chat with your interviewer: “My typical work process in a technical challenge like this is to spend a minute or two thinking quietly about the problem and writing down notes, I’ll share those thoughts with you in a moment to get your input. Then, while I code, I tend to work quietly as well, but I’ll be sure to pause now and then to share my thought process as I go, and then walk you through the code more thoroughly before we run it the first time. Would that be okay with you, or do you have different expectations of how you’d like me to communicate or work through the problem?”

I promise you’ll blow their mind. Most interviewers won’t be ready for you to take their expectations into consideration like this. It shows that you’ll work well on a team. You’re setting the environment where you’re advocating for yourself, but also being considerate of others. You’re stating your intentions up front, and giving them the opportunity to collaborate on the process.

As your interviewer, I have a small amount of help that I’m likely able to provide during a technical challenge. I can’t coach you through everything, obviously, but I’d rather give you a hint, deduct a point on a rubric, and see you ultimately succeed at the problem, than to struggle silently and spin in circles and make us both feel like the interview is a waste of time.

As a professional interviewer and an instructor at a software school, I’ve become pretty good at asking leading questions to guide you to a realization or answer without me giving you the solution.

It’s okay to admit when you’re stuck. It doesn’t make you a failure, it makes you human. Let your interviewer know what you’re thinking and  where you’re having problems. Listen very carefully to their response, they might be offering a clue to the problem, or might give you more thorough advice on how to proceed.

When our interviews at IIO are over, I like to dive into a lot of feedback on their process and where I think they could use extra practice to improve. Generally, I spend 10 to 20 minutes, sometimes going way beyond my one-hour expected time slot, to answer questions for someone, and going into more detail on things. I LOVE to help people on IIO.

Here are a few common areas of advice I offer to folks.

There’s nothing worse than listening to your own recorded voice. But all IIO interviews are recorded, and I often tell folks in my feedback and in the review notes I type up afterward to listen to the last few minutes of the interview recording to review the feedback I give them. You can also pause those recordings and grab a copy of your code at any time. (These recordings are of course private to your and your interviewer.)

During the playback, listen to your own thought process and how you communicate your ideas. As you work through other challenges, find a way to record yourself talking through the problem out loud if possible, and play that back for yourself. You’ll get better at articulating full and complete thoughts.

The more common practice sites like HackerRank, CodeWars, LeetCode, etc, are great for writing a coded algorithm, but don’t give you any way to exercise your design process.

I send my students to Project Euler. Euler was a mathematician, so the problems on the website will generally be pretty math-heavy, but you can change the problems to be whatever you’re comfortable building. If you don’t know how to calculate a prime number, that’s fine, swap that out for whether a number is equally divisible by 17 or something instead.

I like Project Euler because the challenges there are just word problems. You have to think of everything: the algorithm, which data structure(s) to use, and especially how to break the problem into smaller pieces.

One of my favorite problems is #19 in their archive: counting how many months between January 1901 and December 1999 began on a Sunday. They give you the number of days in each calendar month, tell you January 1st 1900 is a Monday, and how to calculate a leap year. The rest is up to you.

The more you expose yourself to different types of problems, the better you’ll get at spotting patterns.

One piece of advice we give our students is to practice each technical challenge several times. Our executive director, Jeff Casimir, tells students to practice something 10 times. That feels like a big effort. I aim more for 3 to 4 times, and here’s my reasoning:

The first time you solve a problem, all you’ve done is solve the problem. You might have struggled through certain parts, but your only real achievement here is finishing.

If you erase your work and start it a second time, you might think of a different approach to solving the problem, maybe a more efficient solution. Maybe not, but at least you’re getting practice with this kind of problem.

Now erase your work and do it a third time. Then a fourth time. These are the times when you will start to actively build a memory of the strategy it takes to solve this particular problem. This “muscle memory” will help you when you see other technical challenges, where you’ll start to spot similarities. “Oh, this looks like the knapsack problem” and because you’ve solved that several times, the time you take on high level design and mid-level design just shortened quite a lot.

One of my favorite technical challenges can be solved using a handful of different algorithms (DFS, BFS, DP, etc). If you think you can solve a problem in a similar fashion, solve it 3 or 4 times with each of those algorithms as well. You’ll get REALLY good at spotting similarities, and have a great collection of strategies to approach other technical problems.

I’ve been writing up notes for aspiring new developers at https://techinterview.guide. It’s not complete, but I have a lot of my own thoughts on preparing for technical interviews, networking and outreach, resumes and cover letters, and so on. I still have a few chapters to write about negotiation tactics and graceful resignations, but I’m happy to take feedback from others on the content.

I also have a daily email series covering several kinds of interview questions, but not from a perspective of how to answer the questions perfectly, there are plenty of resources out there to do that. Instead, I examine questions from an interviewer’s perspective — what am I really asking, what do I hope you’ll tell me, what do I hope you won’t say, and so on. A small preview, for example: when you’re asked “Tell me about yourself” they’re not really asking for your life story. They’re really asking “Tell me a summary of things about you that will make you a valuable employee here”.

Uncategorized

6 red flags I saw while doing 60+ technical interviews in 30 days

Posted by Guest Author on September 15th, 2020.

Hey, Aline (founder of interviewing.io) here. We’re trying something new. Up till now, all posts on this blog have been written by interviewing.io employees or contractors. Why? Frankly, it’s hard to find great content in the recruiting space. There’s so much fluff and bad advice out there, and we didn’t want any part of that.

The other day though, I was reading Hacker News and saw an article by Uduak Obong-Eren about how he did over 60 technical interviews in 30 days and what he learned from that gauntlet of an experience. I thought it was honest, vulnerable, well-written, and brimming with actionable advice. So, I reached out to him to see if he’d want to write something else. Fortunately, he did, and the article below is the inaugural post in what I hope will become our Guest Author series. You can read more about Uduak in the bio below.

A quick note because this is the first time we’re doing this. One of the things I’m most excited about with this new Guest Author series is the diversity of opinions it will bring to our blog. Technical interviewing and hiring is fraught with controversy, and not everything these posts contain will be in line with my opinions or the official opinions of interviewing.io. But that’s what’s great about it. After over a decade in this business, I *still* don’t think there’s a right way to conduct interviews, and I think hiring is always going to be a bit of a mess because it’s a fundamentally human process. Even if we don’t always agree, I do promise that the content we put forth will be curated, high quality, and written by smart people who are passionate about this space.

Now, off we go!

Uduak Obong-Eren is a Software Engineer based in the San Francisco Bay Area who is passionate about architecting and building scalable software systems. He has about five years of industry experience and holds a Masters in Software Engineering from Carnegie Mellon University. He is also an open source enthusiast and writes technical articles – you can view some of his writings at https://meekg33k.dev. He especially enjoys conducting free mock technical interviews to help folks get better at technical interviewing. You can follow him on Twitter @meekg33k.

What is the one thing you would look out for if you had to join a company?

Sometime between January and February 2020, I wanted to change jobs and was looking to join a new company. This, among other reasons, led me to embark on a marathon of technical interviews – 60+ technical interviews in 30 days. 

Doing that many number of interviews in such a short time meant I had an interesting mix of experiences from the various companies I interviewed with, each with their unique culture and values that often reflected in the way their interviews were conducted, intentionally or not.

In this article, I will be sharing some of the red flags I observed while I was on this marathon of technical interviews. I will not be mentioning names of any companies because that’s not the intent behind this article. 

The goal of this article is also not to make you paranoid and be on the hunt for red flags in your next interview, far from it. Rather the goal is to equip you with knowledge to help you immediately identify exactly the same or similar red flags in your next interview, and hopefully identifying them will set you up to better handle them. 

Even though the stories I’ll be sharing come from my marathon of technical interviews, these red flags do not apply only to technical interviews. They apply to all kinds of interviews and so there’s a lot to learn here for everyone.

In the world of computing and in life generally, for any given problem, there is typically more than one way to solve that problem. For example, given a sorting problem, you could solve it using a merge-sort algorithm or a heap sort algorithm. 

Having this rich number of techniques to solve a problem makes it even more interesting and the general expectation in technical interviews is that you should have the flexibility to solve a problem using your preferred technique.

I had an interview where the interviewer asked me to solve an algorithmic problem. I had started solving the problem using a specific technique when the interviewer stopped me in my tracks and asked that I use another technique. 

When I probed a bit further to know why, it appeared that the reason he asked that I used the second wasn’t to test my knowledge of that second technique; it was because he was more ‘comfortable’ with that approach.

It is different if the interviewer wants to test your knowledge of something very specific. For example, given a problem that can be solved using iteration and recursion, the interviewer may want to test your knowledge of recursion and can ask you to solve the problem recursively. That wasn’t the case here.

I ended up using both techniques and discussed the trade-offs but frankly that experience left a bad taste in my mouth especially because that interview was with the hiring manager — my would-be manager, who is someone that can significantly influence your career growth and trajectory. 

It’s very exciting and fulfilling when you go through all preliminary stages of a technical interview, through to the onsite interview (remote or in-person) and then you receive that “Congratulations <insert name here>, we are pleased to offer you…” email.

However, that excitement often becomes short-lived when there is some form of pressure from your soon-to-be employer to accept the offer. It’s a bit more manageable when the pressure comes from someone in HR or the recruiter, but when it’s from the hiring manager, that can be harder to manage.

That was the case for me when I interviewed with a startup based in Palo Alto. They were a small company in terms of staff strength. My onsite interview with them had gone quite well. I had a good conversation with the hiring manager and an even better conversation with the VP of Engineering, so much so that I could tell that I was going to be extended an offer. I asked to know how long I had to accept the offer letter and I was told seventy-two hours.

The offer letter arrived later that evening, and it looked great — a six-figure offer definitely didn’t seem like a bad start. I was also at the final stage of the interview process with other companies too and thankfully, I had enough time to negotiate and accept the offer, or so I thought. 

Then the pressure started, incessant calls from the hiring manager and the VP of Engineering, back-to-back emails, all within the allotted time. So much was the pressure that it got to a point where I wasn’t sure I wanted to negotiate the offer anymore. I turned down the offer.

I turned down the offer because the experience got me thinking about the company’s work culture. Were the methods employed by the company to get me to accept the offer indicative of their work culture? If they needed to get something done, how far would they go? 

Now don’t get me wrong, yes the company wants to employ you, yes the recruiting team wants to ‘close the deal’ however, it’s very important to pay attention to how the company does this. Do they remain professional about it?

A company’s values go beyond what they say, it shows in what they do and how they do it.

Among the many reasons why you would join a company is your desire to be involved in valuable work. I had the opportunity to join a US company based in Boulder, Colorado. They had contracted a recruiting agency to help them find someone to fill a Software Engineer position in their firm. 

The hiring process started with an exploratory interview with the recruiting agency closely followed by a second interview with a recruiter from the company. In both interviews, I couldn’t get a clear sense of the details of my specific role was — what team I would be on, what kinds of projects I’d be working on, what the career growth pathway was, etc. 

I understand that sometimes companies can be going through restructuring, but that didn’t seem to be the case here. It seemed more like the company was focused on completing their headcount. Even though there’s nothing wrong with completing a headcount, I think there is everything wrong with not having a clear purpose for a role for a couple of reasons:

• It means the role may not be critical to the company’s core business.
• If the role isn’t that important, it may mean when a layoff comes, your position may be impacted.

On a more personal note, I don’t want to be just a number. I want to work at a place where I have the opportunity to contribute in an impactful way, and I like to believe you would too. So it’s important to get clarity about your role, for where you are today and for future career growth.

When looking to join a company, one of the things you simply must care about is the team you will end up working on. At least 25% of your waking hours will be spent interacting with that team whether in-person or virtually.

Interviews offer you an opportunity to experience firsthand what it will look like to work with your prospective teammates, especially since, unless you’re interviewing at a huge company, your interviewers are likely to become your teammates.

If through all the different stages of the interview process, you experience a consistent lack of interest or low morale from your interviewers, you might want to pay attention . 

When I experienced that during one of my interviews, I couldn’t exactly tell what the cause was, but I knew something just wasn’t right. After some internal tussle, I decided to trust my gut feelings and ended the interview process with the company. 

Fast forward to two months after, two of my interviewers (would-be teammates) had left the company and joined another company (no I wasn’t stalking, I just checked on LinkedIn). 

Now I’m not saying that during the interview process, there won’t be one or two people, who because of their busy schedules, would have preferred to be doing something else rather than interviewing. Yet, when all of the interviewers don’t want to be there, you certainly want to pay attention to that.

A lack of interest or low morale could be pointers to a combination of any of the following:

• Your prospective team-mates may be experiencing burnout.
• Some internal dissatisfaction with company — culture, policies, something, anything.
• The team isn’t that interested in you (hard pill to swallow?), maybe they don’t see you as a long-term hire.

Or it could be for reasons that I have not included here, but I implore you to not ignore this red flag if you see it in your next interview.

Have you been in an interview before where the interviewer doesn’t seem to have any questions to ask you? Trust me it can get really awkward.

That was my experience during a technical phone-screen interview with an educational technology company based in California. The interviewer wasn’t prepared for our interview and didn’t have any questions at hand. He wasn’t even sure of who he was interviewing and what role I was interviewing for. It wasn’t a pleasant experience.

I understand that there are a myriad of reasons why interviewers may not be prepared for an interview. Some of which include:

• Lack of proper planning by the HR/recruiting team.
• Last-minute changes on the interviewee.
• Busy schedules for the interviewer.
• The interviewer just wasn’t prepared.

I typically won’t act on this red flag in isolation. I will be looking for other red flags in a bid to form a cluster of patterns before making any decision.

It’s fulfilling to be a part of a company that is involved in meaningful work that creates value for its users. Joining such a company would mean you have the desire to contribute  in helping the organization meet its goals. This invariably means the organization must have some goals right?

I was contacted by a startup based in San Francisco via AngelList. I had a first introductory call with a recruiter from the company, closely followed by a phone screen technical interview. 

In both interviews, even though the interviewers shared some details about the company, there was a lot of vagueness and about the company’s direction and where the company was headed. 

I particularly remember that one question I asked at the time, was about how the company would deal with its growing competition. Sadly, the answers I got didn’t seem convincing and the company later got acquired by the competition.

When you are interviewing to join a company, you are selling more than just your skills, but also yourself — your unique experience. While it’s important to do that, I think it’s equally important that the company should be able to sell you on its vision and what it hopes to achieve.

When I think of joining a company, I picture myself in that company for the next 2–5 years. If my vision for where I want to be in my career doesn’t align with the company’s vision, that is a mismatch that shouldn’t be ignored.

We sometimes focus more on securing the job and even though that is very important, even more important than getting the job is staying fulfilled on the job. For me, fulfillment meant joining a company that had a clear vision of where they were headed, working in a role that was critical to the company’s business while being equipped with a lot of growth opportunities. 

Hopefully, these red flags I have shared will equip you to make better decisions on what companies you choose to grow your career with. I would generally not advise making a decision based on one or two red flags, but if you see a cluster of red flags, you shouldn’t ignore them. I wish you the best in your career journey.

If you ever need someone to do a mock interview with you, feel free to schedule one here or you can reach out directly to me on Twitter @meekg33k.

And if you’d like a list of things to ask companies while you’re interviewing that may help you identify these red flags (and others!) sooner, take a look at this one.

If you have something to say about your adventures in interviewing or hiring, write a guest post on our blog! Please email me at aline@interviewing.io to get started.

Uncategorized

Announcing the interviewing.io Technical Interview Practice Fellowship

Posted by Aline Lerner on July 23rd, 2020.

I started interviewing.io because I was frustrated with how inefficient and unfair hiring was and how much emphasis employers placed on resumes.

But the problem is bigger than resumes. We’ve come to learn that interview practice matters just as much. The resume gets you in the door, and your interview performance is what gets you the offer. But, even though technical interviews are hard and scary for everyone — many of our users are senior engineers from FAANG who are terrified of getting back out there and code up the kinds of problems they don’t usually see at work while someone breathes down their neck — interview prep isn’t equitably distributed.

This inequity never really sat right with me (that’s why interviewing.io exists), but when we started charging for interview practice post-COVID, it really didn’t sit right with me.

As you may have read, if you follow interviewing.io news, COVID-19 turned our world upside down. In its wake, the pandemic left a deluge of hiring slowdowns and freezes. For a recruiting marketplace, this was an existential worst nightmare — in a matter of weeks, we found ourselves down from 7-figure revenue to literally nothing. Companies didn’t really want or need to pay for hiring anymore, and we were screwed.

Then, we pivoted and started charging our users, who had previously been able to practice on our platform completely for free (albeit with some strings, more on that in a moment). While this pivot was the right thing to do — without it, we would have had to shut down the company, unable to provide any practice at all — charging people, especially those from underrepresented backgrounds, didn’t sit right with us, and in our last post announcing our model, we made the following promises:

• We’d ALWAYS have a free tier 
• We’d immediately start working on a fellowship for engineers from underrepresented backgrounds or in a visa crisis/experiencing financial hardship ← That’s what this post is about!
• We’d find a way to let people defer their payments

We launched with a free tier, and it’s still there and going strong. We’re still working on deferred payments and are in the thick of user research and price modeling.

But, the rest of this post is about the 2nd promise. To wit, I’m so proud to tell you that we’ve officially launched the first (pilot) cohort of the interviewing.io Technical Interview Practice Fellowship. This cohort will be focused on engineers from backgrounds that are underrepresented in tech. We are acutely aware, of course, that our first cohort couldn’t capture everyone who’s underrepresented, that gender and race isn’t enough, and that we need to do more for our users who can’t afford our price tags, regardless of who they are or where they come from.

Our hope is to expand this Fellowship to anyone who needs it.

We’re also working on the much harder problem of how to navigate the visa situation we’re in right now (different than when we wrote the first post, sadly… but especially important to me, given that I’m an immigrant myself).

What is the Fellowship, and why does it exist?

Before we tell you a little bit about the Fellows in our inaugural cohort and what the Fellowship entails, a quick word about why this matters.

In order to get a job as a software engineer, it’s not enough to have a degree in the field from a top school. However you learned your coding skills, you also have to pass a series of rigorous technical interviews, focusing on analytical problem solving, algorithms, and data structures.

This interview style is controversial, in part because it’s not entirely similar to the work software engineers do every day but also because 1) like standardized testing, it’s a learned skill and 2) unlike standardized testing, interview results are not consistent or repeatable — the same candidate can do well in one interview and fail another one in the same day. According to our data, only about 25% of candidates are consistent in their performance from interview to interview, and women quit 7X more often than men after a poor performance.

To account for both of these limitations, the best strategy to maximize your chances of success is to practice a lot so you can 1) get better and 2) accept that the results of a single interview are not the be-all and end-all of your future aptitude as a software engineer and that it’s ok to keep trying.

The main problem created by modern interview techniques is that, despite interview practice being such a critical prerequisite to success in this field, access to practice isn’t equitably distributed. We want to fix this, and we’re well equipped to do so. Based on our data, engineers are twice as likely to pass a real interview after they’ve done 3-5 practice sessions on our platform.

Our Fellows will get these practice sessions completely for free. These will be 1:1 hour-long sessions with senior engineers from a top company who have graciously volunteered their time and expertise. Huge thank you and a big shout-out to them all.

After each session, Fellows will get actionable feedback that will help them in their upcoming job search, and we will be helping Fellows connect with top companies as well.

Note: We’d like to be able to offer even more support – and are actively seeking more partners to do so. Please see the How you can help section below if you or your organization would like to get involved!

Why now?

The world seems to be in a place, now more than ever, to have the conversation about race, gender, socioeconomic, and other kinds of equity, in hiring. This is our small part of that conversation.

Who are the Fellows?

After opening up our application process, we close to 1,000 submissions in a week, and (though it was really, really hard) we culled those down to 56 Fellows.

Our first cohort is:

• 82% Black, Latinx, and/or Indigenous
• 53% women
• 55% senior (4+ years of experience) & 45% junior (0-3 years of experience)

Here are some of their (anonymized) stories. There were a lot of stories like these.

My goal is to keep pressing as well as to share and give to underrepresented communities because the journey in tech can be isolating. Often I am the only one. It is critical that there are more people that look like me that are engineers *and* ascend the leadership ladder.

My parents immigrated from [redacted] to The Bronx without a formal education. I’m the first individual in my household to graduate from college and I’m the only Software Engineer in my family. I grew up in a poor neighborhood where many individuals had limited economic and educational opportunity. I aim to make the path to become a Software Engineer easier for those who were in my situation.

My journey to becoming a software engineer almost never happened. Throughout my undergraduate studies I was faced with having to drop out multiple times, due to the immigration status of my parents…. I was tasked with assisting in my family’s living situation and paying for school. I worked full time and started my own construction company in order to take care of my family and studies. It was always tough having to work 8-10 hours a day and then going to class or doing homework… Becoming a software engineer was always a goal of mine, and realizing that goal was well worth the struggle, given the struggle my parents went through to bring us here in the first place.

I spent 5 years in public education working directly with marginalized communities in the struggle for equity. My journey through software engineering is a continuation of this spirit of advocacy and changemaking. Software engineering is a tool to be put at the service of advocacy.

What can I do to help?

There are a number of ways you can help and get involved!

Help sponsor future Fellowship cohorts & create scholarships for underrepresented engineers!

Every Fellow in this first cohort represents at least 100X who are not. We have the tech to scale the hell out of this program, and all we need is backing and resources from people or organizations who recognize there’s a need (donations are tax-deductible). Please email fellowship-sponsors@interviewing.io if you’d like to get involved or want more information.

Hire through us!

Despite mounting evidence that resumes are poor predictors of aptitude, companies were obsessed with where people had gone to school and worked previously. On interviewing.io, software engineers, no matter where they come from or where they’re starting, can book anonymous mock interviews with senior interviewers from top companies. We use data from these interviews to identify top performers much more reliably than a resume, and fast-track them to real job interviews with employers on our platform through the same anonymous, fair process. Because we use data, not resumes, our candidates end up getting hired consistently by companies like Facebook, Uber, Twitch, Lyft, Dropbox, and many others, and 40% of the hires we’ve made to date have been candidates from non-traditional backgrounds. Many of our candidates have literally been rejected based on their resumes by the same employer who later hired them when they came through our anonymous platform (one notable candidate was rejected 3 times from a top-tier public company based on his resume before he got hired at that same company through our anonymous interview format).

Please email sales@interviewing.io to get rolling.

Buy an individual practice session for someone who can’t afford it

If you know individual engineers who need interview practice but can’t afford it, use our handy interview gifting feature. Interviews are $100 each. They’re not cheap, but we have to price them that way to pay for interviewer time (interviewers are senior FAANG engineers) and cover our costs. Sadly that means practice interviews are not affordable to everyone. Even if you can’t get involved to help us fund interviews at scale, if you know someone who needs practice but can’t afford it, you can buy them an anonymous mock interview or two individually. It’s the best gift you can give to an engineer who’s starting their job search.

Uncategorized

interviewing.io is finally out of beta. Anonymous technical interview practice for all!

Posted by Aline Lerner on June 4th, 2020.

First, for the brevity-minded, a TL;DR:

• interviewing.io is now open to all engineers in North America and the UK, regardless of seniority level
• We have both free and paid, premium mock interviews
• No more limits on how many practice interviews you can do
• You can now give (or receive!) practice interviews as gifts

I started interviewing.io 5 years ago. After working as both an engineer and a recruiter, my frustration with how inefficient and unfair hiring had reached a boiling point. What made me especially angry was that despite mounting evidence that resumes are poor predictors of aptitude, employers were obsessed with where people had gone to school and worked previously. In my mind, any great engineer, regardless of how they look on paper, should have the opportunity to get their foot in the door wherever they choose.

So, we set out to build a better system. On interviewing.io, software engineers can book anonymous mock interviews with senior engineers from companies like Facebook, Google, and others, and if they do well in practice, get fast-tracked with top employers regardless of how they look on paper. Fast-tracking means that you bypass resume screens, scheduling emails, and recruiter calls, and go straight to the technical interview (which, by the way, is still anonymous¹) at companies of your choice. Because we use interview data, not resumes, our candidates end up getting hired consistently by companies like Facebook, Uber, Twitch, Lyft, Dropbox, and many others, and 40% of the hires we’ve made to date have been candidates from non-traditional backgrounds. What’s nuts is that many of our candidates have literally been rejected based on their resumes by the same employer who later hired them when they came through interviewing.io. One notable candidate was rejected three times from a top-tier public company based on his resume before he got hired at that same company through us.

Over the past 5 years, we’ve hosted over 50,000 technical interviews (both practice and real) on our platform. Our YouTube channel, where you can watch other people interview, has gotten over 3.5M views, and, most importantly, we have helped thousands of engineers get great jobs.

Despite that for our entire, multi-year existence, we’ve been in beta. Over the past year or so, this increasingly inaccurately named “beta” became kind of a smoke screen. Our product was stable, we had plenty of interviewers, but sadly, we couldn’t serve many of the people who needed us most. Because we made money by charging companies for hires, despite a growing waitlist of >180,000 engineers, we could only serve the ones whom we had a shot at placing, i.e. engineers who 1) were located in a city where we had customers and 2) had 4 or more years of experience⁠—sadly, despite our best efforts, employers across the board were not willing to pay for junior hires.

Then, COVID-19 happened and with it, a deluge of hiring slowdowns and freezes. In a matter of weeks, we found ourselves down from 7-figure revenue to literally nothing. Companies didn’t really want or need to pay for hiring anymore.

But these hiring slowdowns freezes weren’t just affecting us. In parallel, we saw a growing sea of layoffs, and we realized that, soon, more and more candidates would be vying for a shrinking number of jobs. On top of that, because a disproportionate amount of layoffs targeted recruiters, an overworked in-house skeleton recruiting team would go back to relying on resumes and other old-school proxies, unintentionally marginalizing non-traditional candidates once again. We also realized that many of the folks getting laid off would be here on visas, which meant that they’d have a paltry 60 days to find their next job or risk deportation.

So, we made a hard call. You may know that historically, we’ve offered completely free mock interviews. What you may not know, is that we pay our professional interviewers, as it’s the only way to ensure that we have seasoned, experienced engineers delivering a consistent, realistic, and high-quality experience. This is often our largest expense.

Since we previously funded practice by charging companies for hires, we had to find another revenue stream to continue. In the face of either shutting down the company, unable to provide any practice at all, or to begin to charge for premium practice interviews, we made the choice to launch a paid tier. But, because charging for practice felt anathema to our mission, we knew we needed some ground rules in place. I started this company to fix hiring, after all, and that’s why the people who work at interviewing.io are here, too. 

After 50,000 interviews, our data shows that where someone goes to school has no correspondence to their interview performance. Despite that, we are all too aware that just because aptitude is uniformly distributed among populations, resources are not. We understand that paying for interviews will be prohibitive to many of the people who need help most, so our ground rules and goals for this pivot were as follows:

• We’d ALWAYS have a free tier
• We’d immediately start working on a fellowship for engineers from underrepresented backgrounds or in a visa crisis experiencing financial hardship
• We’d find a way to let people defer their payments

There is an upside to our new model though. Now that we’re no longer strictly beholden to employers, we’re able to open up interviewing.io to engineers of all seniority levels and many more locations. And because our revenue isn’t coming from placements but directly from practice, we don’t have to constrain our users to a limited number of practice interviews.

So, as of today, interviewing.io is open to engineers of all experience levels in North America and the UK. Engineers who sign up can book mock interviews 24 hours out, and top performers will still get fast-tracked for great jobs.²

Here’s how it works:

• If you’re able to pay, you can now book unlimited interviews with professional interviewers from FAANG and other top companies. We’re no longer constraining you to 2 or 3, and we have the interviewer supply to make this work. Interviews cost between $100 and $225.
• If you can’t pay, there’s a free tier where you can do peer practice with other users.

What about the goals and ground rules above? We’ve already made some headway against these. The free tier was there from day one. Moreover, a number of our professional interviewers have stepped up and volunteered their time to help the people who need it most prepare for free (if you’d like to volunteer your time to help engineers from underrepresented groups practice interviewing, please email interviewers@interviewing.io with the subject line ‘Volunteer’), and a formal fellowship is in the works. Lastly, we’re working on a deferred payment plan, where users who buy premium interviews will not have to pay us for them until they find their next job.

Look, whether you got laid off, are a student who’s reeling from your internship being withdrawn, lost your visa, whether you’re fortunate enough to still be employed but worry about your job stability, or if you just want to help making hiring fairer, we hope you’ll take us for a spin. Technical interviews are hard, and hiring is broken. And whether you’re new to interviewing or are just rusty after being off the market, and whether you can pay or not, we have your back. 

The coming months are going to be hard. We know that in the current climate more people than ever are feeling helpless, and the world feels like it’s burning. And it might not even seem like technical interviews matter that much. But they do to us… because this is our way of creating a world where access to opportunity isn’t determined by who you are or where you come from but by what you can do.

P.S. If you don’t need practice for yourself but you or your organization want to help others get awesome at technical interviews, check out our new gifting feature.