Why we’re pausing our Pay Later Program

If you’ve been following the history of interviewing.io, then you’ll know we took a bunch of sharp turns during our journey. I’ll spare you the long version (you can find that in our announcement that we’re out of beta, as well as our Series A announcement, but the TL;DR is that we used to make money from employers, and mock interviews were free for engineers. Then hiring basically froze when COVID-19 happened, and to survive, we started charging engineers. We didn’t feel great about it, but the company would have shut down otherwise. The silver lining was that this allowed us to open up interviewing.io to engineers of all experience levels and in more locations and also meant that we no longer needed to throttle how many mock interviews people could do.

When I announced that we were starting to charge our users, I promised 2 things: (1) We would start a Fellowship program for engineers from non-traditional backgrounds, where they could receive mentorship and practice for free, and (2) We would create a program where you didn’t have to pay for practice till you got a job.

We launched our Technical Interview Practice Fellowship in 2020 and have completed 2 successful cohorts so far. We also launched our beta deferred payment program in September 2020 and ran it for a year to see what portion of users would get jobs and pay us back. When that was successful, we followed it in January 2021 with a much bigger, fully productized version called the Pay Later Program.

Unfortunately, we recently made the very difficult decision to pause our Pay Later Program. In this post, we’ll talk about why we made that call and what we’ll be doing instead to ensure that engineers who can’t afford to pay for practice will still be able to get it. We’ll also explain some things we’ve learned along the way about funnel optimization, some mistakes we made while iterating on this program, and what we’ll do differently when we hopefully unpause it in the future.

But first, some history.

Our beta deferral program

Ever since we started charging, we wanted to give users a way to de-risk buying interview practice. However, we didn’t want to subject our users to lengthy and invasive loan applications. I had seen what some Income Share Agreement (ISA) application processes looked like, and they turned my stomach – many demanded bank account info so they could periodically check on your purchases. In one demo we saw, one of these firms bragged that if a student was spending too much money at Starbucks, it could be a sign that it was time to collect.

We didn’t want any part of that. We also wanted to distance ourselves from ISAs in general – mock interviews don’t cost tens of thousands of dollars, so with us, you wouldn’t have to give us a portion of your income. You’d just defer paying us for some number of mock interviews, which would usually run you somewhere between $500-$1000 depending on the number of interviews. So, we decided we’d ideally backstop this program ourselves, and as a precursor, we ran the beta deferral program for a year to see what kinds of payback rates we’d get.

Below are the terms of the original deferral program. While this program was in beta, most of our enrollment flow was in Typeform. It looked like this:

Then we’d ask a few questions, and the final step of the form was to sign a contract, like so:

After 4 months, we saw a 70% payback rate, and after 6 months, it was around 90%. Between September 2020 and September 2021, over 500 beta users enrolled in this program, and feedback was overwhelmingly positive.

We felt great about this; however, we knew that the execution of the beta was still clunky. Typeform was our whole UI, and our back-end included a bunch of manual work held together with chewing gum and a massive spreadsheet, where we tracked enrollments, statuses, and paybacks.

When we saw that the deferral program was clearly working, we decided to go all in, remove the manual work, and productize it properly.

The shiny new Pay Later Program

The new Pay Later Program would be a fully automated, shiny version of what we had cobbled together in the beta. The terms would be streamlined, enrollment would be easier, check-in emails would be automatic, and everything would just… work. As such, we took a critical look at the deferral program flow and all of the operational pain points and decided to streamline them as much as possible.

Because so many people in the old deferral program reached out asking to upgrade their algorithmic interviews to something more specialized (e.g., “I want to practice machine learning instead,” “I want an interviewer specifically from Google/Amazon/Facebook”), we decided to make the next version easier for everyone by just giving users interviewing.io credits.

When you enrolled in the program, you’d start with $512 or $1024 in credits (your choice) that you could spend on any mock interview or mentorship session that you liked.

We launched our Pay Later Program at the end of December 2021. As we usually do with new features, we A/B tested a few different variants, with the intent of increasing enrollment as much as possible each time.

Our first iteration of Pay Later was almost identical to its predecessor, the deferral program, except that instead of having to fill out a Typeform, the enrollment flow was in our app. Here’s how it looked:

The only out-of-app part of the experience was having to sign a contract in HelloSign. We also had a very wordy explanation of the terms. It was a wall of text, much like the first page of the old Typeform.

Over time, as we continued to A/B test, we pruned more and more steps, tried different UIs, and even changed up the terms. We were ultimately able to more than double enrollment in the Pay Later Program. The 3 biggest and most impactful changes we made were:

• Removing the contract-signing step entirely. Instead, we just asked users to check a box that said, “I agree to the terms of the Pay Later Program,” where clicking into the terms opened them in a new tab (see the screenshot below). This was by far the most impactful change.
• Making our terms more lax. Instead of charging you 4 months later, if you stayed at your job, we amended the terms to make it so you could get as many extensions as you needed, as long as you were still looking for a job (in good faith).
• Condensing the terms summary from the wordy bullets in the original Typeform to the blue box below:

The dangers of funnel hacking

Our new Pay Later Program was objectively better for users than our old deferral program for 2 reasons. First, it was more flexible – users got credits they could spend however they wished, instead of 5 algorithmic interviews that they’d have to pay to upgrade. Second, the terms were more generous and user-friendly – you didn’t have to pay us back if you stayed at your old job, as long as you promised us you were still looking, whereas in the old deferral program if you stayed at your current job, you’d have to pay back the full amount after 4 months.

Despite the offering being objectively better, user feedback was objectively worse.

In our quest to increase funnel throughput, we made enrollment too frictionless. By no longer requiring people to sign a contract and by basically creating 2-click enrollment with an abbreviated TL;DR that glossed over the mechanics of payback (we’ll send you an email, we’ll charge if we don’t hear back from you, you’re on the hook for the full amount and any unused credits are yours to keep in perpetuity), we failed to sufficiently explain the program to users, many of whom were justifiably shocked when they got billed.

Reading a contract and signing it forced users to think through whether they were willing to be on the hook for a non-trivial amount of money. When we made enrollment as easy as clicking a few buttons, many users (understandably) no longer read the terms and made assumptions about how the program worked. These assumptions varied from “I thought these interviews were free” to “I thought I only had to pay if I got a job specifically at Google” to “I never signed up for this.”

With that in mind, of course it made sense that some portion of our users were irate and called us a scam when we emailed to ask if they were ready to pay us back. What sucked is that, even though we created this program with the best of intentions and modified the terms to make them more permissive, because of how we packaged and presented it, we experienced a very different response than from the original deferral program.

Though the majority of users still paid us back within the 4 month period, there was now an angry and disappointed vocal minority, and our ops team began to dread the end of the month, when we’d invariably receive a number of irate emails stating that interviewing.io was a scam.

I don’t feel great that I got caught up in the funnel hacking and in seeing enrollment numbers increase. If you’ve read my writing over the years, you’ll know that I take pride in transparency, candor, and clarity. You may also know that our one core value as a company is putting our users (i.e., engineers practicing for interviews) first. I didn’t live up to that this time, and I am sorry.

Practically speaking, I’ve learned that creating some amount of friction is necessary when you’re asking people to promise to pay you ~$1000 in the future. Removing that friction can create short-term wins but may hurt you (and disappoint your users) in the long run.

Now, you might expect that if we just revert the flow and bring back the friction, everything will be great again. Unfortunately not.

Why we’re pausing

In the current economic climate, we’ve had to take a hard look at our business and, like many other companies, cut our burn and tighten our belts. When we launched the program and committed to backstopping it ourselves, it was with the expectation that hiring would continue at a healthy clip. We anticipated a hiring slowdown in the recession, but multiple FAANGs freezing hiring was beyond what we had predicted – all of our models and assumptions had been based on COVID-era economics. This was something new.

The reality is that because the payback period is at least 4 months, regardless, as long as we’re running it, we are losing 4 months of runway. In this climate that’s significant.

Despite that, we knew that for some portion of our users, the existence of this program would be the difference between being able to practice and not, so we looked into financing it through a third party. Here’s how we did the math:

1. Estimate our enrollment rate in the Pay Later Program if we go back to the old UI/UX and terms.
2. Estimate our payback rate for the Pay Later Program moving forward
3. Cross-reference (1) and (2) with the interest rates we’d be paying to finance the program

Unfortunately, no matter how we ran the numbers, we couldn’t find a way to make this program ROI positive in the near term, especially with the reduction in enrollment we’d expect upon adding friction back into our enrollment process, which we decided would be a hard requirement for bringing back the program – we learned to be careful with growth hacks through the mistakes we made with this program, and we will not repeat them.

In summary, the old flow had a lot of friction, and in the current climate, enrollment rates would be too low to justify its existence. The new flow had significantly more enrollment, but it made it too easy for users to enroll without grokking the terms, and this generated enough negative feedback from users to make us want to kill it, especially given our company’s core value of putting our engineering community first. Finally, even if we kept tweaking the old flow to find the perfect balance, payback rates within a reasonable time window would be too low to be ROI positive when we factor in the cost of financing this program by a 3rd party, which we’d need to do to conserve cash.

It’s possible we’ll bring this program back in the future, but if we do, it’ll look substantially different from what we released in 2022.

What about users who are already enrolled in the Pay Later Program?

If you’re already enrolled, you’re good! Your credits will continue to work in perpetuity, just like we promised, and we’ll honor the terms: You get extensions as long as you haven’t found a job and are still actively looking.

What we’re doing instead to help those who can’t afford to pay for practice

We realize that many of you were depending on this program to be able to practice, especially during a recession. Even though we’ve paused our Pay Later Program, until we hopefully bring it back in the future, we have a few other ways that we can make it easier for you to get the practice you need.

• We still have peer-to-peer practice interviews, where you get matched with other interviewing.io users and work problems together. We introduced these right after COVID-19, and they’re not going anywhere. You get 1 interview with a peer when you join the platform, and after that, you can unlock more by conducting interviews yourself. We’ll be looking at this program to see how we can make it more accessible.
• We’ll be starting the next cohort of our Fellowship for engineers from non-traditional backgrounds soon.
• We’re experimenting with a few other ideas that may, together, fill the void created by the Pay Later Program. One of the more promising items is implementing Affirm, which will let our users defer payment for practice or pay it in smaller chunks.

This was a hard post to write, and I welcome feedback. You can always reach me at aline@interviewing.io.