Lenny's Podcast: Product | Career | Growth
00:06:30 · Lenny Rachitsky, Sean Ellis
And the idea is that if you... 4040% or more of people say they'd be very disappointed if they can no longer use the product, you essentially have product market fit. [00:06:40] S1: I would say it's a leading indicator of product market fit. The the the lagging indicator is do they actually keep using it? So probably retention cohorts are are more more accurate. But the problem is, like your time at Airbnb, how long do you have to look at a retention cohort before you know that you've actually long term retained someone? And so with this question, can kind of find out day one. You don't actually even need a good analytics system in place to be able to see if product market fit exists. So, yeah, the 40% was not something I originally had in there. Also originally I was trying to have just a filter so that I was not treating all feedback from customers the same, but I was trying to find feedback from customers who actually really cared about the product. And then was over time as I, at the time I was working for a couple of YC backed companies. And so those companies were all pretty connected. And so I would share the question with a lot of other startups in Silicon Valley. So over time, I started to see there was a pattern that once you got a high enough percentage of users saying they'd be very disappointed, most of those very disappointed without the product, most of those products did pretty well. And then if if you felt too low, those products tended to suffer.
00:08:04 · Lenny Rachitsky, Sean Ellis
[00:08:04] S0: Okay, there's two things I want to definitely follow-up on here. The first is such an important point that you made at the beginning when I introduced this test is that you described it as a leading indicator of product market fit and actually retention. People actually using your product. The product actually being used by the market is the actual ultimate test. [00:08:21] S1: Right. [00:08:21] S0: So the idea here is this is a good way to get a sense of before you actually have data, are we headed in a good direction? Can you just speak more about that? Like how to... When to use this and when it's most useful?
00:08:21 · Lenny Rachitsky, Sean Ellis
[00:08:21] S0: So the idea here is this is a good way to get a sense of before you actually have data, are we headed in a good direction? Can you just speak more about that? Like how to... When to use this and when it's most useful? [00:08:33] S1: Mean, for me in particular, when I come into a company, my my my goal is to help them grow. And so I don't want to put myself in a situation where I'm going to fail because no one actually cares about the product. So it can really be asked at a company of any stage. It's helpful to understand who your must have users are. But essentially once you have an even an MVP, very first MVP on the product, you can still get some useful feedback about the product if it's resonating with anyone. I actually had a company where I had committed to work with them. It was right after I left Dropbox and I committed to work with these guys for six months to help them grow. I ran the question and it came back at only 7% of users saying they'd be very disappointed without the product. So I'm like, I have six months to help them grow and they're only at 7% right now. Knows? It might take six months to get the 40%. Am I doing them a disservice by being in a growth role and being on payroll during this period of time? Fortunately, with the signal and the information we got from the initial survey, we were able to get them at 40% in two weeks. [00:09:50] S0: Wow. What did you do there just as a case study?
00:09:53 · Sean Ellis
[00:09:53] S1: Yeah. So the company called Lookout, it's a mobile security company. And now most of the things in Lookout are built into iPhones and Androids. But at that time, the product had everything from backup my data to find my lost phone to protecting your phone a firewall and antivirus. And so when we ran this initial survey, I dug into the 7% who said they'd be very disappointed without the product and found that most of that 7% were focused on the antivirus functionality. So they were like, they know they need to protect their computer from viruses. Smartphones were becoming more like computers. So it just made a lot of sense for them that they'd need to protect their phone. And Interesting, at the time, I think there was only one phone virus that had ever even happened, but it was a pretty easy mental leap for people. So now we knew, okay, it's antivirus that people really valued. And so step one was just reposition the product on antivirus so that that kind of creates a filter. So anyone who now is coming in to sign up for the product who doesn't care about antivirus is not going to convert. And those who are excited about antivirus are going to convert. We already know from the initial survey that people value that after they convert. By setting the right expectations around it upfront, you're going to bring people in with the right expectations. But then the second thing that we did was we streamlined onboarding so that the first thing that they did after signing up for the product was to set up the antivirus and then get a message, you're now protected from viruses. And so it's really the combination of those two things. It's set the right expectations and then speed the value. And so the next cohort of people that we surveyed were at 40% saying they'd be very disappointed without the product. That literally took two weeks to make those changes. Six months later, it was 60% on the score. Then I think they hit the billion dollar valuation four or five years later on ultimately being one of the early unicorns. Interestingly, as all of those things were built into mobile phones now, they've completely changed the business, but they continue to do really well, But they've continued to iterate the business.
00:12:22 · Sean Ellis, Lenny Rachitsky
I think that having that kind of finger on the pulse early in the business was important to build the muscle in the business to be really responsive as the market changed. [00:12:34] S0: Sean, this is already amazing. There's just like a fractal of topics I want to explore from this very short conversation already. So the first is just follow this thread of basically you're sharing kind of a growth strategy that I imagine you execute is look for the percentage of people that would be very disappointed if your product went away. See who they are, see what they're excited about, and lean into that both positioning wise, onboarding wise, and probably also cut out stuff from your product that they don't care about. [00:13:02] S1: Yeah. And I was coming at it from a marketing perspective initially. Over time, I positioned myself more in a growth role with product and marketing as areas I could influence. But as a marketer, probably didn't have a lot of influence on a engineering founded company to say, let's cut out stuff. So it made more sense to say, let's just sequence the onboarding so that we're highlighting this and and onboarding to this. That was a little easier to sell. [00:13:33] S0: And just hearing that you can move this score so quickly without even changing the product substantially, I imagine what surprised a lot of people when you think about, like, removing retention often is really hard, and maybe we talk about that. But I guess it sounds like there's often something you can do that's not very hard that might significantly shift this product market fit test.
00:13:33 · Lenny Rachitsky, Sean Ellis
[00:13:33] S0: And just hearing that you can move this score so quickly without even changing the product substantially, I imagine what surprised a lot of people when you think about, like, removing retention often is really hard, and maybe we talk about that. But I guess it sounds like there's often something you can do that's not very hard that might significantly shift this product market fit test. [00:13:57] S1: Right. And then that ultimately, like, yeah, moving retention is really hard, but it's it's usually much more a function of onboarding to the right user experience than it is about the kind of the tactical things that people try to do to improve retention. [00:14:11] S0: Okay. I wanna put a pin in that and come back to that because that's a really important topic. I wanna come back to, say someone runs this survey and they get 40%, what should they have in their mind of, like, this is what this is telling me? Because I think a lot of people are like, I got product market fit. I got this. Let's go. Go. Go. What's the best way to think about what this tells you?