Lenny's Podcast: Product | Career | Growth
00:17:40 · Lenny Rachitsky, Sean Ellis
[00:17:40] S0: Awesome. And the 40% threshold, so what you shared is you basically emerged from just looking at tons of startups doing the survey and finding a pattern. How firm is that 40%? Like how big of a deal is it? 39 versus 41? [00:17:52] S1: I don't think it's that firm. You know, to me, I think the real power is having some kind of some kind of target for the team to be shooting for that basically says, we're not going to aggressively start to grow until we hit this target. And I think that as just a focusing piece is really important because I think one of the biggest challenges in an early stage startup is half the people feel like we're years from having this product ready to grow and half the people are like, what are we waiting for? Where if you can actually get people on the same page of what does product market fit look like for our business? And it's at that point that we're going to... Yeah. Before I ever heard the term product market fit, I remember the conversations back at LogMeIn in the mid-2000s of kind of like, when do we step on the gas? What is the combination of factors that need to be in place before we start pouring fuel on the early fire? And so, I think that kind of nail it and scale it, it's probably been term that's been around for decades now. But, you know, it's it's all kind of pointing to that same concept of product market fit.
00:17:52 · Sean Ellis, Lenny Rachitsky
[00:17:52] S1: I don't think it's that firm. You know, to me, I think the real power is having some kind of some kind of target for the team to be shooting for that basically says, we're not going to aggressively start to grow until we hit this target. And I think that as just a focusing piece is really important because I think one of the biggest challenges in an early stage startup is half the people feel like we're years from having this product ready to grow and half the people are like, what are we waiting for? Where if you can actually get people on the same page of what does product market fit look like for our business? And it's at that point that we're going to... Yeah. Before I ever heard the term product market fit, I remember the conversations back at LogMeIn in the mid-2000s of kind of like, when do we step on the gas? What is the combination of factors that need to be in place before we start pouring fuel on the early fire? And so, I think that kind of nail it and scale it, it's probably been term that's been around for decades now. But, you know, it's it's all kind of pointing to that same concept of product market fit. [00:19:03] S0: How often do you do you... Have you seen false positives with this test where someone gets 40% and something is not not right? They... They're actually far from it. Or is it generally pretty accurate?
00:19:14 · Sean Ellis, Lenny Rachitsky
[00:19:14] S1: If if you're having people say that they'd be very disappointed without your product, that... That's that's a really good sign. What I can tell you is that not not just a... Not necessarily a false positive, but, like, what is driving people to say they'd be very disappointed. One of my favorite books is Hooked by Nir Eyal and he talks about in the engagement loop that your last step is investment. And so I ran the survey on a business that I thought was a fairly commoditized business part of that I wanted to see, could I use the same go to market approach on a later stage company and use it to accelerate growth? And so this was a business called webs.com and they eventually got acquired by Vistaprint, but they'd been pretty flat for the year before I went in there. And then I started to kind of use this approach to try to dial in their their growth engine. And I ran the survey thinking, you know, you've you've had products like Wix and Weebly that have have come on to the market since this, you know, more legacy website building product has been around. And I personally think they're a little easier to use. They're a little better. And so I didn't have high hopes when I ran the survey, but it came back with one of the highest scores I'd ever seen. It was like 90% of the people saying they'd be very disappointed if Oh, they could no longer sure. Use [00:20:38] S0: the Never seen that.
00:20:38 · Lenny Rachitsky, Sean Ellis
[00:20:38] S0: the Never seen that. [00:20:39] S1: And I was like, how could that be possible? Product is kind of a commoditized category. I wouldn't even say it's one of the best. And then when I dug into it again, it comes back to that nearly all hooked model is that the investment people have made in building that website, they put so much into that. They know exactly how to make the changes in the CMS kind of side of things. They have spent a lot of time just making it beautiful. So ultimately it was something that that was why they were saying they'd be very disappointed. I mean, but for kind of fast forward and on when I initially went in, still doing these things help the business resume growth and have significant growth over the next twelve months after we did these things. So still the signal we got from why people would be very disappointed without the product was important and speed to value and kind of all the other things I think about in go to market for an early stage product still were relevant, but just the I think they were a little stronger on the percentage, you'd be very disappointed. Even Eventbrite when I was there when we ran it was probably the second highest I'd ever seen. But with event organizers, if they've already set their event up on that platform and they've sent it out to their list and all those people are coming in and they're managing their event, again, they've invested a lot in the platform. Sort of switching costs, I think, can factor in there. So it's a function of both switching costs and utility of the product.
00:22:22 · Lenny Rachitsky, Sean Ellis
[00:22:22] S0: So that question I wanted to ask is what's your guidance on when to ask this question? What I'm hearing is if you ask it very far along the journey, when they're very invested, you'll get a much higher score. Is there any advice on the timing on the best time to ask this question of your users? [00:22:38] S1: What I recommend is a random sample of people who've really used your product. So they've gone in, they didn't just sign up, but they went in and hopefully hit that activation moment. They've used it twice, two plus times, and they've ideally used it, say, within the last week or two weeks so that they haven't churned yet. So if it's a random sample of those people, that's that's kind of the ideal time to ask it. [00:23:08] S0: Got it. So basically, it's people that have activated whatever that means to you and have been using it for a couple weeks. Not not people, like, landing in your homepage and not people just signing up. Right. Not people months later. [00:23:19] S1: Not people who've seen like a demo of your product, but it's Right. It's people who actually have experienced the product. But it's okay if they... If if you're hitting people who've used it months later. But like in that Lookout example that I gave, if I'm if I'm testing people's perception of the product after I made updates to the onboarding, I'm gonna only want to I'm gonna only wanna survey people who went through the new onboarding.
00:23:19 · Sean Ellis, Lenny Rachitsky
[00:23:19] S1: Not people who've seen like a demo of your product, but it's Right. It's people who actually have experienced the product. But it's okay if they... If if you're hitting people who've used it months later. But like in that Lookout example that I gave, if I'm if I'm testing people's perception of the product after I made updates to the onboarding, I'm gonna only want to I'm gonna only wanna survey people who went through the new onboarding. [00:23:43] S0: Yep. In the in the experimental group? Yeah. Okay. So I asked people on Twitter what to ask you. A lot of people had a lot of awesome questions. I'm gonna ask... I'm gonna sprinkle in a couple of these questions throughout the chat. One came in from Shiraz Doshi, popular guest of the podcast. [00:23:58] S1: One of the ones that I listened to recently. [00:24:00] S0: Amazing. Yeah. Sec... I think it's the second most popular episode behind Brian Chesky. Okay. So he had a question of just what are the limitations of the score? When does it break down? When should you not use it if ever? Is there anything of just, like, here's when it's not gonna work for you?