Lenny's Podcast: Product | Career | Growth
00:20:38 · Lenny Rachitsky, Sean Ellis
[00:20:38] S0: the Never seen that. [00:20:39] S1: And I was like, how could that be possible? Product is kind of a commoditized category. I wouldn't even say it's one of the best. And then when I dug into it again, it comes back to that nearly all hooked model is that the investment people have made in building that website, they put so much into that. They know exactly how to make the changes in the CMS kind of side of things. They have spent a lot of time just making it beautiful. So ultimately it was something that that was why they were saying they'd be very disappointed. I mean, but for kind of fast forward and on when I initially went in, still doing these things help the business resume growth and have significant growth over the next twelve months after we did these things. So still the signal we got from why people would be very disappointed without the product was important and speed to value and kind of all the other things I think about in go to market for an early stage product still were relevant, but just the I think they were a little stronger on the percentage, you'd be very disappointed. Even Eventbrite when I was there when we ran it was probably the second highest I'd ever seen. But with event organizers, if they've already set their event up on that platform and they've sent it out to their list and all those people are coming in and they're managing their event, again, they've invested a lot in the platform. Sort of switching costs, I think, can factor in there. So it's a function of both switching costs and utility of the product.
00:22:22 · Lenny Rachitsky, Sean Ellis
[00:22:22] S0: So that question I wanted to ask is what's your guidance on when to ask this question? What I'm hearing is if you ask it very far along the journey, when they're very invested, you'll get a much higher score. Is there any advice on the timing on the best time to ask this question of your users? [00:22:38] S1: What I recommend is a random sample of people who've really used your product. So they've gone in, they didn't just sign up, but they went in and hopefully hit that activation moment. They've used it twice, two plus times, and they've ideally used it, say, within the last week or two weeks so that they haven't churned yet. So if it's a random sample of those people, that's that's kind of the ideal time to ask it. [00:23:08] S0: Got it. So basically, it's people that have activated whatever that means to you and have been using it for a couple weeks. Not not people, like, landing in your homepage and not people just signing up. Right. Not people months later. [00:23:19] S1: Not people who've seen like a demo of your product, but it's Right. It's people who actually have experienced the product. But it's okay if they... If if you're hitting people who've used it months later. But like in that Lookout example that I gave, if I'm if I'm testing people's perception of the product after I made updates to the onboarding, I'm gonna only want to I'm gonna only wanna survey people who went through the new onboarding.
00:23:19 · Sean Ellis, Lenny Rachitsky
[00:23:19] S1: Not people who've seen like a demo of your product, but it's Right. It's people who actually have experienced the product. But it's okay if they... If if you're hitting people who've used it months later. But like in that Lookout example that I gave, if I'm if I'm testing people's perception of the product after I made updates to the onboarding, I'm gonna only want to I'm gonna only wanna survey people who went through the new onboarding. [00:23:43] S0: Yep. In the in the experimental group? Yeah. Okay. So I asked people on Twitter what to ask you. A lot of people had a lot of awesome questions. I'm gonna ask... I'm gonna sprinkle in a couple of these questions throughout the chat. One came in from Shiraz Doshi, popular guest of the podcast. [00:23:58] S1: One of the ones that I listened to recently. [00:24:00] S0: Amazing. Yeah. Sec... I think it's the second most popular episode behind Brian Chesky. Okay. So he had a question of just what are the limitations of the score? When does it break down? When should you not use it if ever? Is there anything of just, like, here's when it's not gonna work for you?
00:24:00 · Lenny Rachitsky, Sean Ellis
[00:24:00] S0: Amazing. Yeah. Sec... I think it's the second most popular episode behind Brian Chesky. Okay. So he had a question of just what are the limitations of the score? When does it break down? When should you not use it if ever? Is there anything of just, like, here's when it's not gonna work for you? [00:24:14] S1: I think one off products would probably... You know, like, how would you feel if you could no longer watch the movie you just watched? Like, I wouldn't care. Even like when I run a workshop, I don't run this as part of my survey after I do a workshop because, like how would you feel that you can no longer attend the workshop you just attended? It doesn't make sense. So I'll ask an NPS question as my filtering question so that I'm focusing in on feedback of people who love it. Also then through a separate lens, looking at people maybe who would be my detractors. So I think one off products are probably not good products to run the question on. There may be other places as well that I'm not thinking of right now, but Okay. [00:24:59] S0: But it sounds like not many. What I'm hearing is it's generally widely applicable. [00:25:04] S1: Yeah. Think it is. Like, at least from from my perspective, like, it's it's been really useful for me anyway.
00:25:04 · Sean Ellis, Lenny Rachitsky
[00:25:04] S1: Yeah. Think it is. Like, at least from from my perspective, like, it's it's been really useful for me anyway. [00:25:11] S0: Awesome. Okay. And then the follow-up question from Shiraz is... And I kind of asked this, but I'm curious if there's anything more here. Just have you seen any instances of startups over relying on the score prematurely declaring product market fit when in reality they haven't reached it yet? And just... Are there any other caveats of, like, cool. I got 40%. Is there anything else you should know? Okay, but maybe check this one thing.
00:25:31 · Sean Ellis
[00:25:31] S1: Yeah. I mean, I think to me, it's kind of like, what really is the definition of product market fit is the definition that people who get through my crappy onboarding and actually experience the product love it. And if I'm able to retain those people, that means I have product market fit. Or is fixing that crappy onboarding part of getting to product market fit as well? I think that's up for debate. To me, the hardest... I wouldn't obsess on onboarding if I know those who kind of get through the challenge of getting started with a product still don't like the product, then feels like it's a core product issue or wrong people using it in the wrong way issue. But once you have that, then ultimately it doesn't mean that you're ready to grow. I focus on growth, then customer acquisition is almost the last step. Once I validate that it's a must have for those early users, then I'm thinking about, Okay, how do I optimize speed to value? How do I make sure that people have the right prompts to come back and use the product at the right time? It's kind of more of that engagement loop. How do I get my existing users to bring in more users if there's something that makes sense on that end? Even how do optimize my revenue model? And once all of those things are working well, then I'll obsess on the customer acquisition side. But customer acquisition is so hard that if you're not really efficient at converting and retaining and monetizing people, you're gonna really struggle on the customer acquisition side.