Lenny's Podcast: Product | Career | Growth
00:22:22 · Lenny Rachitsky, Sean Ellis
[00:22:22] S0: So that question I wanted to ask is what's your guidance on when to ask this question? What I'm hearing is if you ask it very far along the journey, when they're very invested, you'll get a much higher score. Is there any advice on the timing on the best time to ask this question of your users? [00:22:38] S1: What I recommend is a random sample of people who've really used your product. So they've gone in, they didn't just sign up, but they went in and hopefully hit that activation moment. They've used it twice, two plus times, and they've ideally used it, say, within the last week or two weeks so that they haven't churned yet. So if it's a random sample of those people, that's that's kind of the ideal time to ask it. [00:23:08] S0: Got it. So basically, it's people that have activated whatever that means to you and have been using it for a couple weeks. Not not people, like, landing in your homepage and not people just signing up. Right. Not people months later. [00:23:19] S1: Not people who've seen like a demo of your product, but it's Right. It's people who actually have experienced the product. But it's okay if they... If if you're hitting people who've used it months later. But like in that Lookout example that I gave, if I'm if I'm testing people's perception of the product after I made updates to the onboarding, I'm gonna only want to I'm gonna only wanna survey people who went through the new onboarding.
00:23:19 · Sean Ellis, Lenny Rachitsky
[00:23:19] S1: Not people who've seen like a demo of your product, but it's Right. It's people who actually have experienced the product. But it's okay if they... If if you're hitting people who've used it months later. But like in that Lookout example that I gave, if I'm if I'm testing people's perception of the product after I made updates to the onboarding, I'm gonna only want to I'm gonna only wanna survey people who went through the new onboarding. [00:23:43] S0: Yep. In the in the experimental group? Yeah. Okay. So I asked people on Twitter what to ask you. A lot of people had a lot of awesome questions. I'm gonna ask... I'm gonna sprinkle in a couple of these questions throughout the chat. One came in from Shiraz Doshi, popular guest of the podcast. [00:23:58] S1: One of the ones that I listened to recently. [00:24:00] S0: Amazing. Yeah. Sec... I think it's the second most popular episode behind Brian Chesky. Okay. So he had a question of just what are the limitations of the score? When does it break down? When should you not use it if ever? Is there anything of just, like, here's when it's not gonna work for you?
00:24:00 · Lenny Rachitsky, Sean Ellis
[00:24:00] S0: Amazing. Yeah. Sec... I think it's the second most popular episode behind Brian Chesky. Okay. So he had a question of just what are the limitations of the score? When does it break down? When should you not use it if ever? Is there anything of just, like, here's when it's not gonna work for you? [00:24:14] S1: I think one off products would probably... You know, like, how would you feel if you could no longer watch the movie you just watched? Like, I wouldn't care. Even like when I run a workshop, I don't run this as part of my survey after I do a workshop because, like how would you feel that you can no longer attend the workshop you just attended? It doesn't make sense. So I'll ask an NPS question as my filtering question so that I'm focusing in on feedback of people who love it. Also then through a separate lens, looking at people maybe who would be my detractors. So I think one off products are probably not good products to run the question on. There may be other places as well that I'm not thinking of right now, but Okay. [00:24:59] S0: But it sounds like not many. What I'm hearing is it's generally widely applicable. [00:25:04] S1: Yeah. Think it is. Like, at least from from my perspective, like, it's it's been really useful for me anyway.
00:25:04 · Sean Ellis, Lenny Rachitsky
[00:25:04] S1: Yeah. Think it is. Like, at least from from my perspective, like, it's it's been really useful for me anyway. [00:25:11] S0: Awesome. Okay. And then the follow-up question from Shiraz is... And I kind of asked this, but I'm curious if there's anything more here. Just have you seen any instances of startups over relying on the score prematurely declaring product market fit when in reality they haven't reached it yet? And just... Are there any other caveats of, like, cool. I got 40%. Is there anything else you should know? Okay, but maybe check this one thing.
00:25:31 · Sean Ellis
[00:25:31] S1: Yeah. I mean, I think to me, it's kind of like, what really is the definition of product market fit is the definition that people who get through my crappy onboarding and actually experience the product love it. And if I'm able to retain those people, that means I have product market fit. Or is fixing that crappy onboarding part of getting to product market fit as well? I think that's up for debate. To me, the hardest... I wouldn't obsess on onboarding if I know those who kind of get through the challenge of getting started with a product still don't like the product, then feels like it's a core product issue or wrong people using it in the wrong way issue. But once you have that, then ultimately it doesn't mean that you're ready to grow. I focus on growth, then customer acquisition is almost the last step. Once I validate that it's a must have for those early users, then I'm thinking about, Okay, how do I optimize speed to value? How do I make sure that people have the right prompts to come back and use the product at the right time? It's kind of more of that engagement loop. How do I get my existing users to bring in more users if there's something that makes sense on that end? Even how do optimize my revenue model? And once all of those things are working well, then I'll obsess on the customer acquisition side. But customer acquisition is so hard that if you're not really efficient at converting and retaining and monetizing people, you're gonna really struggle on the customer acquisition side.
00:27:08 · Lenny Rachitsky, Sean Ellis
[00:27:08] S0: Yeah. Cool. And we'll talk about customer acquisition slash growth. Sure. Another question I wanted to ask and a couple listeners asked is the 40%. So I had a Jag from Nubank on the podcast. I think you may have worked with them, and they use 50% as their threshold because apparently Brazilians are very nice and they just [00:27:28] S1: Optimistic. He's not gonna use what he said. Yeah. [00:27:30] S0: Yeah. So I guess the question is, you find instances where you should increase that percentage and slash in b two b is anything different? Do you change the percentage in b two b? Any advice there just like when you adjust the threshold?