Lenny's Podcast: Product | Career | Growth
00:37:30 · Lenny Rachitsky
[00:37:30] S0: Such an important topic. Yeah. Just like, again, to remind people why this is so important. One of the most common questions founders ask is, do I have product market fit? Have I built something people want? Like, that's just an endless series of I don't know. How do I know? When do I know? And this is telling you in a really interesting way. So your advice is this is a leading indicator. You don't actually know until people actually start using it and whether they retain and continue using it. Is there just, like, advice on the shift you make from relying on the survey to actually looking at retention cohorts? Is it just once you have enough data, once you have a couple cohorts, then start looking at that, forget about the survey?
00:38:05 · Sean Ellis, Lenny Rachitsky
[00:38:05] S1: Yeah. What I would say is, like, but retention cohorts don't don't give you any of the qualitative insights into the why. So that's why we continue to do the survey. So initially, I would say if the survey comes back and it shows whatever your target number is, if you want to be like Nubank, it'd be a 50% or I spent a few years with two of the companies I launched, we launched in Hungary and I would say it's kind of the opposite end of the spectrum of Brazil, maybe more pessimistic than average kind of culture. And so maybe 30% is good enough there. But that ultimately, whatever your target is, you have the signal that says, Okay, we have enough value here. Let's start working on growing the business. But while you're working on growing the business, I would be paying attention to those retention cohorts. And if you're churning out all the customers who were saying that they'd be very disappointed without the product, then okay, let's retrench and rethink, we really have product market fit here and what do we need to do to get it if we don't? [00:39:12] S0: Awesome. And speaking of Nubank, if anyone wants to see how a company has actually operationalized this in the way they operate, that... There's an episode that we'll link to in the show notes where every new product at Nubank they build, before they launch it, they wait for a 50% threshold for people to say 50% of people would be disappointed if this product did not exist as they're developing it. And only then do they launch it publicly.
00:39:12 · Lenny Rachitsky, Sean Ellis
[00:39:12] S0: Awesome. And speaking of Nubank, if anyone wants to see how a company has actually operationalized this in the way they operate, that... There's an episode that we'll link to in the show notes where every new product at Nubank they build, before they launch it, they wait for a 50% threshold for people to say 50% of people would be disappointed if this product did not exist as they're developing it. And only then do they launch it publicly. [00:39:36] S1: Yeah. I think they even do it down to the feature level. [00:39:39] S0: Wow. [00:39:39] S1: So if think about it, you know, how would you feel if you can no longer use this feature? It starts to give you, again, the signal, is feature a must have feature? And if it's not, maybe we shouldn't have it. And so, yeah, I was super excited when I when I saw how they were using the the survey, and they they were doing it before I engaged with them. [00:39:59] S0: Oh, wow. [00:40:00] S1: But... Yeah. That's awesome. They they were doing it, I think, from pretty early on in the business. [00:40:05] S0: The reason they can do this is they have a lot of users. They have millions and millions of users, so they can ask some small percentage of people this question. Because people hearing this might be like, oh my god. How many times am I gonna be asked this question when I'm using this feature? But they have a lot of users, so it's easier. [00:40:17] S1: Yeah. Yeah.
00:40:17 · Sean Ellis, Lenny Rachitsky
[00:40:17] S1: Yeah. Yeah. [00:40:18] S0: Okay. Last question, I promise, along these lines. Say someone is listening and they're like, okay. Man, I'm getting, like, 10%. I'm getting 15%. I don't know what to do to increase my product market fit. You shouldn't just like a strategy of just dig into the people that are very disappointed and see what they have to say. But any other advice slash what do you find often works in helping people move from, say, 10% to 40%?
00:40:43 · Sean Ellis
[00:40:43] S1: Yeah. So one of the things that's kind of cool about, almost like open sourcing this survey approach is, is, again, watching, like, how how Nubank has, has evolved their usage. But, one of the other companies that I think used it in an interesting way is Superhuman. I would say that they basically ended up probably putting a lot more momentum behind the question than it had even before. They posted something about how they did it on First Round Capital's blog. And what I have always said, and again, it's me coming at it from probably initially a marketing background, which is I'm kind of taking the product as a fixed thing. And how do I actually figure out how to market and grow this product? Product changes are going to take a long time. So what are the variables that I can control with a marketing background? But so one of the things I've always said is just ignore the people who say they'd be somewhat disappointed. They're telling you it's nice to have. They're as good as gone. So just ignore those guys. And then, but what Superhuman did was and the reason that yeah, I'll put one piece in the middle there before I say what Superhuman did. The reason that I say ignore those guys is that if you start paying attention to what your somewhat disappointed users are telling you, and then you start tweaking onboarding and product based on their feedback, maybe you're going to dilute it for your must have users. And that ultimately it becomes kind of good for everyone, but not great for anyone. And so that was my fear of trying to read too much into the users who say they'd be somewhat disappointed. But the Superhuman guys actually found, I think, a good way around that where they said, okay, what is the benefit that my must have users are focused on? And then of the users who say they'd be somewhat disappointed, so the nice to have users, Of those users who are also focused on that benefit, what do they need in the product for it then to become a must have for them? And so they're staying true to that kind of core benefit, but they're trying to essentially take those on the fence users and moving them up. And so I think their way of approaching that addressed what my concern was, which is are we gonna are we gonna kinda break it for the must have users?
00:43:04 · Lenny Rachitsky, Sean Ellis
[00:43:04] S0: That's an awesome insight. By the way, did Rahul and the team there just do this on their own or were you involved in any way in this? [00:43:10] S1: No. Mean, that's the same thing. Like I said, I I wasn't initially involved with Nubank. I wasn't involved with them. And that's the benefit of... You know, we wrote about it in our book in 2017. And so I think that I got it out there. I actually teamed up with the KISSmetrics team in 2012 and essentially published this survey on survey. Where we just made it freely available for people and a really easy template to prepare and send out and the how to guide on it. And it was all just free. I think Kissmetrics kind of using it as maybe lead gen. For me, I just wanted a way to kind of put something out for the community. And so it's been out there for a long time, so it's not surprising that different companies have found different unique ways to use it. [00:44:01] S0: That's awesome. I think that post is one of most popular in the first round. Really had an impact on a lot of people. [00:44:05] S1: Yeah. [00:44:06] S0: So just to repeat the approach you recommend for when you're digging into, I wrote this down when you were talking, for how to dig into what benefit people are finding. Your advice is, it's basically a follow-up survey to the extremely disappointed people asking them what is the primary benefit you get. It's a open text initially. Then once you get a collection, you do... It sounds like another survey as multiple choice. Here's, like, [00:44:31] S1: five benefits. Group of people, to be clear.