Lenny's Podcast: Product | Career | Growth
00:38:05 · Sean Ellis, Lenny Rachitsky
[00:38:05] S1: Yeah. What I would say is, like, but retention cohorts don't don't give you any of the qualitative insights into the why. So that's why we continue to do the survey. So initially, I would say if the survey comes back and it shows whatever your target number is, if you want to be like Nubank, it'd be a 50% or I spent a few years with two of the companies I launched, we launched in Hungary and I would say it's kind of the opposite end of the spectrum of Brazil, maybe more pessimistic than average kind of culture. And so maybe 30% is good enough there. But that ultimately, whatever your target is, you have the signal that says, Okay, we have enough value here. Let's start working on growing the business. But while you're working on growing the business, I would be paying attention to those retention cohorts. And if you're churning out all the customers who were saying that they'd be very disappointed without the product, then okay, let's retrench and rethink, we really have product market fit here and what do we need to do to get it if we don't? [00:39:12] S0: Awesome. And speaking of Nubank, if anyone wants to see how a company has actually operationalized this in the way they operate, that... There's an episode that we'll link to in the show notes where every new product at Nubank they build, before they launch it, they wait for a 50% threshold for people to say 50% of people would be disappointed if this product did not exist as they're developing it. And only then do they launch it publicly.
00:39:12 · Lenny Rachitsky, Sean Ellis
[00:39:12] S0: Awesome. And speaking of Nubank, if anyone wants to see how a company has actually operationalized this in the way they operate, that... There's an episode that we'll link to in the show notes where every new product at Nubank they build, before they launch it, they wait for a 50% threshold for people to say 50% of people would be disappointed if this product did not exist as they're developing it. And only then do they launch it publicly. [00:39:36] S1: Yeah. I think they even do it down to the feature level. [00:39:39] S0: Wow. [00:39:39] S1: So if think about it, you know, how would you feel if you can no longer use this feature? It starts to give you, again, the signal, is feature a must have feature? And if it's not, maybe we shouldn't have it. And so, yeah, I was super excited when I when I saw how they were using the the survey, and they they were doing it before I engaged with them. [00:39:59] S0: Oh, wow. [00:40:00] S1: But... Yeah. That's awesome. They they were doing it, I think, from pretty early on in the business. [00:40:05] S0: The reason they can do this is they have a lot of users. They have millions and millions of users, so they can ask some small percentage of people this question. Because people hearing this might be like, oh my god. How many times am I gonna be asked this question when I'm using this feature? But they have a lot of users, so it's easier. [00:40:17] S1: Yeah. Yeah.
00:40:17 · Sean Ellis, Lenny Rachitsky
[00:40:17] S1: Yeah. Yeah. [00:40:18] S0: Okay. Last question, I promise, along these lines. Say someone is listening and they're like, okay. Man, I'm getting, like, 10%. I'm getting 15%. I don't know what to do to increase my product market fit. You shouldn't just like a strategy of just dig into the people that are very disappointed and see what they have to say. But any other advice slash what do you find often works in helping people move from, say, 10% to 40%?
00:40:43 · Sean Ellis
[00:40:43] S1: Yeah. So one of the things that's kind of cool about, almost like open sourcing this survey approach is, is, again, watching, like, how how Nubank has, has evolved their usage. But, one of the other companies that I think used it in an interesting way is Superhuman. I would say that they basically ended up probably putting a lot more momentum behind the question than it had even before. They posted something about how they did it on First Round Capital's blog. And what I have always said, and again, it's me coming at it from probably initially a marketing background, which is I'm kind of taking the product as a fixed thing. And how do I actually figure out how to market and grow this product? Product changes are going to take a long time. So what are the variables that I can control with a marketing background? But so one of the things I've always said is just ignore the people who say they'd be somewhat disappointed. They're telling you it's nice to have. They're as good as gone. So just ignore those guys. And then, but what Superhuman did was and the reason that yeah, I'll put one piece in the middle there before I say what Superhuman did. The reason that I say ignore those guys is that if you start paying attention to what your somewhat disappointed users are telling you, and then you start tweaking onboarding and product based on their feedback, maybe you're going to dilute it for your must have users. And that ultimately it becomes kind of good for everyone, but not great for anyone. And so that was my fear of trying to read too much into the users who say they'd be somewhat disappointed. But the Superhuman guys actually found, I think, a good way around that where they said, okay, what is the benefit that my must have users are focused on? And then of the users who say they'd be somewhat disappointed, so the nice to have users, Of those users who are also focused on that benefit, what do they need in the product for it then to become a must have for them? And so they're staying true to that kind of core benefit, but they're trying to essentially take those on the fence users and moving them up. And so I think their way of approaching that addressed what my concern was, which is are we gonna are we gonna kinda break it for the must have users?
00:43:04 · Lenny Rachitsky, Sean Ellis
[00:43:04] S0: That's an awesome insight. By the way, did Rahul and the team there just do this on their own or were you involved in any way in this? [00:43:10] S1: No. Mean, that's the same thing. Like I said, I I wasn't initially involved with Nubank. I wasn't involved with them. And that's the benefit of... You know, we wrote about it in our book in 2017. And so I think that I got it out there. I actually teamed up with the KISSmetrics team in 2012 and essentially published this survey on survey. Where we just made it freely available for people and a really easy template to prepare and send out and the how to guide on it. And it was all just free. I think Kissmetrics kind of using it as maybe lead gen. For me, I just wanted a way to kind of put something out for the community. And so it's been out there for a long time, so it's not surprising that different companies have found different unique ways to use it. [00:44:01] S0: That's awesome. I think that post is one of most popular in the first round. Really had an impact on a lot of people. [00:44:05] S1: Yeah. [00:44:06] S0: So just to repeat the approach you recommend for when you're digging into, I wrote this down when you were talking, for how to dig into what benefit people are finding. Your advice is, it's basically a follow-up survey to the extremely disappointed people asking them what is the primary benefit you get. It's a open text initially. Then once you get a collection, you do... It sounds like another survey as multiple choice. Here's, like, [00:44:31] S1: five benefits. Group of people, to be clear.
00:44:31 · Sean Ellis, Lenny Rachitsky
[00:44:31] S1: five benefits. Group of people, to be clear. [00:44:33] S0: Group. Got it. Awesome. And then it's like, which of these four or five benefits is is what you're getting out of this product? And then the question is, why is this benefit important to you? [00:44:41] S1: Yeah. And then you'll see, we have, like, eventually the the survey.io got closed down, but essentially the template that I typically used was then moved to pmfsurvey.com. And so you'll see some other questions that I have on there as well. Like what would you use instead if this product were no longer available? And that's one of the interesting things as you start to see people who say they'd be somewhat disappointed. Usually, they're focused on a commodity use case, and they know an easy alternative to switch to. So to be a must have, it needs to be both valuable and unique. [00:45:18] S0: Okay. Anything else on this topic of the Sean Ellis test, Product Market Fit test before we move on to growth strategy advice? [00:45:24] S1: No. I think that's good.