Lenny's Podcast: Product | Career | Growth
00:29:07 · Lenny Rachitsky, Jason Cohen
[00:29:07] S0: Oh, man. I'm so happy we're spending so much time on this very specific first step of logo churn, because the way you described it is so visceral. It took so much. It's impossible how far this customer got already. They are using your product and understand it mostly, and then they still decide to leave. So brutal the way- [00:29:29] S1: And you're going to believe them when they say it's because of the cost? It just doesn't even make sense when you put it that way. [00:29:36] S0: So let me summarize the advice you shared here, because this is so good. So step one is look at logo churn. The way to understand, and essentially to understand how big of a problem this is and why you need to spend time here is basically do the math, how many new customers you're getting divided by the cancellation rate, and that essentially tells you, if that doesn't change, what's the maximum number of customers you will ever have? [00:30:01] S1: Exactly. [00:30:02] S0: That's going to be a sad number. And then the question is, okay, cool. How do I reduce the cancellation rate? Obviously, as you said, everyone wants new customers, more new customers. [00:30:08] S1: Yeah. And I know you're going to do that anyway, but you've got this cap. [00:30:12] S0: Exactly. Okay, so a few things you've shared here. One is instead of asking people a multiple choice, why did you decide to cancel? You make it freeform and you make the question, how would you say it? Was it what made you cancel? [00:30:26] S1: What Made you cancel? [00:30:27] S0: What made you cancel? [00:30:28] S1: Yeah.
00:30:28 · Jason Cohen, Lenny Rachitsky
[00:30:28] S1: Yeah. [00:30:28] S0: Great. [00:30:28] S1: Yeah. [00:30:29] S0: And then you could use AI to help summarize these things, I imagine, instead of ...
00:30:32 · Jason Cohen
[00:30:32] S1: Yeah. I think what I find with AI is this, with this sort of thing, with surveys is this. AI is good at picking out themes. It is bad at picking out details that are actionable. When I say AI, of course, I mean LLMs, which is probably what we mean when we're looking at natural language. And if you think about it, it makes sense because the LLM is an averaging machine. It's predicting the most likely. That's an averaging kind of a thing. And so when what you're looking for is a kind of average, it's usually pretty good, so summarization, topics, themes, but when you're asking for what is interesting and not average, it's actually pretty bad at it. One way that I've found that's sort of useful is, yes, I'll ask it about themes, but then I'll say, "Now pick out every specific detail that goes under one of these themes. Put it along with which customer said it and the link to blah, blah, blah." So you have to play with this to tune it right, but that kind of thing so that a human being can then still see the detail, which is what triggers in your mind, "Wait a minute. But that means we should do this," because the topics won't do that. The topics will be ... I already know what the topics will be. It'll be stuff like I couldn't figure out how to do this, this integration. The topics are actually not going to be that surprising probably. It's the details that are going to be the triggers for actionable stuff or patterns or something like that. So yeah, AI is not useless, but it's not as useful as it sounds. It's probably still a good idea to just read all this stuff. Although AI might be able to clean up, maybe people's grammar's bad, it's a weird language. Okay, yes, that's annoying. You could clean that up, but I wouldn't rely on AI to do the thinking for that reason.
00:32:20 · Lenny Rachitsky, Jason Cohen
[00:32:20] S0: That's such good advice. I actually have a really cool guest post coming out soon that gives a bunch of really specific techniques to avoid AI hallucinating or just giving you really bad results from this very specific synthesis work, because it turns out AI is very not great at actually being honest about some of the stuff, so I'll link to it if it comes out before this. And I think in real life, most people don't have that much. The volume of these cancellations, unless it's a super consumer app, is not that high, so you don't even need AI for this. Just read it. And then this is a good segue to your next piece of advice, which is essentially the five whys but not the five whys, where you force yourself to dig into what's the real reason that forced them to cancel? It's probably not pricing. It's probably not the project ended. There's something deeper. [00:33:06] S1: Yeah. [00:33:07] S0: And then advice number three is try to catch people early, try to catch them before they churn, and if you don't have a lot of customers, it's a lot easier if you have a lot. It's obviously harder. There's always been this holy grail idea of a product that just watches metrics and tells you this person's going to cancel. I haven't seen that before.
00:33:23 · Jason Cohen, Lenny Rachitsky
[00:33:23] S1: What I would say is it is not hard. You don't need a lot of customers to go talk to the ones who are in trouble. You do need a lot of data or customers to mathematically know what behaviors are correlated with canceling, and therefore, to spend your time wisely. Then you need a lot more data. But to your point, even if you have that data, it's not entirely clear whether some kind of mechanistic thing is all that important. One way I look at it is it's very common advice, you should try to get more good customers and fewer bad customers. Of course you should. And so therefore, they say you should see what the good customers have in common, but that's not the end of the sentence because a lot of the things that good customers have in common, they also have in common with the bad customers, because it's just what your customers do, just what anybody does. So it's what the good customers have in common that are different from what the bad customers have in common. Okay. So with that in mind, this it has to be both or else you're just getting correlations that are not helpful, the cancellations or talking to people who are in trouble is another application of that. So what is correlated with people who actually end up canceling, not just what ... And so I think that mindset is correct, if you add the other side of that to it. [00:34:44] S0: Really important nuance. [00:34:46] S1: Yeah.
00:34:46 · Jason Cohen, Lenny Rachitsky
[00:34:46] S1: Yeah. [00:34:46] S0: Okay. And then the final step just to close this out is onboarding, work on onboarding activation. One of the most recurring themes on this podcast is just the power across every dimension of improving onboarding, improving activation. [00:34:59] S1: Yeah. [00:34:59] S0: Sweet. Okay, so this is just step one, which is already full of gold if your growth has slowed. So step one is focus on your logo churn, the number of customers leaving, people leaving, actually canceling your product.