Lenny's Podcast: Product | Career | Growth
00:27:48 · Jason Cohen
So in that crazy-looking curve, for the people that have watched it for 15 minutes, maybe there's something you could do to keep a few of them staying to the end, but that's not going to change very much how many people get to the end. Whereas for me, I've only done a few, but what I see is about 50% fall off in the first 30 seconds. Well, if I can get that from 50% to 55% stay, that's an additional... And at the end of the line, I only have 20% still there, which is pretty good for a longer video. But if I get it from 50 to 55, I might go from 20 to 25% staying. In other words, if I shifted 10% at the front, which maybe I could do, I can't be dramatic but maybe a little, then in the output, I might be able to increase it by 20, 30%. So that's a huge change, and so the SaaS equivalent is, as we all know, if they leave early, not only is it bad but it's super unprofitable, because you spend all this money to acquire them and then they never stayed around long enough to pay it back, much less to be profitable. So if you can do a little bit in the onboarding or shift the onboarding percentage a little bit, it pays off enormously in revenue and profit over time by making them successful. So again, if you don't know what to do, onboarding is a good bet, and even if you do know what to do, I'll still bet that onboarding is a good bet for where to go.
00:29:07 · Lenny Rachitsky, Jason Cohen
[00:29:07] S0: Oh, man. I'm so happy we're spending so much time on this very specific first step of logo churn, because the way you described it is so visceral. It took so much. It's impossible how far this customer got already. They are using your product and understand it mostly, and then they still decide to leave. So brutal the way- [00:29:29] S1: And you're going to believe them when they say it's because of the cost? It just doesn't even make sense when you put it that way. [00:29:36] S0: So let me summarize the advice you shared here, because this is so good. So step one is look at logo churn. The way to understand, and essentially to understand how big of a problem this is and why you need to spend time here is basically do the math, how many new customers you're getting divided by the cancellation rate, and that essentially tells you, if that doesn't change, what's the maximum number of customers you will ever have? [00:30:01] S1: Exactly. [00:30:02] S0: That's going to be a sad number. And then the question is, okay, cool. How do I reduce the cancellation rate? Obviously, as you said, everyone wants new customers, more new customers. [00:30:08] S1: Yeah. And I know you're going to do that anyway, but you've got this cap. [00:30:12] S0: Exactly. Okay, so a few things you've shared here. One is instead of asking people a multiple choice, why did you decide to cancel? You make it freeform and you make the question, how would you say it? Was it what made you cancel? [00:30:26] S1: What Made you cancel? [00:30:27] S0: What made you cancel? [00:30:28] S1: Yeah.
00:30:28 · Jason Cohen, Lenny Rachitsky
[00:30:28] S1: Yeah. [00:30:28] S0: Great. [00:30:28] S1: Yeah. [00:30:29] S0: And then you could use AI to help summarize these things, I imagine, instead of ...
00:30:32 · Jason Cohen
[00:30:32] S1: Yeah. I think what I find with AI is this, with this sort of thing, with surveys is this. AI is good at picking out themes. It is bad at picking out details that are actionable. When I say AI, of course, I mean LLMs, which is probably what we mean when we're looking at natural language. And if you think about it, it makes sense because the LLM is an averaging machine. It's predicting the most likely. That's an averaging kind of a thing. And so when what you're looking for is a kind of average, it's usually pretty good, so summarization, topics, themes, but when you're asking for what is interesting and not average, it's actually pretty bad at it. One way that I've found that's sort of useful is, yes, I'll ask it about themes, but then I'll say, "Now pick out every specific detail that goes under one of these themes. Put it along with which customer said it and the link to blah, blah, blah." So you have to play with this to tune it right, but that kind of thing so that a human being can then still see the detail, which is what triggers in your mind, "Wait a minute. But that means we should do this," because the topics won't do that. The topics will be ... I already know what the topics will be. It'll be stuff like I couldn't figure out how to do this, this integration. The topics are actually not going to be that surprising probably. It's the details that are going to be the triggers for actionable stuff or patterns or something like that. So yeah, AI is not useless, but it's not as useful as it sounds. It's probably still a good idea to just read all this stuff. Although AI might be able to clean up, maybe people's grammar's bad, it's a weird language. Okay, yes, that's annoying. You could clean that up, but I wouldn't rely on AI to do the thinking for that reason.
00:32:20 · Lenny Rachitsky, Jason Cohen
[00:32:20] S0: That's such good advice. I actually have a really cool guest post coming out soon that gives a bunch of really specific techniques to avoid AI hallucinating or just giving you really bad results from this very specific synthesis work, because it turns out AI is very not great at actually being honest about some of the stuff, so I'll link to it if it comes out before this. And I think in real life, most people don't have that much. The volume of these cancellations, unless it's a super consumer app, is not that high, so you don't even need AI for this. Just read it. And then this is a good segue to your next piece of advice, which is essentially the five whys but not the five whys, where you force yourself to dig into what's the real reason that forced them to cancel? It's probably not pricing. It's probably not the project ended. There's something deeper. [00:33:06] S1: Yeah. [00:33:07] S0: And then advice number three is try to catch people early, try to catch them before they churn, and if you don't have a lot of customers, it's a lot easier if you have a lot. It's obviously harder. There's always been this holy grail idea of a product that just watches metrics and tells you this person's going to cancel. I haven't seen that before.
00:33:23 · Jason Cohen, Lenny Rachitsky
[00:33:23] S1: What I would say is it is not hard. You don't need a lot of customers to go talk to the ones who are in trouble. You do need a lot of data or customers to mathematically know what behaviors are correlated with canceling, and therefore, to spend your time wisely. Then you need a lot more data. But to your point, even if you have that data, it's not entirely clear whether some kind of mechanistic thing is all that important. One way I look at it is it's very common advice, you should try to get more good customers and fewer bad customers. Of course you should. And so therefore, they say you should see what the good customers have in common, but that's not the end of the sentence because a lot of the things that good customers have in common, they also have in common with the bad customers, because it's just what your customers do, just what anybody does. So it's what the good customers have in common that are different from what the bad customers have in common. Okay. So with that in mind, this it has to be both or else you're just getting correlations that are not helpful, the cancellations or talking to people who are in trouble is another application of that. So what is correlated with people who actually end up canceling, not just what ... And so I think that mindset is correct, if you add the other side of that to it. [00:34:44] S0: Really important nuance. [00:34:46] S1: Yeah.