Lenny's Podcast: Product | Career | Growth
00:30:32 · Jason Cohen
[00:30:32] S1: Yeah. I think what I find with AI is this, with this sort of thing, with surveys is this. AI is good at picking out themes. It is bad at picking out details that are actionable. When I say AI, of course, I mean LLMs, which is probably what we mean when we're looking at natural language. And if you think about it, it makes sense because the LLM is an averaging machine. It's predicting the most likely. That's an averaging kind of a thing. And so when what you're looking for is a kind of average, it's usually pretty good, so summarization, topics, themes, but when you're asking for what is interesting and not average, it's actually pretty bad at it. One way that I've found that's sort of useful is, yes, I'll ask it about themes, but then I'll say, "Now pick out every specific detail that goes under one of these themes. Put it along with which customer said it and the link to blah, blah, blah." So you have to play with this to tune it right, but that kind of thing so that a human being can then still see the detail, which is what triggers in your mind, "Wait a minute. But that means we should do this," because the topics won't do that. The topics will be ... I already know what the topics will be. It'll be stuff like I couldn't figure out how to do this, this integration. The topics are actually not going to be that surprising probably. It's the details that are going to be the triggers for actionable stuff or patterns or something like that. So yeah, AI is not useless, but it's not as useful as it sounds. It's probably still a good idea to just read all this stuff. Although AI might be able to clean up, maybe people's grammar's bad, it's a weird language. Okay, yes, that's annoying. You could clean that up, but I wouldn't rely on AI to do the thinking for that reason.
00:32:20 · Lenny Rachitsky, Jason Cohen
[00:32:20] S0: That's such good advice. I actually have a really cool guest post coming out soon that gives a bunch of really specific techniques to avoid AI hallucinating or just giving you really bad results from this very specific synthesis work, because it turns out AI is very not great at actually being honest about some of the stuff, so I'll link to it if it comes out before this. And I think in real life, most people don't have that much. The volume of these cancellations, unless it's a super consumer app, is not that high, so you don't even need AI for this. Just read it. And then this is a good segue to your next piece of advice, which is essentially the five whys but not the five whys, where you force yourself to dig into what's the real reason that forced them to cancel? It's probably not pricing. It's probably not the project ended. There's something deeper. [00:33:06] S1: Yeah. [00:33:07] S0: And then advice number three is try to catch people early, try to catch them before they churn, and if you don't have a lot of customers, it's a lot easier if you have a lot. It's obviously harder. There's always been this holy grail idea of a product that just watches metrics and tells you this person's going to cancel. I haven't seen that before.
00:33:23 · Jason Cohen, Lenny Rachitsky
[00:33:23] S1: What I would say is it is not hard. You don't need a lot of customers to go talk to the ones who are in trouble. You do need a lot of data or customers to mathematically know what behaviors are correlated with canceling, and therefore, to spend your time wisely. Then you need a lot more data. But to your point, even if you have that data, it's not entirely clear whether some kind of mechanistic thing is all that important. One way I look at it is it's very common advice, you should try to get more good customers and fewer bad customers. Of course you should. And so therefore, they say you should see what the good customers have in common, but that's not the end of the sentence because a lot of the things that good customers have in common, they also have in common with the bad customers, because it's just what your customers do, just what anybody does. So it's what the good customers have in common that are different from what the bad customers have in common. Okay. So with that in mind, this it has to be both or else you're just getting correlations that are not helpful, the cancellations or talking to people who are in trouble is another application of that. So what is correlated with people who actually end up canceling, not just what ... And so I think that mindset is correct, if you add the other side of that to it. [00:34:44] S0: Really important nuance. [00:34:46] S1: Yeah.
00:34:46 · Jason Cohen, Lenny Rachitsky
[00:34:46] S1: Yeah. [00:34:46] S0: Okay. And then the final step just to close this out is onboarding, work on onboarding activation. One of the most recurring themes on this podcast is just the power across every dimension of improving onboarding, improving activation. [00:34:59] S1: Yeah. [00:34:59] S0: Sweet. Okay, so this is just step one, which is already full of gold if your growth has slowed. So step one is focus on your logo churn, the number of customers leaving, people leaving, actually canceling your product.
00:35:11 · Jason Cohen
[00:35:11] S1: So I look at it like a question. So the first question is are people leaving too much? Because if your monthly cancellation is 2% for S&B, that's good. So you could try to work on it, but since it's already good, it's still probably a good idea to work... It's probably a good ROI for you to work on it, but it's possible that you've got diminishing returns and that this isn't really the reason or it's not really reasonable for it to go. How low can it go for S&B? There's some floor and you might be near it. So the first question is is logo churn too high? And trying to set a threshold lower than what people normally want to do. So the next question I have is is the pricing correct? Which of course, pricing is a perennially interesting topic, I know. There's this funny thing, especially with newer companies, that the pricing is always too low. It's not always, but that's the common thing. Patrick Campbell, who has 4,200 data points about startups, let that sink in a little, has this great quote which goes like this: "Your prices are way too low because you just guessed and you haven't changed them." Yeah, if you really look deep within, you realize like, yeah, or we just picked whatever our competitors are doing and that's it, or we added or subtracted something because reasons. Right, that's probably not good. And people are scared to rate prices for obvious reasons, but if we set aside the emotional reasons, whether they're correct or not, the economic reason people normally give is they have in their mind this microeconomic supply and demand curve thing, and the demand curve says that if you raise the price, demand goes down. That's why demand curve is always going that way. And so they understand, I think everyone understands, right, but maybe you raise prices by 10%, but signups go down only 5%, so overall, it's better. But the opposite could happen too if I'm on the other side of the demand curve, and okay. So that's how most people think of it. However, this is not how it works. So that's how it works in microeconomics 101 textbooks, that's not how it works in the real world often. So what often happens is you raise prices and signups don't change. When I say signups, I mean signups per month, the rate at signup. Or signups go up. This happens all the time.
00:37:31 · Jason Cohen
Even for solopreneurs on Twitter who have strange projects or everything, it happens all the time. They raise prices, they're like, "I was scared," but then signups went up. I once talked to a guy, this is really funny. I'm going to not say the name to protect the name. So he had a product that he was selling essentially to enterprise and government, so larger companies, and it was to me way too cheap. So he said something like, "Yeah, I charge $300." I'm like, "$300 a month, that's not enough." He goes, "No, per year." They're like, "Okay, wait." I said, "Okay, how many signups do you get a week?" And he's like, "One or two," because this is enterprise and it was a startup. I said, "Okay. Just for fun, just change it from per year to per month," So in other words, we're 12x-ing the price. So he did, and he still got one or two per week. Nothing changed. I'm like, "Okay, what are you going to do next?" And he goes, "Oh my gosh, well, now I have so much more money and profit, so I'm going to hire an engineer. I'm going to do this marketing." And I'm like, "Time out. What you're going to do is raise prices again. You just told me you 12x-ed the price and nothing observable changed. That means you're not near the price yet, right? You don't have to 10x it again necessarily. Maybe 2x, maybe 50%, but you're not done. You can do those other things too, but you're not done with the price." It didn't even occur to him still. Okay. So why does this happen? The reason is that pricing selects the market. So if you only think of the market as people with very limited budgets, barely can do anything, not getting much value out of it, then it is true that if you raise prices, you'll get fewer of them, because they were never getting that much value out of it anyway. They don't have that much money so if you raise prices, they're gone. But think about just even a midsize company. Forget about enterprise, just think about a company with a thousand employees and 400 million in revenue or whatever, and if they see a product that's $2 a month or even $100 a month, the thought is like, well, that can't be good enough. They're not mature enough, it's not going to do enough. The support's not going to be good enough. They probably don't have good governance policies or other things that we need, et cetera.