Lenny's Podcast: Product | Career | Growth
01:32:15 · Lenny Rachitsky, Jason Cohen
[01:32:15] S0: Yeah. [01:32:16] S1: They will stay as long as possible. And these things will probably be right, if that. So this idea of creating value for the customer and then figuring out how to split with them is probably the root idea. And of course, I hesitate because platitudes like that are actually not actionable, not very actionable. Like, "All right, well, I'll move on with my day." And that's why Twitter's not so useful. But given that we've gone into so much detail, perhaps that's a nice way of summarizing it. [01:32:39] S0: I think that's such an important point. I think what's also interesting is some of your advice is the value, you may be picking the wrong customer, the wrong market, you may be positioning it wrong. So the value may be there, you're just trying to convince the wrong people about it. [01:32:54] S1: Yeah. There's so many ways to get it wrong because like we said, all these things have to be right. And you just said another one, which is, and you have to say in a way that when this person hits the homepage, they know it. It's true, but do they know that? It's just so many things have to go right. [01:33:11] S0: What a tough job we've got over here, just solving people's problems. Come on. Okay. [01:33:15] S1: Well, at least we're growing. [01:33:18] S0: So now we will be after this conversation. Okay. So I'm going to take us to a recurring corner, a recurring segment on the podcast that I call AI Corner. What's one way that you have discovered using AI in your work or in your life that might be helpful for folks to hear?
01:33:18 · Lenny Rachitsky, Jason Cohen
[01:33:18] S0: So now we will be after this conversation. Okay. So I'm going to take us to a recurring corner, a recurring segment on the podcast that I call AI Corner. What's one way that you have discovered using AI in your work or in your life that might be helpful for folks to hear? [01:33:37] S1: There's a lot of data on the internet and it's often in things like images, which makes it hard to do your own analysis, or plug it in, or et cetera, come up with your own models or apply it. But I found that AI is really good actually, especially Gemini, at say, you just give a chart to it and say, "Make this into a table that I can paste," literally say, "that I can paste into Google Sheets." And it will do in a way that literally you can copy, and it will actually paste correctly into Google Sheets, and then you can do stuff. So especially with the book and my articles, I love to use real data whenever I can, of course. And so I do that all the time. So I think that kind of interpretation is very useful. And so all of a sudden you can get 10 examples of something and test a theory, where before it was just too hard and you didn't. [01:34:24] S0: That's an awesome tip because people know you can generate all these infographics, especially with Gemini and all these things. That's really cool to know you can just feed it, "Here's a chart and make a text." [01:34:35] S1: Yeah. [01:34:36] S0: Okay. I'm going to now take us to Contrarian Corner. The question here is, what's something that you believe that most other people don't?
01:34:43 · Jason Cohen
[01:34:43] S1: A/B testing doesn't work very well and it doesn't work on most things. It won't work on strategy, or vision, or insights, nothing actually important to the success of the company. You don't A/B test whether Uber's a good idea. And then, even when you do A/B test the details, where I agree, sometimes that can work. What happens is people will try things like, "Oh, I'll just try this verb, and that verb, and this ad, and that ad." And then like, "Oh, the seventh or eighth one, I got a positive result. That must be good." And what happens is you keep doing that. You pick the best one, and then you go on, and you find another one, and pick the best one. And then a year later you look back and you should be like 50 or a hundred percent better because you've stacked these things, and you look back, and like nothing's different. The conversion rates are the same as they've always been. You say, "What the hell happened? I thought I picked the winner." And the answer is in a combination of the tools not being statistically accurate, which they're not. And the fact that you will get false positives even if the tool is statistically accurate means that most of them are false positives. Even if the tool's 95% accurate, when the thing you're looking for is rare, which it is in the case of A/B testing, the false positives happen more often than the actual thing happens. And so most of the results you get are false positives anyway. So as a result, this isn't true of all A/B testing, but for what most people do when they just do the mundane A/B testing, you can't A/B test the important things and the details are mostly false positives, so it's an enormous waste of time unless you're incredibly sophisticated. I know there's special groups that actually are very sophisticated. Fine. If you're not doing that, it's like the poker table. If you don't know who the patsy is, it's you. If you don't have all of this information and knowledge about A/B testing, then you're the patsy.
01:36:31 · Lenny Rachitsky, Jason Cohen
[01:36:31] S0: Bam. All right. [01:36:32] S1: Yeah. [01:36:33] S0: I will just say that I have found A/B testing useful in my career. I think it's maybe at a certain scale when you're just trying to optimize and continue to grow, you know- [01:36:43] S1: Yeah. [01:36:43] S0: ... where it's like millions of users, like a percentage gain is like millions of dollars. [01:36:47] S1: True. [01:36:47] S0: Most people are not working at that scale, most people. [01:36:49] S1: Right. [01:36:50] S0: Yeah. So just wanted to, for folks that find it valuable. [01:36:54] S1: That's true. [01:36:55] S0: But I love- [01:36:56] S1: However, even so, on your own podcast when you were interviewing the guy from Shopify, and he was saying how maybe a third of the things that they found with their systems just disappear, they just magically disappear. And they have a team of a hundred people, and they're really good at it, and their effects disappear all the time. So they double check later whether the immediate effect goes away because even then. [01:37:19] S0: Right. I think that was the CTO of Shopify conversation. Yeah. [01:37:19] S1: Oh, okay. Yeah. [01:37:22] S0: Yeah. Sweet. Okay. [01:37:24] S1: I just go off of memory because I listen to a lot of episodes, but I don't- [01:37:26] S0: That's such a good one. [01:37:26] S1: ... no, it's- [01:37:27] S0: I love that. Yeah. Where they leave like a holdout group essentially, and then they just look back, "Was the effect something that lasted?" And most times it didn't. [01:37:34] S1: Yeah. [01:37:34] S0: Such a good one.
01:37:34 · Lenny Rachitsky, Jason Cohen
[01:37:34] S0: Such a good one. [01:37:35] S1: There you go. And that's with a lot of end. So that's what I mean. If you're doing that level of stuff, good for you. But if you're not, I don't know, man. [01:37:42] S0: There we go. Well, Jason, it's always a really good sign when I'm just like, "I can't wait to get this conversation out the door and into people's minds because there's so much value here." I'm just already anticipating all people are going to reply and just like, "I got so many ideas for what to do with my product," which is exactly the goal. And with that, we have reached our very exciting lightning round. I've got five questions for you. Are you ready? [01:38:06] S1: Boy. Yeah. I mean, I don't like talking a long time anyway, so lightning's great. [01:38:10] S0: Here we go. What are two or three books that you find yourself recommending most to other people? [01:38:16] S1: For writing, On Writing Well by William Zinsser. I know I'm not the only one, but that's the point. On my best day, I write like that. And then for product, I actually like Crossing the Chasm, which of course everyone's heard of, but what I find is no one's read it, so that you know the little picture, and you think you know what the chasm is, what I find is very quickly I realize, "Oh, you haven't read the book. You saw a blog post." And there's so much good stuff in there, how to define a market and really what to do with this model. It's fantastic. So I highly recommend reading the book.
01:38:16 · Jason Cohen, Lenny Rachitsky
[01:38:16] S1: For writing, On Writing Well by William Zinsser. I know I'm not the only one, but that's the point. On my best day, I write like that. And then for product, I actually like Crossing the Chasm, which of course everyone's heard of, but what I find is no one's read it, so that you know the little picture, and you think you know what the chasm is, what I find is very quickly I realize, "Oh, you haven't read the book. You saw a blog post." And there's so much good stuff in there, how to define a market and really what to do with this model. It's fantastic. So I highly recommend reading the book. [01:38:51] S0: I have had Geoffrey Moore on the podcast. We dove into a lot of this stuff. One of the things that always stuck with me is when early companies are looking for someone like them to adopt the thing. That's something that really stuck with me. It's not like they're looking for an early adopter to be like, "Oh, this is awesome." They're looking for someone that feels like them to say, "This is great." And so the early adopters are just going to spread to other early adopters and there's work to do beyond that. [01:39:14] S1: Yeah. Part of that's because he defines a market and among other things as, and the people in the market respect the opinions of the other people in the market. [01:39:23] S0: Exactly. [01:39:23] S1: And that's when you realize, "Oh, so jumping to a different market, it's not impossible. It's just like case studies aren't going to work." Yeah. [01:39:30] S0: It's a new thing. [01:39:31] S1: Yeah.