Lenny's Podcast: Product | Career | Growth
01:33:18 · Lenny Rachitsky, Jason Cohen
[01:33:18] S0: So now we will be after this conversation. Okay. So I'm going to take us to a recurring corner, a recurring segment on the podcast that I call AI Corner. What's one way that you have discovered using AI in your work or in your life that might be helpful for folks to hear? [01:33:37] S1: There's a lot of data on the internet and it's often in things like images, which makes it hard to do your own analysis, or plug it in, or et cetera, come up with your own models or apply it. But I found that AI is really good actually, especially Gemini, at say, you just give a chart to it and say, "Make this into a table that I can paste," literally say, "that I can paste into Google Sheets." And it will do in a way that literally you can copy, and it will actually paste correctly into Google Sheets, and then you can do stuff. So especially with the book and my articles, I love to use real data whenever I can, of course. And so I do that all the time. So I think that kind of interpretation is very useful. And so all of a sudden you can get 10 examples of something and test a theory, where before it was just too hard and you didn't. [01:34:24] S0: That's an awesome tip because people know you can generate all these infographics, especially with Gemini and all these things. That's really cool to know you can just feed it, "Here's a chart and make a text." [01:34:35] S1: Yeah. [01:34:36] S0: Okay. I'm going to now take us to Contrarian Corner. The question here is, what's something that you believe that most other people don't?
01:34:43 · Jason Cohen
[01:34:43] S1: A/B testing doesn't work very well and it doesn't work on most things. It won't work on strategy, or vision, or insights, nothing actually important to the success of the company. You don't A/B test whether Uber's a good idea. And then, even when you do A/B test the details, where I agree, sometimes that can work. What happens is people will try things like, "Oh, I'll just try this verb, and that verb, and this ad, and that ad." And then like, "Oh, the seventh or eighth one, I got a positive result. That must be good." And what happens is you keep doing that. You pick the best one, and then you go on, and you find another one, and pick the best one. And then a year later you look back and you should be like 50 or a hundred percent better because you've stacked these things, and you look back, and like nothing's different. The conversion rates are the same as they've always been. You say, "What the hell happened? I thought I picked the winner." And the answer is in a combination of the tools not being statistically accurate, which they're not. And the fact that you will get false positives even if the tool is statistically accurate means that most of them are false positives. Even if the tool's 95% accurate, when the thing you're looking for is rare, which it is in the case of A/B testing, the false positives happen more often than the actual thing happens. And so most of the results you get are false positives anyway. So as a result, this isn't true of all A/B testing, but for what most people do when they just do the mundane A/B testing, you can't A/B test the important things and the details are mostly false positives, so it's an enormous waste of time unless you're incredibly sophisticated. I know there's special groups that actually are very sophisticated. Fine. If you're not doing that, it's like the poker table. If you don't know who the patsy is, it's you. If you don't have all of this information and knowledge about A/B testing, then you're the patsy.
01:36:31 · Lenny Rachitsky, Jason Cohen
[01:36:31] S0: Bam. All right. [01:36:32] S1: Yeah. [01:36:33] S0: I will just say that I have found A/B testing useful in my career. I think it's maybe at a certain scale when you're just trying to optimize and continue to grow, you know- [01:36:43] S1: Yeah. [01:36:43] S0: ... where it's like millions of users, like a percentage gain is like millions of dollars. [01:36:47] S1: True. [01:36:47] S0: Most people are not working at that scale, most people. [01:36:49] S1: Right. [01:36:50] S0: Yeah. So just wanted to, for folks that find it valuable. [01:36:54] S1: That's true. [01:36:55] S0: But I love- [01:36:56] S1: However, even so, on your own podcast when you were interviewing the guy from Shopify, and he was saying how maybe a third of the things that they found with their systems just disappear, they just magically disappear. And they have a team of a hundred people, and they're really good at it, and their effects disappear all the time. So they double check later whether the immediate effect goes away because even then. [01:37:19] S0: Right. I think that was the CTO of Shopify conversation. Yeah. [01:37:19] S1: Oh, okay. Yeah. [01:37:22] S0: Yeah. Sweet. Okay. [01:37:24] S1: I just go off of memory because I listen to a lot of episodes, but I don't- [01:37:26] S0: That's such a good one. [01:37:26] S1: ... no, it's- [01:37:27] S0: I love that. Yeah. Where they leave like a holdout group essentially, and then they just look back, "Was the effect something that lasted?" And most times it didn't. [01:37:34] S1: Yeah. [01:37:34] S0: Such a good one.
01:37:34 · Lenny Rachitsky, Jason Cohen
[01:37:34] S0: Such a good one. [01:37:35] S1: There you go. And that's with a lot of end. So that's what I mean. If you're doing that level of stuff, good for you. But if you're not, I don't know, man. [01:37:42] S0: There we go. Well, Jason, it's always a really good sign when I'm just like, "I can't wait to get this conversation out the door and into people's minds because there's so much value here." I'm just already anticipating all people are going to reply and just like, "I got so many ideas for what to do with my product," which is exactly the goal. And with that, we have reached our very exciting lightning round. I've got five questions for you. Are you ready? [01:38:06] S1: Boy. Yeah. I mean, I don't like talking a long time anyway, so lightning's great. [01:38:10] S0: Here we go. What are two or three books that you find yourself recommending most to other people? [01:38:16] S1: For writing, On Writing Well by William Zinsser. I know I'm not the only one, but that's the point. On my best day, I write like that. And then for product, I actually like Crossing the Chasm, which of course everyone's heard of, but what I find is no one's read it, so that you know the little picture, and you think you know what the chasm is, what I find is very quickly I realize, "Oh, you haven't read the book. You saw a blog post." And there's so much good stuff in there, how to define a market and really what to do with this model. It's fantastic. So I highly recommend reading the book.
01:38:16 · Jason Cohen, Lenny Rachitsky
[01:38:16] S1: For writing, On Writing Well by William Zinsser. I know I'm not the only one, but that's the point. On my best day, I write like that. And then for product, I actually like Crossing the Chasm, which of course everyone's heard of, but what I find is no one's read it, so that you know the little picture, and you think you know what the chasm is, what I find is very quickly I realize, "Oh, you haven't read the book. You saw a blog post." And there's so much good stuff in there, how to define a market and really what to do with this model. It's fantastic. So I highly recommend reading the book. [01:38:51] S0: I have had Geoffrey Moore on the podcast. We dove into a lot of this stuff. One of the things that always stuck with me is when early companies are looking for someone like them to adopt the thing. That's something that really stuck with me. It's not like they're looking for an early adopter to be like, "Oh, this is awesome." They're looking for someone that feels like them to say, "This is great." And so the early adopters are just going to spread to other early adopters and there's work to do beyond that. [01:39:14] S1: Yeah. Part of that's because he defines a market and among other things as, and the people in the market respect the opinions of the other people in the market. [01:39:23] S0: Exactly. [01:39:23] S1: And that's when you realize, "Oh, so jumping to a different market, it's not impossible. It's just like case studies aren't going to work." Yeah. [01:39:30] S0: It's a new thing. [01:39:31] S1: Yeah.
01:39:31 · Jason Cohen, Lenny Rachitsky
[01:39:31] S1: Yeah. [01:39:31] S0: Okay. All right. And then On Writing Well, such a huge trend in the book. That's like the book that most helped me write. And if you summarize the book, for me, it's just cut. Cut more and more of your stuff. There's always to cut. [01:39:43] S1: I love this phrase where he's on a panel with this guy who is like an amateur writer, and his summary to that guy is, the guy told him, "I never knew writing could be hard." And Zinsser says, " I never knew writing could be easy." I think both of those summarize the turmoil of being a writer. [01:40:07] S0: Yeah. The classic, maybe Hemingway, maybe not quote, "Writing is easy. I just sit at the typewriter and bleed." [01:40:14] S1: And bleed, yeah. [01:40:16] S0: So good. Okay. Moving on. Favorite recent movie or TV show? [01:40:21] S1: ER from 1994- [01:40:22] S0: Can- [inaudible 01:40:23] [01:40:24] S1: ... 15 seasons. Why do I say that? Besides the fact I think is good, I have a 16-year-old daughter and we're now on season, I think 13, watching this whole thing. She says it holds up after 25 years. And being Gen Alpha or whatever, I don't even know what it is. And so if this is an era when shows were an hour long, and seasons were forever, and she says it's great TV, it must be great TV. [01:40:52] S0: Wow. I've not had this podcast yet. Also, The Pitt. I don't know. If you enjoy ER, you'll enjoy The Pitt, which has won all these awards- [01:40:59] S1: Yeah. Pitt's good.