Lenny's Podcast: Product | Career | Growth
00:33:47 · Amol Avasare
[00:33:47] S1: I think if your core product value is very backed by AI, then it is shifting where you're skewing more towards larger bets as opposed to smaller, medium experiments. I think in other things, it may be a big thing related to this that I think will accelerate this, which I am really interested to see how it will play out, is we are starting to look at how do we automate growth, which I think is a really interesting area. So our growth platform team, we're very lucky we have Alexey Komissarouk who teaches growth engineering at Reforge, and he's just the guy on the team. And so he is driving this effort. The name is, it's a little cringey. I didn't come up with it. And it's called CASH, which is Claude Accelerates Sustainable Hypergrowth. I did not come up with that, but really it's an effort to look at how can we use Claude to automate growth experimentation? And it is still very, very small. It's still very, very early. We kicked it off only, I think, a couple of months ago. I think before Opus 4.5, it wasn't really possible. We were just not seeing the results. And more recently with Opus 4.6 we're like, "Okay, this is headed in the right direction." And so this, I think, it will happen more and more across the industry, where basically if you think about, okay ... I think this can happen all across product, but growth teams in particular because there's this whole body of work that is very small optimizations, I think are just more inherently suited to tackle this earlier. I think if you think about the lifecycle of shipping, there's four parts to it. One is identifying opportunities. How good is Claude at actually identifying opportunities based on different trends, based on previous trends that Claude has seen in the past? Second is then building the actual feature and getting it ready to ship. Third is testing and ensuring that it meets your quality bar and your brand bar. And then fourth is then once you've actually shipped the thing, analyzing the data, gathering the learnings. If you think about that as the loop of, "Okay, these are four things that you can eval and hill climb on each of these areas and understand how good is a model doing for you there." And we basically think about it in that four ways and we are scoring how good is Claude doing in each of those areas. And so we've been testing this at pretty small scale right now.
00:36:25 · Amol Avasare, Lenny Rachitsky
It's been a lot of copy changes and some very minor UI tweaks. It's delivering results and it's like you can push it, press play with it. And it ultimately prints money where I'd say that the win rate is like, I would expect a senior PM to do better. I would say this is a junior PM, two, three years in. I would say this is the win rate that I would expect from a junior PM, but it's not quite at the senior PM level. Although, I think you look at the exponential, this wasn't available at all a couple of months ago. So it's getting better rapidly. And I think that it's going to change where you'll be able to do this for larger and larger types of experiments. But then you think about the largest types of experiments, I think that I mentioned the four pieces around identifying opportunities, building, testing and shipping. The one I didn't mention there, Lenny, is cross-functional stakeholder management. [00:37:26] S0: There's still a need for human brains. There is. [00:37:31] S1: And I think that that one is going to mean that in my eyes, the work of PMs is not going away actually, anytime soon. And that piece, especially for larger projects. Usually, you don't need to do as much of it for smaller stuff. You can skip it. But for larger stuff, that piece is not going away. [00:37:51] S0: Until the other stakeholders have their own little agents running around.
00:37:51 · Lenny Rachitsky, Amol Avasare
[00:37:51] S0: Until the other stakeholders have their own little agents running around. [00:37:54] S1: I think that's right. I think that would be the point where it changes. It's funny, we had a difficult meeting a couple of weeks ago, and me and our head of design, Joel, we were debriefing afterwards. And he pings me, he's just like, "Amol, we will have AGI and it will still be impossible to get six people in a room to get to align." And I'm like, "Yeah, I think I can say that." [00:38:16] S0: What's the harder alignment problem? Oh, okay. This is so interesting and this is exactly where I feel like things are going. So just to be clear what you shared here, there's basically this tool that comes up with experiments to run, to help grow Claude and all the tools. So it comes up with idea, somebody looks at them, approves, "Cool, let's do these things." Builds it, ships it, tests the results, see how it's doing, and then comes back of, "Here's things that are working." Is that roughly right?
00:38:48 · Amol Avasare
[00:38:48] S1: It's roughly right. And right now, we have human in the loop approving, but the amount of time, I kind of think about scale in this way of just week on week, are things getting better in each of the areas? Are people spending less time on each of the areas? Are the results getting better in each of the areas? And as long as week on week, that's getting better, then you're like, "Okay, this whole initiative is scaling." And so that's roughly right, but you can think about it as, I think a lot of this can be automated where human review is not needed. So we care a lot about our brand, right? So that is something that we do look at right now. We don't want to be shipping something that goes against the brand, but then you can have a skill that contains your brand guidelines and very clear, yet dos and don'ts on brand. And so all of these types of accompaniments, I think are going to get better. The model's going to get better at understanding how to use them. And so over time, I think that the need to human review this really, really decreases significantly.
00:39:50 · Lenny Rachitsky, Amol Avasare
[00:39:50] S0: Yeah, and you could always unship it if it's like, "Oh yeah, that was actually not a great idea." And that's such a good point that we think we need people to do these things forever. And it turns out, "Okay, a skill could do this really well. Here's our brand guidelines, here's our vision, here's our mission, here's our goals, here's what matters to us. Okay, no, let's not ship that thing." So the reason I think this is so interesting, I've just been watching the expansion of AI doing more of the product development process, in this case the growth process of just, okay, it went from helping you write code, to writing all your code, to reviewing your code. Now it feels like, what are the two ends around this? It's kind of going from the middle up. The top of that is coming up with what to do and then there's the alignment stuff's still very hard. And then on the other end, it's reviewing the code and then shipping it and then get distribution. That's a whole other thing I want to talk to you about. So what I'm hearing here, and this is exactly what I thought was going to start happening, is AI is now getting really good at telling us what to do, not just taking our orders and building it. And it feels like the growth version of this is where it starts because it's so much simpler, not that growth is easy, but just like it's data-driven, there's this loop that you talk about. So I think this is such an interesting sign of things to come across just generally, product. [00:41:04] S1: Yeah.
00:41:04 · Amol Avasare, Lenny Rachitsky
[00:41:04] S1: Yeah. [00:41:05] S0: Just putting that out there. Okay. So something that I'm constantly thinking about along these lines is just the future of product PM engineering, how those roles shift over time based on the stuff we're talking about. How are you working together as a triad and where do you see these roles going? What's most going to change, do you think, across these three roles?