Lenny's Podcast: Product | Career | Growth
00:36:25 · Amol Avasare, Lenny Rachitsky
It's been a lot of copy changes and some very minor UI tweaks. It's delivering results and it's like you can push it, press play with it. And it ultimately prints money where I'd say that the win rate is like, I would expect a senior PM to do better. I would say this is a junior PM, two, three years in. I would say this is the win rate that I would expect from a junior PM, but it's not quite at the senior PM level. Although, I think you look at the exponential, this wasn't available at all a couple of months ago. So it's getting better rapidly. And I think that it's going to change where you'll be able to do this for larger and larger types of experiments. But then you think about the largest types of experiments, I think that I mentioned the four pieces around identifying opportunities, building, testing and shipping. The one I didn't mention there, Lenny, is cross-functional stakeholder management. [00:37:26] S0: There's still a need for human brains. There is. [00:37:31] S1: And I think that that one is going to mean that in my eyes, the work of PMs is not going away actually, anytime soon. And that piece, especially for larger projects. Usually, you don't need to do as much of it for smaller stuff. You can skip it. But for larger stuff, that piece is not going away. [00:37:51] S0: Until the other stakeholders have their own little agents running around.
00:37:51 · Lenny Rachitsky, Amol Avasare
[00:37:51] S0: Until the other stakeholders have their own little agents running around. [00:37:54] S1: I think that's right. I think that would be the point where it changes. It's funny, we had a difficult meeting a couple of weeks ago, and me and our head of design, Joel, we were debriefing afterwards. And he pings me, he's just like, "Amol, we will have AGI and it will still be impossible to get six people in a room to get to align." And I'm like, "Yeah, I think I can say that." [00:38:16] S0: What's the harder alignment problem? Oh, okay. This is so interesting and this is exactly where I feel like things are going. So just to be clear what you shared here, there's basically this tool that comes up with experiments to run, to help grow Claude and all the tools. So it comes up with idea, somebody looks at them, approves, "Cool, let's do these things." Builds it, ships it, tests the results, see how it's doing, and then comes back of, "Here's things that are working." Is that roughly right?
00:38:48 · Amol Avasare
[00:38:48] S1: It's roughly right. And right now, we have human in the loop approving, but the amount of time, I kind of think about scale in this way of just week on week, are things getting better in each of the areas? Are people spending less time on each of the areas? Are the results getting better in each of the areas? And as long as week on week, that's getting better, then you're like, "Okay, this whole initiative is scaling." And so that's roughly right, but you can think about it as, I think a lot of this can be automated where human review is not needed. So we care a lot about our brand, right? So that is something that we do look at right now. We don't want to be shipping something that goes against the brand, but then you can have a skill that contains your brand guidelines and very clear, yet dos and don'ts on brand. And so all of these types of accompaniments, I think are going to get better. The model's going to get better at understanding how to use them. And so over time, I think that the need to human review this really, really decreases significantly.
00:39:50 · Lenny Rachitsky, Amol Avasare
[00:39:50] S0: Yeah, and you could always unship it if it's like, "Oh yeah, that was actually not a great idea." And that's such a good point that we think we need people to do these things forever. And it turns out, "Okay, a skill could do this really well. Here's our brand guidelines, here's our vision, here's our mission, here's our goals, here's what matters to us. Okay, no, let's not ship that thing." So the reason I think this is so interesting, I've just been watching the expansion of AI doing more of the product development process, in this case the growth process of just, okay, it went from helping you write code, to writing all your code, to reviewing your code. Now it feels like, what are the two ends around this? It's kind of going from the middle up. The top of that is coming up with what to do and then there's the alignment stuff's still very hard. And then on the other end, it's reviewing the code and then shipping it and then get distribution. That's a whole other thing I want to talk to you about. So what I'm hearing here, and this is exactly what I thought was going to start happening, is AI is now getting really good at telling us what to do, not just taking our orders and building it. And it feels like the growth version of this is where it starts because it's so much simpler, not that growth is easy, but just like it's data-driven, there's this loop that you talk about. So I think this is such an interesting sign of things to come across just generally, product. [00:41:04] S1: Yeah.
00:41:04 · Amol Avasare, Lenny Rachitsky
[00:41:04] S1: Yeah. [00:41:05] S0: Just putting that out there. Okay. So something that I'm constantly thinking about along these lines is just the future of product PM engineering, how those roles shift over time based on the stuff we're talking about. How are you working together as a triad and where do you see these roles going? What's most going to change, do you think, across these three roles?
00:41:28 · Amol Avasare
[00:41:28] S1: This is something that we talk about and think about frequently, and the picture changes rapidly. So sometimes when things break, execution, like the bottlenecks break, historically in the past it's like, "Okay, now you need to hire more engineers, you need to hire more designers," et cetera. And it's more just a lifecycle thing of, where is the lifecycle of your team and that specific pod to identify where the bottleneck is? But now when things break, you need to still look at it, of, "Okay, is it the actual ratio or is there also underlying technological shifts that are also causing this to break?" And so that's an interesting thing. I think that smaller companies, if I'm at a 15 or 20-person company, I'm the only PM there working with some designers and engineers, I think in the smaller companies, you'll see probably the biggest blend, where the PM will be doing all sorts of ... They'll be designing, shipping, et cetera, and I think you just have extreme bifurcation. You know, at larger organizations, more scaled organizations, I think the jury is still very much out. You speak to people even internally here, and different people and different teams will have different views of, "Okay, how much are these roles coming together versus how much are these roles going to be separate?" I think that it's not to say even the PMs, number of PMs are shipping, and themselves shipping and pushing PRs, et cetera. But if I look at, okay, across what I'm seeing, I think that it's clear that while PMs and designers are getting more leverage from AI, engineering is getting the most leverage right now. I look at tools like Claude Code, and the amount of leverage engineers are getting from them is higher than, I think the leverage that designers and PMs are getting from them today. Now, this is rapidly also changing, but that to me is my view today. And so if you think about, okay, a default team, which is, say five engineers, one designer, one PM. With Claude Code, that five engineers is two to 3xed. And the PMs and designers have also increased, but now they're managing what is effectively a much larger group of engineers. And so even though the headcount and the org structure hasn't changed, you're now just dealing with a situation of maybe 15 to 20 engineers in the old world, one and a half to two PMs, and maybe one and a half to two designers.