Lenny's Podcast: Product | Career | Growth
00:14:55 · Lenny Rachitsky, Anish Acharya
[00:14:55] S0: ... a lot of spreadsheets. [00:14:56] S1: So, the loop version of that should be that every variant gets generated, every variant gets measured. Once you get to stat sig with a high enough P-value, you converge and ship that variant. You then have a long-term holdout and you start working on the next experiment. And then you're going to hit some local maxima, and I think this is really important. The loop will help you climb to the local maxima, but then it plateaus and you need some sort of out of distribution thinking, you need human intuition, you need somebody to actually help you land at the base of the next hill. [00:15:26] S0: Yeah, you have this chart. I don't know, maybe we'll show it over later as we talk about this, which is such an interesting way of thinking about it. This idea that agents will help you hill climb and reach some new plateau, and then you need a human there to think about a bigger idea, kind of unlock it, and then it keeps going and going and there's kind of this agent to human kind of back and forth. [00:15:47] S1: Yes. Yeah. And you know what really illustrates that? If you've ever tried to have an agent come up with a business idea for you, like, "Hey Claude, make me a million dollars, make no mistakes," why doesn't that work? And it's because you sort of need to set it in the right direction, and nothing in the technology has shown us that that is not needed.
00:15:47 · Anish Acharya, Lenny Rachitsky
[00:15:47] S1: Yes. Yeah. And you know what really illustrates that? If you've ever tried to have an agent come up with a business idea for you, like, "Hey Claude, make me a million dollars, make no mistakes," why doesn't that work? And it's because you sort of need to set it in the right direction, and nothing in the technology has shown us that that is not needed. [00:16:06] S0: Yeah, it's interesting. I've been hearing more and more. I just saw a tweet that I think at OpenAI, the go-to-market team now is using Codex more. More of the go-to-market team is using Codex more often than even the engineering team. [00:16:19] S1: Yes. And think of how happy that makes them. What does the go-to-market team want to do? I mean, this is a caricature but I'm going to stand behind it, which is they want to hit the gym, they want to go to steak dinners, and they want to raise the trophy up for being salesperson of the quarter or of the year. So, I actually think that this is a distillation of their job into the thing that they're the best in the world at, that they're the most interested in, and all the administration that goes around doing that core work is now handled for them. So, that's where go-to-market is going, and it's going to be awesome.
00:16:19 · Anish Acharya, Lenny Rachitsky
[00:16:19] S1: Yes. And think of how happy that makes them. What does the go-to-market team want to do? I mean, this is a caricature but I'm going to stand behind it, which is they want to hit the gym, they want to go to steak dinners, and they want to raise the trophy up for being salesperson of the quarter or of the year. So, I actually think that this is a distillation of their job into the thing that they're the best in the world at, that they're the most interested in, and all the administration that goes around doing that core work is now handled for them. So, that's where go-to-market is going, and it's going to be awesome. [00:16:51] S0: This reminds me of a PM friend who had this really funny take that as a PM, you're constantly having to say no to all these ideas that are coming at you and you have to like, "I'll put it on the roadmap, we'll prioritize it." He's like, "Okay, I'm going to flip this. I'm going to say yes to everything. I'm going to build everything and then simulate every idea with Simile," or all these products that are launching where you could simulate how a user will react, and then that'll tell you, "Should we build this?" What a hilarious way to rethink PM.
00:16:51 · Lenny Rachitsky, Anish Acharya
[00:16:51] S0: This reminds me of a PM friend who had this really funny take that as a PM, you're constantly having to say no to all these ideas that are coming at you and you have to like, "I'll put it on the roadmap, we'll prioritize it." He's like, "Okay, I'm going to flip this. I'm going to say yes to everything. I'm going to build everything and then simulate every idea with Simile," or all these products that are launching where you could simulate how a user will react, and then that'll tell you, "Should we build this?" What a hilarious way to rethink PM. [00:17:21] S1: And you know what's so beautiful about that? Actually, there's two things, and I'll tell you maybe the one that's less obvious to me, which is I feel like every PM at every company feels like they're a true zero to one thinker, but they're held back by the heavy hand of management, and executives, and founders, and engineering capacity. And in a world where every story gets told, every product story gets told, every feature gets tried, I think a lot of PMs are going to realize they're actually not that good at zero to one and it's much more fulfilling to work on someone else's good idea than your own bad idea. So, I think even things like that are going to lead to a lot more organizational health than we've had in the past. Not to mention the fact that the idea that ends up winning doesn't have to be the one that's came up with by the person who can sell it best to an executive. It just gets tried, and the best idea wins.
00:18:09 · Lenny Rachitsky
[00:18:09] S0: So coming back to this loops idea, the way I'm thinking about it is how can every function start to think of their function as setting up an agent to be able to just go from input to some kind of impact? And what this makes me think about is something actually Claire tweeted recently, this point that people always used to joke that soft skills are the least valuable and engineering skills the most important valuable because they're so concrete, and turns out that's what AI is the best at, the things that are verifiable and you know what success looks like. And so, the question I think about now is just which skills can you not just turn into a loop because the output is so hard to verify?
00:18:50 · Anish Acharya, Lenny Rachitsky
[00:18:50] S1: Yeah, I think that's right, and I think that those are going to be the rate limiting factors because you can only do one steak dinner a night. I guess you could do a steak lunch, but to some extent there's going to be these rate limiting factors in every system. I think a useful way to think about it is anytime the model's making a mistake or doing something you wouldn't do, what do you know that it doesn't know? And there's a really interesting example I heard from Ale at Kavak. He's probably the most sophisticated thinker on this stuff that I get to hang out with, where he said anytime their agent, they have an agent per customer, they sell used cars online, and when the agent gets stuck it actually calls a human, and the human will coach the agent through. Now, the magic of that is not only does it unblock the agent, but of course the agent then captures all the traces and learns from it. That's a really interesting mental model. It's either a knowledge gap or a data gap that you have to give the agent, and the next time it shouldn't have to call you. [00:19:42] S0: I love that example. Basically, the takeaway here is you need to start thinking about every function as an agent loop, and the job is to figure out where it gets blocked, where it goes wrong, and give it more context, more insight, more direction, basically.