Lenny's Podcast: Product | Career | Growth
00:11:36 · Anish Acharya
[00:11:36] S1: Yeah, yeah. Well, I think the broad concept and the loops concept kind of gets teased a little on X because at sometimes it feels like maybe we're big braining it. I know there was a big meme around graphs, like the next stage of loops is graphs, but let me make the kind of steel man for it, which is we had prompts and then we invented agents. Agents of course are just models in a loop with tools and memory and skill files. And then we had sort of loops, which are sets of agents that are doing tasks. If you look at a lot of work in coding, coding is such a great domain because you have the best models and you have the most sort of technically apt customer, plus you have these established loops, like bug fix, or a bug report comes in, repro gets generated, bug fix gets created, bug fix gets reviewed. If high risk, then humans should confirm that it's okay to ship to prod. And if low risk, it just gets shipped. And maybe you even email the customer and say, "Hey, we fixed the bug you reported." That happens in five minutes. There are many loops like that in engineering, everything from bug fixes to customer feedback, to sales demos, to new feature development. So, coding sets itself up well. So, you have a coding loop and the kind of change it makes is to the code base. My question is what are the business loops? So if you're the GM of a business, you're looking across many job functions and you've got loops running in coding, and marketing, and sales, and support, and legal, the output of all of those loops is something that is itself a loop that you should be able to optimize for. And I think the strong form of this is that it sends a message to the CEO saying, "Hey, we need to actually make a change to one of the physical aspects of the business, or to our business model, or to our strategy." So, I think we're going to see this sort of cascading set of everything from a loop per person, loop per job function, loop across entire business units to loops that can run large parts of the company. With that said, I think humans are a critical ingredient. I just don't think that most work in the organization can be done fully autonomously. When you think of what a human will do in this AI native company, sales, support, strategy and exceptions. And all those things are super critical.
00:13:42 · Anish Acharya, Lenny Rachitsky
We've seen one thing, Lenny, it's that the ability for models to do new thinking out of distribution thinking is still really limited. And I don't actually take the point that some of the new thinking in math is actually representative of new thinking in domains like business. So, you're still going to need a person to say, "Hey, here's the thing I think we should make," and have them be right about it. [00:14:05] S0: Let me just kind of make sure this point is really clear, because it's so interesting. What you're saying here is engineering and building more and more is becoming this loop of input, feedback, support ticket, whatever, input of just what to build. And then AI more and more is taking that, deciding, "Here's a PR. Here, is this ready?" And then shipping it. And you're saying that you expect that to spread to say go to market, legal, growth, support. So, maybe describe what that loop looks like or may look like within a company. [00:14:38] S1: I mean, a great example is a growth team. You worked on the growth team at Airbnb, right? [00:14:41] S0: Yeah, yeah. [00:14:41] S1: Supply growth. [00:14:42] S0: Yeah, that's right. [00:14:42] S1: Awesome. [00:14:42] S0: That's right. [00:14:45] S1: I don't know how you ran your team, but I'm guessing it was something like you got everyone together, you built a list of possible experiments, you prioritized them, you built them, you ship them, you measure them- [00:14:53] S0: Brainstorming. Yeah. A lot of brainstorming- [00:14:54] S1: Correct, right? [00:14:55] S0: ... a lot of spreadsheets.
00:14:55 · Lenny Rachitsky, Anish Acharya
[00:14:55] S0: ... a lot of spreadsheets. [00:14:56] S1: So, the loop version of that should be that every variant gets generated, every variant gets measured. Once you get to stat sig with a high enough P-value, you converge and ship that variant. You then have a long-term holdout and you start working on the next experiment. And then you're going to hit some local maxima, and I think this is really important. The loop will help you climb to the local maxima, but then it plateaus and you need some sort of out of distribution thinking, you need human intuition, you need somebody to actually help you land at the base of the next hill. [00:15:26] S0: Yeah, you have this chart. I don't know, maybe we'll show it over later as we talk about this, which is such an interesting way of thinking about it. This idea that agents will help you hill climb and reach some new plateau, and then you need a human there to think about a bigger idea, kind of unlock it, and then it keeps going and going and there's kind of this agent to human kind of back and forth. [00:15:47] S1: Yes. Yeah. And you know what really illustrates that? If you've ever tried to have an agent come up with a business idea for you, like, "Hey Claude, make me a million dollars, make no mistakes," why doesn't that work? And it's because you sort of need to set it in the right direction, and nothing in the technology has shown us that that is not needed.
00:15:47 · Anish Acharya, Lenny Rachitsky
[00:15:47] S1: Yes. Yeah. And you know what really illustrates that? If you've ever tried to have an agent come up with a business idea for you, like, "Hey Claude, make me a million dollars, make no mistakes," why doesn't that work? And it's because you sort of need to set it in the right direction, and nothing in the technology has shown us that that is not needed. [00:16:06] S0: Yeah, it's interesting. I've been hearing more and more. I just saw a tweet that I think at OpenAI, the go-to-market team now is using Codex more. More of the go-to-market team is using Codex more often than even the engineering team. [00:16:19] S1: Yes. And think of how happy that makes them. What does the go-to-market team want to do? I mean, this is a caricature but I'm going to stand behind it, which is they want to hit the gym, they want to go to steak dinners, and they want to raise the trophy up for being salesperson of the quarter or of the year. So, I actually think that this is a distillation of their job into the thing that they're the best in the world at, that they're the most interested in, and all the administration that goes around doing that core work is now handled for them. So, that's where go-to-market is going, and it's going to be awesome.
00:16:19 · Anish Acharya, Lenny Rachitsky
[00:16:19] S1: Yes. And think of how happy that makes them. What does the go-to-market team want to do? I mean, this is a caricature but I'm going to stand behind it, which is they want to hit the gym, they want to go to steak dinners, and they want to raise the trophy up for being salesperson of the quarter or of the year. So, I actually think that this is a distillation of their job into the thing that they're the best in the world at, that they're the most interested in, and all the administration that goes around doing that core work is now handled for them. So, that's where go-to-market is going, and it's going to be awesome. [00:16:51] S0: This reminds me of a PM friend who had this really funny take that as a PM, you're constantly having to say no to all these ideas that are coming at you and you have to like, "I'll put it on the roadmap, we'll prioritize it." He's like, "Okay, I'm going to flip this. I'm going to say yes to everything. I'm going to build everything and then simulate every idea with Simile," or all these products that are launching where you could simulate how a user will react, and then that'll tell you, "Should we build this?" What a hilarious way to rethink PM.
00:16:51 · Lenny Rachitsky, Anish Acharya
[00:16:51] S0: This reminds me of a PM friend who had this really funny take that as a PM, you're constantly having to say no to all these ideas that are coming at you and you have to like, "I'll put it on the roadmap, we'll prioritize it." He's like, "Okay, I'm going to flip this. I'm going to say yes to everything. I'm going to build everything and then simulate every idea with Simile," or all these products that are launching where you could simulate how a user will react, and then that'll tell you, "Should we build this?" What a hilarious way to rethink PM. [00:17:21] S1: And you know what's so beautiful about that? Actually, there's two things, and I'll tell you maybe the one that's less obvious to me, which is I feel like every PM at every company feels like they're a true zero to one thinker, but they're held back by the heavy hand of management, and executives, and founders, and engineering capacity. And in a world where every story gets told, every product story gets told, every feature gets tried, I think a lot of PMs are going to realize they're actually not that good at zero to one and it's much more fulfilling to work on someone else's good idea than your own bad idea. So, I think even things like that are going to lead to a lot more organizational health than we've had in the past. Not to mention the fact that the idea that ends up winning doesn't have to be the one that's came up with by the person who can sell it best to an executive. It just gets tried, and the best idea wins.