Lenny's Podcast: Product | Career | Growth
00:13:42 · Anish Acharya, Lenny Rachitsky
We've seen one thing, Lenny, it's that the ability for models to do new thinking out of distribution thinking is still really limited. And I don't actually take the point that some of the new thinking in math is actually representative of new thinking in domains like business. So, you're still going to need a person to say, "Hey, here's the thing I think we should make," and have them be right about it. [00:14:05] S0: Let me just kind of make sure this point is really clear, because it's so interesting. What you're saying here is engineering and building more and more is becoming this loop of input, feedback, support ticket, whatever, input of just what to build. And then AI more and more is taking that, deciding, "Here's a PR. Here, is this ready?" And then shipping it. And you're saying that you expect that to spread to say go to market, legal, growth, support. So, maybe describe what that loop looks like or may look like within a company. [00:14:38] S1: I mean, a great example is a growth team. You worked on the growth team at Airbnb, right? [00:14:41] S0: Yeah, yeah. [00:14:41] S1: Supply growth. [00:14:42] S0: Yeah, that's right. [00:14:42] S1: Awesome. [00:14:42] S0: That's right. [00:14:45] S1: I don't know how you ran your team, but I'm guessing it was something like you got everyone together, you built a list of possible experiments, you prioritized them, you built them, you ship them, you measure them- [00:14:53] S0: Brainstorming. Yeah. A lot of brainstorming- [00:14:54] S1: Correct, right? [00:14:55] S0: ... a lot of spreadsheets.
00:14:55 · Lenny Rachitsky, Anish Acharya
[00:14:55] S0: ... a lot of spreadsheets. [00:14:56] S1: So, the loop version of that should be that every variant gets generated, every variant gets measured. Once you get to stat sig with a high enough P-value, you converge and ship that variant. You then have a long-term holdout and you start working on the next experiment. And then you're going to hit some local maxima, and I think this is really important. The loop will help you climb to the local maxima, but then it plateaus and you need some sort of out of distribution thinking, you need human intuition, you need somebody to actually help you land at the base of the next hill. [00:15:26] S0: Yeah, you have this chart. I don't know, maybe we'll show it over later as we talk about this, which is such an interesting way of thinking about it. This idea that agents will help you hill climb and reach some new plateau, and then you need a human there to think about a bigger idea, kind of unlock it, and then it keeps going and going and there's kind of this agent to human kind of back and forth. [00:15:47] S1: Yes. Yeah. And you know what really illustrates that? If you've ever tried to have an agent come up with a business idea for you, like, "Hey Claude, make me a million dollars, make no mistakes," why doesn't that work? And it's because you sort of need to set it in the right direction, and nothing in the technology has shown us that that is not needed.
00:15:47 · Anish Acharya, Lenny Rachitsky
[00:15:47] S1: Yes. Yeah. And you know what really illustrates that? If you've ever tried to have an agent come up with a business idea for you, like, "Hey Claude, make me a million dollars, make no mistakes," why doesn't that work? And it's because you sort of need to set it in the right direction, and nothing in the technology has shown us that that is not needed. [00:16:06] S0: Yeah, it's interesting. I've been hearing more and more. I just saw a tweet that I think at OpenAI, the go-to-market team now is using Codex more. More of the go-to-market team is using Codex more often than even the engineering team. [00:16:19] S1: Yes. And think of how happy that makes them. What does the go-to-market team want to do? I mean, this is a caricature but I'm going to stand behind it, which is they want to hit the gym, they want to go to steak dinners, and they want to raise the trophy up for being salesperson of the quarter or of the year. So, I actually think that this is a distillation of their job into the thing that they're the best in the world at, that they're the most interested in, and all the administration that goes around doing that core work is now handled for them. So, that's where go-to-market is going, and it's going to be awesome.
00:16:19 · Anish Acharya, Lenny Rachitsky
[00:16:19] S1: Yes. And think of how happy that makes them. What does the go-to-market team want to do? I mean, this is a caricature but I'm going to stand behind it, which is they want to hit the gym, they want to go to steak dinners, and they want to raise the trophy up for being salesperson of the quarter or of the year. So, I actually think that this is a distillation of their job into the thing that they're the best in the world at, that they're the most interested in, and all the administration that goes around doing that core work is now handled for them. So, that's where go-to-market is going, and it's going to be awesome. [00:16:51] S0: This reminds me of a PM friend who had this really funny take that as a PM, you're constantly having to say no to all these ideas that are coming at you and you have to like, "I'll put it on the roadmap, we'll prioritize it." He's like, "Okay, I'm going to flip this. I'm going to say yes to everything. I'm going to build everything and then simulate every idea with Simile," or all these products that are launching where you could simulate how a user will react, and then that'll tell you, "Should we build this?" What a hilarious way to rethink PM.
00:16:51 · Lenny Rachitsky, Anish Acharya
[00:16:51] S0: This reminds me of a PM friend who had this really funny take that as a PM, you're constantly having to say no to all these ideas that are coming at you and you have to like, "I'll put it on the roadmap, we'll prioritize it." He's like, "Okay, I'm going to flip this. I'm going to say yes to everything. I'm going to build everything and then simulate every idea with Simile," or all these products that are launching where you could simulate how a user will react, and then that'll tell you, "Should we build this?" What a hilarious way to rethink PM. [00:17:21] S1: And you know what's so beautiful about that? Actually, there's two things, and I'll tell you maybe the one that's less obvious to me, which is I feel like every PM at every company feels like they're a true zero to one thinker, but they're held back by the heavy hand of management, and executives, and founders, and engineering capacity. And in a world where every story gets told, every product story gets told, every feature gets tried, I think a lot of PMs are going to realize they're actually not that good at zero to one and it's much more fulfilling to work on someone else's good idea than your own bad idea. So, I think even things like that are going to lead to a lot more organizational health than we've had in the past. Not to mention the fact that the idea that ends up winning doesn't have to be the one that's came up with by the person who can sell it best to an executive. It just gets tried, and the best idea wins.
00:18:09 · Lenny Rachitsky
[00:18:09] S0: So coming back to this loops idea, the way I'm thinking about it is how can every function start to think of their function as setting up an agent to be able to just go from input to some kind of impact? And what this makes me think about is something actually Claire tweeted recently, this point that people always used to joke that soft skills are the least valuable and engineering skills the most important valuable because they're so concrete, and turns out that's what AI is the best at, the things that are verifiable and you know what success looks like. And so, the question I think about now is just which skills can you not just turn into a loop because the output is so hard to verify?