Lenny's Podcast: Product | Career | Growth
00:18:50 · Anish Acharya, Lenny Rachitsky
[00:18:50] S1: Yeah, I think that's right, and I think that those are going to be the rate limiting factors because you can only do one steak dinner a night. I guess you could do a steak lunch, but to some extent there's going to be these rate limiting factors in every system. I think a useful way to think about it is anytime the model's making a mistake or doing something you wouldn't do, what do you know that it doesn't know? And there's a really interesting example I heard from Ale at Kavak. He's probably the most sophisticated thinker on this stuff that I get to hang out with, where he said anytime their agent, they have an agent per customer, they sell used cars online, and when the agent gets stuck it actually calls a human, and the human will coach the agent through. Now, the magic of that is not only does it unblock the agent, but of course the agent then captures all the traces and learns from it. That's a really interesting mental model. It's either a knowledge gap or a data gap that you have to give the agent, and the next time it shouldn't have to call you. [00:19:42] S0: I love that example. Basically, the takeaway here is you need to start thinking about every function as an agent loop, and the job is to figure out where it gets blocked, where it goes wrong, and give it more context, more insight, more direction, basically.
00:19:42 · Lenny Rachitsky, Anish Acharya
[00:19:42] S0: I love that example. Basically, the takeaway here is you need to start thinking about every function as an agent loop, and the job is to figure out where it gets blocked, where it goes wrong, and give it more context, more insight, more direction, basically. [00:19:58] S1: That's right. Yeah, that's right. And then hopefully a lot of your day-to-day work, when you come up with a new idea that works, the kind of implications of that idea, if it's a product, there's marketing work, there's sales work, there's product marketing, there's communication, there's legal, all of that should be largely handled for you. And your job is to go take a hike and dream the dreams and come up with the next hill to climb. [00:20:20] S0: And this begs the question a bit of just what will separate the companies that win in this world where AI is kind of doing a lot of this? I imagine part of the answer is the human in this local maxima coming in with a better idea. Is there anything else that you think becomes a differentiator in this world where AI is doing so much of the work that humans are currently doing?
00:20:20 · Lenny Rachitsky, Anish Acharya
[00:20:20] S0: And this begs the question a bit of just what will separate the companies that win in this world where AI is kind of doing a lot of this? I imagine part of the answer is the human in this local maxima coming in with a better idea. Is there anything else that you think becomes a differentiator in this world where AI is doing so much of the work that humans are currently doing? [00:20:39] S1: I mean, I think one thing that's under discussed is that it's unclear that the sort of competitive equilibria that exists in a lot of industries will really change. So, let's say Pizza Hut, and Domino's, and Papa John's, and Round Table all get a data center of PhDs, and they're all going to either adopt it or have a CEO change and adopt it. So, it'll be a rocky period and there'll be some relative shuffling, but I think that they're all going to embrace the new technology because most companies do. I don't know that one of them is going to have 99% of the market. So, I think what's under discussed is that yes, in the near term I think there'll be winners and losers based on adoption of the technology and how ambitiously you adopt it. But I do think there's a lot of industries that will sort of maintain their current competitive dynamics because they're not intelligence bound. I think the most useful question to ask yourself as a founder CEO is just, "Hey, if we assume these things are infinitely intelligent and astonishingly cheap, how would we reorganize the company?" Because that's where we're going?
00:20:39 · Anish Acharya, Lenny Rachitsky
[00:20:39] S1: I mean, I think one thing that's under discussed is that it's unclear that the sort of competitive equilibria that exists in a lot of industries will really change. So, let's say Pizza Hut, and Domino's, and Papa John's, and Round Table all get a data center of PhDs, and they're all going to either adopt it or have a CEO change and adopt it. So, it'll be a rocky period and there'll be some relative shuffling, but I think that they're all going to embrace the new technology because most companies do. I don't know that one of them is going to have 99% of the market. So, I think what's under discussed is that yes, in the near term I think there'll be winners and losers based on adoption of the technology and how ambitiously you adopt it. But I do think there's a lot of industries that will sort of maintain their current competitive dynamics because they're not intelligence bound. I think the most useful question to ask yourself as a founder CEO is just, "Hey, if we assume these things are infinitely intelligent and astonishingly cheap, how would we reorganize the company?" Because that's where we're going? [00:21:42] S0: Kind of along those lines, you have this interesting take about this kind of split that might be coming within companies between these generalists and specialists, and how AI plays into that. Talk about that. [00:21:52] S1: Yeah. Well, I think that there's so many interesting things that this touches on. One is the question of open weight versus frontier. So I'm sure, are you familiar with Pareto efficiency? I'm sure you are.
00:21:52 · Anish Acharya, Lenny Rachitsky
[00:21:52] S1: Yeah. Well, I think that there's so many interesting things that this touches on. One is the question of open weight versus frontier. So I'm sure, are you familiar with Pareto efficiency? I'm sure you are. [00:22:03] S0: Please explain.
00:22:04 · Anish Acharya
[00:22:04] S1: Yeah, so Pareto efficiency is just for price performance, for example, the efficient frontier is what's considered the optimal trade-off of a unit of performance for a unit of price. And if you're along that curve, you're always paying the rational amount for the performance. And what's interesting is that frontier models are actually irrationally priced in that, first of all, Mythos is infinite dollars a token, you can't use it. But if you look at even something like Fable 5, for one IQ, conceptually one IQ of extra intelligence you're paying 100X more than Opus 4.8. So it's not rational, but I think there's a lot of jobs in which you have unbounded upside, like drug discovery. And if that one IQ point lets you discover the next statin, it's a trillion dollar outcome. So, it's sort of rational to pay for the highest intelligence in these types of jobs and industries. Now on the other hand, I'm going to pick on maybe legal. There's perhaps only so much upside to be had in legal or finance. And for those jobs, you actually do want to be very Pareto efficient and probably pay for good performance at a good price, not infinitely priced, infinite potential upside performance. So, I think what we're going to see is a split between job functions that demand kind of mid-IQ intelligence, and those will often be open weight, sort of biased with reinforcement learning, things that make the models even cheaper, more performant for a narrow job, along with incredibly "expensive but performant" frontier tokens for sales, support, research, engineering. So, I think you're going to end up having both architectures, and we can touch on this in a little bit but I do think there are these sort of comparative advantages amongst model families, and then also amongst, of course, individual models that we're seeing more and more of. It's not going to be one or the other.