Lenny's Podcast: Product | Career | Growth
00:22:04 · Anish Acharya
[00:22:04] S1: Yeah, so Pareto efficiency is just for price performance, for example, the efficient frontier is what's considered the optimal trade-off of a unit of performance for a unit of price. And if you're along that curve, you're always paying the rational amount for the performance. And what's interesting is that frontier models are actually irrationally priced in that, first of all, Mythos is infinite dollars a token, you can't use it. But if you look at even something like Fable 5, for one IQ, conceptually one IQ of extra intelligence you're paying 100X more than Opus 4.8. So it's not rational, but I think there's a lot of jobs in which you have unbounded upside, like drug discovery. And if that one IQ point lets you discover the next statin, it's a trillion dollar outcome. So, it's sort of rational to pay for the highest intelligence in these types of jobs and industries. Now on the other hand, I'm going to pick on maybe legal. There's perhaps only so much upside to be had in legal or finance. And for those jobs, you actually do want to be very Pareto efficient and probably pay for good performance at a good price, not infinitely priced, infinite potential upside performance. So, I think what we're going to see is a split between job functions that demand kind of mid-IQ intelligence, and those will often be open weight, sort of biased with reinforcement learning, things that make the models even cheaper, more performant for a narrow job, along with incredibly "expensive but performant" frontier tokens for sales, support, research, engineering. So, I think you're going to end up having both architectures, and we can touch on this in a little bit but I do think there are these sort of comparative advantages amongst model families, and then also amongst, of course, individual models that we're seeing more and more of. It's not going to be one or the other.
00:24:01 · Lenny Rachitsky, Anish Acharya
[00:24:01] S0: Such an interesting insight. So, just to make sure I understand what you're describing here, you're thinking there's going to be this split between within an org of function and model, where specific functions that have a lot more upside and potential leverage go for the frontier models, and you described these kind of product sales, engineering research roles. And then, for other functions you don't need Mythos, you don't need Astra, I think is that the latest one that's coming? [00:24:29] S1: Yes, Astra. Yeah, yeah. [00:24:30] S0: And so it's both, you're saying the AI doesn't need to be the frontier model and the people don't have to be the smartest people in the world to do really well in that world. [00:24:37] S1: Yeah, and I don't want to be diminutive. There's extraordinary people in those job functions. I just think that they're bounded upside problems that they work on. You can only kind of close the books correctly, you can't close the books 100X better. So yeah, that's exactly what I'm saying. [00:24:53] S0: I wonder if it's connected back to that discussion we had earlier about it's verifiable. If it's a lot more verifiable, you don't need the frontier model versus, I don't know, the potential upside.
00:24:53 · Lenny Rachitsky, Anish Acharya
[00:24:53] S0: I wonder if it's connected back to that discussion we had earlier about it's verifiable. If it's a lot more verifiable, you don't need the frontier model versus, I don't know, the potential upside. [00:25:04] S1: I'm not sure. I mean, I think for a ... Verifiability is a good question because I think for a drug development company, you have infinite upside, it is verifiable, but closing the books is also a verifiable problem, which has limited upside. I think the question is really how much upside is there and how hard is it to calculate it? Here's a nuance. Customer support, a customer may call in and report a bug. That bug may actually be the first breadcrumb to a thing that changes our entire organization. And if the CEO was on that call or the smartest person, they could follow the trail. But if we actually have this "mid-IQ generalist," then not, that actually makes the case for as a basket of problems always use frontier intelligence. And I can of course make the case for the mid-IQ basket as well, which is simply that we've crossed an intelligence threshold for almost every economically useful problem, and anything beyond that threshold is simply waste. [00:25:57] S0: Yeah. And I think the point people forget is that the models that are not the frontier models, that was what the frontier was, I don't know, six months ago, and we were so impressed and we loved it. It was like, "Holy shit, it can do all this." And now just because there's something better, we don't give those models as much credit.
00:25:57 · Lenny Rachitsky, Anish Acharya
[00:25:57] S0: Yeah. And I think the point people forget is that the models that are not the frontier models, that was what the frontier was, I don't know, six months ago, and we were so impressed and we loved it. It was like, "Holy shit, it can do all this." And now just because there's something better, we don't give those models as much credit. [00:26:12] S1: You're right, because it's sort of like, "This thing is so crazy. This is AGI." And then, a day later it's like the old thing that you throw in the dustbin. [00:26:22] S0: So along those lines, I hear people jokingly call you a model sommelier. Tell us with your sommelier credentials, what's the current state of the model art? What is each model great at? What are model's terrible at? [00:26:38] S1: Yes. Well, as you know, the secret of every sommelier is 10,000 hours or maybe 10,000 bottles. So, I think that the secret of being a model sommelier is just using them all. [00:26:47] S0: Drinking them all.
00:26:47 · Lenny Rachitsky, Anish Acharya
[00:26:47] S0: Drinking them all. [00:26:48] S1: I push myself really hard to ship something with every new model that comes out, and I think you learn so much. I think for people who believe the models are commodities or totally fungible, you just haven't actually used the models. And for example, in the last few weeks I've been obsessed with Qwen 3.8-Max. Qwen is an awesome model. It's very good at long horizon tasks, and it's actually really creative. It's a great storyteller. So, I've been using it to create these impossible documentaries, mostly of planets of the Star Wars universe. I did Tatooine, which turned out great. I did Bespin last night, and it can just work for four or five hours. It tells a great story, it generates all the video using the MiniMax model via Fal, generates audio via Eleven. It actually directs the movie, the five-minute movie, and then it cuts all of the clips in. It overlays it, it watches it. It's just extraordinary. It's got a totally different shape than GLM-5.2. GLM-5.3 just came out, which I used for a bunch of product work. It doesn't have a vision component, and it's sort of like this neurotic PhD you put in the corner, and they both have the role. This is a little bit of these tension of models. It's not that one is ahead of another, one is more intelligent, it's rather one has a mind that's shaped in one direction, perhaps creativity and openness for Qwen, and others that are shaped in other directions like neuroticism and precision like GLM-5.3. So, I just make something with every model and that's how I build my intuition.
00:28:22 · Lenny Rachitsky, Anish Acharya
[00:28:22] S0: How important is this habit, do you think, for people? Because I hear a lot that it's really important to be using these models. And a secondary question is how do you come up with what to do with these models? Because a lot of people want to try these things. They're like, "Okay, what do I do?" [00:28:37] S1: I know. [00:28:37] S0: Any suggestions? [00:28:38] S1: Well, I think almost all of us have got ... The most insufferable thing for a long time was your app idea friend. Every time you went to have a beer they're like, "Let me tell you my app idea." You're like, "Oh, here we go again." We have got to be the app idea guys now, and all the silly ideas, actually, especially the silly ideas, because those are the ones that often have the most alpha, are the ones that we should all be building. So, I've probably got two dozen apps that I've built, I've got one or two large apps that I iterate on. And I think that if you don't have a chassis with which to use the models, it's just really hard to come up with an idea from scratch every time. So, I'd say work on something. It's actually better if it's not important with a capital I, and then keep finding new ways to invest in it and add to it with a model as a kind of tool rather than as a goal. Does that make sense? [00:29:27] S0: Yeah. Maybe even zooming out, I've been thinking more and more one of the most important habits to build right now is to, whenever you're about to do something, ask yourself, "How can AI do this for me?" [00:29:38] S1: Totally.