Invest Like the Best with Patrick O'Shaughnessy
· Dylan Patel
[00:00:00] S1: There's this concept of software-only singularity, which is that the world has AI singularity, but only in software. And now what about the rest of the world? The vast majority of the world is physical and you can see the world orient around hardware, not software. That's actually why I think software-only singularity is just a blip and not like a, we do get everything else. Because software is super easy, what makes robots really hard? It's programming microcontrollers and actuators and controlling all this stuff is very difficult. And right now, the interesting thing about models, AI models, is they're actually really inefficient in learning. It's just, we're able to give them so much data that they're able to learn and pass us in certain ways. Currently, the robot models – VLAs, Vision Language Action Models, which is very popular right now, is probably not going to be the thing that ultimately scales beyond. They are inefficient in data and we can't scale the data for them fast enough. There is going to be some way to large-scale pre-trained robot models where, just like humans, see all this data throughout their lives. And what's interesting is humans, the reason why we're so good is we're sample efficient. One example, two example, we're good. So applying that to robotics, so once you have this software-only singularity implementation is super cheap, anyone can start to build these models that now robots are actually useful. And so I think in the next six to 18 months, we'll start seeing real breakthroughs in robotics that enable few-shot learning, i.e. there's a pre-trained robot model and now there's a robot that you have hired or bought or whatever. You show it a few examples and it's able to do it. Right now, there's a lot of companies doing robots for advertisement or robots for simple stuff like that, but it'll be like, oh, folding clothes, sure, sure, sure. No, but it's going to get really niche. Robots just for cleaning chalkboards, and it's a rental service. Or it'll be a model package that you download onto your standard robot that then does that. In any ways, there'll be a huge explosion in physical good acceleration and deflationary effects there, but that's ultimately going to keep token demand going crazy. I don't think token demand slows down personally.
· Patrick O'Shaughnessy
[00:00:00] S0: Did you learn anything else about the world based on Mythos' results and how it was built? It's my way of asking if you break down the components of the scaling laws.
· Dylan Patel
[00:00:00] S1: Mythos is a materially larger model than prior models. And so yes, it is a much larger model. What chip it's trained on is not really relevant. It's the scale. And obviously to a hundred thousand black wells is equivalent to hundreds of thousands of prior generation chips. TPUs and Trainium have their different release cadence. So it's not exactly like mirrored one-to-one, but ultimately, yes, Mythos is a significantly larger model. It's proof that the scaling laws still work. Everything about it shows the trend line continues of models – more compute into model makes model better. And along the whole way, it's not just more compute into model makes model better. Along the whole way, we're also getting these compute efficiency wins, which are as all this research compute that the labs are spending is actually turning into, if I want X capability to your model every six months, that cost or every two months, that cost is dramatically decreasing. But then if I scale it up massively, I get a humongous capability jump as well. And so yes, it's proof that this is still happening. Google and Anthropic are not heavy, heavy users of GPUs on the training side, but OpenAI, they'll start having their new class of models. I think they're taking a more sensible principled approach to scaling in small steps. Anthropic really went for a huge jump. We'll see better and better models throughout the year. And the release cadence is only going to get faster.
· Patrick O'Shaughnessy
[00:00:00] S0: We've gone a long way in the conversation with saying almost nothing about OpenAI, which would have been so strange.
· Dylan Patel
[00:00:00] S1: So this is the interesting thing. Everyone's like, okay, so Anthropic's just won, right? They had Mythos in February. They never even released it because they didn't feel the need to. They're already sold out. Their revenue is already adding $10 billion a month. And then you've got Opus 4.7 today, all before OpenAI's alleged Spud release, which media such as The Information and others have posted about. So clearly Anthropic is in the lead and OpenAI is cooked. What's interesting is because Anthropic has such bounds on compute, and they can only grow it so fast and to the point of – Dario used to gloat about how OpenAI was being too aggressive on compute and Anthropic was more sensible in their scaling. And now Anthropic is like, fuck, I wish we had a lot more compute. OpenAI is able to pay the bills perfectly fine. In fact, they've raised a ton of money to get incremental compute in addition to the irresponsible levels of compute that they were buying from Oracle and CoreWeave and SoftBank and all these people in Microsoft, such as Trainium. Now they're getting Trainium as well from Amazon. They've done this insane thing on compute. They also know they need more. But what's interesting is if you were to say Opus 4.6, let's ignore models getting better over time. Let's just take diffusion of this technology. You and I may jump on the model immediately day one, but other businesses take time and it takes time for people to learn. And the spark of, oh shit, quad psychosis moment doesn't hit everyone at the same time. And so by the end of the year, let's say a 4.6 Opus tier model, the economy would spend $100 billion on. I don't think that's unreasonable. It's spending $40 billion right now.
· Patrick O'Shaughnessy, Dylan Patel
[00:00:00] S0: That's like a linear extrapolation. [00:00:00] S1: It's a linear extrapolation, not an exponential. To get the exponential, you need the better models. Anthropic won't have enough compute to do that. And presumably OpenAI and Google will hit that tier soon enough. Whoever hits that tier next – sure, Anthropic may get to charge 70 plus percent gross margins. But if OpenAI hits it next, they charge 50% gross margins. They still get all of this incremental demand. And probably they also won't have enough compute to serve all the users. Sure, maybe Mythos is a model where if the world had enough compute, it'd be $500 billion of revenue or something crazy. There is such demand for these tokens and such limitations on compute. We see this with H100 prices skyrocketing and all these other things. The useful life of these GPUs continue to extend and extend. It's pretty clear even the tier 2 lab is going to be sold out of tokens, let alone the tier 1 lab. The tier 1 lab will have better margins, but the tier 2 lab will be sold out. And probably the tier 3 lab will also be close to sold out. The economic value that the best model can deliver is growing faster than our ability to actually serve those tokens to people via the infrastructure. And so this gap will continue to grow and the model labs will continue to have expanding margins. Until people in the hardware supply chain, infrastructure supply chain are like – wait, no, why don't I just jack up my margins?