Invest Like the Best with Patrick O'Shaughnessy
· Dylan Patel, Patrick O'Shaughnessy
What used to matter a lot was execution was very, very fucking difficult and ideas were cheap. Now ideas are cheap and plentiful, but execution is very easy. So really only the good ideas are the ones that can justify the spend on super cheap implementation. [00:00:00] S0: So are you actually scared or does it just introduce some uncertainty that's hard to grapple with?
· Dylan Patel
[00:00:00] S1: Uncertainty is there, but I do think that causes some fear in terms of how does society reform itself? How does one exist in a world where actually your ability to implement something is not actually that important? Your ability to choose the correct idea for AI to implement and then your ability to sell that idea or sell what the AI has implemented is what matters. Your ability to garner capital towards that is what matters. And going back to the point of it's very important to have the newest model always, who's going to have access to the newest model? Anthropic's project, I know it's not called Earwig, but I troll Anthropic people by calling it Earwig. Glasswing. Anthropic Earwig – where they only released Mythos to certain companies for cyber, that's just going to be something that continues. Models will have less broad and less broad deployment. I know OpenAI and Anthropic and all these people are like, we want to have great AI for everyone. AI is very fucking expensive. Who's going to pay for the trillion dollars of infrastructure? People who have money and can build useful things with AI. And then you don't want people to distill your model so you don't release them broadly. You release them to fewer and fewer set of customers. Those customers are also now wrestling over the tokens unless Anthropic jacks them. They could double their pricing on Opus and I would continue to pay and most users would continue to pay. I bet that wouldn't solve their humongous capacity problem that they have. So then the question becomes, where does this cycle end where token usage and therefore the benefits of those tokens, the additional value generated on top of those tokens, aggregates among fewer and fewer and fewer companies. I don't have Mythos. You know who has Mythos? Top freaking banks. Now they're only using it for cybersecurity, but at some point I can envision a world where – hey, maybe I, because I have an enterprise Anthropic contract and Anthropic people kind of like me, they're willing to give us slightly earlier access or slightly higher rate limits or something for a model. I hope that's what happens. And then my competitor, whoever that is, doesn't have that and I'm able to fucking crush them. There are people like Ken Griffin of Citadel, who's super well connected and super rich.
· Dylan Patel, Patrick O'Shaughnessy
He goes and signs a deal with OpenAI Anthropic that's like, yeah, I'm going to get access to your models and I'll buy the first 10 billion dollars worth of tokens each year. So whenever you release the model, I'll spend the 10 billion tokens and then everyone else can get the model after that. And it's like, okay, well now what does that do? Now he's going to crush everyone in the markets. That's just an example. It could be any number of things. It could be cyber like Anthropic is worried about. Oh, now I can hack people. It can be information services business like myself where I crush someone else. I think it's such a broad base. We don't know what these models can do. Anthropic doesn't know what these models can do. No one knows what these models can do. It's up to the end-user to figure out where they can leverage the tokens to see what they can build and imagine, which is tremendously productive and uplifting for humanity. But then what happens to the concentration of resources and usage of it? [00:00:00] S0: Presumably right now, robotics or robots consume relatively zero tokens versus everything else. What's your view of that? If that's like a second demand curve that could start to ratchet. There's a new startup every single day within a mile of here trying to build something interesting in robotics.
· Dylan Patel
[00:00:00] S1: There's this concept of software-only singularity, which is that the world has AI singularity, but only in software. And now what about the rest of the world? The vast majority of the world is physical and you can see the world orient around hardware, not software. That's actually why I think software-only singularity is just a blip and not like a, we do get everything else. Because software is super easy, what makes robots really hard? It's programming microcontrollers and actuators and controlling all this stuff is very difficult. And right now, the interesting thing about models, AI models, is they're actually really inefficient in learning. It's just, we're able to give them so much data that they're able to learn and pass us in certain ways. Currently, the robot models – VLAs, Vision Language Action Models, which is very popular right now, is probably not going to be the thing that ultimately scales beyond. They are inefficient in data and we can't scale the data for them fast enough. There is going to be some way to large-scale pre-trained robot models where, just like humans, see all this data throughout their lives. And what's interesting is humans, the reason why we're so good is we're sample efficient. One example, two example, we're good. So applying that to robotics, so once you have this software-only singularity implementation is super cheap, anyone can start to build these models that now robots are actually useful. And so I think in the next six to 18 months, we'll start seeing real breakthroughs in robotics that enable few-shot learning, i.e. there's a pre-trained robot model and now there's a robot that you have hired or bought or whatever. You show it a few examples and it's able to do it. Right now, there's a lot of companies doing robots for advertisement or robots for simple stuff like that, but it'll be like, oh, folding clothes, sure, sure, sure. No, but it's going to get really niche. Robots just for cleaning chalkboards, and it's a rental service. Or it'll be a model package that you download onto your standard robot that then does that. In any ways, there'll be a huge explosion in physical good acceleration and deflationary effects there, but that's ultimately going to keep token demand going crazy. I don't think token demand slows down personally.
· Patrick O'Shaughnessy
[00:00:00] S0: Did you learn anything else about the world based on Mythos' results and how it was built? It's my way of asking if you break down the components of the scaling laws.
· Dylan Patel
[00:00:00] S1: Mythos is a materially larger model than prior models. And so yes, it is a much larger model. What chip it's trained on is not really relevant. It's the scale. And obviously to a hundred thousand black wells is equivalent to hundreds of thousands of prior generation chips. TPUs and Trainium have their different release cadence. So it's not exactly like mirrored one-to-one, but ultimately, yes, Mythos is a significantly larger model. It's proof that the scaling laws still work. Everything about it shows the trend line continues of models – more compute into model makes model better. And along the whole way, it's not just more compute into model makes model better. Along the whole way, we're also getting these compute efficiency wins, which are as all this research compute that the labs are spending is actually turning into, if I want X capability to your model every six months, that cost or every two months, that cost is dramatically decreasing. But then if I scale it up massively, I get a humongous capability jump as well. And so yes, it's proof that this is still happening. Google and Anthropic are not heavy, heavy users of GPUs on the training side, but OpenAI, they'll start having their new class of models. I think they're taking a more sensible principled approach to scaling in small steps. Anthropic really went for a huge jump. We'll see better and better models throughout the year. And the release cadence is only going to get faster.