Invest Like the Best with Patrick O'Shaughnessy
01:16:58 · Gavin Uberti
[01:16:58] S1: One of the most important ideas that we believe in is that machines don't think like people think. You look at airplanes, for example, airplanes don't fly like birds fly, that when you think about how mechanical devices have to work, it's often very different. And in much the same way, for people, storing data and loading memory is very cheap for neurons and doing math is relatively expensive. It is the exact opposite for chips. Generally, modern data is very expensive and doing math is very cheap and as time goes on, then you'll end up finding that math gets cheaper at a rate that is faster than memory gets cheaper due to this fundamental limit on any kind of DRAM device. You should think about how can I make my model use a huge, huge amount of compute? What if I had, for example, many copies running at the same time? What if I activated a huge number of experts? What if I had gigantic experts that I could go ahead and run on multiple server racks at the same time? That is how I think you'll build models that are the next generation of intelligence. And context too, there's been a lot of work on a very efficient inference. What if I don't load the full context in the memory? And most of the time, I think that makes a lot of sense. We don't even build a super intelligence. Why can't it go look at a billion tokens of context? Why can't it spend a huge amount of compute to go ahead and read all in a super fast? I would love to be able to talk to a machine that was able to go attend to every book ever written and short term memory, and I think that's really going get to a point where you can't.
01:18:31 · Rob Wachen
[01:18:31] S3: A theme in models right now is this focus on something called dynamism, which is this ability to control the level of computation and memory spent at a per token or per user level when doing attention, as well as this ability to dynamically, in your chip, on the fly, send data to other chips for different MOE models doing certain types of operations. And, you know, the reason is fundamentally as we are scaling context length, as we're scaling model size, as we're scaling the amount of computation per user, we're looking for ways to be more efficient. So, you know, the first thing is like Gavin says, you know, a mixture of experts, you know, architectures where maybe we don't need every parameter being used for every token. But maybe there's things where even at a token level we can say, well, this token needs this context from this other token, they can share that memory so we don't have to have overhead of using a memory as much. Maybe this token is really important, so we should spend more compute. We should have longer context on that token. So hardware that really accelerates these types of very dynamic computation is extremely important. And as you can imagine, current hardware that was designed before those types of architectures have lots of overheads in doing them. So you basically end up in these really bad worlds where you have inefficient hardware at doing this dynamism, so therefore you can't run it very well, or you have these very blocky architectures that are applying blunt force to many different tokens that all need more or less computation.
01:19:53 · Patrick O'Shaughnessy
[01:19:53] S0: I have two questions about the future. We've talked a lot about what you've built so far and how you've built it. The first is about the new ways that people might start using these systems. The raw technology, longer run times, things of this nature. When inference gets much cheaper, faster, more accessible, there's more total supply and it's better. What are the things that you think people will use that capacity to do that are the most interesting and exciting to both of you?
01:20:18 · Gavin Uberti
[01:20:18] S1: Yeah, there was a viral tweet by Noah Brown, which said that as these models are having longer and longer time horizons, they can do tasks that take, say, six months, and there's often not enough time to go and evaluate them for such a long period of time because by that point, you'll have a new model, you'll want to go evaluate instead. And with tech like we built our cluster scale memory, you can go ahead and run that six month job much faster. But there's a second piece of this too, talking to him about it, he's now an angel as well, where it's not just the time, it's also the number of people or agents who are working on this. If you're trying to go and evaluate, can a human build a rocket? You will find that the answer is no. No one person can build a rocket, instead you have to go put a team together. And I believe the same thing will be true of agents too. If you want to go ask, can an agent go out and build some crazy futuristic piece of software, you'll probably need a very large team. Maybe that's 10, maybe that's a million. You have to go and have this enormous amount of both cluster scale memory to go ahead and have that very short time per token and a huge amount of flops to be able to go run that whole fleet.
01:21:27 · Rob Wachen
[01:21:27] S3: I'm going be a little futuristic. I firmly believe we are on a global march of inference becoming majority of global GDP. It may take more than ten years, but it's gonna happen. And right now, we measure productivity as a society as GDP per capita. But really, it's going to look much more like agents per megawatt or it may be agents per gigawatt by then. And while we're being futuristic, I think this is the second to last year where a majority of the workforce is going to be human. I think in 2027, you're going to see there's going to be more agents doing knowledge work than humans, And it's going to be extremely interesting to see what happens. You could imagine a world where, for countries, a majority of their energy ends up going into data centers doing inference. And the energy efficiency of those data centers basically governs how many agents and therefore, you know, how big their workforce is. So you're going to see, like as Gavin is saying, one agent or a team of five to 10 agents working on group projects for a couple days. So you can do pretty cool stuff because they're smart, but it's not going to be civilization scale. What happens when you have countries that can have literally a billion concurrent agents, like a billion people in the workforce working twenty four seven concurrently on the same stuff? It's just kind of unfathomable what's going to happen. And it's going to be the biggest proliferation of technology humanity's ever seen.
01:22:45 · Gavin Uberti, Patrick O'Shaughnessy
[01:22:45] S1: I think as well, like, when you have these huge huge amounts of demand, you get this idea of economies of scale again. Yep. Or think about people. I have a brain. I'm not using the whole thing all at the same time. That's only part of us gonna be active and this is the way healthy brains work. MOE models have worked much the same way. On MOE model, only a small fraction of parameters is being used for any given token at any given moment. But if you have a large number of users on a piece of hardware, you can go kind of take that brain, cut it up into many different experts on many different servers, and run a huge amount of volume through it. So you'll have a bunch of different pieces of traffic, you'll have many of them using each part of the brain at any given point in time, and you'll also make the cost per thought, cost per token way, way lower. So I think you're going to end up with these giant scale distributed brains. The form factor of this is a big data center with a bunch of chips, a huge amount of flops, a huge amount of scale up interconnect. [01:23:38] S0: You think we'll see a trillion dollar individual data center? [01:23:41] S1: Absolutely. It is a matter of time. It's like asking, will you see a billion dollar fab or a $10,000,000,000 fab or a $100,000,000,000 fab? It is inevitable that the economies of scale don't stop at, oh, 40,000,000,000 is the magic number for fabs. No, the cost wafer keeps going down as you keep spending more money. And the same thing will be true of plants that go out and make steel or plants that