Invest Like the Best with Patrick O'Shaughnessy
01:15:08 · Patrick O'Shaughnessy, Rob Wachen
[01:15:08] S0: What about other problems in the ecosystem that are outside of your control, such as capacity at the leading nanometer at TSMC or availability of HBM4 memory or some of these other things where everyone is fighting for a scarce unit of capacity or whatever. How do you face up against those realities when you're trying to produce as much as humanly possible? [01:15:31] S3: The people deploying the most compute in the world do think about supply a bit zero sum, which is there's only so many wafers being produced on a given nanometer node on a given fab, right? And there's only so much memory being produced, and that's why actually for our first gen product, we built it on a different supply chain than the Rubens. We're on four nanometer, Rubens on three nanometer. Yeah. We're on a different HPM than Rubens and so forth. So it actually is not a zero something. It's a positive something where more is more. So often when we're talking to people deploying at scale, it's not a decision between a gigawatt of a GPU and a gigawatt of us, it's two gigawatts. And I think as much as possible, thinking about supply chain early in the design decisions because if you have the most performing product and you can't produce it, then you're just a podcast.
01:15:31 · Rob Wachen, Gavin Uberti
[01:15:31] S3: The people deploying the most compute in the world do think about supply a bit zero sum, which is there's only so many wafers being produced on a given nanometer node on a given fab, right? And there's only so much memory being produced, and that's why actually for our first gen product, we built it on a different supply chain than the Rubens. We're on four nanometer, Rubens on three nanometer. Yeah. We're on a different HPM than Rubens and so forth. So it actually is not a zero something. It's a positive something where more is more. So often when we're talking to people deploying at scale, it's not a decision between a gigawatt of a GPU and a gigawatt of us, it's two gigawatts. And I think as much as possible, thinking about supply chain early in the design decisions because if you have the most performing product and you can't produce it, then you're just a podcast. [01:16:15] S1: That's the other big thing about vertical integration too, is well, certain things like for the chips and for the memory, have to go ahead and partner. For most of the other stuff, those are also very highly in demand components. And the more that you build yourself, the more stuff you can go do on top of what the world can currently build. It is not, oh you're taking compatibility with somebody else, you're adding way, way more. I think that's how you win.
01:16:15 · Gavin Uberti, Patrick O'Shaughnessy
[01:16:15] S1: That's the other big thing about vertical integration too, is well, certain things like for the chips and for the memory, have to go ahead and partner. For most of the other stuff, those are also very highly in demand components. And the more that you build yourself, the more stuff you can go do on top of what the world can currently build. It is not, oh you're taking compatibility with somebody else, you're adding way, way more. I think that's how you win. [01:16:37] S0: One of the things that I realized we haven't talked at all about is the models themselves, which is kind of crazy, the things behind all of this demand. Anything interesting that you would say about the way that you see models progressing based on what we've seen so far? I guess I'm more interested in how you as thinkers about hardware think hardware might impact where the models themselves go in the future.
01:16:58 · Gavin Uberti
[01:16:58] S1: One of the most important ideas that we believe in is that machines don't think like people think. You look at airplanes, for example, airplanes don't fly like birds fly, that when you think about how mechanical devices have to work, it's often very different. And in much the same way, for people, storing data and loading memory is very cheap for neurons and doing math is relatively expensive. It is the exact opposite for chips. Generally, modern data is very expensive and doing math is very cheap and as time goes on, then you'll end up finding that math gets cheaper at a rate that is faster than memory gets cheaper due to this fundamental limit on any kind of DRAM device. You should think about how can I make my model use a huge, huge amount of compute? What if I had, for example, many copies running at the same time? What if I activated a huge number of experts? What if I had gigantic experts that I could go ahead and run on multiple server racks at the same time? That is how I think you'll build models that are the next generation of intelligence. And context too, there's been a lot of work on a very efficient inference. What if I don't load the full context in the memory? And most of the time, I think that makes a lot of sense. We don't even build a super intelligence. Why can't it go look at a billion tokens of context? Why can't it spend a huge amount of compute to go ahead and read all in a super fast? I would love to be able to talk to a machine that was able to go attend to every book ever written and short term memory, and I think that's really going get to a point where you can't.
01:18:31 · Rob Wachen
[01:18:31] S3: A theme in models right now is this focus on something called dynamism, which is this ability to control the level of computation and memory spent at a per token or per user level when doing attention, as well as this ability to dynamically, in your chip, on the fly, send data to other chips for different MOE models doing certain types of operations. And, you know, the reason is fundamentally as we are scaling context length, as we're scaling model size, as we're scaling the amount of computation per user, we're looking for ways to be more efficient. So, you know, the first thing is like Gavin says, you know, a mixture of experts, you know, architectures where maybe we don't need every parameter being used for every token. But maybe there's things where even at a token level we can say, well, this token needs this context from this other token, they can share that memory so we don't have to have overhead of using a memory as much. Maybe this token is really important, so we should spend more compute. We should have longer context on that token. So hardware that really accelerates these types of very dynamic computation is extremely important. And as you can imagine, current hardware that was designed before those types of architectures have lots of overheads in doing them. So you basically end up in these really bad worlds where you have inefficient hardware at doing this dynamism, so therefore you can't run it very well, or you have these very blocky architectures that are applying blunt force to many different tokens that all need more or less computation.
01:19:53 · Patrick O'Shaughnessy
[01:19:53] S0: I have two questions about the future. We've talked a lot about what you've built so far and how you've built it. The first is about the new ways that people might start using these systems. The raw technology, longer run times, things of this nature. When inference gets much cheaper, faster, more accessible, there's more total supply and it's better. What are the things that you think people will use that capacity to do that are the most interesting and exciting to both of you?