Invest Like the Best with Patrick O'Shaughnessy
00:12:31 · Gavin Uberti
[00:12:31] S1: is all a memory game. More memory bound, you can load the model faster, load the KD cache faster, and serve more tokens per second per user. We think people ask the wrong question here. People often ask how much memory bandwidth is on your chip. You should be asking how much memory bandwidth is on your full scale up cluster. What we were able to do is add way, way more bandwidth and a much lower latency from chip to chip to our interconnects that allows us to be able to go serve models at this much higher speed because you can go use the SRAM and the HBM from the full scale up cluster as a single pool. And that's our second key technical bet, what we call cluster scale memory. And on GPUs today, the cluster memory bandwidth is often very badly utilized because the time to go hop from one GPU to another is extremely long. For example, on Blackwell chips, it can be about four thousand nanoseconds to go point to point. And that means that if you go ahead and go to an 8x TP setup, you will get way, way less than an 8x improvement in your tokens per second per user. And what we did is built our own totally custom interconnect stack. We took everything above the second layer Ethernet, built it full custom, and we can go out and do far, far better latencies and imbalance this way too. We can go ahead and cut this by more than a factor of five x, and that allows us to then use the memory of other chips much more effectively. As you scale the world size, your time per token goes down proportionally.
00:13:57 · Rob Wachen, Patrick O'Shaughnessy
[00:13:57] S3: Yeah. And it's not that surprising given all these architectures were built before ChantiPT. So if we're trying to build a chip for the modern workloads, it's gonna look very different. The way we, you know, organize our flops, the way we do our voltage domains, the way we do our power planes are gonna look super different. The way we do the packaging is gonna look super different. The way we do the board design is gonna look different. And then the decode side, the way we connect everything is going to look very different. So we're now bringing forward our first generation of this low voltage inference technology, which is running at under half the voltage of any other AI chip. If you zoom all the way out, why is this so important? Like, why is the delivery of much higher throughput, much lower cost per token, better tokens per watt, like [00:14:37] S0: all of these metrics that the universe is going to start talking about more and more? Everyone knows the supply side of the equation is a big problem right now. Why is this in the bigger picture looking at a decade the bottleneck in the technology world?
00:14:49 · Rob Wachen
[00:14:49] S3: Well, think it comes down to productivity, where we are at this extremely interesting moment in the history of civilization where there's real artificial intelligence, Not like sci fi stuff, but like these models can solve problems that most humans can't. And, like, it's going to create new scientific discoveries. It's going to create instant access to medical care, instant access to education. And now it's just about how many people can use this at the same time, how many products can serve this at the same time, and also the speed of doing different tasks. So when you think about wall clock time, if we can take an agent that, you know, can run at a certain model quality and could take a year to solve a certain task using inference time compute, if you have way faster decode speed, you can compress that into a month. So the amount of scientific innovation and the amount of actual proliferation of technology will happen much faster. And then the second part is concurrency, where today it's just not possible for a billion people to use these models concurrently. Ultimately, some people are gonna get downgraded, some people's model is gonna be slower, some people just won't be able to access the hardware. A few years from now, there's going to be giant models serving billions of users. We're very much in the early innings of AI today where the paid plans there's only a few million users in the world using paid plans of AI models. So we're at one one thousandth of the global population actually using this stuff. So if you wanna serve a giant scale, a lot of things change, and one of them is the number of chips that communicate together, where people usually think about this in the context of training. You know, you have these giant training clusters. You have Colossus with over a 100,000 GPUs that are all networked together. And the inference side, today, people usually think about it as an eight chip cluster or maybe just, you know, NVL 72, the scale up domain. But very quickly, this is going to become thousands of chips and tens of thousands of chips. And the way to get the most performance there, the time between sending data from one chip to another, that primitive matters way more than is getting credit right now.
00:16:34 · Rob Wachen
So when we think about optimizing memory bandwidth for the system, you have to think about how fast these chips can communicate together because if they can only communicate really quickly with themselves and very slowly with other chips, you're not going to actually be able to serve giant models at 10,000, 20,000 tokens per second. So we need multiple orders of magnitude of infrastructure build out throughout the entire stack, the wafer to the watt, transistor to the token, to actually bring this stuff to the world. And I think that
00:16:59 · Gavin Uberti
[00:16:59] S1: if you look at most other goods, like the iPhone, for example, they've gotten into these economies of scale where as a result, more money does not really buy a better iPhone. That if you're a billionaire or if you're just the average American, you buy the same phone. And tokens aren't like that yet. We're still in the very early days where a general purpose system, relatively small one, is kind of handcrafting these tokens like they made screws back in, like, the Renaissance. And I wanted to live in a world where you have the same economies of scale for token making that you do for making, say, iPhones or cars or anything else. I think that is one of the huge unlocks that allows a huge group of people to go use the best quality models. Economies of scale have all made capitalism very, I don't know, fair. I think that allows you to go ahead and have the same product in many, many different hands, and you're able to go then serve way more users on a single scale up cluster, allows you to get closer to that point for token serving too.
00:17:54 · Rob Wachen
[00:17:54] S3: Yeah. And also just certain products aren't usable if they're slow. So if you want to serve coding models and you want people to actually use them, like, there's a certain number of tokens per second you need to hit. So the question is, while maintaining that per token speed, how many users can I serve at the same time? And you can basically decide, I'm gonna shut off a bunch of the world from using this stuff or everyone's gonna get a worse experience. So fundamentally, you need to find ways to push out the curve, and that's why there's such a pressure for new hardware.