Invest Like the Best with Patrick O'Shaughnessy
00:08:24 · Gavin Uberti, Patrick O'Shaughnessy
[00:08:24] S1: Yeah, I think that you can start with the product. We're not just building a chip, we're building a full inference solution, and that means a rack. That means the chip, that means the power delivery into the chip, that means the board in which it sits, that means the interconnect at which the chip talk to each other, That means the production for this mass volume of racks. Really, the production is the product. We think about how we get our advantage. There are two key parts of running inference. There's prefill and there's decode. Now, we have two key text messages in both of these things. Prefill is reading in a huge volume of text and decode is then using that data to generate output tokens. When you go out and run a prefill, your key job is not to go predict tokens, you already know the text. Your job is to go ahead and get the model's memory, what we call as KV cache, into the right state. Then you can go ahead and run decode with that same KV cache. So we will often go do is we call PD disaggregation, prefilled decode disag. You'll have more cluster of servers running these pre fills. You'll then transfer those model memories, those KV caches over to the decode cluster and then go ahead and use that cluster to go generate the next tokens. [00:09:29] S0: So sort of like loading the gun and then firing it, like if I think about it in super simple terms. [00:09:33] S1: Yeah, you got it. It's getting the model to remember the right things and then using those things to go do tasks.
00:09:38 · Rob Wachen
[00:09:38] S3: Candidly, people think about this market a bit lazily. They say, are you prefilled chip? Are you decode chip? Is your decode chip? Are you an HPM chip? Are you an SRAM chip? Are you three d DRAM chip? Are you using optics using copper? When we started this, we just wanted to understand why extremely smart people were working on these different directions. We seriously looked at architectures like having a bunch of DDR memory and like a shared memory pool and looking at advanced packaging to basically break out of the shoreline. We looked at things like, is there ways to put memory dies on top of compute dies? In doing so, we realized that there's no free lunch. Everything has a trade off. Right? Three d DRAM, you have a thermal issue, you have a supply chain issue, you have to figure out hybrid bonding, you have to figure out the flops, and now you're a decode chip. So we went through everything, both on the prefold and the decode side. In doing so, we realized there's a few design spaces that nobody had seriously tried to explore because they were never done in AI chips. And we asked ourselves, what are the actual metrics that are going to matter the most? On the prefill side, the thing that matters is flops and FLOPS density. And people talk about FLOPS often as a headline number, but in reality, you should care about the FLOPS you're getting when you're running real workloads. There's this concept called MFU or model FLOPS utilization, which is, you know, for every peak FLOP advertised, how many cents on the dollar are you actually getting? And on GPUs, you often get somewhere between 2050% depending on the workload. And actually, you can probably not run at a 100% because you have a thermal issue, where as you increase the flop utilization, you have more transistors going on and off, you draw more power, and the chip will self regulate and actually lower its clock speed to make sure it doesn't overheat. So as we looked at inference, we said if we want way more flops because we wanna run at way higher throughputs, we fundamentally need to solve the thermal problem before we even think about adding FLOPS to the chip. If I just add more FLOPS to a GPU today or another AI chip, I'm not actually gonna get more performance because it's just gonna thermal throttle.
00:11:26 · Rob Wachen
So fundamentally, the essence of that is this concept of Dennard scaling, which is voltage is quadratically proportional to power. So if I two x my voltage, my power goes up by four x. If I cut my voltage in half, my power goes down by quarter. So we asked ourselves, how could we run voltages lower than GPUs? And we talked to a lot of people about this. We flew out to to Silicon Valley after dropping out and basically asked, you know, dozens of people in semiconductors and all these different chip companies how they did it. And the answer we got was like, you can't. You can't run at voltages lower than GPUs. And this was very dissatisfying because there was many different industries of chips that run at voltages lower than GPUs. Bitcoin miners run at under a quarter of the voltage of GPUs. So this is obviously physically possible. The question is, are there issues with GPU architectures that make it unable to run at these voltages? And when we looked at the problem for a long time, we were able to create a new mechanism of running at much lower voltages, a new type of power delivery that we call low voltage inference. And we think all AI chips in the future are gonna be low voltage chips. They're going to have to cram way more FLOPS in the same silicon area and without thermal throttling run at way lower voltages. So that's pretty simple. For decode, it
00:12:31 · Gavin Uberti
[00:12:31] S1: is all a memory game. More memory bound, you can load the model faster, load the KD cache faster, and serve more tokens per second per user. We think people ask the wrong question here. People often ask how much memory bandwidth is on your chip. You should be asking how much memory bandwidth is on your full scale up cluster. What we were able to do is add way, way more bandwidth and a much lower latency from chip to chip to our interconnects that allows us to be able to go serve models at this much higher speed because you can go use the SRAM and the HBM from the full scale up cluster as a single pool. And that's our second key technical bet, what we call cluster scale memory. And on GPUs today, the cluster memory bandwidth is often very badly utilized because the time to go hop from one GPU to another is extremely long. For example, on Blackwell chips, it can be about four thousand nanoseconds to go point to point. And that means that if you go ahead and go to an 8x TP setup, you will get way, way less than an 8x improvement in your tokens per second per user. And what we did is built our own totally custom interconnect stack. We took everything above the second layer Ethernet, built it full custom, and we can go out and do far, far better latencies and imbalance this way too. We can go ahead and cut this by more than a factor of five x, and that allows us to then use the memory of other chips much more effectively. As you scale the world size, your time per token goes down proportionally.
00:13:57 · Rob Wachen, Patrick O'Shaughnessy
[00:13:57] S3: Yeah. And it's not that surprising given all these architectures were built before ChantiPT. So if we're trying to build a chip for the modern workloads, it's gonna look very different. The way we, you know, organize our flops, the way we do our voltage domains, the way we do our power planes are gonna look super different. The way we do the packaging is gonna look super different. The way we do the board design is gonna look different. And then the decode side, the way we connect everything is going to look very different. So we're now bringing forward our first generation of this low voltage inference technology, which is running at under half the voltage of any other AI chip. If you zoom all the way out, why is this so important? Like, why is the delivery of much higher throughput, much lower cost per token, better tokens per watt, like [00:14:37] S0: all of these metrics that the universe is going to start talking about more and more? Everyone knows the supply side of the equation is a big problem right now. Why is this in the bigger picture looking at a decade the bottleneck in the technology world?
00:14:49 · Rob Wachen
[00:14:49] S3: Well, think it comes down to productivity, where we are at this extremely interesting moment in the history of civilization where there's real artificial intelligence, Not like sci fi stuff, but like these models can solve problems that most humans can't. And, like, it's going to create new scientific discoveries. It's going to create instant access to medical care, instant access to education. And now it's just about how many people can use this at the same time, how many products can serve this at the same time, and also the speed of doing different tasks. So when you think about wall clock time, if we can take an agent that, you know, can run at a certain model quality and could take a year to solve a certain task using inference time compute, if you have way faster decode speed, you can compress that into a month. So the amount of scientific innovation and the amount of actual proliferation of technology will happen much faster. And then the second part is concurrency, where today it's just not possible for a billion people to use these models concurrently. Ultimately, some people are gonna get downgraded, some people's model is gonna be slower, some people just won't be able to access the hardware. A few years from now, there's going to be giant models serving billions of users. We're very much in the early innings of AI today where the paid plans there's only a few million users in the world using paid plans of AI models. So we're at one one thousandth of the global population actually using this stuff. So if you wanna serve a giant scale, a lot of things change, and one of them is the number of chips that communicate together, where people usually think about this in the context of training. You know, you have these giant training clusters. You have Colossus with over a 100,000 GPUs that are all networked together. And the inference side, today, people usually think about it as an eight chip cluster or maybe just, you know, NVL 72, the scale up domain. But very quickly, this is going to become thousands of chips and tens of thousands of chips. And the way to get the most performance there, the time between sending data from one chip to another, that primitive matters way more than is getting credit right now.