Invest Like the Best with Patrick O'Shaughnessy
00:06:04 · Gavin Uberti
[00:06:04] S1: There are some folks who went purely on heuristics. Of, hey, young founders, they claim they can go beat the biggest company in world of performance, it cannot happen. And there is no thing you could go say to me that would make me change my mind. But there's also people out there who are of course skeptical, but are willing to go ahead and say, I'll spend the time, I'll do the work, and is it actually possible? Like for example, one of our earliest, earliest supporters was Mark Ross, and Mark was a very prestigious semiconductor expert. He used to be CTO at Cypress Semi that sold for $9,000,000,000. And when we met him, we were just a couple of guys in a dorm room and we came to him and said, hey, do you want to go build hardware for inference? We think we can be much faster than NVIDIA. And Mark's like, no you can't, it will not work, but if you wanted to go convince me, should write a white paper, you should go ahead and build a functional simulation and show me. And so after a lot of very long nights, went back to Mark and said, hey, here's a simulation, what do you think? And he was like, this works, but to go do a company like this, you'll need a large amount of capital and at least $3,000,000 even to get started, end up went ahead and raised 5, raised a lot more after that and then he was again surprised but got more involved and then he became an advisor, a half time advisor and eventually a full time CTO as he saw more and more of the development progress and I think in general, this has filtered really heavily, folks who want to go ahead and be right regardless or who want to be very truth seeking and say, sure, I'm skeptical, but I will go ahead and work through the numbers myself. And if I can go figure out why this is possible, well, let's go build it.
00:07:43 · Patrick O'Shaughnessy
[00:07:43] S0: The specifics that you've made bets on, the way that you built this system are immensely interesting to me. And because so many people are trying to do this now, build new chips that will do a better job of serving inference at massive scale, The world is interested in the research approaches, the different architecture approaches that people are taking to building a new AI chip. And I'd love you to just start by describing what this thing is, what it does, but maybe more interestingly and more importantly, the process that you went through to decide what bets to take, what technologies to invent, and compare and contrast those with what you've seen the rest of the marketplace try to do.
00:08:24 · Gavin Uberti, Patrick O'Shaughnessy
[00:08:24] S1: Yeah, I think that you can start with the product. We're not just building a chip, we're building a full inference solution, and that means a rack. That means the chip, that means the power delivery into the chip, that means the board in which it sits, that means the interconnect at which the chip talk to each other, That means the production for this mass volume of racks. Really, the production is the product. We think about how we get our advantage. There are two key parts of running inference. There's prefill and there's decode. Now, we have two key text messages in both of these things. Prefill is reading in a huge volume of text and decode is then using that data to generate output tokens. When you go out and run a prefill, your key job is not to go predict tokens, you already know the text. Your job is to go ahead and get the model's memory, what we call as KV cache, into the right state. Then you can go ahead and run decode with that same KV cache. So we will often go do is we call PD disaggregation, prefilled decode disag. You'll have more cluster of servers running these pre fills. You'll then transfer those model memories, those KV caches over to the decode cluster and then go ahead and use that cluster to go generate the next tokens. [00:09:29] S0: So sort of like loading the gun and then firing it, like if I think about it in super simple terms. [00:09:33] S1: Yeah, you got it. It's getting the model to remember the right things and then using those things to go do tasks.
00:09:38 · Rob Wachen
[00:09:38] S3: Candidly, people think about this market a bit lazily. They say, are you prefilled chip? Are you decode chip? Is your decode chip? Are you an HPM chip? Are you an SRAM chip? Are you three d DRAM chip? Are you using optics using copper? When we started this, we just wanted to understand why extremely smart people were working on these different directions. We seriously looked at architectures like having a bunch of DDR memory and like a shared memory pool and looking at advanced packaging to basically break out of the shoreline. We looked at things like, is there ways to put memory dies on top of compute dies? In doing so, we realized that there's no free lunch. Everything has a trade off. Right? Three d DRAM, you have a thermal issue, you have a supply chain issue, you have to figure out hybrid bonding, you have to figure out the flops, and now you're a decode chip. So we went through everything, both on the prefold and the decode side. In doing so, we realized there's a few design spaces that nobody had seriously tried to explore because they were never done in AI chips. And we asked ourselves, what are the actual metrics that are going to matter the most? On the prefill side, the thing that matters is flops and FLOPS density. And people talk about FLOPS often as a headline number, but in reality, you should care about the FLOPS you're getting when you're running real workloads. There's this concept called MFU or model FLOPS utilization, which is, you know, for every peak FLOP advertised, how many cents on the dollar are you actually getting? And on GPUs, you often get somewhere between 2050% depending on the workload. And actually, you can probably not run at a 100% because you have a thermal issue, where as you increase the flop utilization, you have more transistors going on and off, you draw more power, and the chip will self regulate and actually lower its clock speed to make sure it doesn't overheat. So as we looked at inference, we said if we want way more flops because we wanna run at way higher throughputs, we fundamentally need to solve the thermal problem before we even think about adding FLOPS to the chip. If I just add more FLOPS to a GPU today or another AI chip, I'm not actually gonna get more performance because it's just gonna thermal throttle.
00:11:26 · Rob Wachen
So fundamentally, the essence of that is this concept of Dennard scaling, which is voltage is quadratically proportional to power. So if I two x my voltage, my power goes up by four x. If I cut my voltage in half, my power goes down by quarter. So we asked ourselves, how could we run voltages lower than GPUs? And we talked to a lot of people about this. We flew out to to Silicon Valley after dropping out and basically asked, you know, dozens of people in semiconductors and all these different chip companies how they did it. And the answer we got was like, you can't. You can't run at voltages lower than GPUs. And this was very dissatisfying because there was many different industries of chips that run at voltages lower than GPUs. Bitcoin miners run at under a quarter of the voltage of GPUs. So this is obviously physically possible. The question is, are there issues with GPU architectures that make it unable to run at these voltages? And when we looked at the problem for a long time, we were able to create a new mechanism of running at much lower voltages, a new type of power delivery that we call low voltage inference. And we think all AI chips in the future are gonna be low voltage chips. They're going to have to cram way more FLOPS in the same silicon area and without thermal throttling run at way lower voltages. So that's pretty simple. For decode, it
00:12:31 · Gavin Uberti
[00:12:31] S1: is all a memory game. More memory bound, you can load the model faster, load the KD cache faster, and serve more tokens per second per user. We think people ask the wrong question here. People often ask how much memory bandwidth is on your chip. You should be asking how much memory bandwidth is on your full scale up cluster. What we were able to do is add way, way more bandwidth and a much lower latency from chip to chip to our interconnects that allows us to be able to go serve models at this much higher speed because you can go use the SRAM and the HBM from the full scale up cluster as a single pool. And that's our second key technical bet, what we call cluster scale memory. And on GPUs today, the cluster memory bandwidth is often very badly utilized because the time to go hop from one GPU to another is extremely long. For example, on Blackwell chips, it can be about four thousand nanoseconds to go point to point. And that means that if you go ahead and go to an 8x TP setup, you will get way, way less than an 8x improvement in your tokens per second per user. And what we did is built our own totally custom interconnect stack. We took everything above the second layer Ethernet, built it full custom, and we can go out and do far, far better latencies and imbalance this way too. We can go ahead and cut this by more than a factor of five x, and that allows us to then use the memory of other chips much more effectively. As you scale the world size, your time per token goes down proportionally.