Invest Like the Best with Patrick O'Shaughnessy
00:04:21 · Rob Wachen
[00:04:21] S3: I think there's a certain level of naivety required to think that you could build a chip better than every other AI chip ever built and build a company to do it way faster than ever has been done. And we have the naivety. But there's many times where we would say, like, why isn't this possible? And, you know, really push on it. And it turns out that, like, everybody's answers are extremely siloed to a set of constraints that aren't true anymore. And the reality is the entire semiconductors and data center industry is built on buffer. And what I mean by that is every part of the stack from the EDA tools to the power modules to the circuit boards to the chip design and standard cells, everything is built to be general purpose for everything, not just in the data center, but IoT on the edge and so forth. And when you have a specific use case you're really trying to design for, you can change the constraints a lot. And I'll give you just a very simple example that we're not the only one who does, which is one of the things you care a lot about is the clock speed of your chip. It's proportional to the throughput of your system. When you are doing sign off for different timing, what clock speed you're actually gonna be able to run on when you tape out your chip, there's this concept called corners, which is, you know, what temperatures are you going to be able to run at this clock speed. The default configurations for a lot of these EVA tools assume that you're going to be running your chips in freezing temperatures. Now I don't know about you, but I've never seen an AI data center with ice in it. So, you know, we can feel pretty confident that our chips don't need to run at full speed at zero degrees Celsius. In fact, like, they're never really gonna be running below eight degrees Celsius anyway. And just by knowing that that's a constraint that doesn't matter, we can make a ton of changes throughout the entire system, that's like a very simple one, but there's many more that you get, 20% here, 50% there, two x here, and these compound to a system that can be radically better for inference. I think you found two kinds of people.
00:06:04 · Gavin Uberti
[00:06:04] S1: There are some folks who went purely on heuristics. Of, hey, young founders, they claim they can go beat the biggest company in world of performance, it cannot happen. And there is no thing you could go say to me that would make me change my mind. But there's also people out there who are of course skeptical, but are willing to go ahead and say, I'll spend the time, I'll do the work, and is it actually possible? Like for example, one of our earliest, earliest supporters was Mark Ross, and Mark was a very prestigious semiconductor expert. He used to be CTO at Cypress Semi that sold for $9,000,000,000. And when we met him, we were just a couple of guys in a dorm room and we came to him and said, hey, do you want to go build hardware for inference? We think we can be much faster than NVIDIA. And Mark's like, no you can't, it will not work, but if you wanted to go convince me, should write a white paper, you should go ahead and build a functional simulation and show me. And so after a lot of very long nights, went back to Mark and said, hey, here's a simulation, what do you think? And he was like, this works, but to go do a company like this, you'll need a large amount of capital and at least $3,000,000 even to get started, end up went ahead and raised 5, raised a lot more after that and then he was again surprised but got more involved and then he became an advisor, a half time advisor and eventually a full time CTO as he saw more and more of the development progress and I think in general, this has filtered really heavily, folks who want to go ahead and be right regardless or who want to be very truth seeking and say, sure, I'm skeptical, but I will go ahead and work through the numbers myself. And if I can go figure out why this is possible, well, let's go build it.
00:07:43 · Patrick O'Shaughnessy
[00:07:43] S0: The specifics that you've made bets on, the way that you built this system are immensely interesting to me. And because so many people are trying to do this now, build new chips that will do a better job of serving inference at massive scale, The world is interested in the research approaches, the different architecture approaches that people are taking to building a new AI chip. And I'd love you to just start by describing what this thing is, what it does, but maybe more interestingly and more importantly, the process that you went through to decide what bets to take, what technologies to invent, and compare and contrast those with what you've seen the rest of the marketplace try to do.
00:08:24 · Gavin Uberti, Patrick O'Shaughnessy
[00:08:24] S1: Yeah, I think that you can start with the product. We're not just building a chip, we're building a full inference solution, and that means a rack. That means the chip, that means the power delivery into the chip, that means the board in which it sits, that means the interconnect at which the chip talk to each other, That means the production for this mass volume of racks. Really, the production is the product. We think about how we get our advantage. There are two key parts of running inference. There's prefill and there's decode. Now, we have two key text messages in both of these things. Prefill is reading in a huge volume of text and decode is then using that data to generate output tokens. When you go out and run a prefill, your key job is not to go predict tokens, you already know the text. Your job is to go ahead and get the model's memory, what we call as KV cache, into the right state. Then you can go ahead and run decode with that same KV cache. So we will often go do is we call PD disaggregation, prefilled decode disag. You'll have more cluster of servers running these pre fills. You'll then transfer those model memories, those KV caches over to the decode cluster and then go ahead and use that cluster to go generate the next tokens. [00:09:29] S0: So sort of like loading the gun and then firing it, like if I think about it in super simple terms. [00:09:33] S1: Yeah, you got it. It's getting the model to remember the right things and then using those things to go do tasks.
00:09:38 · Rob Wachen
[00:09:38] S3: Candidly, people think about this market a bit lazily. They say, are you prefilled chip? Are you decode chip? Is your decode chip? Are you an HPM chip? Are you an SRAM chip? Are you three d DRAM chip? Are you using optics using copper? When we started this, we just wanted to understand why extremely smart people were working on these different directions. We seriously looked at architectures like having a bunch of DDR memory and like a shared memory pool and looking at advanced packaging to basically break out of the shoreline. We looked at things like, is there ways to put memory dies on top of compute dies? In doing so, we realized that there's no free lunch. Everything has a trade off. Right? Three d DRAM, you have a thermal issue, you have a supply chain issue, you have to figure out hybrid bonding, you have to figure out the flops, and now you're a decode chip. So we went through everything, both on the prefold and the decode side. In doing so, we realized there's a few design spaces that nobody had seriously tried to explore because they were never done in AI chips. And we asked ourselves, what are the actual metrics that are going to matter the most? On the prefill side, the thing that matters is flops and FLOPS density. And people talk about FLOPS often as a headline number, but in reality, you should care about the FLOPS you're getting when you're running real workloads. There's this concept called MFU or model FLOPS utilization, which is, you know, for every peak FLOP advertised, how many cents on the dollar are you actually getting? And on GPUs, you often get somewhere between 2050% depending on the workload. And actually, you can probably not run at a 100% because you have a thermal issue, where as you increase the flop utilization, you have more transistors going on and off, you draw more power, and the chip will self regulate and actually lower its clock speed to make sure it doesn't overheat. So as we looked at inference, we said if we want way more flops because we wanna run at way higher throughputs, we fundamentally need to solve the thermal problem before we even think about adding FLOPS to the chip. If I just add more FLOPS to a GPU today or another AI chip, I'm not actually gonna get more performance because it's just gonna thermal throttle.
00:11:26 · Rob Wachen
So fundamentally, the essence of that is this concept of Dennard scaling, which is voltage is quadratically proportional to power. So if I two x my voltage, my power goes up by four x. If I cut my voltage in half, my power goes down by quarter. So we asked ourselves, how could we run voltages lower than GPUs? And we talked to a lot of people about this. We flew out to to Silicon Valley after dropping out and basically asked, you know, dozens of people in semiconductors and all these different chip companies how they did it. And the answer we got was like, you can't. You can't run at voltages lower than GPUs. And this was very dissatisfying because there was many different industries of chips that run at voltages lower than GPUs. Bitcoin miners run at under a quarter of the voltage of GPUs. So this is obviously physically possible. The question is, are there issues with GPU architectures that make it unable to run at these voltages? And when we looked at the problem for a long time, we were able to create a new mechanism of running at much lower voltages, a new type of power delivery that we call low voltage inference. And we think all AI chips in the future are gonna be low voltage chips. They're going to have to cram way more FLOPS in the same silicon area and without thermal throttling run at way lower voltages. So that's pretty simple. For decode, it