Invest Like the Best with Patrick O'Shaughnessy
00:24:54 · Neil Movva
[00:24:54] S2: Cerebras, Grok and a couple others that are coming out of Stealth now, I think have made a very interesting bet on not just building another GPU, but actually building a different kind of accelerator that focuses on a different memory hierarchy. They want to maximize the amount of SRAM on the chip and use that as very, very fast memory for weights and kv cache. So SRAM versus DRAM, there's two ways to make memory for a chip. One is to integrate the memory on the logic die itself. Meaning you tell TSMC, I want this many megabytes of storage on my chip and there's a way to build that. TSMC has a standard cell library you can use and you can just print out a bunch of cells of SRAM. The problem with SRAM is it takes a lot of area on the silicon die. So if you wanna build a large die, like let's say the NVIDIA black well at 800 millimeter square, if you made that whole die SRAM, it would be in the maybe like single digit gigabytes, it feels like. It's not a crazy amount of data storage. Compare that to if you're willing to take a different process technology entirely, so not TSMC anymore, but now Micron, SK Hynix, Samsung. They build DRAM, which is a whole different way to build memory that's more focused on capacitors than transistor cells. The standard way to build SRAM is what's called the 6T transistor cell. It's a stable transistor arrangement that allows you to write a bit to it and then it holds that state in that bit regardless of whether you keep it playing. Well, you had to play some power, but it's holding that bit without any sort of like active management. It's static. Now dynamic RAM, DRAM, it's dynamic because what you do to write some data is you write a charge onto a capacitor. And as soon as you write that charge into that capacitor, the charge is dissipating. The dynamic part of DRAM is that you must every fifty milliseconds or so refresh every bit you've written. So you're constantly juggling billions of balls in the air, essentially billions of bits that we managed by a memory controller, which is reading and refreshing every bit on the DRAM. Now the benefit of that is you can get much much higher density. And it's a whole different process technology. There's a ton of different trade offs. Hence, where we split the DRAM manufacturing into an entirely different company like Micron, SK Hynix and Samsung.
00:26:57 · Neil Movva
These are the best companies in the world to do this. They build DRAM and if you take DRAM from those companies and you stack it into many layers and you kind of print them or solder them around the main logic die that you get from Nvidia, you can now get hundreds of gigabytes. Like Blackwell has 288 gigabytes of HBM capacity around the logic die. And the logic die itself maybe only has like 500 megabytes of SRAM. So it's possibly multiple orders of magnitude, three orders of magnitude difference in density for DRAM versus SRAM. So let's go back to Cerberus. What are they doing? They see this problem. There's not really an obvious way to increase SRAM density on the chip. But thing with SRAM is because it's so physically close to the logic gates that actually do the computation, The arithmetic logic units are right next to the SRAM that they're gonna pull from. The compute units that are doing the matrix multiplies can pull data from SRAM at mind boggling speeds. Serivas quits petabytes per second, 21 petabytes per second for their wafer scale engine three. Compare that to HBM on an NVIDIA Blackwell is, 10 terabytes per second or so in that range. So once again, many orders of magnitude difference. More capacity but proportionally less bandwidth essentially. What Cerberus does is they say that they were gonna take as many of these dies as we can. We're not gonna limit ourselves to the 800 millimeter reticle limit that TSMC... 800 square millimeter limit that TSMC imposes on us. We're gonna take the entire wafer and have every die connect to every other die over scribe lines and we're just going to try to get as much SRAM as we can on the whole wafer. And we can get to like, let's say 50 gigabytes of SRAM per wafer. And then we're gonna stack many wafers together in a pipeline or similar. And now we can have up to a terabyte of memory, very, very fast memory. You do all that work just to get to the ability to read data from SRAM at 21 petabytes per second per wafer. Therefore, you can now serve these language models at extremely high tokens per second because you can move the entire parameter count of a large model like Kimi. You can move all that data in and off the logic cores in about a millisecond or something like that.
00:28:56 · Patrick O'Shaughnessy, Neil Movva
[00:28:56] S0: So there you go. You have a path to a thousand tokens per second. So what is your prediction for, like, that segment of the market? [00:29:02] S2: What happens to them is some hybrid, we had to pair the Cerberus chip where it's very strong, it's very, very good at fast access to memory with something that has more capacity for memory. It's true that you can take a 1,000,000,000,000 parameter model like Kimi and fit it on a large number of Cerberus Diode wafers. But you can't do something about the KBChash very easily. The KBChash is something that grows as people use the model more and that is always dynamic. You don't even know how much KBChash you're gonna need. It depends on how many users you have and how many users you want to serve. Can you explain KBChash? KBChash, when you ever use a language model, every token you send through the language model stays in the context window of the language model for as long as you're having a conversation. We talk for a 100,000 tokens, the 100,001 token is still in the conversation behind us and the model is referencing all the past conversation history in order to make better predictions about what the next thing we're gonna say is. That KvCache, it's a bunch of memory. You have to store a representation for every token that we send through the language model and it frequently gets to be larger than the weights of the model themselves. You have this crystallized knowledge in the model weights and you have the dynamic knowledge of the exact conversation we're having in the KvCache is the way I like to think about it.
00:29:02 · Neil Movva, Patrick O'Shaughnessy
[00:29:02] S2: What happens to them is some hybrid, we had to pair the Cerberus chip where it's very strong, it's very, very good at fast access to memory with something that has more capacity for memory. It's true that you can take a 1,000,000,000,000 parameter model like Kimi and fit it on a large number of Cerberus Diode wafers. But you can't do something about the KBChash very easily. The KBChash is something that grows as people use the model more and that is always dynamic. You don't even know how much KBChash you're gonna need. It depends on how many users you have and how many users you want to serve. Can you explain KBChash? KBChash, when you ever use a language model, every token you send through the language model stays in the context window of the language model for as long as you're having a conversation. We talk for a 100,000 tokens, the 100,001 token is still in the conversation behind us and the model is referencing all the past conversation history in order to make better predictions about what the next thing we're gonna say is. That KvCache, it's a bunch of memory. You have to store a representation for every token that we send through the language model and it frequently gets to be larger than the weights of the model themselves. You have this crystallized knowledge in the model weights and you have the dynamic knowledge of the exact conversation we're having in the KvCache is the way I like to think about it. [00:30:16] S0: And this is why sometimes people would observe deep in a conversation things start to degrade because there's some sort
00:30:16 · Patrick O'Shaughnessy, Neil Movva
[00:30:16] S0: And this is why sometimes people would observe deep in a conversation things start to degrade because there's some sort [00:30:21] S2: of technical problem? So the KB cache is quite interesting in that regard. The KB cache is an exact representation of everything that came before. We store all the information that we've seen in the conversation. However, during training, the model did not get trained primarily on very long context conversations. It got trained primarily on, let's say, 8,000 token conversations or 16,000 token conversations. So if you take the model to 200,000 tokens, there was some training that happened at that context length, but it's not the models like core strength. And so there's always been a challenge for the frontier labs to figure out how do we make the model exactly as intelligent at 10,000 tokens as we expect it to be at 200,000 tokens. And it's gonna be a perennial battle for us. We've had 1,000,000 context windows as a concept for years now. Anthropic was, I think, the first to hit that 1,000,000 context window length length. I still, you know, use slash compact in my cloud code well before 1,000,000 context length. I don't think it's actually great to hit the full length. [00:31:12] S0: These extremely fast, extremely low latency approaches ultimately are limited by this factor. [00:31:18] S2: Yes. You can do whatever you want for the weights. It's very possible to have much unbeatable performance on weight storage. However, the KBChash is gonna be a big thorn in
00:31:18 · Neil Movva, Patrick O'Shaughnessy
[00:31:18] S2: Yes. You can do whatever you want for the weights. It's very possible to have much unbeatable performance on weight storage. However, the KBChash is gonna be a big thorn in [00:31:26] S0: your side. So three years from now, five years from now, what role do you think these kinds of chips play? Like, what sort of market share do they have in the heterogeneous chip market like...