Invest Like the Best with Patrick O'Shaughnessy
00:22:00 · Patrick O'Shaughnessy
[00:22:00] S0: It's a great analogy. And step one for what you're trying to do is create the best possible bus on top of NVIDIA GPUs, like that's step one of your optimization?
00:22:08 · Neil Movva
[00:22:08] S2: That's exactly right. It means we explore things like different parallelism schemes. And maybe that's another example I can give you is with NVIDIA GPUs, one of the things that they've really innovated on and done a great job with is the NVLink interconnect between GPUs. And in fact, that NVLink system is so good that if you have a large matrix multiply that you want to perform faster, you can actually cut that matrix multiply in half and chart it across two or more up to eight, let's say NVIDIA GPUs and have them all work on pieces of that larger matrix multiply and have them connect to their results together at the end, reduce their results back together at the end. This is a great great way to cut the minimum latency of an operation. Each GPU is ending one eighth as much work, let's say, and therefore can finish faster but not eight times faster. It's sublinear scaling. You'll use eight times more hardware, but you won't get eight times the speed. You might get like four to five x the speed. You're not gonna get strong scaling. This is because of communication overhead, it's because every GPU is gonna be a little less efficient working on a smaller tile of work than a larger tile of work. It's the only way to speed up. If you want the minimum latency possible, you can do that. But it is not the choice I would make. For example, I would prefer to use a different parallelism scheme like expert parallelism or pipeline parallelism. We may do interesting things to overlap and hide the communication latency in a way that you would have less ability to do that for a low latency server.
00:23:26 · Patrick O'Shaughnessy, Neil Movva
[00:23:26] S0: So is there way to think about NVLink as a technology which improves latency performance [00:23:31] S2: Yes. [00:23:31] S0: And only latency performance? [00:23:32] S2: Which will segue into the next segment of what we do differently as a company. But yes, NVLink is mandatory for low latency inference. So Nvidia is excellent at low latency inference and I'm telling you that we don't really care that much about low latency inference. So where does that leave us? I'm not holding my breath for other companies broadly to figure out NVLink quickly. It's a challenging technology to figure out. It's hard to scale. It's hard to productionize. If I do have some other vendor's chip and it is good at the foundational compute components, it can still do matrix multiplies really well, it just can't communicate those results across its peers quickly. Well, maybe there's room for that other chip in my stack as a really, really good compute per dollar option. That's what I actually optimize for in most cases is how many flops does this chip have and how much is it gonna cost me per hour to own and operate. There are other chips that definitely rank higher than Nvidia on flops per dollar, but they may not have as much interconnect. So it's my job to figure out what parallelism scheme am I gonna use that's going to make this chip suitable for inference. It's not gonna be tensor parallelism. Nvidia is basically mandatory for that. But other techniques may work well for me.
00:24:36 · Patrick O'Shaughnessy
[00:24:36] S0: So before we leave the latency part of the story, can you comment on companies like Cerebras or others that can perform incredibly fast operations? I'm curious, like, what you think about those approach of those companies, what might happen in the future? What is your prediction for the future of very low latency focused hardware?
00:24:54 · Neil Movva
[00:24:54] S2: Cerebras, Grok and a couple others that are coming out of Stealth now, I think have made a very interesting bet on not just building another GPU, but actually building a different kind of accelerator that focuses on a different memory hierarchy. They want to maximize the amount of SRAM on the chip and use that as very, very fast memory for weights and kv cache. So SRAM versus DRAM, there's two ways to make memory for a chip. One is to integrate the memory on the logic die itself. Meaning you tell TSMC, I want this many megabytes of storage on my chip and there's a way to build that. TSMC has a standard cell library you can use and you can just print out a bunch of cells of SRAM. The problem with SRAM is it takes a lot of area on the silicon die. So if you wanna build a large die, like let's say the NVIDIA black well at 800 millimeter square, if you made that whole die SRAM, it would be in the maybe like single digit gigabytes, it feels like. It's not a crazy amount of data storage. Compare that to if you're willing to take a different process technology entirely, so not TSMC anymore, but now Micron, SK Hynix, Samsung. They build DRAM, which is a whole different way to build memory that's more focused on capacitors than transistor cells. The standard way to build SRAM is what's called the 6T transistor cell. It's a stable transistor arrangement that allows you to write a bit to it and then it holds that state in that bit regardless of whether you keep it playing. Well, you had to play some power, but it's holding that bit without any sort of like active management. It's static. Now dynamic RAM, DRAM, it's dynamic because what you do to write some data is you write a charge onto a capacitor. And as soon as you write that charge into that capacitor, the charge is dissipating. The dynamic part of DRAM is that you must every fifty milliseconds or so refresh every bit you've written. So you're constantly juggling billions of balls in the air, essentially billions of bits that we managed by a memory controller, which is reading and refreshing every bit on the DRAM. Now the benefit of that is you can get much much higher density. And it's a whole different process technology. There's a ton of different trade offs. Hence, where we split the DRAM manufacturing into an entirely different company like Micron, SK Hynix and Samsung.
00:26:57 · Neil Movva
These are the best companies in the world to do this. They build DRAM and if you take DRAM from those companies and you stack it into many layers and you kind of print them or solder them around the main logic die that you get from Nvidia, you can now get hundreds of gigabytes. Like Blackwell has 288 gigabytes of HBM capacity around the logic die. And the logic die itself maybe only has like 500 megabytes of SRAM. So it's possibly multiple orders of magnitude, three orders of magnitude difference in density for DRAM versus SRAM. So let's go back to Cerberus. What are they doing? They see this problem. There's not really an obvious way to increase SRAM density on the chip. But thing with SRAM is because it's so physically close to the logic gates that actually do the computation, The arithmetic logic units are right next to the SRAM that they're gonna pull from. The compute units that are doing the matrix multiplies can pull data from SRAM at mind boggling speeds. Serivas quits petabytes per second, 21 petabytes per second for their wafer scale engine three. Compare that to HBM on an NVIDIA Blackwell is, 10 terabytes per second or so in that range. So once again, many orders of magnitude difference. More capacity but proportionally less bandwidth essentially. What Cerberus does is they say that they were gonna take as many of these dies as we can. We're not gonna limit ourselves to the 800 millimeter reticle limit that TSMC... 800 square millimeter limit that TSMC imposes on us. We're gonna take the entire wafer and have every die connect to every other die over scribe lines and we're just going to try to get as much SRAM as we can on the whole wafer. And we can get to like, let's say 50 gigabytes of SRAM per wafer. And then we're gonna stack many wafers together in a pipeline or similar. And now we can have up to a terabyte of memory, very, very fast memory. You do all that work just to get to the ability to read data from SRAM at 21 petabytes per second per wafer. Therefore, you can now serve these language models at extremely high tokens per second because you can move the entire parameter count of a large model like Kimi. You can move all that data in and off the logic cores in about a millisecond or something like that.