Invest Like the Best with Patrick O'Shaughnessy
00:19:30 · Patrick O'Shaughnessy, Neil Movva
[00:19:30] S0: the company. And so coming out of this time there, you get this experience of what it's like to develop more efficient usage of the underlying hardware through software. So link that to today's environment. [00:19:41] S2: The GPU is fundamentally a throughput machine. The GPU is happiest when you give it a lot of work to do and let it chew through that work at peak utilization of its compute units. But that's actually not the way that we've taken AI in the last couple years. We've pushed AI to be an interactive chatbot tool, is the most common form of AI usage today. In that world, you care a lot about actually spitting answers out to the person at the keyboard as quickly as possible. To your point about don't make the user wait, I want things as fast as possible. That's actually quite interesting for the GPU. It's very difficult to put the GPU in its happy path of being fully compute utilized when you're trying to spit out tokens quickly. There's a fundamental trade off on the GPU between being throughput oriented or latency optimized. And everyone has chosen latency optimization because the shape of usage was chatbot oriented. I believe that's the most profound change we're going to see in the next year. We're gonna move away from chatbots to more proactive or background agents. In that world, it makes a lot more sense to build a stack around throughput. [00:20:35] S0: Can you explain technically why the trade off between throughput and latency is unbreakable? Why can't we have both from the same hardware?
00:20:42 · Neil Movva
[00:20:42] S2: It's quite foundational in almost every system that you could ever possibly look at. There's always a trade off between getting a small amount of data through the system as quickly as possible and leaving a lot of buffer room for that or trying to run wide and slow. Narrow and fast or wide and slow is like a classic trade off in all computer science. But for GPU specifically, I think there's one thing to focus on which is there's this concept of like batching on the GPU. We want to group many users work together into a batch that we can run all at once on the GPU. That's the parallel processing of the GPU. We'd like to have a lot of parallel work to do. The thing is though, you're doing net more work when you run a large batch of compute together. So you might be filling all the units but every step along the way as you carry a batch of work through the GPU, there's more work to be done. Any individual token or any individual user's request in that batch, it's gonna spend a longer time on the GPU being carried with other people's traffic. Maybe the way to say it is, if you wanna get downtown in SF, you can take the bus or you can take a private transit. And the private transit is gonna have its own direct path, like, as the crow flies or using exactly the roads that you want from point a to point b. A bus, it's gonna have to serve many more people. And it has to fundamentally do something that works for everyone. So it takes a slower path and it stops and and waits for other people to get on and off. I think the bus versus car analogy is pretty accurate.
00:22:00 · Patrick O'Shaughnessy
[00:22:00] S0: It's a great analogy. And step one for what you're trying to do is create the best possible bus on top of NVIDIA GPUs, like that's step one of your optimization?
00:22:08 · Neil Movva
[00:22:08] S2: That's exactly right. It means we explore things like different parallelism schemes. And maybe that's another example I can give you is with NVIDIA GPUs, one of the things that they've really innovated on and done a great job with is the NVLink interconnect between GPUs. And in fact, that NVLink system is so good that if you have a large matrix multiply that you want to perform faster, you can actually cut that matrix multiply in half and chart it across two or more up to eight, let's say NVIDIA GPUs and have them all work on pieces of that larger matrix multiply and have them connect to their results together at the end, reduce their results back together at the end. This is a great great way to cut the minimum latency of an operation. Each GPU is ending one eighth as much work, let's say, and therefore can finish faster but not eight times faster. It's sublinear scaling. You'll use eight times more hardware, but you won't get eight times the speed. You might get like four to five x the speed. You're not gonna get strong scaling. This is because of communication overhead, it's because every GPU is gonna be a little less efficient working on a smaller tile of work than a larger tile of work. It's the only way to speed up. If you want the minimum latency possible, you can do that. But it is not the choice I would make. For example, I would prefer to use a different parallelism scheme like expert parallelism or pipeline parallelism. We may do interesting things to overlap and hide the communication latency in a way that you would have less ability to do that for a low latency server.
00:23:26 · Patrick O'Shaughnessy, Neil Movva
[00:23:26] S0: So is there way to think about NVLink as a technology which improves latency performance [00:23:31] S2: Yes. [00:23:31] S0: And only latency performance? [00:23:32] S2: Which will segue into the next segment of what we do differently as a company. But yes, NVLink is mandatory for low latency inference. So Nvidia is excellent at low latency inference and I'm telling you that we don't really care that much about low latency inference. So where does that leave us? I'm not holding my breath for other companies broadly to figure out NVLink quickly. It's a challenging technology to figure out. It's hard to scale. It's hard to productionize. If I do have some other vendor's chip and it is good at the foundational compute components, it can still do matrix multiplies really well, it just can't communicate those results across its peers quickly. Well, maybe there's room for that other chip in my stack as a really, really good compute per dollar option. That's what I actually optimize for in most cases is how many flops does this chip have and how much is it gonna cost me per hour to own and operate. There are other chips that definitely rank higher than Nvidia on flops per dollar, but they may not have as much interconnect. So it's my job to figure out what parallelism scheme am I gonna use that's going to make this chip suitable for inference. It's not gonna be tensor parallelism. Nvidia is basically mandatory for that. But other techniques may work well for me.
00:24:36 · Patrick O'Shaughnessy
[00:24:36] S0: So before we leave the latency part of the story, can you comment on companies like Cerebras or others that can perform incredibly fast operations? I'm curious, like, what you think about those approach of those companies, what might happen in the future? What is your prediction for the future of very low latency focused hardware?