Invest Like the Best with Patrick O'Shaughnessy
00:14:45 · Neil Movva, Patrick O'Shaughnessy
[00:14:45] S2: should [00:14:45] S0: be part of that conversation, and you can request a demo at ridgeline.ai. Alright. Now let's talk about the very clever stack of solutions that you hope to build ultimately to have this giant token factory, supplier of extremely low cost intelligence. I think you think about this in terms of software, hardware, and power. Talk through what your master plan is to approach this challenge that's so different from what others are thinking about doing. [00:15:09] S2: We always have to start with software. Where is the opportunity on today's data centers to improve efficiency? And the first thing we did was we tried to build the entire LLM software stack around peak GPU efficiency, meaning we're using NVIDIA GPUs. We wanted to squeeze out more tokens from the same chip than anyone else in the world. And that starts with the lowest level of programming, kernels. It's actually my background. I spent my whole professional life working on GPUs and kernels. NVIDIA was my first job while I was in college. I got to see how the tensor cores got to earn their right to be on the chip. [00:15:39] S0: What does that mean? Like, what is a tensor core? [00:15:40] S2: Tensor core is a specialized unit on the GPU that accelerates matrix multiplication. Simple as that. There's been a long history of how we evolve that tensor core over time that we'll get into. [00:15:49] S0: And why is matrix multiplication so important?
00:15:51 · Neil Movva
[00:15:51] S2: I cannot say that there is a divine truth of the inverse that explains why matrix multiplies seem to be the atomic unit of computation. But one way I've heard it described to me is, well, it's a really succinct way to mix two blocks of numbers together and have them interact in some interesting way. That's as much as I can say about it. It is really convenient that linear algebra turns out to be a very compact representation of arbitrary relationships and data. So Nvidia, great graphics company, had market share dominance in GPUs and gaming graphics for quite some time. And then starting in like the mid twenty tens, they started to actually start these like Skunkworks projects to make the graphics processor more suitable for machine learning tasks that they were tracking. I remember actually reading some of the lab notebooks of some of my managers when I was at Nvidia. They would visit these small ML conferences like ICML or NeurIPS at the time. They would just take note of these papers like, oh, this deep learning thing seems to be catching on. And what's really interesting is that these grad students are using gaming Nvidia GPUs in order to train their large models. We should double click on this and figure out what's going on here. By 2015, 2016, at least Jensen had the conviction to kind of double down on, hey, this usage of our chips is only gonna grow. Let's start allocating more and more precious silicon die area to this capability that seems to be emerging. Let's put the first version of tensor cores on the chip. So we're talking about taking this gaming chip, which is designed for painting pixel on the screen and adapting it to do mission multiplies. It was early. You would be competing against the graphics teams. Essentially, you ask for more silicon area at any chip company, there's always competition for that. It is something that the designers guard so carefully. You don't ever want to invest in the wrong technology because that's opportunity cost that you could have allocated to some other functionality. We fought tooth and nail and got just a tiny bit of diary, maybe like five, 10%, something like that, for the first generation of these chips to get some amount of acceleration for basic convolutions, which were the fundamental operation for computer vision models of the day.
00:17:43 · Neil Movva, Patrick O'Shaughnessy
And then we had a software team that was trying to squeeze all the performance we could out of the chip. And I think on that software team, which is where I worked, that's what actually taught me the most about the ethos that Nvidia has around this term called speed of light. They always chase the speed of light for any piece of hardware that they make. It is so ingrained in every engineer's mind that if the machine can do it, we're going to push the machine to the frontier until it does what we think... [00:18:08] S0: And the speed of light is the edge of what's possible. [00:18:10] S2: The speed of light is the edge of what's possible. Exactly. We think the chip can run at this frequency and produce this many multipliers per cycle, we're gonna get there. We're gonna break every bottleneck and get to that peak level of performance. To this day, I tell all my engineers, we're chasing a 100% speed of light. Don't I care about relative numbers versus the competition. I only care about absolute numbers. What are we able to do on the chip? [00:18:29] S0: How do we achieve that? Before we leave that chapter of your time at NVIDIA, anything else beyond that cultural touch point that changed the way you think about things or that stood out the most about how the business ran back then or its culture?
00:18:29 · Patrick O'Shaughnessy, Neil Movva
[00:18:29] S0: How do we achieve that? Before we leave that chapter of your time at NVIDIA, anything else beyond that cultural touch point that changed the way you think about things or that stood out the most about how the business ran back then or its culture? [00:18:40] S2: I have a ton of stories about NVIDIA. I could tell you a few of them. One of my favorites is that on the tenure side, a lot of people I worked with NVIDIA in 2015, 2016 are still there today. That company has incredible retention and these are the best engineers on on the silicon side at least I've worked with in my whole career. They're extremely, extremely motivated and passionate. They believed in parallel computing as a concept through its various incarnations and have loved seeing the chip evolve. This is their life's work and they're extremely competent in that direction. They're also a very frugal company. NVIDIA and all of Silicon Valley companies after 2008, they had some cutbacks and like perks. So no free lunch, for example. NVIDIA took it one step further. There was no free milk in the fridge. So if you wanted to drink coffee at NVIDIA and you wanted some milk, you actually had to chip in a dollar every month to the milk club and the milk club would stock Costco milk in the fridge. And I remember that distinctly. We don't do that at sale. It's a frugality that permeates [00:19:30] S0: the company. And so coming out of this time there, you get this experience of what it's like to develop more efficient usage of the underlying hardware through software. So link that to today's environment.
00:19:30 · Patrick O'Shaughnessy, Neil Movva
[00:19:30] S0: the company. And so coming out of this time there, you get this experience of what it's like to develop more efficient usage of the underlying hardware through software. So link that to today's environment. [00:19:41] S2: The GPU is fundamentally a throughput machine. The GPU is happiest when you give it a lot of work to do and let it chew through that work at peak utilization of its compute units. But that's actually not the way that we've taken AI in the last couple years. We've pushed AI to be an interactive chatbot tool, is the most common form of AI usage today. In that world, you care a lot about actually spitting answers out to the person at the keyboard as quickly as possible. To your point about don't make the user wait, I want things as fast as possible. That's actually quite interesting for the GPU. It's very difficult to put the GPU in its happy path of being fully compute utilized when you're trying to spit out tokens quickly. There's a fundamental trade off on the GPU between being throughput oriented or latency optimized. And everyone has chosen latency optimization because the shape of usage was chatbot oriented. I believe that's the most profound change we're going to see in the next year. We're gonna move away from chatbots to more proactive or background agents. In that world, it makes a lot more sense to build a stack around throughput. [00:20:35] S0: Can you explain technically why the trade off between throughput and latency is unbreakable? Why can't we have both from the same hardware?
00:20:42 · Neil Movva
[00:20:42] S2: It's quite foundational in almost every system that you could ever possibly look at. There's always a trade off between getting a small amount of data through the system as quickly as possible and leaving a lot of buffer room for that or trying to run wide and slow. Narrow and fast or wide and slow is like a classic trade off in all computer science. But for GPU specifically, I think there's one thing to focus on which is there's this concept of like batching on the GPU. We want to group many users work together into a batch that we can run all at once on the GPU. That's the parallel processing of the GPU. We'd like to have a lot of parallel work to do. The thing is though, you're doing net more work when you run a large batch of compute together. So you might be filling all the units but every step along the way as you carry a batch of work through the GPU, there's more work to be done. Any individual token or any individual user's request in that batch, it's gonna spend a longer time on the GPU being carried with other people's traffic. Maybe the way to say it is, if you wanna get downtown in SF, you can take the bus or you can take a private transit. And the private transit is gonna have its own direct path, like, as the crow flies or using exactly the roads that you want from point a to point b. A bus, it's gonna have to serve many more people. And it has to fundamentally do something that works for everyone. So it takes a slower path and it stops and and waits for other people to get on and off. I think the bus versus car analogy is pretty accurate.