Invest Like the Best with Patrick O'Shaughnessy
00:44:06 · Rob Wachen
[00:44:06] S3: Here's how customers think about deploying models generally, which is when I'm building a data center or I'm building a cluster, it's not like in the abstract of like, oh, I like these chips and like, yeah, this is the power of footprint and so forth. It's like, I have a real production workload I'm trying to serve and for my product to be useful, there's a certain speed I need to serve for that. And for certain products, it's really fast and certain products, it's really slow. Whatever the speed is, this is my speed. The question is, in a given amount of power, how many users can I serve while guaranteeing that speed? So another way to put it is ISO, what's called interactivity, what is my throughput? We are just finishing kind of the early innings of the AI infrastructure boom where people really just cared about speed. You know, GPUs were not able to reach a lot of the speeds of other types of chips like all these SRAM chips, thousands of tokens per second, and that enabled tons of new use cases that got people very excited. There's an entirely new wave of AI chips, us being one of them, that are all going to be able to hit these speeds. The question then is, if you're hitting these speeds, what is the number of users you can serve at the same time? And by proxy, if I have a 100 megawatt data center, how many software agents can I run at the same time? So when people are doing that evaluation, our hardware is going to generally be able to get you an order of magnitude more concurrency at a given level of interactivity. So that directly translates into tokens per watt, tokens per dollar, all the things people care about when they're actually serving these giant mixture of expert models at scale. There's these now famous interactivity curves. Not many people publish them, but you
00:45:32 · Patrick O'Shaughnessy, Gavin Uberti
[00:45:32] S0: can see a Blackwell curve. You can see an A and B curve, which is a little bit worse than Blackwell's, and it's still an $800,000,000,000 company. So if you think about what then the impacts are of shifting that curve, not just a little bit further out, but much further out, what are the things that most excite you about what this will enable? I I want to go out and [00:45:52] S1: solve some of the hardest problems, and I want to go solve these in much less time. There were things growing up that I was not sure I'd be able to live to see. For example, the immunodysics conjecture, one of the things that we talked about in college, and I was not sure I'd see that proven in my life, And this was done by an AI model and it was done over a long period of time. But if you're able to then run the same model 10 times faster, you can go shrink the time to go have these breakthroughs. And there's a huge number of other problems in math like this as well that I worry it will take a thousand years to go prove a thing like this. You'd either have a much smarter model or a model of the same intelligence running much faster. You can then shrink that and I can see it. It's so cool seeing these breakthroughs get made. I am so, so excited to see much more of this happen.
00:46:36 · Rob Wachen
[00:46:36] S3: I think too often people think about tasks and applications and stuff in these very short time horizons. Doing a chat and it's like 50% faster is nice, but it's not like game changing. As these agents go longer and longer time horizon and the models get more and more capable, you're going to see gigantic bodies of work that would take months of compute. And we think about this in wall clock time. Like if you talk to a pre training researcher at a lab, they'll tell you that wall clock time often is one of the most important things that matters, and what wall clock time means is the time from, you know, starting your run to finishing it to actually get data back. If you can shrink this time from a six month run to a two month experiment, you're gonna be able to do many more iterations and people will make changes on the model architectures to actually improve the wall clock time. Very similar here in terms of how we think about the use cases, which is the exciting part about super low latency decode is wall clock time on long horizon tasks becomes much shorter. So a year long compute build would now take a month and that month long compute build will now take three days and that three day compute build will now take seven hours and so forth and so forth. So that's the thing that I think is really hard to internalize because the models are just getting capable enough to do this stuff. I thought it was really cool months ago when Cursor published that they had a bunch of coding agents build an entire browser from scratch in a week. Totally nuts. And that will soon happen in under an hour. And there's going to be many of those types of things that are going to happen with these massive parallel agents all working on a given task.
00:48:05 · Patrick O'Shaughnessy, Gavin Uberti, Rob Wachen
[00:48:05] S0: What are the ultimate limitations of these systems? Is this just like a physics question? Like, how many times faster, cheaper can we get theoretically? There's a lot [00:48:16] S1: of room at the bottom, as they say. If you think about chip to chip latencies, on an NVIDIA product, you're looking at four thousand nanoseconds to go from one chip to another. We'll be able to do much better than that. [00:48:26] S3: What's the mathematical limit? [00:48:27] S1: Is speed of light. You can do it in just a handful, like two, three nanoseconds. And they're 4,004 today. There is a lot of room at the bottom. The same thing with things like power efficiency that, sure, we're able to go and save huge amount by bringing the voltage down by so much, but you can go lower, you can go much, much lower. It's very challenging, but when I think about twenty, thirty years in the future, then I think it's inevitable. And also for economies of scale, for cluster scale up, for a long time, eight chips was the biggest scale up domain and it even had the NL-seventy two, bringing it to, well, 72, but you can be way, way bigger. You look at like a fab, for example, you have a $40,000,000,000 single monolithic building with only a handful of lines running through it. You could have the same kind of thing for some futuristic mega cluster, $40,000,000,000, $100,000,000,000 as a giant mega token factory serving one or a handful of models for a massive number of users to get that same economies of scale thing, same model, massive number of people.
00:49:33 · Patrick O'Shaughnessy
[00:49:33] S0: You mentioned kernels engineering and that being your first job, That has emerged as a thing that nobody had ever heard of in their lives to now something that you hear about all the time, the importance of it, to eke more performance out of the bare raw metal. When will that just be something that AI does entirely as well? Are humans still the best kernel engineers? Are they doing it with the assistance of AI systems? Like, how far down will humans still be in the loop of designing these things? Like, when will that go away?
00:50:02 · Gavin Uberti
[00:50:02] S1: Today, all very hybrid. And the best kernels are still written by human AI collaborations. Also, any AI model is built up of these fundamental primitives like matmoles, like convolutions, like chip to chip operations collectives. And making these overlap and making these really fast matters enormously. And it's the kernel of the matter's job to go ahead and figure out, where can I overlap? How do I allocate memory? How do I verify that if there's some issue, like a retransmit, doesn't stall the whole pipeline? These things are very challenging, but they can go and make your model performance be, say, 3%, 4% better per optimization, and you can do so many. And we thought about our software stack, we wanted to go skate it where the puck is going to be. And three years ago, there were kind of two ways you could build software. One of them was to invest heavily in graph compilers. These things are not very performant, but they work out of the box. They don't require a human to go come in and tweak all the kernels, but we went the opposite direction. We are kernels first programming and that means that it, for a long time, did not work out of the box. But if you were a kernel's expert, you could get incredibly, incredibly high performance. And the thing about this is that now, as the coding models get better and better, they're doing more and more of the kernel generation task. And when the models keep getting smarter, it'll eventually do all of it, it will become superhuman. So we're going to build where the world is going. And even today, we think about our profiling tools or debugging stack, we think about it from how will the model use these tools more than we think about how