Invest Like the Best with Patrick O'Shaughnessy
00:48:05 · Patrick O'Shaughnessy, Gavin Uberti, Rob Wachen
[00:48:05] S0: What are the ultimate limitations of these systems? Is this just like a physics question? Like, how many times faster, cheaper can we get theoretically? There's a lot [00:48:16] S1: of room at the bottom, as they say. If you think about chip to chip latencies, on an NVIDIA product, you're looking at four thousand nanoseconds to go from one chip to another. We'll be able to do much better than that. [00:48:26] S3: What's the mathematical limit? [00:48:27] S1: Is speed of light. You can do it in just a handful, like two, three nanoseconds. And they're 4,004 today. There is a lot of room at the bottom. The same thing with things like power efficiency that, sure, we're able to go and save huge amount by bringing the voltage down by so much, but you can go lower, you can go much, much lower. It's very challenging, but when I think about twenty, thirty years in the future, then I think it's inevitable. And also for economies of scale, for cluster scale up, for a long time, eight chips was the biggest scale up domain and it even had the NL-seventy two, bringing it to, well, 72, but you can be way, way bigger. You look at like a fab, for example, you have a $40,000,000,000 single monolithic building with only a handful of lines running through it. You could have the same kind of thing for some futuristic mega cluster, $40,000,000,000, $100,000,000,000 as a giant mega token factory serving one or a handful of models for a massive number of users to get that same economies of scale thing, same model, massive number of people.
00:49:33 · Patrick O'Shaughnessy
[00:49:33] S0: You mentioned kernels engineering and that being your first job, That has emerged as a thing that nobody had ever heard of in their lives to now something that you hear about all the time, the importance of it, to eke more performance out of the bare raw metal. When will that just be something that AI does entirely as well? Are humans still the best kernel engineers? Are they doing it with the assistance of AI systems? Like, how far down will humans still be in the loop of designing these things? Like, when will that go away?
00:50:02 · Gavin Uberti
[00:50:02] S1: Today, all very hybrid. And the best kernels are still written by human AI collaborations. Also, any AI model is built up of these fundamental primitives like matmoles, like convolutions, like chip to chip operations collectives. And making these overlap and making these really fast matters enormously. And it's the kernel of the matter's job to go ahead and figure out, where can I overlap? How do I allocate memory? How do I verify that if there's some issue, like a retransmit, doesn't stall the whole pipeline? These things are very challenging, but they can go and make your model performance be, say, 3%, 4% better per optimization, and you can do so many. And we thought about our software stack, we wanted to go skate it where the puck is going to be. And three years ago, there were kind of two ways you could build software. One of them was to invest heavily in graph compilers. These things are not very performant, but they work out of the box. They don't require a human to go come in and tweak all the kernels, but we went the opposite direction. We are kernels first programming and that means that it, for a long time, did not work out of the box. But if you were a kernel's expert, you could get incredibly, incredibly high performance. And the thing about this is that now, as the coding models get better and better, they're doing more and more of the kernel generation task. And when the models keep getting smarter, it'll eventually do all of it, it will become superhuman. So we're going to build where the world is going. And even today, we think about our profiling tools or debugging stack, we think about it from how will the model use these tools more than we think about how
00:51:30 · Rob Wachen
[00:51:30] S3: will humans use these tools. We sometimes run experiments internally and we had Codecs actually get GPTOSS running from scratch just based off of our docs completely by itself and it did it I think overnight. We think about game selection a lot, and what we mean by that is making sure we're investing our energy in the right bets. Because regardless of what you choose to work on, it will take tremendous effort. And one of the things that we started with was, you know, the decision explicitly not to build an arbitrary graph compiler, not to support arbitrary PyTorch, not to support arbitrary CUDA, not to support arbitrary ONNX graphs. But instead, we envisioned a world where there was going to be under a 100 models that actually mattered, and they were all going to look very similar from the underlying mathematical perspective, and that we were going to build primitives using physics that were going to accelerate these as much as humanly possible, and we were going to allow the most sophisticated customers to have direct access to the hardware and do whatever they want. And that has saved us a tremendous amount of time not having to build a compiler and that has allowed us to actually get much more performance. And funnily enough, when we started, a lot of people dismissed this idea and the only people that took us seriously were in high frequency trading. They all hate compilers too. They all write their own kernels. And we've had dozens of people from high frequency trading join the team because they saw this philosophy too.
00:52:46 · Patrick O'Shaughnessy
[00:52:46] S0: What are the limits to vertical integration? Like, how do you know where to draw the line? And I'm starting with this question to talk a bit about the broader market. The circumstances of the broader market are really interesting to me, where the vast majority of chips, of AI chips, get bought by a very small set of customers. Many of those customers are themselves trying to design their own AI chips. OpenAI announced TALAPENO. It seems like this very funny circumstance for like the most valuable thing in the world all kind of flows through a couple chip makers, a couple chip buyers. They all seem to be thinking about doing each other's job. And then you've got the circumstance where like, okay, then these things go into data center and you've got neo clouds and inference providers in this other part of the stack. You've got model builders and providers. Like, I could imagine a world where, because you have the best hardware, you design models and you build data centers. You leak outside of your current vertical. So, like, how do you think about where to draw the lines for the business?
00:53:41 · Rob Wachen
[00:53:41] S3: We have a saying that production is the product. Ultimately, matters here is we know inference is going to be the biggest market in the world. Whoever produces the most tokens is going to be the most valuable company in the world. So all the decisions we make is how do we get the most token capacity online as possible? And part of that is building a really good product that has way more throughput, that can run at way better latencies and so forth so we can per chip we make get way more tokens online. Another part of it is like not doing parts of the stack unless we absolutely have to to get to giant scale. So there are parts that we decided to do because it was absolutely required to get the scale, like building the rack instead of just building the chips, like doing a Centimeters model instead of doing a JDM model. But there are parts of it that are kind of noise to us right now. Like, we're not going and building our own data centers today. That doesn't actually help us get more capacity online. In general, our customers are actually making power and moving their clusters around to get our chips online because they're such high throughput. If there was a world where other things are constrained, we would totally go and integrate with them, but the reality is we're just purely focused on getting as many tokens online as possible.