Invest Like the Best with Patrick O'Shaughnessy
00:36:26 · Neil Movva
[00:36:26] S2: I like the phrase that Internet was a one time subsidy on data. We got it for free. It's extremely high quality. There are about 30,000,000,000,000 tokens of high quality text. 300,000,000,000,000 tokens if you could take a wider view on what qualifies as good text. And we've basically looked at it all already. Models have seen the entire Internet many times over at this point. And there is not a whole lot more to be done on human data from the internet. The next phase of data in my mind is model self improvement through RL environment, GIMs, basically. Now, in fact, we don't even benefit from getting more random user interactions with AI. It used to be that the new type of data that we cared about a lot was the interaction data from people using ChatGPT and giving ChatGPT signals on what they liked and didn't like. I like the argument now that the median model that we serve is so much more advanced than like a random human giving feedback that the signal you get from random human preference, unconditioned human preference is not actually worth anything anymore. You want expert human preference at this point. The model has outgrown Everyday Joe. Generic. Yeah, Everyday Joe, exactly. So the feature of data to me is giving the model a hard verifiable task and letting it run-in this gym where it's kind of isolated and it just has a problem and it can make progress on it, get measurement of whether it made progress on that problem or not. You can imagine coding problems are in this category, math problems are also in this category. More and more, you can just give the agent a computer essentially and have it act like it's a human worker and give it feedback on whether it's making progress towards the target outcome. That environment becomes the data. I think this is not a super differentiated take, but it's been really, really productive from what I've seen so far.
00:37:59 · Patrick O'Shaughnessy, Neil Movva
[00:37:59] S0: And you think that just goes on for a really long period of time? Or is that another, like, if I think about the Internet as this one big block, like, is another big block that will have its day in the sun and will kind of get it all Yeah. And then we'll have to move on to something else? [00:38:12] S2: I think it's actually more profound than that. Basically, the idea is that if you want artificial general intelligence, the best way to get there is to just keep stacking specialized intelligences until you have no more gaps to fill. The only thing you need to make sure you do to make this work is you must make sure that your task is verifiable. You need to give the model a self grading system. If you have that, you have the recipe for self improvement on any task you like. I think you've seen this held up by the way frontier labs spend. They used to spend that much on data. Now they spend a lot more on RL environments. These environments absolutely capture that relationship of recursive self improvement on a verifiable task. [00:38:48] S0: Coming back to your initial task of making existing hardware more efficient by being more in control of what's going on at the hardware level through software. Keep going on what you've done so far and what you wanna do and then we're gonna jump to hardware and then jump to energy finally.
00:39:06 · Neil Movva
[00:39:06] S2: Mentioned kernels. It's surprising, people think kernels are done. There are great people like Trudeau who write excellent kernels and they form the bedrock of all of our modern deep learning. It's built on flash attention. Modern transformers are built on flash attention. But if you deviate from the happy path at all, if there's a new model that comes out that has a slightly different way to embed positional information, like the change of the rope system, suddenly the kernel that we had is not suitable for this new model and we may have to make a patch to this kernel. I wouldn't say we're in the phase where we had to invent new kernels from scratch but having the ability to quickly modify existing GPU kernel... Sorry, a kernel, by the way, is a general term for any program you run on the GPU. Historically, kernels tend to be put into a library where every kernel has a very, very scoped purpose. Typically, you have a kernel for a matrix multiply. You have another kernel for even something as simple as addition. You're add two tensors together, that's another kernel. And then increasingly, we've started to fuse those kernels together. So if I do a matrix multiply and then I want to add it to another matrix that I've also multiplied, maybe those two become one kernel and I just fuse the operations where sort of writing the data out to DRAM and then reading it back in just to do the addition, maybe I can just do this easily.
00:40:16 · Patrick O'Shaughnessy, Neil Movva
[00:40:16] S0: Why are humans still doing this? Like, seems like the sort of thing that AIs would be exceptionally good engineering, more efficient kernels. Maybe that's where we're going and we're just not quite there yet. But if we aren't there yet, is that where we're going? If we're not there yet, why humans still doing this? Why why is TreeHouse so well known? You know, it's a name I know. [00:40:32] S2: I don't want to speak for Tree, but what he taught me was you shouldn't write kernels by hand anymore necessarily. I like to say we write kernels on the whiteboard. We go to the whiteboard, we describe what we think the machine should be doing, then we succinctly describe that in natural language to a model And then the model is able to do the execution of here's my input and output, here's the strategy of how we want to dispatch this work onto the GPU. I'm gonna go implement... [00:40:56] S0: So we're doing the conceptual design. [00:40:57] S2: Exactly. I'm not sure exactly why models are not superb at doing this. I don't think this is like our mode or anything like that. I'm sure in six months time, we'll have much better models on kernel engineering. And I'm sure the labs would tell you that they already do a lot of their kernel engineering in a fully automated way. So software as an edge. If I [00:41:13] S0: think about software as maximally near speed of light efficient usage of an underlying piece of hardware is going to trend towards not being an advantage for a company like yours over time.
00:41:13 · Patrick O'Shaughnessy, Neil Movva
[00:41:13] S0: think about software as maximally near speed of light efficient usage of an underlying piece of hardware is going to trend towards not being an advantage for a company like yours over time. [00:41:22] S2: I think that's right. The rising tide of something like Mythos or GPT 5.6 SOL, that lifts all boats. It really does. I actually don't think there's a point in specializing to say we work on making the model better for just kernel engineering. I think that's actually not the most meaningful subset of coding in general, kernel engineering in particular. Maybe there's some privileged information you inject into the prompt that's like a useful way to steer the model to be better at writing kernels. But broadly speaking, yes, we're all downstream of the frontier in terms of this capability. [00:41:52] S0: I always love this from the history of energy. There's always this pendulum between the raw source, let's say coal, and then if there's a certain amount of energy available inside of a hunk of coal, what percent of it we can harness and use? A big part of the history of energy was getting that number from 10% to 95% or whatever. If I just think about it at Blackwell or something and Blackwell is the piece of coal, what percent do you think we're at? How efficiently can we use an existing piece today?
00:41:52 · Patrick O'Shaughnessy, Neil Movva
[00:41:52] S0: I always love this from the history of energy. There's always this pendulum between the raw source, let's say coal, and then if there's a certain amount of energy available inside of a hunk of coal, what percent of it we can harness and use? A big part of the history of energy was getting that number from 10% to 95% or whatever. If I just think about it at Blackwell or something and Blackwell is the piece of coal, what percent do you think we're at? How efficiently can we use an existing piece today? [00:42:19] S2: There's a lot of different ways to analyze that. In some level, we are really efficient at optimizing the performance when the GPU is doing the thing that it's most happy doing, which is a large dimension matrix will apply. That operation runs at 70, 80% of peak utilization and it's limited not by software but by power. The way NVIDIA goes peak FLOPS is a little optimistic. You never hit that because of power throttling. [00:42:42] S0: Because of heat.