Invest Like the Best with Patrick O'Shaughnessy
00:40:16 · Patrick O'Shaughnessy, Neil Movva
[00:40:16] S0: Why are humans still doing this? Like, seems like the sort of thing that AIs would be exceptionally good engineering, more efficient kernels. Maybe that's where we're going and we're just not quite there yet. But if we aren't there yet, is that where we're going? If we're not there yet, why humans still doing this? Why why is TreeHouse so well known? You know, it's a name I know. [00:40:32] S2: I don't want to speak for Tree, but what he taught me was you shouldn't write kernels by hand anymore necessarily. I like to say we write kernels on the whiteboard. We go to the whiteboard, we describe what we think the machine should be doing, then we succinctly describe that in natural language to a model And then the model is able to do the execution of here's my input and output, here's the strategy of how we want to dispatch this work onto the GPU. I'm gonna go implement... [00:40:56] S0: So we're doing the conceptual design. [00:40:57] S2: Exactly. I'm not sure exactly why models are not superb at doing this. I don't think this is like our mode or anything like that. I'm sure in six months time, we'll have much better models on kernel engineering. And I'm sure the labs would tell you that they already do a lot of their kernel engineering in a fully automated way. So software as an edge. If I [00:41:13] S0: think about software as maximally near speed of light efficient usage of an underlying piece of hardware is going to trend towards not being an advantage for a company like yours over time.
00:41:13 · Patrick O'Shaughnessy, Neil Movva
[00:41:13] S0: think about software as maximally near speed of light efficient usage of an underlying piece of hardware is going to trend towards not being an advantage for a company like yours over time. [00:41:22] S2: I think that's right. The rising tide of something like Mythos or GPT 5.6 SOL, that lifts all boats. It really does. I actually don't think there's a point in specializing to say we work on making the model better for just kernel engineering. I think that's actually not the most meaningful subset of coding in general, kernel engineering in particular. Maybe there's some privileged information you inject into the prompt that's like a useful way to steer the model to be better at writing kernels. But broadly speaking, yes, we're all downstream of the frontier in terms of this capability. [00:41:52] S0: I always love this from the history of energy. There's always this pendulum between the raw source, let's say coal, and then if there's a certain amount of energy available inside of a hunk of coal, what percent of it we can harness and use? A big part of the history of energy was getting that number from 10% to 95% or whatever. If I just think about it at Blackwell or something and Blackwell is the piece of coal, what percent do you think we're at? How efficiently can we use an existing piece today?
00:41:52 · Patrick O'Shaughnessy, Neil Movva
[00:41:52] S0: I always love this from the history of energy. There's always this pendulum between the raw source, let's say coal, and then if there's a certain amount of energy available inside of a hunk of coal, what percent of it we can harness and use? A big part of the history of energy was getting that number from 10% to 95% or whatever. If I just think about it at Blackwell or something and Blackwell is the piece of coal, what percent do you think we're at? How efficiently can we use an existing piece today? [00:42:19] S2: There's a lot of different ways to analyze that. In some level, we are really efficient at optimizing the performance when the GPU is doing the thing that it's most happy doing, which is a large dimension matrix will apply. That operation runs at 70, 80% of peak utilization and it's limited not by software but by power. The way NVIDIA goes peak FLOPS is a little optimistic. You never hit that because of power throttling. [00:42:42] S0: Because of heat.
00:42:42 · Patrick O'Shaughnessy, Neil Movva
[00:42:42] S0: Because of heat. [00:42:43] S2: Exactly, thermals. In practice, you don't spend the majority of your time in a transformer in that happy path where you're doing a large batch. Matrix will apply. Our job is to basically build the engine around the chip such that we are feeding the GPU these large batches of work at all times. One of the most profound transitions we've had in the GPU world in the last year has been this moving of you don't program one GPU at a time anymore. You should think about the whole rack and maybe you should think about the whole cluster, the entire data center at a time. With NVIDIA, they've started shipping not just a single GPU or a single motherboard, but actually the whole rack system is something that they prescribe. They call it NVL 72. Their latest chip, the Grace Blackwell 300, that chips as a rack of 72 units and it's an open race to figure out who can program the whole rack scale computer as efficiently as possible. And my belief is that that shape of compute is the future of efficiency and speed. In fact, NVIDIA does a great job of if you want the lowest possible latency, you should be using that chip. And if you want the highest possible throughput, you should probably also be using that chip as of right now. And it all comes down to like, this is a very new paradigm of programming. [00:43:43] S0: One of the things you hear is that the market for the best chips, Blackwell's let's say [00:43:48] S2: Yes.
00:43:48 · Neil Movva, Patrick O'Shaughnessy
[00:43:48] S2: Yes. [00:43:48] S0: Is like a drug market or something right now. There's all sorts of fascinating things happening to get as many of them as possible because everyone's so short. It would be to react to that analogy, is that what it feels like? Then also to talk about what the market is like for, like, not the bleeding edge chips. If I am willing to accept a slightly or moderately inferior chip, what's that market like? Let us into that world. Media has
00:44:10 · Neil Movva
[00:44:10] S2: a long term view on all their chips. They see this immense demand for the black hole chips. They can do what other suppliers have done in the past, which is crank prices, made the market supply and demand curves well correct. They'll intersect at some point and everyone will be technically happier. But NVIDIA sees if they just let the most deep pockets buy all the chips that maybe hurts them in the long term if that customer ends up accruing a lot of more power. They understand that compute is power today. They're quite strategic about how they allocate compute. That's the first thought. The second thought is that relationships matter a lot. Nobody wants to have a huge order of a chip rental come in from this new startup that says, oh yeah, I'm gonna rent 10,000 black wells for three years or five years. The startup has only been operating for months, typically. Who knows that they're good for the money? The way you convince someone to give you access to compute is quite challenging these days and requires some pretty great relationships or just incredible financial backing to make this happen on the NVIDIA side. And it's all because the scarcity is so high and demand is just off the charts. Now for other chips, I wouldn't even call them inferior. I like to say there's no bad chips, there's really bad pricing. I will make any chip work at the right price. That's one of the ethos of the company. Let's talk about AMD. AMD, I think great chips overall. The challenge is that people don't understand how to program them very well. I've been talking to you about how we have such a great kernel team. We're so serious about squeezing the performance out of the hardware. NVIDIA is pretty good at doing that for their own chips, frankly. There's some alpha that we can squeeze out, but there's a lot more to be done other chips because the vendor does a little bit less work than NVIDIA does to make the best kernels out of the box. Or even better, there is alpha and just other people have this perception that AMD is not as good as NVIDIA. That's music to my ears. I'm very happy for them to sleep on this chip and for me to buy as much as I can. Now I think that that's not actually super true anymore. I think AMD is actually somewhat popular amongst some large buyers. I think publicly, Meta and OpenAI have bought a ton of AMD chips.