Invest Like the Best with Patrick O'Shaughnessy
01:11:49 · Gavin Uberti, Rob Wachen, Patrick O'Shaughnessy
[01:11:49] S1: Another one. Another one. [01:11:51] S3: Yeah. There's definitely a find a way mentality. If you're here, you're here because you assume it's possible. So, like, we can't be saying it's impossible. Everything is solvable, and we're just going to work at it until we figure it out. A favorite story I have, this guy who's kind of a legend in silicon validation, who joined our team and we were doing the early stages of what's called wafer sort. When your chips are coming out of fab, they go out on these wafers and you have this thing called a probe card that attaches to the wafer before you dice it with these probe pads and you send these electrical signals to basically test which chips are good and bad. So when I slice the wafer into a bunch of chips, I can package it and only package the good ones. We go through our first wafer. It's like two or 3AM because we're doing it with TSMC over the phone in Taiwan. We have the screen with the wafer that's all gray and each ship is gray. And then as you start running the patterns, the squares are supposed to turn green or red. And they all turn red. We're like, fuck. Like, this is really bad. Like, everybody's like, guys, take a breath. He leads back. He's like, the puzzle begins. [01:12:52] S1: Like that you have to have the attitude of like, yes, you will go out and you will go stare into the abyss. You will go see scary things, and we'll solve them. [01:13:01] S0: When did you see the first Green Square?
01:13:01 · Patrick O'Shaughnessy, Rob Wachen
[01:13:01] S0: When did you see the first Green Square? [01:13:03] S3: Within a day of that. But in the moment, it's extremely scary, and there's a certain type of person who just like is addicted to that feeling, of just feeling the fear and solving it, and we're lucky to have lot of those people here. [01:13:15] S0: If you think about applying all of this earned know how from this last several years and now thinking ahead to Gen two, Gen three, beyond, what will you be doing most differently as a result of everything that you've learned? Just like from a conceptual standpoint, like the way that you will attack designing and producing this next one, based on what you learned doing it the first time?
01:13:36 · Rob Wachen
[01:13:36] S3: It took us a while to get to the primitives that we think are really what matters for scaling inference. We tried a bunch of things early on from compilers that would turn different models into FPGAs, to burning weights in silicon, to splitting your HBM, to KV cache and weights, and all of these different things. And there was a lot of cycles of learning till we got to the point that we realized that like, fundamentally, if you want to run a majority of tokens in the world, you need to do three things. You need to build a chip with the most flops and given power budget, you need to build a chip that has the lowest latency between other chips, so the biggest scale up domain possible, and you need to produce as much of it as possible. And I think probably in the first half of our journey so far, we learned the first two and that informed the design a lot and that informs a lot about the bets we're making in the future with the low voltage inference and the cluster scale memory. But the production part, I think, in the past year has become extremely obvious how much people want to deploy this stuff if you can have it available today. The best ability is availability. If I have a thousand chips a day, someone's gonna use them. And, you know, we need to build a chip that's not just like way better than what's been built before, but it needs to be available at many gigawatt scale. We need to be able to be building a product that is producible at gigawatts per month than the limit. As we think about that, a lot of the design decisions we're making with our next gen, which you've seen already, is just about simplicity. Removing tons of parts, trying to assemble and disassemble a thing again and again, and learning how to make it as quick in the cycle times as possible in production, making sure it's gonna be reliable, making sure it's gonna be serviceable, and making sure it's gonna be producible at gigantic scales.
01:15:08 · Patrick O'Shaughnessy, Rob Wachen
[01:15:08] S0: What about other problems in the ecosystem that are outside of your control, such as capacity at the leading nanometer at TSMC or availability of HBM4 memory or some of these other things where everyone is fighting for a scarce unit of capacity or whatever. How do you face up against those realities when you're trying to produce as much as humanly possible? [01:15:31] S3: The people deploying the most compute in the world do think about supply a bit zero sum, which is there's only so many wafers being produced on a given nanometer node on a given fab, right? And there's only so much memory being produced, and that's why actually for our first gen product, we built it on a different supply chain than the Rubens. We're on four nanometer, Rubens on three nanometer. Yeah. We're on a different HPM than Rubens and so forth. So it actually is not a zero something. It's a positive something where more is more. So often when we're talking to people deploying at scale, it's not a decision between a gigawatt of a GPU and a gigawatt of us, it's two gigawatts. And I think as much as possible, thinking about supply chain early in the design decisions because if you have the most performing product and you can't produce it, then you're just a podcast.
01:15:31 · Rob Wachen, Gavin Uberti
[01:15:31] S3: The people deploying the most compute in the world do think about supply a bit zero sum, which is there's only so many wafers being produced on a given nanometer node on a given fab, right? And there's only so much memory being produced, and that's why actually for our first gen product, we built it on a different supply chain than the Rubens. We're on four nanometer, Rubens on three nanometer. Yeah. We're on a different HPM than Rubens and so forth. So it actually is not a zero something. It's a positive something where more is more. So often when we're talking to people deploying at scale, it's not a decision between a gigawatt of a GPU and a gigawatt of us, it's two gigawatts. And I think as much as possible, thinking about supply chain early in the design decisions because if you have the most performing product and you can't produce it, then you're just a podcast. [01:16:15] S1: That's the other big thing about vertical integration too, is well, certain things like for the chips and for the memory, have to go ahead and partner. For most of the other stuff, those are also very highly in demand components. And the more that you build yourself, the more stuff you can go do on top of what the world can currently build. It is not, oh you're taking compatibility with somebody else, you're adding way, way more. I think that's how you win.
01:16:15 · Gavin Uberti, Patrick O'Shaughnessy
[01:16:15] S1: That's the other big thing about vertical integration too, is well, certain things like for the chips and for the memory, have to go ahead and partner. For most of the other stuff, those are also very highly in demand components. And the more that you build yourself, the more stuff you can go do on top of what the world can currently build. It is not, oh you're taking compatibility with somebody else, you're adding way, way more. I think that's how you win. [01:16:37] S0: One of the things that I realized we haven't talked at all about is the models themselves, which is kind of crazy, the things behind all of this demand. Anything interesting that you would say about the way that you see models progressing based on what we've seen so far? I guess I'm more interested in how you as thinkers about hardware think hardware might impact where the models themselves go in the future.