Invest Like the Best with Patrick O'Shaughnessy
00:51:15 · Neil Movva
[00:51:15] S2: I think one of the themes in our conversation has come back to what is training versus inference? Like, what is the difference between these two categories? And, you know, what was different about two years ago being training oriented and today being inference oriented? And I think the most conservative players in the entire AI stack have got to be the infra players, whether it's data centers or even more conservatives TSMC, that chip in for people. Data centers, they were built for training. Training is a super set workload over inference. You can make any training cluster work for inference, but maybe not vice versa. The difference there is networking. How much do you invest in bandwidth between chips and how large of a cluster do you need? There's actually a diseconomy of scale to data centers in some way. It's way more expensive and difficult to build a 100,000 GPUs in one data center than it is to build 10,000 than it is to build 1,000. Now we just talk about how many megawatts or gigawatts do you have. And basically, there's no way to build a gigawatt data center in The United States easily anymore. Even a 100 megawatts is increasingly hard. It's basically impossible unless you're a very special set of customers. 10 megawatts is probably on the edge of what's possible today. And one megawatt, I would argue, is plentiful. So there's this incredible floor on the market where you can find lots of aggregate power, but it will not be concentrated. And that was not interesting to anyone who's building training data centers because you just assume...
00:52:30 · Patrick O'Shaughnessy, Neil Movva
[00:52:30] S0: They all be in [00:52:30] S2: one spot. They all be in one spot for... No one wants to deal with cross data center training. The market has some lag in it. I think that the market still assumes that we had to go shake down those 100 megawatt and 10 megawatt data centers wherever we can bind them. It's still the attitude I hear from a lot of data center developers. But increasingly, we're seeing a few new thinkers realize that inference is gonna be suitable for these distributed one megawatt data centers. We're quite in agreement with that and we are very happy to buy small pools of compute across The United States and use that as our inference fleet. [00:53:00] S0: Give us a sense of literal physical size of Yeah. One megawatt versus 10. [00:53:04] S2: Yeah. Well, so this got really wonky with the advent of liquid cooling. Now you can pack insane levels of power density into a single physical rack. A megawatt of compute, you'd imagine this, like, massive data hall, like a huge warehouse, basically. And now you can actually pack that into around, like, eight racks worth of compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up.
00:53:04 · Neil Movva, Patrick O'Shaughnessy
[00:53:04] S2: Yeah. Well, so this got really wonky with the advent of liquid cooling. Now you can pack insane levels of power density into a single physical rack. A megawatt of compute, you'd imagine this, like, massive data hall, like a huge warehouse, basically. And now you can actually pack that into around, like, eight racks worth of compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up. [00:53:26] S0: That's a nugget. Your view would be that the future that you wanna help build is a whole bunch of different chips that can be used together that you can buy your buyer to eke out the most per chip. And that those chips can then be coupled in very small data centers to just do inference. Those two steps of a whole bunch of random compute, some of which is cheaper than it should be, your ability to eke more out of it and then small units of expression in the data center equals way cheaper intelligence. I certainly think so.
00:53:58 · Neil Movva, Patrick O'Shaughnessy
[00:53:58] S2: Yes. There's a lot of ways to access cheaper flops if you're able to be creative with what you take. And so one of the ways that I describe what we do is we will buy any chip anywhere in the world for any duration of time. That is a level of flexibility and liquidity that no one else has right now. We're very aggressive about putting our money where our mouth is and we will take any capacity and find a way to make it work in our fleet. And that is a big part of our advantage today. Long term, we had to create more of that advantage by investing in these data centers that other people are gonna be skeptical of. Because what's gonna happen when you set this army of a thousand small data centers versus the one big gigawatt data center? Few things, you're not going to have power redundancy quite often. You're not going have backup diesel generators on-site. Those are all very expensive. We cut all that overhead. We're not even going have redundant networking in a lot of cases. We're going to put these in facilities where we have good access to power, a single source of power. And we're gonna trench one line of fiber to these data centers. But we're not gonna have three lines of fiber with redundancy and failover in SLAs. It's just gonna go down sometimes. I won't be surprised if some of them get down to like 95% uptime. Which is bad. Very bad. That's fatal, atrocious for anyone else... [00:55:02] S0: Couldn't survive in a a big gig Watson. You'd have
00:55:02 · Patrick O'Shaughnessy, Neil Movva
[00:55:02] S0: Couldn't survive in a a big gig Watson. You'd have [00:55:04] S2: risk zero buyers for a data center that has 95% uptime. I'm that first buyer. I will buy 95% uptime. [00:55:10] S0: And the reason for that is because this background engine thing. If things are running in the background, you don't care? [00:55:15] S2: Partially, it's actually two things. One is that we have a really robust control plane that is gonna be fine handling any single failure in any single data center as long as it's not correlated with other data centers and I can just move the workload somewhere else, I'm cool with that. The failures happen at some rate and I am basically linearly happy with a data center that's 95% uptime versus 98% versus 99%. It's just linearly good or bad for me. You do need that async piece that I mentioned. We serve these long horizon agents because what happens when a request fails is that I'm gonna have to go find a new GPU to put that request on. And that means that for that single turn of the agent's work, it's working for an hour but then hits a roadblock because its GPU got pulled away. At that moment in time, that agent is gonna experience maybe like an extra minute or two or three, maybe even ten of latency. My argument is that my customers don't care because their agent was running for hours. [00:56:07] S0: They're sleeping.
00:56:07 · Patrick O'Shaughnessy, Neil Movva
[00:56:07] S0: They're sleeping. [00:56:08] S2: It doesn't matter. It doesn't matter if like a single turn occasionally becomes a little bit longer. We tell our customers, look, our average throughput is gonna be very competitive, but our p 99, our 99 percentile latency, is not gonna be controlled. It cannot be. And in return, I'll give you unbeatable economics. And I think that's right fit for background agents. [00:56:25] S0: Talk about power as a category. What the innovation that you're seeing is? Where do you think it goes from here? What are you seeing that's interesting, innovative? Where do you think this goes?