Invest Like the Best with Patrick O'Shaughnessy
00:52:30 · Patrick O'Shaughnessy, Neil Movva
[00:52:30] S0: They all be in [00:52:30] S2: one spot. They all be in one spot for... No one wants to deal with cross data center training. The market has some lag in it. I think that the market still assumes that we had to go shake down those 100 megawatt and 10 megawatt data centers wherever we can bind them. It's still the attitude I hear from a lot of data center developers. But increasingly, we're seeing a few new thinkers realize that inference is gonna be suitable for these distributed one megawatt data centers. We're quite in agreement with that and we are very happy to buy small pools of compute across The United States and use that as our inference fleet. [00:53:00] S0: Give us a sense of literal physical size of Yeah. One megawatt versus 10. [00:53:04] S2: Yeah. Well, so this got really wonky with the advent of liquid cooling. Now you can pack insane levels of power density into a single physical rack. A megawatt of compute, you'd imagine this, like, massive data hall, like a huge warehouse, basically. And now you can actually pack that into around, like, eight racks worth of compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up.
00:53:04 · Neil Movva, Patrick O'Shaughnessy
[00:53:04] S2: Yeah. Well, so this got really wonky with the advent of liquid cooling. Now you can pack insane levels of power density into a single physical rack. A megawatt of compute, you'd imagine this, like, massive data hall, like a huge warehouse, basically. And now you can actually pack that into around, like, eight racks worth of compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up. [00:53:26] S0: That's a nugget. Your view would be that the future that you wanna help build is a whole bunch of different chips that can be used together that you can buy your buyer to eke out the most per chip. And that those chips can then be coupled in very small data centers to just do inference. Those two steps of a whole bunch of random compute, some of which is cheaper than it should be, your ability to eke more out of it and then small units of expression in the data center equals way cheaper intelligence. I certainly think so.
00:53:58 · Neil Movva, Patrick O'Shaughnessy
[00:53:58] S2: Yes. There's a lot of ways to access cheaper flops if you're able to be creative with what you take. And so one of the ways that I describe what we do is we will buy any chip anywhere in the world for any duration of time. That is a level of flexibility and liquidity that no one else has right now. We're very aggressive about putting our money where our mouth is and we will take any capacity and find a way to make it work in our fleet. And that is a big part of our advantage today. Long term, we had to create more of that advantage by investing in these data centers that other people are gonna be skeptical of. Because what's gonna happen when you set this army of a thousand small data centers versus the one big gigawatt data center? Few things, you're not going to have power redundancy quite often. You're not going have backup diesel generators on-site. Those are all very expensive. We cut all that overhead. We're not even going have redundant networking in a lot of cases. We're going to put these in facilities where we have good access to power, a single source of power. And we're gonna trench one line of fiber to these data centers. But we're not gonna have three lines of fiber with redundancy and failover in SLAs. It's just gonna go down sometimes. I won't be surprised if some of them get down to like 95% uptime. Which is bad. Very bad. That's fatal, atrocious for anyone else... [00:55:02] S0: Couldn't survive in a a big gig Watson. You'd have
00:55:02 · Patrick O'Shaughnessy, Neil Movva
[00:55:02] S0: Couldn't survive in a a big gig Watson. You'd have [00:55:04] S2: risk zero buyers for a data center that has 95% uptime. I'm that first buyer. I will buy 95% uptime. [00:55:10] S0: And the reason for that is because this background engine thing. If things are running in the background, you don't care? [00:55:15] S2: Partially, it's actually two things. One is that we have a really robust control plane that is gonna be fine handling any single failure in any single data center as long as it's not correlated with other data centers and I can just move the workload somewhere else, I'm cool with that. The failures happen at some rate and I am basically linearly happy with a data center that's 95% uptime versus 98% versus 99%. It's just linearly good or bad for me. You do need that async piece that I mentioned. We serve these long horizon agents because what happens when a request fails is that I'm gonna have to go find a new GPU to put that request on. And that means that for that single turn of the agent's work, it's working for an hour but then hits a roadblock because its GPU got pulled away. At that moment in time, that agent is gonna experience maybe like an extra minute or two or three, maybe even ten of latency. My argument is that my customers don't care because their agent was running for hours. [00:56:07] S0: They're sleeping.
00:56:07 · Patrick O'Shaughnessy, Neil Movva
[00:56:07] S0: They're sleeping. [00:56:08] S2: It doesn't matter. It doesn't matter if like a single turn occasionally becomes a little bit longer. We tell our customers, look, our average throughput is gonna be very competitive, but our p 99, our 99 percentile latency, is not gonna be controlled. It cannot be. And in return, I'll give you unbeatable economics. And I think that's right fit for background agents. [00:56:25] S0: Talk about power as a category. What the innovation that you're seeing is? Where do you think it goes from here? What are you seeing that's interesting, innovative? Where do you think this goes?
00:56:33 · Neil Movva, Patrick O'Shaughnessy
[00:56:33] S2: Okay. So I said I want 95 uptime on my data centers. Could I even take 80% uptime at the right price? Probably. And what does that mean? Well, I'm a son of California. I love solar and wind. I think solar and wind power is way under tapped in The United States and the challenge has always been this intermittency. You wouldn't even consider solar and wind unsuitable for data centers because you have a persistent base load and the intermittent power source. What are you gonna do? Well, I think we're actually not that far from solving that problem. I am totally capable of tolerating a outage from a data center that's measured in even days or weeks, which is like the worst case nightmare scenario for a data center is that we're gonna have long term outage because the wind doesn't blow and the clouds are in the sky. Fog is hanging over the valley for some time. That's worst case scenario. It's in fact highly predictable and I can just call in capacity in some other place of the world whenever that happens. I'll just model the weather and figure out when my data center is gonna be offline, move my network with somewhere else and it's fine. The trick is that it's gonna give me better access to power that no one else is gonna touch because it is so annoying to deal with that kind of outage. And if my chips are cheap enough, they're probably not gonna be NVIDIA racks. But if my chips are cheap enough, I don't mind the capital cost of having idle chips. [00:57:41] S0: I've heard you describe this entire system as like a scavenger strategy.