Invest Like the Best with Patrick O'Shaughnessy
00:57:41 · Patrick O'Shaughnessy, Neil Movva
[00:57:41] S0: I've heard you describe this entire system as like a scavenger strategy. [00:57:45] S2: Yeah, that's right. [00:57:45] S0: Unpack that analogy a little bit. [00:57:46] S2: Well, first we scavenge chips and then we scavenge power for those chips. The idea is in both cases, I do not wanna be bidding against Anthropic or OpenAI for compute capacity. I'm not gonna win against them and I don't want to. I want to be more creative and use the supply that they don't find legible today. And over time, I amass enough aggregate supply. I'm never gonna get concentrated supply. I will only get aggregate supply. And over time, I build my aggregate factory that is unbeatable in economics. We are building a factory. We're trying to build the best steel factory in the world, but it will come through mini mills, not through large monolithic steel plants.
00:57:46 · Neil Movva, Patrick O'Shaughnessy
[00:57:46] S2: Well, first we scavenge chips and then we scavenge power for those chips. The idea is in both cases, I do not wanna be bidding against Anthropic or OpenAI for compute capacity. I'm not gonna win against them and I don't want to. I want to be more creative and use the supply that they don't find legible today. And over time, I amass enough aggregate supply. I'm never gonna get concentrated supply. I will only get aggregate supply. And over time, I build my aggregate factory that is unbeatable in economics. We are building a factory. We're trying to build the best steel factory in the world, but it will come through mini mills, not through large monolithic steel plants. [00:58:20] S0: And if I imagine the different versions of this, how vertically integrated you can be, the extreme would be you own everything. So that's a very capital intensive business. You own the power source. You build the data centers. You design your own chips, you control the software that eats the most out of those chips and you sell the endfinished token to your user. Your user is me and you just own the whole stack. But you can imagine many other permutations of the business where draw the line anywhere, you could be incredibly capital light, own nothing, and just be like the coordination plane across all this stuff, the virtual scavenger. How do you think about that question of which type type of these businesses to be?
00:58:20 · Patrick O'Shaughnessy, Neil Movva
[00:58:20] S0: And if I imagine the different versions of this, how vertically integrated you can be, the extreme would be you own everything. So that's a very capital intensive business. You own the power source. You build the data centers. You design your own chips, you control the software that eats the most out of those chips and you sell the endfinished token to your user. Your user is me and you just own the whole stack. But you can imagine many other permutations of the business where draw the line anywhere, you could be incredibly capital light, own nothing, and just be like the coordination plane across all this stuff, the virtual scavenger. How do you think about that question of which type type of these businesses to be? [00:59:01] S2: You know, there's actually two parts to me to receive that question. One is the CEO of a company that needs to work every day and grow sustainable and as quickly as it possibly can. The other is the founder. And the founder is much more imaginative and loves this stuff. The founder in me wants to do everything. This is my entire life. I spent my entire life thinking about chips, power, energy. All I care about is this stuff. So of course, I wanna be maximally ambitious. I don't wanna stop ever. I will never stop until I have built the most efficient system from soup to nuts. [00:59:27] S0: You're doing real life factorio, basically.
00:59:27 · Patrick O'Shaughnessy, Neil Movva
[00:59:27] S0: You're doing real life factorio, basically. [00:59:29] S2: Very much so. For intelligence. Very much so. So that's like the emotional from the hard answer. On the CEO side, I think we had to be more pragmatic. The capital we're looking at for owning everything is insane. Software has high leverage, so we have to start with software. But ultimately, do we own power generation or can we get great power purchase agreements with utilities? I'm more inclined to pursue letting other people specialize in the things that they're historically good at and then see if we can get to the scale. I think of it as like, I want to get to the scale where I earn the right to take this under our wing. I absolutely think that there is efficiencies to be gained everywhere in the stack if you can break the assumption that people I would be buying from, they made assumptions about who their customers would be and I maybe break those assumptions. It's a pretty optimistic view. I think it's only possible because we're actually trying to underwrite the largest market for compute in the history of computing. We're actually gonna build so many billions, trillions of dollars of investment into inference. Because of that focus, it makes sense to build a lot of things that are custom for inference. And it's my job to seek all the places where that's possible. And then as as it become obvious to me and my partners, I look at my partners to build custom things for me. And if they can't do it for me, I will do it myself. [01:00:40] S0: If you had to just zoom out on this entire system
01:00:40 · Patrick O'Shaughnessy, Neil Movva
[01:00:40] S0: If you had to just zoom out on this entire system [01:00:43] S2: Yes. [01:00:43] S0: Software, hardware, energy, etcetera and stack rank the places that you think that we are the most inefficient today at producing useful intelligent tokens, what does that list look like?
01:00:54 · Neil Movva
[01:00:54] S2: I think compute scaling is actually very efficient. You give me more FLOPS and I will use more FLOPS. I would say we're actually fairly judicious already with our use of FLOPS. If you look at a modern MOE model, there are very few models that are more than 10% dense, meaning 10% of the possible number of experts you can activate are activated. And I think the frontier models are closer to like 1%. Fairly sparse already. I don't think that we're wasting too much on the MOE side. People have been working with MOEs for quite some time. They're pretty good at squeezing MOEs. Where we are not good is attention and it's use of memory specifically. The KV cache is quite uncompressed right now. I think if you look at the entropy in a KV cache, it's not earning its keep. Like we're storing many kilobytes of data in the KV cache per token. And that's probably off by an order of magnitude or two. I don't know the frontier labs do, but DeepSeq certainly publishes really interesting work to compress that further and further. And they're making good progress. And I think the fact that they're able to make an order of magnitude and progress here every year or so signals that there's a lot more room to go. If you zoom out further, I think that we actually don't marshal our compute effectively at all. We have all this compute in the world. Nvidia is pumping out 5,000,000 black wall chips this year. Where are they all going? Are they all being used at all all the time? I certainly doubt it. I think that at some level, we just need better orchestration of compute across the world. This is very difficult to do because a lot of the compute disappears into private pools of compute that will never see the light of day and those GPUs sit very sadly idle. It pains me physically to see that those GPUs are just silicon and power going into that and it's just sitting idle. And I wanna fix that. How we organize and orchestrate the world's compute as a shared resource and pack it more efficiently. I would estimate that we all make fun of x AI for having, you know, some challenges with total flop utilization on its clusters. The reality for the rest of the world is it's far worse. A ton of GPUs just sit in warehouses or sit in private pools allocated to a specific customer just don't