Invest Like the Best with Patrick O'Shaughnessy
01:11:23 · Neil Movva, Patrick O'Shaughnessy
[01:11:23] S2: Yeah. It's millions of dollars. [01:11:25] S0: Yeah. So what's the world in which we consume what currently costs $5,000,000 per person per day? [01:11:29] S2: We were asking for at least three to six orders magnitude improvement in cost per token. Get that into 5,000, you probably have some customers. And in fact, would argue that for some size of model, we are approaching a trillion tokens being measured in tens of thousands of dollars. And that's something that you could imagine running for a single job. [01:11:48] S0: Are you at all worried that just like the average person just can't and won't do that? Doesn't do that now with their own brain? There actually isn't that much demand for intelligence in the world? [01:11:58] S2: I never will believe in that. There's always demand for intelligence in the world. The on ramps to that intelligence are our challenge as a product community. I'm not a product person, so I cannot say I had the best vision. [01:12:08] S0: Okay, you wanna enable those people. [01:12:09] S2: I wanna enable those people. I want them to never be held back by the sense that my free tier users, I can't afford to give them this many tokens. And I hear that from my customers all the time. We wanna fix that. [01:12:18] S0: What about the inverse question? Not what you think is craziest, but like what consensus thing you think is wrong?
01:12:18 · Patrick O'Shaughnessy, Neil Movva
[01:12:18] S0: What about the inverse question? Not what you think is craziest, but like what consensus thing you think is wrong? [01:12:23] S2: One of the things I keep coming back to is this question of NVIDIA. I am bullish on NVIDIA in the short term and NVIDIA, should never bet against them. They're always gonna reinvent themselves. Fundamentally, I think one thing that surprises people is when I tell them that, hey, if you look at Hopper and a Blackwell to Rubin and you compare like for like, what is the performance per watt of b float 16 multiply, it hasn't improved all that much. Or you take that one step further, go to TSMC. If you look at TSMC five nanometer versus four versus three versus two, the performance per watt on these chips doesn't change like a dramatic amount. The consequence of this is people lose their minds over geopolitics like what would happen if we lost access to TSMC for any reason. My contrarian take is that it wouldn't be that bad. Supply would take a shock for sure, but the best processes that we have in the West, like Intel, not that far behind, at worst, like maybe two x, worst performance per watt. The gap is just far smaller than you would make it out to be if you follow like [01:13:18] S0: the chipboard dialogue. What else is happening in the AI world that is not in your path, meaning it's not like a component of this whole system that you would end up doing something in, that interests you most?
01:13:18 · Patrick O'Shaughnessy, Neil Movva
[01:13:18] S0: the chipboard dialogue. What else is happening in the AI world that is not in your path, meaning it's not like a component of this whole system that you would end up doing something in, that interests you most? [01:13:29] S2: Well, we're fully downstream of models. So the model people get to decide how to design their architectures. I don't have any input to OpenAI or Anthropic, but I can only pray that they go in the direction that is amenable to me or I have to like do my best to predict where I think they're gonna go and build my serving architecture accordingly, both software and hardware choices. They have, I think, the most interesting game in some ways to play. Once again, this is game actually like the profundity of the machine thinking and how consequential it is to decide to use something like sparse attention versus dense attention or how consequential it is to like use a different data type. We were training in Vivo at 16, but now we can train in FP eight or FP four, lower precision data types. That is just an arbitrary choice, feels like, but it has profound implications for what chips I can use and how I should build my hardware and think about the future of compute.
01:14:14 · Patrick O'Shaughnessy
[01:14:14] S0: If you had a 100 entrepreneurs in a room, all of whom wanted to create some new compute startup, and let's say they were specifically wanting to make hardware chips or systems or racks or whatever, what advice would you give them on how to orient their companies or like the type of company, not the specific choice they're making on a tech bed or something like this? Because it seems like we're gonna try everything and that will be great for the world. Some stuff will work. But if you had to give them advice on how to orient their business to be successful in this coming world, What advice would you give them?
01:14:42 · Neil Movva, Patrick O'Shaughnessy
[01:14:42] S2: It's all about the bottlenecks on supply chain. So you need to first convince me or convince an investor that you understand the like three to five bottlenecks that dictate modern chip supply. There's TSMC wafer capacity, there's HPM capacity and there's advanced packaging. Maybe a fourth one would be power. Where will you get the power? How will you build these racks? You should have a great answer to each of those four bottlenecks and how you're gonna work around them because it's all arbitraged at the end of the day. You're building a chip because you think that NVIDIA has made some choices that are difficult for them to change, which is true. NVIDIA makes a lot of choices that are difficult for them to change. They're not perfect, they're just really well balanced. You wanna be spiky. You wanna pick something and say, I think they've underpriced the impact of how short we're gonna be on HBM. We're gonna push really hard in some direction instead, which I do think is probably the thing to attack most. Why? There's no easy way to bring on a lot more fabs of memory. [01:15:32] S0: So it's gonna be a while until we have... [01:15:35] S2: The the boys in Boise don't love huge CapEx for cyclical. They've been burned on that many times. [01:15:40] S0: Conceivably, like, because of that shortage, the world is just gonna route around it by making everything else in the system more efficient? I think they're
01:15:40 · Patrick O'Shaughnessy, Neil Movva
[01:15:40] S0: Conceivably, like, because of that shortage, the world is just gonna route around it by making everything else in the system more efficient? I think they're [01:15:47] S2: gonna make everything else more expensive. I think that iPhones will cut their memory. IPhones are gonna go up in price. We're just gonna deal with it. [01:15:52] S0: Why doesn't NVIDIA go all the way to the end and sell tokens, you think? [01:15:57] S2: NVIDIA is really smart about this. They don't compete with their customers. NVIDIA takes a long view on everything. Why don't they even start with the Neo Cloud? Why don't they just sell compute at the back door? Well, NVIDIA is really good. Jensen is really good at making his friends billionaires. He's made CoreWeave a billion dollar company, a many billion dollar company. And there's no need for him to destroy that goodwill. He wants to create a diverse community of Neo Clouds and inference providers who are all jockeying to create demand for NVIDIA such that if any one of them decides to, I don't know, vertically integrate or go with AMD or any other option, he's got three more people hungry to fill that position. It's great to have competition amongst his buyers. [01:16:34] S0: My favorite closing question for everyone is what is the kindest thing that anyone's ever done for you?