Invest Like the Best with Patrick O'Shaughnessy
00:40:24 · Gavin Uberti, Rob Wachen, Patrick O'Shaughnessy
[00:40:24] S1: In that particular case too, that was a big part of it. Also, like, shift, I think, made a big difference too. Totally. We went out and literally had a day shift and a night shift. There were team members who would come in at 10AM and leave at around midnight. She would come in at midnight and leave at 10AM. You're running around the clock to get to that forty days. [00:40:43] S3: Yeah, I mean over half the company lives next to the office, so it makes it easier to do that type of thing. [00:40:47] S0: You pay them to do that, right? Pay them extra, do you still do that? [00:40:50] S3: The invisible hand does wonders. [00:40:52] S1: I mean, works for me too. [00:40:54] S3: We're both there. [00:40:55] S0: I'd love to take one big step back and talk a bit about just the broader ecosystem here. The amount of shortages on the supply side, the exposure of risks in the global system and the supply chain around this stuff has become like everyday Wall Street Journal front page news. Like the stocks that people are watching and investing in and excited about, about the memory stocks. These were boring commodity nothing burgers five years ago. And now they're the center of global attention. If you just assess because you've been building in it, the global connected supply chain that's required to make stuff like this possible, just riff on it. Like what scares you? What's working well? What needs to change? What do you hope you change by virtue of how you build this thing? What's your assessment of this story right now?
00:41:42 · Gavin Uberti
[00:41:42] S1: I think that one of the most undervalued pieces of a supply chain story is almost none of these things are you buy them and they don't talk to the vendor again. You have to go collaborate. That is the most important part, being successful, I think, in ships with TSMC or with memory vendors. You need that partnership. I think that for TSMC in particular, people don't understand why it is so valuable. People look at the tech and the tech is the best in the world. But for me, the real value is all in the service. TSMC customer service is way, way better than I have seen at any other company in any other industry. It's the kind of thing where if you say, hey, you can improve your yield by making this change, You can go make them a recommendation and then we'll go run an experiment on their own dime in our case to see if they could actually get the higher yield. And when we found that we were right and the experiment worked, they moved over the rest of the line. And that kind of thing doesn't happen in most industries. If I go to, like, the steelworks plant and say, hey, I want you to change the composition of the steel, and they'll say, screw you. Not TSMC. It is why they are the number one and why they're going to win.
00:42:51 · Rob Wachen, Patrick O'Shaughnessy
[00:42:51] S3: One of the things that matters a ton is power availability and time to power. And the problem is the more power you want, the more shortage there is. It's actually very similar to chip clusters, which is like, why is Colossus charging $12 an hour for black holes? It's because they're the only place you can buy 20,000 of them at once. Right? Why is the 500 megawatt data center so hard to find? It's the exact same reason. You know, one of the things we need to think about is, like, how do we get way more juice out of each megawatt? People are looking throughout the entire stack, whether it's just improving the PUE, but also entirely new hardware to get the most tokens per megawatt to solve this problem. But fundamentally, building new buildings is hard. It's much easier to go from a 100 megawatt to a gigawatt from a gigawatt to ten and ten to a 100, and we are pushing the limits of what's possible on these timelines. So there's a lot of people trying to scale in their data centers as much as trying to scale them out. [00:43:41] S0: One of the interesting things about a system like this is what it replaces. Yeah. If I think about a rack like this versus, I don't know, a set of Blackwells or something or Rubens or whatever is coming next, how should I conceptualize that? It's not just Watts. It's also physical space. The Rebbe's talked about this in their recent reviews called that literally this is a problem, that there's literally no space to put the systems. How should I conceptualize what this represents or replaces in terms of other units of compute?
00:44:06 · Rob Wachen
[00:44:06] S3: Here's how customers think about deploying models generally, which is when I'm building a data center or I'm building a cluster, it's not like in the abstract of like, oh, I like these chips and like, yeah, this is the power of footprint and so forth. It's like, I have a real production workload I'm trying to serve and for my product to be useful, there's a certain speed I need to serve for that. And for certain products, it's really fast and certain products, it's really slow. Whatever the speed is, this is my speed. The question is, in a given amount of power, how many users can I serve while guaranteeing that speed? So another way to put it is ISO, what's called interactivity, what is my throughput? We are just finishing kind of the early innings of the AI infrastructure boom where people really just cared about speed. You know, GPUs were not able to reach a lot of the speeds of other types of chips like all these SRAM chips, thousands of tokens per second, and that enabled tons of new use cases that got people very excited. There's an entirely new wave of AI chips, us being one of them, that are all going to be able to hit these speeds. The question then is, if you're hitting these speeds, what is the number of users you can serve at the same time? And by proxy, if I have a 100 megawatt data center, how many software agents can I run at the same time? So when people are doing that evaluation, our hardware is going to generally be able to get you an order of magnitude more concurrency at a given level of interactivity. So that directly translates into tokens per watt, tokens per dollar, all the things people care about when they're actually serving these giant mixture of expert models at scale. There's these now famous interactivity curves. Not many people publish them, but you
00:45:32 · Patrick O'Shaughnessy, Gavin Uberti
[00:45:32] S0: can see a Blackwell curve. You can see an A and B curve, which is a little bit worse than Blackwell's, and it's still an $800,000,000,000 company. So if you think about what then the impacts are of shifting that curve, not just a little bit further out, but much further out, what are the things that most excite you about what this will enable? I I want to go out and [00:45:52] S1: solve some of the hardest problems, and I want to go solve these in much less time. There were things growing up that I was not sure I'd be able to live to see. For example, the immunodysics conjecture, one of the things that we talked about in college, and I was not sure I'd see that proven in my life, And this was done by an AI model and it was done over a long period of time. But if you're able to then run the same model 10 times faster, you can go shrink the time to go have these breakthroughs. And there's a huge number of other problems in math like this as well that I worry it will take a thousand years to go prove a thing like this. You'd either have a much smarter model or a model of the same intelligence running much faster. You can then shrink that and I can see it. It's so cool seeing these breakthroughs get made. I am so, so excited to see much more of this happen.
00:46:36 · Rob Wachen
[00:46:36] S3: I think too often people think about tasks and applications and stuff in these very short time horizons. Doing a chat and it's like 50% faster is nice, but it's not like game changing. As these agents go longer and longer time horizon and the models get more and more capable, you're going to see gigantic bodies of work that would take months of compute. And we think about this in wall clock time. Like if you talk to a pre training researcher at a lab, they'll tell you that wall clock time often is one of the most important things that matters, and what wall clock time means is the time from, you know, starting your run to finishing it to actually get data back. If you can shrink this time from a six month run to a two month experiment, you're gonna be able to do many more iterations and people will make changes on the model architectures to actually improve the wall clock time. Very similar here in terms of how we think about the use cases, which is the exciting part about super low latency decode is wall clock time on long horizon tasks becomes much shorter. So a year long compute build would now take a month and that month long compute build will now take three days and that three day compute build will now take seven hours and so forth and so forth. So that's the thing that I think is really hard to internalize because the models are just getting capable enough to do this stuff. I thought it was really cool months ago when Cursor published that they had a bunch of coding agents build an entire browser from scratch in a week. Totally nuts. And that will soon happen in under an hour. And there's going to be many of those types of things that are going to happen with these massive parallel agents all working on a given task.