Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Krishna Rao
[00:00:00] S0: How are scaling laws talked about internally? The consensus has been you've got different components of them. You've got pre-training, post-training, reasoning, and all of these are moving at different paces, and to hit a true wall, they would all need to fall down. That's how the world is now conceptualizing scaling laws. How are they talked about internally? How do you think about them? [00:00:00] S1: We look at models at various points in their development, so we can see during a pre-training run, how does this model compare to a prior model that we did on these loss curves. That gives us a sense for model capability. You can do the same thing as you think about RL. Probably as important is when customers get their hands on it, what are they seeing? Where are they identifying pain points? Those pain points become training targets for us. We don't train on customer data on the enterprise side. On the prosumer side, it's only if you opt in. But customers tell us things like, "Hey, I wish the model were better at this," or, "I had this particular place where it got stuck, and I could build this other product, but the capability needs to be further than that." What we usually tell them is, "Build your product for that, because we're going to, on the R&D side, improve that over time." There is this connected loop, but internally we're always looking at different models being trained, different snapshots that we have, and comparing them internally and to a lesser extent externally against our own measures, and then ultimately how our customers view them as well.
· Krishna Rao, Patrick O'Shaughnessy
[00:00:00] S1: We look at models at various points in their development, so we can see during a pre-training run, how does this model compare to a prior model that we did on these loss curves. That gives us a sense for model capability. You can do the same thing as you think about RL. Probably as important is when customers get their hands on it, what are they seeing? Where are they identifying pain points? Those pain points become training targets for us. We don't train on customer data on the enterprise side. On the prosumer side, it's only if you opt in. But customers tell us things like, "Hey, I wish the model were better at this," or, "I had this particular place where it got stuck, and I could build this other product, but the capability needs to be further than that." What we usually tell them is, "Build your product for that, because we're going to, on the R&D side, improve that over time." There is this connected loop, but internally we're always looking at different models being trained, different snapshots that we have, and comparing them internally and to a lesser extent externally against our own measures, and then ultimately how our customers view them as well. [00:00:00] S0: And it feels like there's just no slowdown in the scaling laws themselves. Is that a fair characterization?
· Patrick O'Shaughnessy, Krishna Rao
[00:00:00] S0: And it feels like there's just no slowdown in the scaling laws themselves. Is that a fair characterization? [00:00:00] S1: For us, that's a fair characterization. Obviously, a bunch of the authors of the scaling laws papers are amongst our founders. Notwithstanding that, we can be a bit of a skeptical bunch. We hold ourselves to a really high standard. It's very scientific method, and people are constantly challenging previously held assumptions. But from what we see, the scaling laws are not slowing down. [00:00:00] S0: So if that's true — you said before it's hard for humans to be exponential in their thinking and not linear. If that continues to be true for however many more turns of the crank here — how do you do that thing of not thinking linear and thinking exponentially, yourself in your job and for the business? The implications are really hard to reason through. Exponential growth rate is one thing, but exponential growth of capability — I don't even know how to get my head around it. How do you get your head around it?
· Krishna Rao, Patrick O'Shaughnessy
[00:00:00] S1: We think about the world as scenarios. It's very hard to have a point estimate in this business. And then having a very low bar for updating your current priors, or your current perspective. It could be the case that something a month ago was true that's just not true today, and that breaks your model, and you have to go back and update it. This old, "Well, we'll forecast once a quarter and we'll revisit this in three months at the next board meeting" — that doesn't work for our business. It's so dynamic that we have to always think about, "Oh, our models couldn't do this before and they could do this now. What does that mean for the TAM?" We've seen this in coding first. Starting with around Sonnet 3.5, 3.6, we started to see this really remarkable jump in capability, which was then followed by adoption and usage and revenue. It was a little hard to predict that, but now we can use coding as an analog for a lot of what's happening elsewhere in the economy and elsewhere in our business. We look at pattern recognition in our own business to try to predict what's going to happen in the future. [00:00:00] S0: Literally 15 minutes before you got here, the news came out about your partnership with xAI and the Tennessee facility. It makes me curious about how you are canvassing the world. That is an opportunity you decided to do. I'm sure there's a universe of things that you've explored. What is the strategy for trying to get more in creative ways? Bring us a little more into that.
· Krishna Rao
[00:00:00] S1: We announced a partnership with SpaceX for their Colossus facility in Memphis. Really excited about that. It's going to allow us to continue to expand, especially on the consumer and prosumer side. That's just one example of us looking for near-term compute wherever we can get it. As the compute base grows, that near-term compute becomes a smaller and smaller fraction of what's available and what's out there. We look at it as, can we deploy that compute that's available productively? Sometimes the answer's yes, and sometimes it's no. But if we can, then we look at the economic return on it based on what it's priced, what duration we have it for, where it's located, what type of compute it is, and how efficiently we can run it. We have a process to assess. That same process, we use to assess longer-term deals as well. Last month, we signed a five-gigawatt deal with Google and Broadcom for TPUs starting in 2027. We also signed a deal with Amazon for Trainium, for up to five gigawatts as well. It was an over $100 billion commitment, and a lot of that compute is already landing and will land in the rest of this year into next year. So if you think about it, it's a layer cake of compute that's starting at different times with different capabilities, and we're very dynamically comparing that compute. Its price-performance over time is really important to us — when it lands, and what we think we can do with it internally in the business. There are so many different variables you have to optimize for around what compute it is, at what cost, and over what time horizon. But we have a pretty dynamic way of looking at near-term compute and then medium- to long-term compute. But the things we're assessing are largely the same. What is different is just the time horizon.
· Patrick O'Shaughnessy, Krishna Rao
[00:00:00] S0: What about the trade-off — you said price to performance — between cost per token or something, throughput, and speed? Both are important, of course. From the customer perspective, they care about both. Speed probably unlocks some capability and use cases that are really interesting that we don't know about yet as these things get faster. Can you talk a little bit about that trade-off in compute as you're assessing it? [00:00:00] S1: As we look across three different chip platforms, we also have multiple generations of chips within it. It could be TPU v5e, v6, and v7, Trainium 2, Trainium 3. All of them are at different places on the price-performance curve. We look at how we will utilize it. Price to performance is important because of efficiency. Speed is also important for certain use cases as well. We look at the compute down to a very granular level in terms of what it can deliver for us and when. Our compute team leads that, but we closely collaborate across the business to say, "Where do we need this compute, and for what?" Okay, we might need CPUs for RL. How much of that do we need and where, and what's the nature of it? We might need this more leading-edge compute, and we're going to deploy it for our best and fastest models, or for training them. From our perspective, it's customer demand, but it's also down quite granular in terms of what each chip is best for, and then what we will have when.