Invest Like the Best with Patrick O'Shaughnessy
· Patrick O'Shaughnessy, Krishna Rao
[00:00:00] S0: Am I thinking about this in the right way — that something like CUDA, which has been a part of NVIDIA's story for a long time now, and allows you to do a lot with the underlying actual hardware — that you want to eke your way into being as close to the bare metal as possible? That's part of this flexibility and being able to control as many of the variables as you can. Is that the journey that you've been on? [00:00:00] S1: That's part of the journey, for sure, but it's also been pretty collaborative. We work really closely with the Annapurna Labs team at Amazon to help influence the roadmap of these chips, because we believe what we're doing is really stressing the limits of what these chips are capable of. And that means that a dollar of compute inside our organization goes further than I think it does anywhere else. Importantly, we want to utilize each chip to its best purpose within the company, and that does mean that we're building our own compilers. We're really building things from the chip level up in order to have that customization and that flexibility to use it internally the way we think will generate the most ROI. [00:00:00] S0: Can you explain this cone of uncertainty thing? I want to ask about all the component parts of this, but that feels like a really key starting point or overall frame for how to think about both sourcing and then the uses of compute. Can you explain what that concept is?
· Krishna Rao
[00:00:00] S1: When you're building and growing a business exponentially, really small movements in monthly or weekly growth rates compound into very different outcomes. As we're thinking ahead, even with our revenue growth, it's really hard to predict this business. Humans mostly think linearly. They think incrementally. I've been at the company for two years. That's a paradigm I've had to break for myself, to stop just thinking linearly and think on this exponential. When you're on this exponential, the range of outcomes starts to be really wide. We look at a range of scenarios, and we look at different points in that cone of uncertainty over a one- to two-year period, and then we work backwards from that. What we want is to be at a place where we can still be at the frontier — that's the most important thing. To be able to serve customers, and then to have enough internal compute to accelerate our employees. It's interesting, if we were to say to our employees, "You can't use our models anymore," we could serve billions of dollars of revenue with that compute that we allocate to employees internally. But we want to take a long-term view and a long-term perspective on that cone of uncertainty because we want to range towards the top end of these outcomes. But we have to plan for that. And as we go, that's how we think about buying compute in a disciplined way. The most important thing is what happens if you are at one point in the cone of uncertainty, but you've only bought compute for a different point. That's where this compute efficiency has really helped us out.
· Patrick O'Shaughnessy
[00:00:00] S0: Can you bring us into the room for the conversations around the trade-offs between those? I'm so interested by those three buckets of training, research, internal use, broadly speaking, and then serving customer demand. Naively, you might think it's a third, a third, a third allocation or something. How much does that range around? What are the trade-offs? What is that discussion like on an ongoing basis?
· Krishna Rao, Patrick O'Shaughnessy
[00:00:00] S1: In addition to meeting about compute procurement, we meet a lot about compute allocation. What's important is it starts with a place where our culture is one that's incredibly collaborative, and that informs how this conversation happens. There's not fiefdoms. It's done in a collaborative, not zero-sum way. There's a level of compute for model development that we will not go below, even if it means it's harder to serve customers or we have to do unnatural things when it comes to that. We want to continue to make that long-term investment in developing the best models because we think the returns to frontier intelligence are extremely high. And it's especially extremely high in enterprise. That puts a floor on the compute allocated to model development. As we think about the internal use of compute, it really helps us to speed up that model development and to speed up finding those compute efficiency multipliers that get us more from each dollar of compute. So when we're talking about it, each team is representing what they would do with that compute, and then we have a really open and frank discussion about how we think about ROI. Because we can allocate that compute so dynamically, we can make changes and adjustments on a relatively short time horizon. [00:00:00] S0: The efficiency thing is so interesting to me. I'm curious if you have a sense of how much more efficient you are versus your own internal benchmarks from a year ago, or versus others that you have some sense of how efficient they are? How do you measure what efficiency means?
· Krishna Rao
[00:00:00] S1: There are a couple of different ways I would think about it. From a model perspective, the analogy people have when these new models come out is they're like cars. You had a sedan before, and then you might have the higher-end version of that sedan, and you're moving up the chain. That is true in terms of model intelligence. The place that analogy breaks down a little bit is people think, "Okay, I'm going from the sedan to the sports car, and we get much less fuel efficiency. I'm not gonna buy the sports car for the gas mileage." In our case, we actually see huge improvements in capability, but also in model efficiency. If you look at going from Opus 4 to 4.5, 4.6, and now 4.7 — they're not equal, but each one of those leaps to a new model has a multiplier in terms of how much more efficient it is at processing tokens effectively. That doesn't just serve customers, that also helps us internally as well. If we're doing reinforcement learning on the model, it's basically inference within a sandbox with a reward function. So if the model's better at more efficient inference, that RL is more efficient as well. We're able to do this win-win where the customer is getting more capability when we release a new model, and we're able to serve that model — sometimes again, a multiple more efficient than the prior generation. When we're in between generations, we're dynamically deploying efficiency improvements in between these more step-function model changes. It is always getting more efficient over time, and what fuels that is the research team. If you think about it, all these things are very connected. These various tasks and workloads that we have internally all fit together in this way. Doing R&D for model capabilities, for compute efficiency, for serving customers, and then having internal workloads that can be sped up by using the best models — sometimes models that we haven't released.
· Patrick O'Shaughnessy
[00:00:00] S0: You said something really important before, which is the returns to being at the frontier are really high. Can you explain that in as much detail as you can? It sounds obvious when you say it, but there's certainly been some camps saying, "I can use the six-month-old model, it's a fraction of the cost, and I'll just use that. That'll be catching up all the time." That just hasn't been the case. The second Opus 4.7 comes out — even me as a consumer — the thing you do is switch it on. Or GPT 5.5 comes out, you switch on the new one right away. You want the best. Talk about the returns to being on the frontier and why it's so high.