Invest Like the Best with Patrick O'Shaughnessy
· Krishna Rao, Patrick O'Shaughnessy
[00:00:00] S1: The compute that we procure is the lifeblood of our business. It is the most important thing in the company. It's the canvas on which everything else gets built. The decisions we make on how much compute to buy are some of the most consequential and hardest decisions to make in the entire company. Think of it this way — if you buy too much compute, you go out of business. If you buy too little compute, you can't serve your customers. [00:00:00] S0: You go out of business. [00:00:00] S1: And you're not at the frontier. Same thing. We talk a lot about this cone of uncertainty, but the idea is that these purchases have real-world implications. You can't just go out and buy a gigawatt of compute and have it delivered next week. You have to really think ahead to plan for this. We take a very disciplined approach to how we think about it. We look bottoms up. We model what we think demand will be. Obviously, we sometimes get that wrong. We think about the compute we need to stay at the frontier, and we really look ahead and try to estimate that. As we go out and do these deals to procure compute, flexibility is really important to us. We build that flexibility into the deals themselves. We build that flexibility into how we use the compute as well, because the way we bridge from the position we are today to where we want to go when the business is growing exponentially, is to use that compute as efficiently as possible. I would say I spend 30 or 40% of my time on compute, even today. [00:00:00] S0: What does flexibility mean in that example?
· Patrick O'Shaughnessy, Krishna Rao
[00:00:00] S0: What does flexibility mean in that example? [00:00:00] S1: It means a couple of different things. Number one, we use three different chip platforms. We are customers of Amazon's Trainium chip, Google's TPUs, and NVIDIA's GPUs. We use these chips fungibly. If you think about the compute we buy, we're using it for model development, we're using it internally to speed up our own product and model development. We're also using it, obviously, to serve customers. Across those three chip platforms, we're using compute for all of those internal and external uses, and that flexibility — it took us a long time to be able to do that. We've invested in that over multiple years, to be what I believe the most efficient users of compute amongst any of the frontier labs. And that's not something that just happened overnight. When we started using TPUs — I think third-generation TPUs were the first ones we used at scale — people thought, "Oh, well, you're crazy. Everyone's using GPUs. Why aren't you using GPUs?" We've invested very heavily to be able to use that compute incredibly flexibly. We look across the different generations of those chip platforms and use each generation for the best workload internally. We've really built this orchestration layer that gives us that flexibility to use all different types of compute, and in doing so, we're able to get the most value out of it.
· Patrick O'Shaughnessy, Krishna Rao
[00:00:00] S0: Am I thinking about this in the right way — that something like CUDA, which has been a part of NVIDIA's story for a long time now, and allows you to do a lot with the underlying actual hardware — that you want to eke your way into being as close to the bare metal as possible? That's part of this flexibility and being able to control as many of the variables as you can. Is that the journey that you've been on? [00:00:00] S1: That's part of the journey, for sure, but it's also been pretty collaborative. We work really closely with the Annapurna Labs team at Amazon to help influence the roadmap of these chips, because we believe what we're doing is really stressing the limits of what these chips are capable of. And that means that a dollar of compute inside our organization goes further than I think it does anywhere else. Importantly, we want to utilize each chip to its best purpose within the company, and that does mean that we're building our own compilers. We're really building things from the chip level up in order to have that customization and that flexibility to use it internally the way we think will generate the most ROI. [00:00:00] S0: Can you explain this cone of uncertainty thing? I want to ask about all the component parts of this, but that feels like a really key starting point or overall frame for how to think about both sourcing and then the uses of compute. Can you explain what that concept is?
· Krishna Rao
[00:00:00] S1: When you're building and growing a business exponentially, really small movements in monthly or weekly growth rates compound into very different outcomes. As we're thinking ahead, even with our revenue growth, it's really hard to predict this business. Humans mostly think linearly. They think incrementally. I've been at the company for two years. That's a paradigm I've had to break for myself, to stop just thinking linearly and think on this exponential. When you're on this exponential, the range of outcomes starts to be really wide. We look at a range of scenarios, and we look at different points in that cone of uncertainty over a one- to two-year period, and then we work backwards from that. What we want is to be at a place where we can still be at the frontier — that's the most important thing. To be able to serve customers, and then to have enough internal compute to accelerate our employees. It's interesting, if we were to say to our employees, "You can't use our models anymore," we could serve billions of dollars of revenue with that compute that we allocate to employees internally. But we want to take a long-term view and a long-term perspective on that cone of uncertainty because we want to range towards the top end of these outcomes. But we have to plan for that. And as we go, that's how we think about buying compute in a disciplined way. The most important thing is what happens if you are at one point in the cone of uncertainty, but you've only bought compute for a different point. That's where this compute efficiency has really helped us out.
· Patrick O'Shaughnessy
[00:00:00] S0: Can you bring us into the room for the conversations around the trade-offs between those? I'm so interested by those three buckets of training, research, internal use, broadly speaking, and then serving customer demand. Naively, you might think it's a third, a third, a third allocation or something. How much does that range around? What are the trade-offs? What is that discussion like on an ongoing basis?
· Krishna Rao, Patrick O'Shaughnessy
[00:00:00] S1: In addition to meeting about compute procurement, we meet a lot about compute allocation. What's important is it starts with a place where our culture is one that's incredibly collaborative, and that informs how this conversation happens. There's not fiefdoms. It's done in a collaborative, not zero-sum way. There's a level of compute for model development that we will not go below, even if it means it's harder to serve customers or we have to do unnatural things when it comes to that. We want to continue to make that long-term investment in developing the best models because we think the returns to frontier intelligence are extremely high. And it's especially extremely high in enterprise. That puts a floor on the compute allocated to model development. As we think about the internal use of compute, it really helps us to speed up that model development and to speed up finding those compute efficiency multipliers that get us more from each dollar of compute. So when we're talking about it, each team is representing what they would do with that compute, and then we have a really open and frank discussion about how we think about ROI. Because we can allocate that compute so dynamically, we can make changes and adjustments on a relatively short time horizon. [00:00:00] S0: The efficiency thing is so interesting to me. I'm curious if you have a sense of how much more efficient you are versus your own internal benchmarks from a year ago, or versus others that you have some sense of how efficient they are? How do you measure what efficiency means?