Decoder with Nilay Patel
00:10:04 · Nilay Patel
[00:10:04] S2: Several years ago, when I talked to Sundar Pichai about the cost of inference, said, how are you ever gonna make money? He was just very clear. He's like, if there's one thing Google's good at, it's bringing this stuff down the cost curve. And it was a very blunt answer. And I understand how Google can do it, they make their own chips. Anthropic is running on everyone else's clouds, mean this is like the core part of the S1 is how much they've committed to spend on other people's clouds, SpaceX, Google, Amazon. Where does their efficiency come from?
00:10:33 · Joey Brookhart
[00:10:33] S4: Really token throughput, and I think they have a very diverse chip fleet. So between Trainiums, TPUs and GPUs, across all of those we hear a ton of token efficiencies. So I think like token throughput, for example, on Trainium chips are up five to 10 X year over year right now. So dealing with both the hardware level efficiencies and the software, you know, how the model interacts with the hardware, there's lot of token throughput gains where, you know, if you're doing x tokens per second, now you're doing 5x, 10x just a year later, And and I think what they'll tell you is they see continued compute efficiencies as well going forward. So we've even published some research, so Max who co leads Tokenomics model, right now if Anthropic is at $65,000,000 of revenue per inference megawatt, this was mid teens last year at this time, and I think what we've said on maybe OPUS 5.5 on a Blackwell, you can get to 100,000,000 to $120,000,000 of revenue per megawatt. But the big thing here is they're paying on average now just a little over $10 per megawatt in cogs, or just in compute costs. So what you can do with those sort of margins and then reinvest in training people don't realize this, but only about 30% of the compute is actually in inference, serving revenue. The rest is being for experimentation and research, training models. So there's a lot of compute that is building the next model, and that's more of like the R and D side and where you actually get your returns and defensibility in this business.
00:11:51 · Nilay Patel
[00:11:51] S2: I actually, what I was expecting to see in the S1 was that dynamic play out much more strongly. The Anthropic makes a bunch of money selling the models they have now and there are more customers in the future for the models they have now. That's going to throw off a whole bunch of margin and then because they're Anthropic and they think they're building digital god, they're gonna train the next models as much as they can. And that is like a very particular Anthropic story. They are chasing AGI, they will happily tell you that in the s one that Claude might be conscious and this could all go to hell. Like Anthropic is a very unique company and it's conception of itself, but you don't really see that dynamic playing out in the S1, right? They are managing their training costs against their inference margin pretty well. Do you see that going up or changing in any way?
00:12:37 · Joey Brookhart
[00:12:37] S4: I agree. I think long term and we've even said this like internally, they want to increase actually the percentage of training if you believe you're building digital God or going towards AGI. And as long as you have the gross margins 85% plus on inference to do so, you can. Think they've grown faster, you know, obviously a lot faster than they thought, and they didn't expect to be at 11% non GAAP EBIT margins. Yeah, I think they'd love to be closer to zero or even below that if they could get that compute and allocate it towards training and continue to see these compute efficiencies. And I think they've messaged that too to investors in the roadshow that, hey, you can look at this S1 and just the curve of profitability over the last few years and how little cash they burned to get to this point. And you can say, you know, hey this is a good business model but we're going towards HEI, don't expect these margin, the margin leverage we're seeing to continue to this level, even though we can, long term margins can be quite a bit higher.
00:13:32 · Nilay Patel, Joey Brookhart
[00:13:32] S2: There's some real shades of Amazon in there, Jeff Bezos famously telling investors he wasn't going make money until he can make a lot of money. Obviously Amazon and Anthropic have a lot of back and forth. Maybe there's some family resemblance there. But I'm curious, there's a point of model capability that would allow you to go into more verticals than just coding. Right? We we might be at it now. Right? The the idea of capability overhang exists. The idea that the models we have today could go and take over bio research. They're taking over mathematics today. Like there's a point in which the smartest business decision is to stop training and to just sell your profitable model into more and more companies. Anthropic as a public company will face that pressure. Do you think they're they're organized or structured to to fend it off? [00:14:17] S4: Yeah, think they're structured well. Think with something like the voting stock in in areas like that, they definitely are. We've seen this in tech before and even like, you know, publicly, you know, when Zuckerberg has made these decisions before, you know, to invest heavily. I just want to point [00:14:31] S2: out, didn't always pay off. [00:14:32] S4: Yeah, did not always pay off. I think [00:14:34] S2: and I technically are in the metaverse right now. Yeah. In some definition.
00:14:34 · Nilay Patel, Joey Brookhart
[00:14:34] S2: and I technically are in the metaverse right now. Yeah. In some definition. [00:14:40] S4: Right. But I think that's something, and there's an interesting, you know, number in the S1 that 80% of revenue in the last six or last three months comes from models released in the last six months. So then you say, okay, 20% of the models are like OPUS, you know, 4.5 in earlier versions. I think one thing going forward is you always need to be if you're a frontier model company, you always need to be at the frontier. Wait, can I ask a really [00:15:03] S2: dumb question about that? Yeah. Why? Like this is I think the heart of my sort of curiosity about the industry right now is if you enabled some bleeding edge technology that could revolutionize the entire world. At some point, the most profitable thing to do is to just go play it out in all the different verticals, right? It is not to go make the next one and it's actually run efficient model in legal services and medicine and wherever else that there hasn't been diffusion of AI yet. And so that this idea that you always have to be at the frontier, that Anthropic always has to be competing with Google and OpenAI and whoever else. I get it right from their self conception. But from a business point of view, an AI enabled business is going to throw out more money and spend more money with me, the AI provider, and that isn't necessarily related to being at the frontier.