Decoder with Nilay Patel
00:06:58 · Joey Brookhart
[00:06:58] S4: And when we look at like coding today for example, and I think we saw this in Q3 with OpenAI kind of catching up, there's areas of coding where there's very low switching costs and you know, I can just move between that. I think you know, what we hear from enterprises and maybe too on the open wait side, like obviously I think they somewhat botched the Fable release, and why we saw some slowdown because of the zero data retention or lack of zero data retention. And is that something they can fix with Fable 5.5 and maybe drive more, enterprise adoption there? I think that's a huge thing for enterprises when we talk with them. We definitely see enterprises now, and more so to a lot of the AI natives, custom training, post training on open weight models, and then running either their inference off of that or even their products for internal or external use cases. So there's that. Then obviously DeepMind at Google and MSL with Meta are very well funded labs that have pretty similar training compute capacity, and they're bringing on a ton of and not even to mention SpaceX as well, with what Elon said on how much 10 net new gigawatts next year, and half of that kind of is the long term, internal use case. I think in the first half of the year, Anthropic ran away with it. They were 90% of the industry's net new ARR, and then we saw in OpenAI was probably actually the majority of industry net new ARR, and then open source has grown from probably a billion dollar ARR in Q1 to 10,000,000,000 plus here in Q3. So yes, the industry has gotten much more competitive, but I think all these new use cases too, and some of these verticalized use cases are really where we're going to see the growth, you know, going forward, because it's obvious coding isn't going to power us to, you know, even triple from here.
00:08:34 · Nilay Patel
[00:08:34] S2: It's interesting, the concept is called tokenomics, you run that practice alongside a co lead at SemiAnalysis. Explain what it is because to me, it's such a smart way of thinking about, okay, we're going put some money in and maybe we'll get some money out.
00:08:47 · Joey Brookhart
[00:08:47] S4: I think of it as like the unit economics or even just the broader everything from a macro to microeconomics of running an AI business or even looking at AI demand. So we cover the hyperscalers, now some of the neo clouds as well, in kind of a separate product, but then too just the AI labs. And I think for investors or just people that are interested in the business, Anthropic had negative 94% gross margins inference two years ago, now they have positive 85% and people didn't understand kind of how do they allocate compute, what are actually how do we think about the inference business model? I think part of that gross margin conversation is the revenue per megawatt in inference, and why are all companies that are running inference seeing massive gains in inference and compute efficiency? So we try to model out these businesses, and obviously when Anthropic was private, we're dealing with information we can get from the press and things like that. But now that we have numbers, concrete audited numbers, it's easy to explain the unit economics of this and two, how the businesses differ. OpenAI is a much different business than Anthropic, just in both their consumption versus subscription and then historical weight versus, you know, consumer versus enterprise. So trying to understand across the demand level, you know, the economics of these businesses.
00:10:04 · Nilay Patel
[00:10:04] S2: Several years ago, when I talked to Sundar Pichai about the cost of inference, said, how are you ever gonna make money? He was just very clear. He's like, if there's one thing Google's good at, it's bringing this stuff down the cost curve. And it was a very blunt answer. And I understand how Google can do it, they make their own chips. Anthropic is running on everyone else's clouds, mean this is like the core part of the S1 is how much they've committed to spend on other people's clouds, SpaceX, Google, Amazon. Where does their efficiency come from?
00:10:33 · Joey Brookhart
[00:10:33] S4: Really token throughput, and I think they have a very diverse chip fleet. So between Trainiums, TPUs and GPUs, across all of those we hear a ton of token efficiencies. So I think like token throughput, for example, on Trainium chips are up five to 10 X year over year right now. So dealing with both the hardware level efficiencies and the software, you know, how the model interacts with the hardware, there's lot of token throughput gains where, you know, if you're doing x tokens per second, now you're doing 5x, 10x just a year later, And and I think what they'll tell you is they see continued compute efficiencies as well going forward. So we've even published some research, so Max who co leads Tokenomics model, right now if Anthropic is at $65,000,000 of revenue per inference megawatt, this was mid teens last year at this time, and I think what we've said on maybe OPUS 5.5 on a Blackwell, you can get to 100,000,000 to $120,000,000 of revenue per megawatt. But the big thing here is they're paying on average now just a little over $10 per megawatt in cogs, or just in compute costs. So what you can do with those sort of margins and then reinvest in training people don't realize this, but only about 30% of the compute is actually in inference, serving revenue. The rest is being for experimentation and research, training models. So there's a lot of compute that is building the next model, and that's more of like the R and D side and where you actually get your returns and defensibility in this business.
00:11:51 · Nilay Patel
[00:11:51] S2: I actually, what I was expecting to see in the S1 was that dynamic play out much more strongly. The Anthropic makes a bunch of money selling the models they have now and there are more customers in the future for the models they have now. That's going to throw off a whole bunch of margin and then because they're Anthropic and they think they're building digital god, they're gonna train the next models as much as they can. And that is like a very particular Anthropic story. They are chasing AGI, they will happily tell you that in the s one that Claude might be conscious and this could all go to hell. Like Anthropic is a very unique company and it's conception of itself, but you don't really see that dynamic playing out in the S1, right? They are managing their training costs against their inference margin pretty well. Do you see that going up or changing in any way?