Invest Like the Best with Patrick O'Shaughnessy
· Gavin Baker
[00:00:00] S1: It makes an incredible difference. The last time I was on here, or one of the other times, I said, "Hey, as an investor, it's very important that you pay for the $250 a month version to get your own intuitive sense." That's no longer possible. To understand what frontier AI is capable of today — even for a non-coding use case — you need to have Claude Code or Codex, and you need to be on an enterprise plan. The reason for this, and this is another dynamic enabled by Google losing their cost leadership, is these AI models just shifted to usage-based pricing. If you were on that $250 or $300 or $280 a month plan or whatever it is, you were getting severely rate limited. You were getting a lobotomized version of the AI because, like we talked about, Claude now produces 70% less tokens. If you want the tokens that Claude and its harness really think it needs to produce to get you a good answer, you need to be on a usage-based plan. By the way, this is so bullish for AI. If we go back to '05 to '07, cellular had been a great growth industry for the last 10 years. The reason was you had a combination of fixed pricing — you had 900 minutes for whatever it was — and then usage-based pricing over that. When did cellular stop being a great growth industry? When everybody just went to all-you-can-eat. And by the way, long distance was the same thing. AI is just shifting from all-you-can-eat to pay-by-the-drink. It turns out people really like to talk to their friends long distance. They really like to talk to their friends on the phone. And people really like to use AI, particularly now that one person can have 100 agents working. So the shift to usage-based pricing is probably why you will see OpenAI and Anthropic exceed well over $200 billion in ARR this year. Not only is more compute going to come online, but they're going to be able to push frontier token pricing with these usage-based enterprise models. It's sad. It's sad for the world because it just means if you can't afford that, you're not at the frontier. And it's going to throw off a lot of investors' intuitive sense of the capabilities of AI. But yeah, continual learning — if we solve that…
· Patrick O'Shaughnessy, Gavin Baker
[00:00:00] S0: How do you conceptualize that? [00:00:00] S1: AI is constantly updating its weights. It may end up being something different. There are so many mysteries about the human mind. We're such sample-efficient learners relative to AI, by many orders of magnitude. We have a crude variant of continual learning today when something is verifiable, and that's just reinforcement learning during training. Continual learning is a model that dynamically adjusts its weights or adjusts in some way, in real time. As a human — the first time I put my hand in a fire, I learned. I never put it in there before. That model today needs to put its hand in the fire a million times and then have the designers effectively put a fire in the next training run or an RL gym for it to learn. I think it has to be dynamically updating the weights, but people are working on really smart techniques beyond this. If we get that, then we have a really fast takeoff. People seem confident that continual learning is just around the corner. This is the third big question. Bitter lesson violation as a result of ASI, or less likely, human ingenuity? Will frontier tokens still command the premium they do? And will we get continual learning, and if so, when?
· Patrick O'Shaughnessy
[00:00:00] S0: What is the role of new chip companies in all of this? We talked a lot about Nvidia and their relationship with TSMC, and Intel, and all these sorts of things. There's a thousand flowers blooming — I think literally probably a thousand flowers blooming — trying to create a new chip to address some part of this bottleneck. I'm curious how you process this space, this opportunity, and what role it'll play.
· Gavin Baker
[00:00:00] S1: I think this is good and healthy for the world. It's good for Jensen too, because a different administration might take a different view. Competition is good for everyone, and seeing different architectures explored is good. The reason is — in tank design, they talk about the Iron Triangle. The Iron Triangle of tank design is that all designers of a tank have to make trade-offs between attack, defense, and mobility for obvious reasons. The more defense you have — which is just armor — the heavier the tank is, the less mobile it is. So you have to live in this triangle and make trade-offs. The Merkava in Israel is optimized for defense. Russian tanks and tanks like the Leopard are generally more optimized for mobility. Chip design is the same. There are these fundamental constraints imposed by the laws of physics, as embedded in the Taiwan Semi design rules that you need to live with. You have TPU, Trainium, and AMD — which are all essentially trying to be a better GPU. Today, I think probably Trainium is doing the best. Now, nobody has a better GPU, but Trainium is tugging on Superman's cape. They haven't started yet. Trainium 3 needs to ramp into production because it has a switch scale-up network, which you really need to economically inference MOE models. A lot of companies have a torus architecture. That's where Google was; Google's developing a switch-scale network. AMD is always kind of flying a little bit behind. AMD, we'll see. The MI450, we don't know yet. We'll see. We probably know more about Trainium 3 than the MI450. But that's a hard game to play, so you have to do something different. And you have to do something different that is also hard to do. The best path for these startups — my rule of thumb is, 1% market share is going to be worth $100 billion. $100 billion is a pretty good venture outcome. I think what Jensen would say is, if somebody does something different and it gets to 1% or 2% or 3% share, we'll make that chip. That's coming for everyone. But if you're trying to make a better GPU, good luck. If you are doing something different, it also needs to be hard to do. And you can make different trade-offs. The disaggregation of prefill and inference really has opened the aperture for making these different trade-offs, because you can make very aggressive trade-offs for decode, and aggressive trade-offs for prefill.
· Patrick O'Shaughnessy
[00:00:00] S0: Prefill being taking in the context, decode being writing the output.
· Gavin Baker
[00:00:00] S1: Yeah. I have a great colleague named Andrew Fox who said, "Picture a British naval ship from the 18th century. Prefill is loading the cannon; decode is firing." What prefill literally is, it's just the model understanding the question, the prompt — and then keeping track of its own answer. That is fundamentally a memory-capacity-bound problem. Decode is the process of generating new tokens, and that is memory bandwidth constrained So if you're a chip designer, this gives you a richer canvas to paint on. But even so, it needs to be hard. If you make different trade-offs in that Iron Triangle to optimize for memory capacity — and they're not hard trade-offs to make — Nvidia is going to make those same trade-offs. They get better prices from Taiwan Semi than you're ever going to get. Good luck. And they have the advantage of working with every model company and optimizing their designs. By the way, another very funny thing — if you're a VC and you are investing in a semiconductor company that is telling you they are going to have an advantage because of a Taiwan Semi process they have special access to — I promise you that Jensen saw that process when it was a twinkle in Taiwan Semi's eyes. They know more about it than this little company with 200 people can imagine. Taiwan Semi, and everybody in the supply chain is showing Jensen everything. The same way they're showing Amazon everything, AMD everything, TPU everything. That's another reason — don't go try to make a better GPU. So you could do something different. You can paint in the prefill canvas, you can paint in the decode canvas. But you also have to do something hard, because if it gets to scale, you're going to have those 4 companies as very fast followers. My firm was a venture investor in Cerebras. What Cerebras has done is something hard and fundamentally different — wafer-scale computing. It comes with a set of trade-offs, but that architectural decision they made was hard and lets them do something that no one else can do. We'll find out how big that is. They're working on really cool things. One of the problems Cerebras has is, once you start needing to glue a lot of chips together and scale up networks or scale out networks, you need a lot of I/O, and I/O is bound by what's called the shoreline — the sides of the chip.