The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:37:47 · Harry Stebbings, George Sivulka
[00:37:47] S1: Why did they not? They did have all... They've got the best data in finance. [00:37:51] S0: Well, so then GPT four was released, I think, like, a few weeks later. I don't know exactly know the right timeline, but it just destroyed Bloomberg GPT at every single finance task. And so you saw the idea of post training or or or or kind of like this refined verticalized model creation just always would lose to scaling loss. And maybe we're at the end of scaling laws at training, but I actually think, you know, Hebia and and now OpenAI and a variety of other companies are starting to pioneer the idea of scaling laws and inference. And I actually think that it will... Like, nothing that that other players can do to fine tune models will ever catch up. [00:38:25] S1: I need to break that down. Sorry. So everyone's like, oh, are we at the end of scaling rules? Aren't we? Yeah. You know, Reid Benioff and Daniel Dines are like, yes, we are. Yes. We are at the upper end of LLM. Yep. Reid Hoffman is like, no. There's so much more room to run. Can you just break down for me the difference between scaling laws at inference and scaling laws at training?
00:38:25 · Harry Stebbings, George Sivulka
[00:38:25] S1: I need to break that down. Sorry. So everyone's like, oh, are we at the end of scaling rules? Aren't we? Yeah. You know, Reid Benioff and Daniel Dines are like, yes, we are. Yes. We are at the upper end of LLM. Yep. Reid Hoffman is like, no. There's so much more room to run. Can you just break down for me the difference between scaling laws at inference and scaling laws at training? [00:38:50] S0: Yeah. I think, you know, there's... It's a bit of a marketing distinction. Right? But, ultimately, the idea is that the way that we got here over the last five, seven years of of training models has been let's build a bigger and bigger model, and let's give it more and more data, more and more clean data. And then maybe we'll do some RLHF or some, you know, reinforcement leaning to... Reinforcement training to to fine tune it after pretraining. And that worked great to get us here, but we're running up against the amount of good data that exists in the world. We're running up against [00:39:20] S1: Are we? Because people push back on this and say there's so much data that we haven't used yet, whether it's video data that can't be translated, whether it's synthetic data. Like, we are not at all exhausted in terms of data supply. [00:39:32] S0: You know, I think that we're starting to run up against the constraints of it. That's a that's a gut feel. I'm not, you know, I'm not looking at particularly in data collection myself, but I I I think we're starting to run up against the limits of of really good data that we can. [00:39:45] S1: What's then the problem?
00:39:45 · Harry Stebbings, George Sivulka
[00:39:45] S1: What's then the problem? [00:39:46] S0: So so, ultimately, that might mean that, hey. We're training larger and larger models. XAI, again, just created the largest GPU cluster of all time, and they're gonna try to train larger and larger models. But regardless of how the scaling laws for training larger models or parameter count and accuracy or performance carry out, I'm starting to believe that you could still get better compute, not by building a larger engine, to use a metaphor, but by actually putting a bunch of smaller engines together. The Hebia, by by orchestrating large amounts of inference to answer one single question, ends up kind of building like a Tesla, where Tesla is made up a bunch of smaller engines or a bunch of smaller electromechanical motors, that make a lot of torque and a really a really amazing larger engine. [00:40:30] S1: Does it not make it incredibly capital inefficient?
00:40:30 · Harry Stebbings, George Sivulka
[00:40:30] S1: Does it not make it incredibly capital inefficient? [00:40:33] S0: You know, I think the one thing that people in my position, will always tell you is that will go to zero. The cost of intelligence will go to... I mean, I think that since Hebia started, the cost of of of inference over a fixed number of parameters has decreased in... By, like, seven orders of magnitude in four years. And so I genuinely believe that scaling compute is like a no brainer. And, yes, we run more large language model calls than anyone might even say would ever be necessary, but we have the best accuracy in the business. We can answer much more complex problems. We're driving real value for enterprises. And I I actually think that every single quarter, like, our margin goes, oh, we're not spending money fast enough. [00:41:14] S1: You mentioned XSOOT AI's GPU cluster. Yes. What they've been able to do in such a short amount of time is miraculous. Yeah. What do you think that tells us about the layer itself?
00:41:14 · Harry Stebbings, George Sivulka
[00:41:14] S1: You mentioned XSOOT AI's GPU cluster. Yes. What they've been able to do in such a short amount of time is miraculous. Yeah. What do you think that tells us about the layer itself? [00:41:24] S0: Ultimately, the model layer, and I think this is not a hot take anymore. I've been saying it for a few years, but I think it'll become commoditized. I think that a lot of value will accrue at the hardware layer, we we could talk about what that means for NVIDIA, especially as NVIDIA has a stranglehold on training, but not not as much stranglehold on inference. And so you might actually see other chip makers, you know, actually start to... Their their chips start to be used in a more meaningful way because CUDA is what all ML scientists were trained on in their PhDs, but then, you know, inference doesn't matter, kind of what you're using. And I think it'll be the infrastructure layer and then actually the application or agent layer that will accrue the most value. Ultimately [00:42:00] S1: Why does it not follow the same vein as cloud where cloud is commoditized? Yep. But Azure, Google Cloud, AWS, I mean, completely commoditized, to be honest, cloud. But it's great business for them. [00:42:12] S0: I think it might. There's probably fewer players and more entrenched players in cloud. And, ultimately, you you know, I think those those players honestly kind of have, like, an an OPEC oligopoly where that, you know, they can control pricing. I just think that, ultimately, cloud is actually more complex than training larger and larger models to
00:42:12 · George Sivulka, Harry Stebbings
[00:42:12] S0: I think it might. There's probably fewer players and more entrenched players in cloud. And, ultimately, you you know, I think those those players honestly kind of have, like, an an OPEC oligopoly where that, you know, they can control pricing. I just think that, ultimately, cloud is actually more complex than training larger and larger models to [00:42:29] S1: And the cloud providers are basically using models as a loss leader happily to build stronger moats in their cloud businesses. And you see this with Anthropic and Amazon. You see this with Microsoft and, OpenAI. [00:42:42] S0: Absolutely. Whoever has the best models will continue to attract, the right amount of investment. The the different thing about clouds too, though, is that the cost of switching is is much higher. So to so to refine my earlier point, right, like, I can switch models readily. Like, I think there's even entire businesses now. There will be an entire industry of being able to switch models from OpenAI to Anthropic when OpenAI goes down. But to switch clouds is like, you know, for any, like, substantially sized startup, like a 10,000,000 to $20,000,000 investment just to switch. It's almost always never worth it. It's much, much, much sickier. Whereas here, it's a very simple API key. You know? It's it's very simple to switch models. And so I think that that's also a differentiator. [00:43:25] S1: OpenAI at one sixty, Anthropic at 40 Yep. Or XAI at 50. Which one do you buy?