The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:35:35 · Harry Stebbings, George Sivulka
[00:35:35] S1: Talk to me about this. [00:35:36] S0: Right now, you actually... So OpenAI is starting to do this with o one, where they'll have a model recursively think about a question over and over and over again before it produces an answer. And so instead of training a larger model, they're using effectively a similarly sized model and just telling it to run multiple cycles, I. Compute more before answering. Hebia is... Actually pioneered something different where a year or almost eighteen months ago, we said, hey. We can't wait for these models to catch up. What we'll do is infer simple single question. Let's actually run hundreds or even thousands of sub models of the best models in the world to compute over every single document to answer the same question. And so ultimately, if you can't train larger and larger models fast enough, you could take whatever state of the art or cutting edge and run it more times to get more compute, I. More computational power, better decision making for the same user right now. And so this is an idea that we pioneered. It doesn't matter if you're using Claude three five or if you're using o one itself, I. E. Scaling at inference, at the orchestration layer with something that is scaled at inference with the training layer, but you get way better results. And it's, it's a it's a way to to drive to more accuracy. [00:36:47] S1: Find provides the best results. We had Des Trainer on from Intercom recently. He spoke about the movement away from OpenAI to Anthropic.
00:36:47 · Harry Stebbings, George Sivulka
[00:36:47] S1: Find provides the best results. We had Des Trainer on from Intercom recently. He spoke about the movement away from OpenAI to Anthropic. [00:36:53] S0: We've seen that for certain types of documents, like the dense legalese or or or more colloquial documents, Anthropic works better. But for other types of documents, like o one or OpenAI four o works better. And it's always trade offs between accuracy and speed and all kinds of things. Actually, a lot of the time when we're decomposing a task, we'll use mixes of OpenAI, Anthropic, even Gemini. [00:37:15] S1: Do you think we live in a world moving forwards of many models that are specialized in different things? As you said, some do legal, some do whatever we wanna talk about, and that's the world we live in, or there's generalist monolith models which really kind of own the whole stack. [00:37:30] S0: This makes me think of the story of Bloomberg, which has the best financial services training set of all time. And they trained a GPT 3.5 class model. It was called the Bloomberg GPT. And they released an archive paper, and everyone on LinkedIn was like, wow. Bloomberg is, you know, cutting edge, and they're gonna steal finance AI. [00:37:47] S1: Why did they not? They did have all... They've got the best data in finance.
00:37:47 · Harry Stebbings, George Sivulka
[00:37:47] S1: Why did they not? They did have all... They've got the best data in finance. [00:37:51] S0: Well, so then GPT four was released, I think, like, a few weeks later. I don't know exactly know the right timeline, but it just destroyed Bloomberg GPT at every single finance task. And so you saw the idea of post training or or or or kind of like this refined verticalized model creation just always would lose to scaling loss. And maybe we're at the end of scaling laws at training, but I actually think, you know, Hebia and and now OpenAI and a variety of other companies are starting to pioneer the idea of scaling laws and inference. And I actually think that it will... Like, nothing that that other players can do to fine tune models will ever catch up. [00:38:25] S1: I need to break that down. Sorry. So everyone's like, oh, are we at the end of scaling rules? Aren't we? Yeah. You know, Reid Benioff and Daniel Dines are like, yes, we are. Yes. We are at the upper end of LLM. Yep. Reid Hoffman is like, no. There's so much more room to run. Can you just break down for me the difference between scaling laws at inference and scaling laws at training?
00:38:25 · Harry Stebbings, George Sivulka
[00:38:25] S1: I need to break that down. Sorry. So everyone's like, oh, are we at the end of scaling rules? Aren't we? Yeah. You know, Reid Benioff and Daniel Dines are like, yes, we are. Yes. We are at the upper end of LLM. Yep. Reid Hoffman is like, no. There's so much more room to run. Can you just break down for me the difference between scaling laws at inference and scaling laws at training? [00:38:50] S0: Yeah. I think, you know, there's... It's a bit of a marketing distinction. Right? But, ultimately, the idea is that the way that we got here over the last five, seven years of of training models has been let's build a bigger and bigger model, and let's give it more and more data, more and more clean data. And then maybe we'll do some RLHF or some, you know, reinforcement leaning to... Reinforcement training to to fine tune it after pretraining. And that worked great to get us here, but we're running up against the amount of good data that exists in the world. We're running up against [00:39:20] S1: Are we? Because people push back on this and say there's so much data that we haven't used yet, whether it's video data that can't be translated, whether it's synthetic data. Like, we are not at all exhausted in terms of data supply. [00:39:32] S0: You know, I think that we're starting to run up against the constraints of it. That's a that's a gut feel. I'm not, you know, I'm not looking at particularly in data collection myself, but I I I think we're starting to run up against the limits of of really good data that we can. [00:39:45] S1: What's then the problem?
00:39:45 · Harry Stebbings, George Sivulka
[00:39:45] S1: What's then the problem? [00:39:46] S0: So so, ultimately, that might mean that, hey. We're training larger and larger models. XAI, again, just created the largest GPU cluster of all time, and they're gonna try to train larger and larger models. But regardless of how the scaling laws for training larger models or parameter count and accuracy or performance carry out, I'm starting to believe that you could still get better compute, not by building a larger engine, to use a metaphor, but by actually putting a bunch of smaller engines together. The Hebia, by by orchestrating large amounts of inference to answer one single question, ends up kind of building like a Tesla, where Tesla is made up a bunch of smaller engines or a bunch of smaller electromechanical motors, that make a lot of torque and a really a really amazing larger engine. [00:40:30] S1: Does it not make it incredibly capital inefficient?
00:40:30 · Harry Stebbings, George Sivulka
[00:40:30] S1: Does it not make it incredibly capital inefficient? [00:40:33] S0: You know, I think the one thing that people in my position, will always tell you is that will go to zero. The cost of intelligence will go to... I mean, I think that since Hebia started, the cost of of of inference over a fixed number of parameters has decreased in... By, like, seven orders of magnitude in four years. And so I genuinely believe that scaling compute is like a no brainer. And, yes, we run more large language model calls than anyone might even say would ever be necessary, but we have the best accuracy in the business. We can answer much more complex problems. We're driving real value for enterprises. And I I actually think that every single quarter, like, our margin goes, oh, we're not spending money fast enough. [00:41:14] S1: You mentioned XSOOT AI's GPU cluster. Yes. What they've been able to do in such a short amount of time is miraculous. Yeah. What do you think that tells us about the layer itself?