The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:16:30 · Adarsh Hiremath, Harry Stebbings
[00:16:30] S0: We've always used OpenAI in some capacity. [00:16:33] S1: If improved in terms of any aspect of ModelAir, what would make the biggest improvement on the business and the product today for you with McCore? [00:16:41] S0: I think a concrete example would be the AI interviewer. We've built the product in such a way that whenever the models improve, the experience for applicants on our platform also improves pretty significantly. In general, this is something that's been, you know, on our mind. Right? There's like this huge wave of models getting better and better and better. And can we ride that wave to make our product better and better and better? So to summarize, well, we leverage LLMs and all these models throughout our product. The whole product gets better as the models get better. [00:17:09] S1: What do you think will be the next generation of models in terms of what they look like first before we get to training data? [00:17:16] S0: The whole market is shifting to reinforcement learning. You're already seeing this with o one, o three, the deep seek models. As a result, I think we're gonna see really, really powerful models in specific domains that can reason extremely well. And I think that'll be really, really exciting and will unlock just a huge number of use cases across a variety of different industries and domains. [00:17:43] S1: Do we live in a world of many, many specialized models, but very fragmented? Or do we live in a world of monoliths with one or two very horizontal platforms?
00:17:43 · Harry Stebbings, Adarsh Hiremath
[00:17:43] S1: Do we live in a world of many, many specialized models, but very fragmented? Or do we live in a world of monoliths with one or two very horizontal platforms? [00:17:53] S0: I think we'll live in a world with many, many models with different use cases. Right? We're already seeing this with a lot of application layer companies, right? Where they all have these specialized use cases for how they want to leverage the models. Right? For us, it's hiring and beating the expert hiring manager. For another company, it might be financial analysis in a specific domain. Across each of these use cases, I think these companies will need to make their models better for for their own purposes. [00:18:22] S1: How fair do you think the analogy is that the model landscape will be very much like the cloud landscape in terms of three or four juggernauts and it being very hard to switch out of? Do you agree that it's hard to switch out of them, or do you think given the model transients, it's actually much easier and much less defensible in that respect? [00:18:41] S0: There will only be a couple of companies that are able to build these foundation models that everyone builds off of. That analogy roughly holds. I don't anticipate there being 20 companies training foundation models that can all be be leveraged in the same way someone might leverage, OpenAI, for example. [00:18:58] S1: In terms of, like, the post training data side, how much will be human data versus how much will be synthetic data moving forward?
00:18:58 · Harry Stebbings, Adarsh Hiremath
[00:18:58] S1: In terms of, like, the post training data side, how much will be human data versus how much will be synthetic data moving forward? [00:19:06] S0: I think a lot of it will be human data going forward. I think a great example of this is Evals. Evals for models definitionally have to be outside of model capability. In order to see whether a model is doing well at a particular task, you need to have an Eval set created by humans that is better than the model at that particular task. And humans are gonna play a huge role in that, for example. And I think there are a whole set of other use cases, whether it be SFT, RLHF, RL environments, like how the models of tomorrow are being trained, that all require these expert humans to essentially teach the model how to get better. [00:19:42] S1: To what extent would you say that data is the bottleneck that prevents model improvement more than compute or algorithms? [00:19:49] S0: Data is the bottleneck. I think that would be an accurate statement. [00:19:52] S1: Why then does so many people tell me, including Jonathan at Grok, that actually synthetic data is often more high quality? It doesn't involve the drags of the Internet like Reddit, in a lot of cases being included. And actually, you'll see this exponential increase in model performance due to actually mostly using high quality synthetic data, not low quality human data. Why is that wrong?
00:19:52 · Harry Stebbings, Adarsh Hiremath
[00:19:52] S1: Why then does so many people tell me, including Jonathan at Grok, that actually synthetic data is often more high quality? It doesn't involve the drags of the Internet like Reddit, in a lot of cases being included. And actually, you'll see this exponential increase in model performance due to actually mostly using high quality synthetic data, not low quality human data. Why is that wrong? [00:20:16] S0: So the first thing is that it's not zero sum. Even in a world where human data is super important for the next generation of models, it doesn't mean that synthetic data won't also be important. So synthetic data will certainly be a part of the equation. But in a lot of ways, the bottleneck to unlocking and unleashing the next level of intelligence will be expert humans, which brings me back to the question and phrase that you used, low quality human data. Low quality human data certainly won't push the models to be better at anything. High quality human data will. Again, that's a talent assessment problem. The biggest lever on data quality for creating these post training sets, for example, is finding the right people, which, again, is is really, really hard to do. [00:20:56] S1: In terms of, like, compute and algorithms, how do you think about where we're at today in terms of them being a bottleneck? We mentioned data is a bottleneck. Are compute and algorithms too? How do you think about that?
00:20:56 · Harry Stebbings, Adarsh Hiremath
[00:20:56] S1: In terms of, like, compute and algorithms, how do you think about where we're at today in terms of them being a bottleneck? We mentioned data is a bottleneck. Are compute and algorithms too? How do you think about that? [00:21:07] S0: All of them are pieces of the same puzzle, but the era that we're entering requires really, really expert humans to make models better at very, very specific use cases. [00:21:18] S1: How long will that be the case for? [00:21:20] S0: For a very, very long time. There's this huge long tail of tasks that models can't do. [00:21:27] S1: Like what? Just help me understand. You're teaching me.
00:21:27 · Harry Stebbings, Adarsh Hiremath
[00:21:27] S1: Like what? Just help me understand. You're teaching me. [00:21:30] S0: Well, maybe we could take a step back. If we reach the point a couple of year... A hundred years from now where the models are able to do every single job and humans no longer have any work to do, society is gonna look really, really different. We're all going to be living on a UBI. We're all going to be playing video games all day, whatever it may be. But until that point comes, there's going to be a whole set of tasks that the models cannot do, whether they be specific economically valuable tasks, like maybe the job that a consultant could do or maybe a specific category of engineering or even like more niche things. Right? Maybe it's making the model better at like some specific hobby. And we're always gonna need to fill in the gaps, particularly in that long tail. And Harry, the other thing I'll say is I think people are really, really in this mindset of, like, this, like, unidec directional relationship between humans and AI. Right? Where I can't do something, I give it to the AI, the AI takes it to completion. But I think the more realistic breakdown is the AI for a specific use case might be able to get us 60, 70, 80% of the way there. But for that remaining 40, 30, 20%, you're going to need a human to be able to to take you all the way there. And the reality of the situation is finding that human will become harder and more valuable to do.