The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:29:31 · Harry Stebbings, Naveen Rao
[00:29:31] S1: I love a question. Like, what do you believe that today looks kooky, weird, nuts, that in five years' time will be very commonplace and agreed. I remember putting my credit card into PayPal and my parents telling me off. And, like, remember, like, OkCupid and being like, you find your husband or wife online? What a ridiculous idea. Nuts at the time. I think that we will trust our agents more than we trust ourselves. [00:29:55] S0: I think at some point, that's true. [00:29:57] S1: But you think that we are far from that today?
00:29:57 · Harry Stebbings, Naveen Rao
[00:29:57] S1: But you think that we are far from that today? [00:29:59] S0: Yeah. I... We're we're not there today. And expertise doesn't really go out of style. It's just gonna take on different forms. And what I think a future will look like is that we start building... We'll call it models, but I actually don't think it's a model. Think it's a whole machine. I think of intelligence as having a physical and having the intelligence layer as one. They're they're not separable. I have a machine. It's almost like an organism that I can teach to be an expert at various tasks. And the value will come from building that expertise up. It's like, which school did you go to? How did you learn how to be a neurosurgeon or a chef or whatever? That provenance matters, right? Like, you went and studied under this person and you learned their methodology, their thinking. That's where the value came from. You or any human is an intelligent machine, top to bottom, but it was the way you were taught that actually made you valuable. And I think we're gonna start to see something similar in machines. And we can replicate that intelligence and all of that, but but the way you were taught is actually gonna be the important thing. And so part of that is actually knowing how to get to an end result that is real, that will work, that is verified.
00:31:03 · Harry Stebbings, Naveen Rao
[00:31:03] S1: When you think about the value in that learning, a lot of that is in the data that one accrues over time in the years that you're doing something. We see data companies today. I'm in Macore, but we see tens hit astronomical levels of revenue that previously were, like, astonishingly large. I mean, that's still astonishingly large. But in such short time, Naveen, dude, you're such a smart dude. It's such a joy talking to you. I just kinda wanna ask you lots of questions about the world. What happens to this market? Does it just consolidate over time? Does it unbundle more and more and you build more defensibility? What happens? [00:31:39] S0: I think it specializes, to be honest. And I think if you specialize and actually start to work on important problems, you have a big market that's defensible. You know, it's it's... Again, it's a power law for skills. You could argue that all humans are capable of learning many skills, but certain skills are more valuable than others. And the same thing will be true with data and curricula that you you build. Right? [00:32:01] S1: And so we see the specialization of data companies. So you don't think they're commodities?
00:32:01 · Harry Stebbings, Naveen Rao
[00:32:01] S1: And so we see the specialization of data companies. So you don't think they're commodities? [00:32:06] S0: Yeah. I I don't think it's commodity. I I think it started as a commodity because it it was sort of like at volume gathering of of observations. Right? But as you start to construct higher quality datasets, that's what we call it in the AI world, like higher quality for coding, higher quality for a chain of thought or whatever, you actually start to have a lot of value in the data. The model is almost just a manifestation of what it learned in a sense. So the data layer, I think, is gonna be pretty important going forward. I I don't see that commoditizing. I think building models and knowing how to choose that data is actually the game right now. The big model companies, it's largely a data problem. I mean, they are making advances on the model itself, but that's not been where most of the capabilities come from. It's been from curating data, using different sorts of loss functions. Like, we had pre training before, which was a very particular kind of loss. Then we moved to what we call mid training now, which is reinforcement learning, supervised fine tuning, all these different sort of mixes. So that curriculum, how you went from pre training to all these different sorts of trainings to modify the behavior of the model is actually the secret sauce.
00:33:11 · Harry Stebbings, Naveen Rao
[00:33:11] S1: Can I ask you? We've seen... You mentioned that kind of pretraining RL, and then, obviously, a real shift has moved to post training. With that shift of emphasis from pretraining to post training, you think that is an enduring permanent shift in the efficiency of model improvements and how we get there? Or will we move back to RL being the primary or pre being the primary? How does that look in terms of importance over time? [00:33:35] S0: Well, I I don't know that it's more efficient. Actually, I think RL is more inefficient. If you look at it in terms of, you know, how much better the model gets per unit compute or per unit dollar, it's actually more inefficient. But it it does keep moving forward. Right? I think the idea is that we want to make the models better at all... At the tasks that are important. And so I think it's it's probably for the next several years, yes, very important. I think we'll start to build specialization in those middle layers, maybe even in the pretraining layer. So I don't think the model companies are thinking very long terms in in terms of performance. Like, I'm gonna make some massive breakthrough on the way I think about an AI system that's gonna give me an advantage. I would say the big labs are not doing that as much. [00:34:15] S1: Do you think they can afford to in the day by day race that is this current ecosystem?
00:34:15 · Harry Stebbings, Naveen Rao
[00:34:15] S1: Do you think they can afford to in the day by day race that is this current ecosystem? [00:34:21] S0: That's the problem. It's really fast. I mean, you've gotta be on top of it. I mean, like, if you're one month late on a model, that's that's disaster if you're OpenAI or Anthropic or Grok or Google. They can't afford to take those big risks just yet. You know, you're seeing this, like, you know, when Elon spun up, x AI, he basically took, okay. Here's the paradigm. Let's go let's go hire the people, and boom. He got to the frontier pretty fast because he had the resources. So what that points out to me was that there wasn't a ton of, like, very deep innovation. It was taking the pieces, putting them together with a team that knows how to build them and executing very fast. I think that's very different than trying to come up with a new paradigm. And, you know, you look at, like, Yanl Kun's company and, you know, Fei Li's company, they were they were trying to do deeper innovations. And I think that's what... That might be in the realm of startups, and it starts starts to look like drug development. Right? Like, all the big risks are taken by startups, and all the incremental stuff is done by big companies. The big companies buy the startups when they show promise. [00:35:16] S1: I sometimes ask simple questions, but I don't know the answers, and you're very smart. Why does everyone get so excited by RL today? You mentioned it kind of the mid learning there, and I just have so many people. I I just... It's all about RL. I'm like, oh, I I agree. Yeah. I agree.