The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:20:59 · Harry Stebbings, Harrison Chase
[00:20:59] S1: So in two to three years' time, you think the majority is within the harness given the benefits that come from that context, context. But for very horizontal tasks, it will sit in gateway. [00:21:09] S0: I think that's exactly right. If you're building a vertical agent, I think routing needs to be in the harness. If you're thinking about a general purpose horizontal task, like just general chat or coding, it maybe be in the gateway. [00:21:21] S1: Is it a commodity? Like, does everyone just build it? [00:21:24] S0: I don't think routing is a commodity. I do think people will build it, but I think building it is quite hard even today. And I think part of the reason for that is the models just change so rapidly. So like, you can argue that certain models are better at different things, but then a new Opus or a new OpenAI model comes out, and it's just better at everything. And so then the value of routing kind of goes down. [00:21:43] S1: Do you trust evals?
00:21:43 · Harry Stebbings, Harrison Chase
[00:21:43] S1: Do you trust evals? [00:21:44] S0: I think evals are useful, but I wouldn't trust them fully. And so what I mean by that is I I I think a lot of the academic evals that are out there that are used when a lot of the models come out are not really that useful anymore. I don't... I think all the models that come out are kind of benchmark maxed against those evals, and so you don't learn a lot. I think internal evals that you have and that you build for your domain absolutely can still be useful. And so I would... I mean, this is the main reason that building evals is such an important part of building a vertical specific agent. I think you have to have some way of measuring things. And and I think you can generally trust those evals. But even then, one of the most important things to do is just look at the data. Just, like, look at the data and see what it's doing. Don't rely too much on kind of, like, a single number. I think this is why you want to have traces. You want to look at what's going on. Maybe your evals aren't perfect. Evals are really hard. You can have some sense, but maybe models are doing better, but in just a different way that you didn't know how to capture before. And so I do think, yes, I think you can trust vertical specific benchmarks, but you have to evolve those as well. And that's one of the main things that we've seen because these models just get better in different ways.
00:22:46 · Harry Stebbings, Harrison Chase
[00:22:46] S1: Can I ask you? You mentioned traces there. If, like, traces are the, like, exhaustive agent work, where is the value then? Is it the app, the observability layer, the model layer? Where is it? [00:22:58] S0: If you're building an agent, I think the value is at what makes that agent good at the domain or vertical that you're you're working in. So what I what I mean by that is, again, I think there's different ways to control what an agent does. There's the context, there's the harness, and there's the model. Having specific proprietary things for for any of those can be an advantage. I think controlling the the context is the easiest and and the most obvious thing to do when starting [00:23:23] S1: And that would
00:23:23 · Harry Stebbings, Harrison Chase
[00:23:23] S1: And that would [00:23:23] S0: be data. That would be data. That would be memory. That would be specific APIs, specific tools that you can use. Controlling the harness is, like, a little bit harder, but still... You can still put a lot of your domain knowledge around what should do, both routing and things in there. And then the model is the hardest control, but potentially the highest ceiling. The thing that lets you iterate on each of those is evals and traces. So there is a ton of value in these, but there's only value in that you, like, capture insights that you bring back to the context or bring back to the harness or bring back to the model. And that exact process can look different. So post training, great for bringing these insights from traces back to the model. For bringing things back to the context or memory, you're probably gonna wanna run like another agent over those traces and then have that agent output kind of like learnings from memory. And then for bringing the... These learnings in the traces back back to the harness, you're probably gonna wanna run an agent over those traces and update the code of the harness itself. And we actually have something that does this for people. It's called engine. It's an agent in Langsmith that runs over traces and tries to update code. And then when you update these things, you're gonna need to know that it's actually better, and this is where Evalues come in handy. [00:24:31] S1: Can I ask, how does continual learning change agents and how they work?
00:24:31 · Harry Stebbings, Harrison Chase
[00:24:31] S1: Can I ask, how does continual learning change agents and how they work? [00:24:36] S0: Continual learning, I don't think so much changes how agents work, but I think it changes the systems that you wanna have running around the agent. So what I mean by that is is is I think the core loop for a continual learning agent, the core inner loop maybe to say, will will still largely be the same. It will be an LLM running in a loop, calling tools, interacting with the model in in an environment. I I think what continual learning adds is an outer loop, an outer loop that looks at the exhaust of agents and then and and then does things with it. And, again, what what it does could be different depending on what continual learning means for you. So I think continual learning is also one of those words that doesn't have a super precise definition or maybe [00:25:18] S1: How would you define it? [00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file.
00:25:19 · Harrison Chase, Harry Stebbings
[00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file. [00:25:44] S1: If we look at the the context layer, we mentioned data being a part of that. We're seeing data companies scale to hundreds of millions of revenue in the fastest time ever. I mean, like, ever, ever, ever, and everyone is selling data. How do you expect this market to play out given your neutrality on not being involved in it, but being adjacent to it? Will this be a market of consolidation in years to come? How do you see it play out?