The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:18:05 · Harrison Chase, Harry Stebbings
[00:18:05] S0: Alex is great. I think a lot of why people use OpenRouter and Vercel and other gateways is largely for the model access. So I think there's also a difference between a router and a gateway. And so I would probably say most of OpenRouter's usage, and Alex obviously knows the numbers a lot better than I do, but if I had to guess, I would guess a lot of it just comes from being a gateway, not a router. So people specify the model they want upfront, and OpenRouter just provides a single API with a single API key for accessing that. In terms of like smart routing, I think that's actually being done like way less than people think. I And think when it's done, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done in terms of how they do routing. But I think right now, I I pretty strongly believe that, like, routing, you have to be harness aware. You have to know what the task is. You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's gonna be very domain and very harness specific. [00:19:05] S1: I'm sorry, when I listened to you, it seemed so unwaveringly obvious that routing should be at the harness layer, given the context and memory that is stored there. When I have such an obvious response, I like to understand why am I stupid? What is the bull case for it being at the gateway layer?
00:19:20 · Harrison Chase
[00:19:20] S0: I think when people talk about it at the gateway layer, I think there's maybe like two things that they're talking about. So one, it makes it way easier to port. So you can take Cloud Code. People love Cloud Code. Right? If you tell people, hey, you can't use Cloud Code. You have to use this other harness. And there are other good harnesses out there, but let's pretend for a second that you love Cloud Code. And you want to let them keep on using that, but you want to do some semblance of kind of, like, routing to maybe, like, a deep seek or or an open source model or something like that, then the only way you can do that is at the gateway layer because you don't control the harness. Side note, this is another argument for open source harnesses that you can control and modify. And then the second thing is when when people talk about kind of like routing at the gateway layer, I think they're oftentimes implicitly assuming that there's a domain. So there's kind of like two general domains that you could maybe argue that you can build general routers for. And I actually think OpenRouter has done this to some extent. You could argue coding. Coding is like the the most popular domain. That's where all the usage is going. Let's build a router that works really good for coding. Maybe we won't be like harness aware, but we'll be like use case aware. So we know different models are good for different things in coding, and we can put it at the gateway. The other thing you could say is like, hey, we've got this, general chatbot. Like, we're not trying to be best in the class at law or best in the class at finance, but we're just gonna be like a general chatbot across the company. And, like, we can know well enough that if you say, hi, how are you doing? You don't need, like, Opus for that. You can use a cheap model. And so I think when people talk about routing at the gateway level, they're implicitly kind of assuming that you're doing either, like, general purpose chat or coding, and you're not doing, like, a domain specific thing. As soon as you get into a domain specific thing, I really think it has to be at the harness layer. You're not gonna use an off the shelf gateway router for that.
00:20:59 · Harry Stebbings, Harrison Chase
[00:20:59] S1: So in two to three years' time, you think the majority is within the harness given the benefits that come from that context, context. But for very horizontal tasks, it will sit in gateway. [00:21:09] S0: I think that's exactly right. If you're building a vertical agent, I think routing needs to be in the harness. If you're thinking about a general purpose horizontal task, like just general chat or coding, it maybe be in the gateway. [00:21:21] S1: Is it a commodity? Like, does everyone just build it? [00:21:24] S0: I don't think routing is a commodity. I do think people will build it, but I think building it is quite hard even today. And I think part of the reason for that is the models just change so rapidly. So like, you can argue that certain models are better at different things, but then a new Opus or a new OpenAI model comes out, and it's just better at everything. And so then the value of routing kind of goes down. [00:21:43] S1: Do you trust evals?
00:21:43 · Harry Stebbings, Harrison Chase
[00:21:43] S1: Do you trust evals? [00:21:44] S0: I think evals are useful, but I wouldn't trust them fully. And so what I mean by that is I I I think a lot of the academic evals that are out there that are used when a lot of the models come out are not really that useful anymore. I don't... I think all the models that come out are kind of benchmark maxed against those evals, and so you don't learn a lot. I think internal evals that you have and that you build for your domain absolutely can still be useful. And so I would... I mean, this is the main reason that building evals is such an important part of building a vertical specific agent. I think you have to have some way of measuring things. And and I think you can generally trust those evals. But even then, one of the most important things to do is just look at the data. Just, like, look at the data and see what it's doing. Don't rely too much on kind of, like, a single number. I think this is why you want to have traces. You want to look at what's going on. Maybe your evals aren't perfect. Evals are really hard. You can have some sense, but maybe models are doing better, but in just a different way that you didn't know how to capture before. And so I do think, yes, I think you can trust vertical specific benchmarks, but you have to evolve those as well. And that's one of the main things that we've seen because these models just get better in different ways.
00:22:46 · Harry Stebbings, Harrison Chase
[00:22:46] S1: Can I ask you? You mentioned traces there. If, like, traces are the, like, exhaustive agent work, where is the value then? Is it the app, the observability layer, the model layer? Where is it? [00:22:58] S0: If you're building an agent, I think the value is at what makes that agent good at the domain or vertical that you're you're working in. So what I what I mean by that is, again, I think there's different ways to control what an agent does. There's the context, there's the harness, and there's the model. Having specific proprietary things for for any of those can be an advantage. I think controlling the the context is the easiest and and the most obvious thing to do when starting [00:23:23] S1: And that would
00:23:23 · Harry Stebbings, Harrison Chase
[00:23:23] S1: And that would [00:23:23] S0: be data. That would be data. That would be memory. That would be specific APIs, specific tools that you can use. Controlling the harness is, like, a little bit harder, but still... You can still put a lot of your domain knowledge around what should do, both routing and things in there. And then the model is the hardest control, but potentially the highest ceiling. The thing that lets you iterate on each of those is evals and traces. So there is a ton of value in these, but there's only value in that you, like, capture insights that you bring back to the context or bring back to the harness or bring back to the model. And that exact process can look different. So post training, great for bringing these insights from traces back to the model. For bringing things back to the context or memory, you're probably gonna wanna run like another agent over those traces and then have that agent output kind of like learnings from memory. And then for bringing the... These learnings in the traces back back to the harness, you're probably gonna wanna run an agent over those traces and update the code of the harness itself. And we actually have something that does this for people. It's called engine. It's an agent in Langsmith that runs over traces and tries to update code. And then when you update these things, you're gonna need to know that it's actually better, and this is where Evalues come in handy. [00:24:31] S1: Can I ask, how does continual learning change agents and how they work?