The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:16:45 · Harry Stebbings, Harrison Chase
[00:16:45] S1: What the difference? I'm sorry. [00:16:47] S0: The difference is in the harness level, it's in kind of the code of how it runs. And in the gateway, it's just an API call. So the benefits of putting it in the gateway level, because it's just an API call, you can drop it into any harness. You could put this this routing that's happening at the gateway level into Claude code, into any harness without having to change the code. The downside, and why I think we're more bullish on routing happening in the harness as are, I think, Factory and Cognition have also written things about this, is that in the harness, you just have more knowledge of what's going on. You have more awareness of the state. When you have an API, it's generally assuming that it's kind of like an OpenAI chat completions API, and so it only takes in a certain amount of state. You have more state in the harness to know, hey, like what's going on with the prompt cache? You want to try to keep the prompt cache warm and available. You can also change the prompt more easily. So a lot of these models have different prompting strategies. And so if you do the routing at the harness layer, you can more easily change the prompt. And so I think a lot of what we've seen is, one, this debate between kind of like gateway and harness and where does routing go. I think we probably believe it belongs in the [00:17:53] S1: And just so we understand, the open Routers and all the other providers, which there are several There are a lot. They are in the gateway.
00:17:53 · Harry Stebbings, Harrison Chase
[00:17:53] S1: And just so we understand, the open Routers and all the other providers, which there are several There are a lot. They are in the gateway. [00:18:00] S0: They are in the gateway. A lot of... Obviously, OpenRouter, incredible company, incredible usage. Had him [00:18:04] S1: on the show. Love Alex. [00:18:05] S0: Alex is great. I think a lot of why people use OpenRouter and Vercel and other gateways is largely for the model access. So I think there's also a difference between a router and a gateway. And so I would probably say most of OpenRouter's usage, and Alex obviously knows the numbers a lot better than I do, but if I had to guess, I would guess a lot of it just comes from being a gateway, not a router. So people specify the model they want upfront, and OpenRouter just provides a single API with a single API key for accessing that. In terms of like smart routing, I think that's actually being done like way less than people think. I And think when it's done, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done in terms of how they do routing. But I think right now, I I pretty strongly believe that, like, routing, you have to be harness aware. You have to know what the task is. You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's gonna be very domain and very harness specific.
00:18:05 · Harrison Chase, Harry Stebbings
[00:18:05] S0: Alex is great. I think a lot of why people use OpenRouter and Vercel and other gateways is largely for the model access. So I think there's also a difference between a router and a gateway. And so I would probably say most of OpenRouter's usage, and Alex obviously knows the numbers a lot better than I do, but if I had to guess, I would guess a lot of it just comes from being a gateway, not a router. So people specify the model they want upfront, and OpenRouter just provides a single API with a single API key for accessing that. In terms of like smart routing, I think that's actually being done like way less than people think. I And think when it's done, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done in terms of how they do routing. But I think right now, I I pretty strongly believe that, like, routing, you have to be harness aware. You have to know what the task is. You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's gonna be very domain and very harness specific. [00:19:05] S1: I'm sorry, when I listened to you, it seemed so unwaveringly obvious that routing should be at the harness layer, given the context and memory that is stored there. When I have such an obvious response, I like to understand why am I stupid? What is the bull case for it being at the gateway layer?
00:19:20 · Harrison Chase
[00:19:20] S0: I think when people talk about it at the gateway layer, I think there's maybe like two things that they're talking about. So one, it makes it way easier to port. So you can take Cloud Code. People love Cloud Code. Right? If you tell people, hey, you can't use Cloud Code. You have to use this other harness. And there are other good harnesses out there, but let's pretend for a second that you love Cloud Code. And you want to let them keep on using that, but you want to do some semblance of kind of, like, routing to maybe, like, a deep seek or or an open source model or something like that, then the only way you can do that is at the gateway layer because you don't control the harness. Side note, this is another argument for open source harnesses that you can control and modify. And then the second thing is when when people talk about kind of like routing at the gateway layer, I think they're oftentimes implicitly assuming that there's a domain. So there's kind of like two general domains that you could maybe argue that you can build general routers for. And I actually think OpenRouter has done this to some extent. You could argue coding. Coding is like the the most popular domain. That's where all the usage is going. Let's build a router that works really good for coding. Maybe we won't be like harness aware, but we'll be like use case aware. So we know different models are good for different things in coding, and we can put it at the gateway. The other thing you could say is like, hey, we've got this, general chatbot. Like, we're not trying to be best in the class at law or best in the class at finance, but we're just gonna be like a general chatbot across the company. And, like, we can know well enough that if you say, hi, how are you doing? You don't need, like, Opus for that. You can use a cheap model. And so I think when people talk about routing at the gateway level, they're implicitly kind of assuming that you're doing either, like, general purpose chat or coding, and you're not doing, like, a domain specific thing. As soon as you get into a domain specific thing, I really think it has to be at the harness layer. You're not gonna use an off the shelf gateway router for that.
00:20:59 · Harry Stebbings, Harrison Chase
[00:20:59] S1: So in two to three years' time, you think the majority is within the harness given the benefits that come from that context, context. But for very horizontal tasks, it will sit in gateway. [00:21:09] S0: I think that's exactly right. If you're building a vertical agent, I think routing needs to be in the harness. If you're thinking about a general purpose horizontal task, like just general chat or coding, it maybe be in the gateway. [00:21:21] S1: Is it a commodity? Like, does everyone just build it? [00:21:24] S0: I don't think routing is a commodity. I do think people will build it, but I think building it is quite hard even today. And I think part of the reason for that is the models just change so rapidly. So like, you can argue that certain models are better at different things, but then a new Opus or a new OpenAI model comes out, and it's just better at everything. And so then the value of routing kind of goes down. [00:21:43] S1: Do you trust evals?
00:21:43 · Harry Stebbings, Harrison Chase
[00:21:43] S1: Do you trust evals? [00:21:44] S0: I think evals are useful, but I wouldn't trust them fully. And so what I mean by that is I I I think a lot of the academic evals that are out there that are used when a lot of the models come out are not really that useful anymore. I don't... I think all the models that come out are kind of benchmark maxed against those evals, and so you don't learn a lot. I think internal evals that you have and that you build for your domain absolutely can still be useful. And so I would... I mean, this is the main reason that building evals is such an important part of building a vertical specific agent. I think you have to have some way of measuring things. And and I think you can generally trust those evals. But even then, one of the most important things to do is just look at the data. Just, like, look at the data and see what it's doing. Don't rely too much on kind of, like, a single number. I think this is why you want to have traces. You want to look at what's going on. Maybe your evals aren't perfect. Evals are really hard. You can have some sense, but maybe models are doing better, but in just a different way that you didn't know how to capture before. And so I do think, yes, I think you can trust vertical specific benchmarks, but you have to evolve those as well. And that's one of the main things that we've seen because these models just get better in different ways.