The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:14:50 · Harrison Chase
[00:14:50] S0: I think what we've seen in regards to post training is that a lot of the post training that practically makes sense right now is probably for, like, simpler tasks that make up a larger harness. So what I mean by that is, like, when you have an agent harness that's doing interesting work, it's actually very rarely just one model running the loop. That might be the core of it, but there's a bunch of other model calls that happen either as like guardrails or routing or sub agents or sub workflows or things like that. And so a lot of the places where we see post training being efficient now are for simpler tasks, like classification or extraction or summarization that might be a tool the agent uses or a guardrail or things like that. I think those types of post training right now make a ton of sense and will continue going forward. I think the question that is more unanswered is for this core model that's driving the loop. Because the the main way an agent works is there's an LLM running in a loop calling tools. And so the question is for this model that's driving that loop, will those be kind of like post trained models, or will those be kind of like frontier models? And I think this is where we see more of like a back and forth, where like every time a new frontier model comes out, the better they are at just running this kind of like core loop. We do see people like Harvey and Ramp and others kind of like experimenting with post training models for this core loop, and so I think Harvey recently in the past month or so announced something exactly in that vein. Question. Think
00:16:09 · Harry Stebbings, Harrison Chase
[00:16:09] S1: If you were building your own vertical application, would you post train your own or not? [00:16:14] S0: Not to start. Absolutely not to start. But like if we're talking like five years in the future, assuming that I'm getting a ton of usage, I think it could be a realistic possibility. [00:16:22] S1: One of the things that harnesses also need to do, I imagine, is model routing, depending on request. How much value is in the model routing layer? [00:16:32] S0: Model routing is so interesting, and I think it's one of the hottest and maybe most divisive topics right now. So a few thoughts. First, there's a question of where the model routing should even be. Should it be at the harness level, or should it be in the gateway level? And this is getting a little bit into the technical weeds. [00:16:45] S1: What the difference? I'm sorry.
00:16:45 · Harry Stebbings, Harrison Chase
[00:16:45] S1: What the difference? I'm sorry. [00:16:47] S0: The difference is in the harness level, it's in kind of the code of how it runs. And in the gateway, it's just an API call. So the benefits of putting it in the gateway level, because it's just an API call, you can drop it into any harness. You could put this this routing that's happening at the gateway level into Claude code, into any harness without having to change the code. The downside, and why I think we're more bullish on routing happening in the harness as are, I think, Factory and Cognition have also written things about this, is that in the harness, you just have more knowledge of what's going on. You have more awareness of the state. When you have an API, it's generally assuming that it's kind of like an OpenAI chat completions API, and so it only takes in a certain amount of state. You have more state in the harness to know, hey, like what's going on with the prompt cache? You want to try to keep the prompt cache warm and available. You can also change the prompt more easily. So a lot of these models have different prompting strategies. And so if you do the routing at the harness layer, you can more easily change the prompt. And so I think a lot of what we've seen is, one, this debate between kind of like gateway and harness and where does routing go. I think we probably believe it belongs in the [00:17:53] S1: And just so we understand, the open Routers and all the other providers, which there are several There are a lot. They are in the gateway.
00:17:53 · Harry Stebbings, Harrison Chase
[00:17:53] S1: And just so we understand, the open Routers and all the other providers, which there are several There are a lot. They are in the gateway. [00:18:00] S0: They are in the gateway. A lot of... Obviously, OpenRouter, incredible company, incredible usage. Had him [00:18:04] S1: on the show. Love Alex. [00:18:05] S0: Alex is great. I think a lot of why people use OpenRouter and Vercel and other gateways is largely for the model access. So I think there's also a difference between a router and a gateway. And so I would probably say most of OpenRouter's usage, and Alex obviously knows the numbers a lot better than I do, but if I had to guess, I would guess a lot of it just comes from being a gateway, not a router. So people specify the model they want upfront, and OpenRouter just provides a single API with a single API key for accessing that. In terms of like smart routing, I think that's actually being done like way less than people think. I And think when it's done, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done in terms of how they do routing. But I think right now, I I pretty strongly believe that, like, routing, you have to be harness aware. You have to know what the task is. You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's gonna be very domain and very harness specific.
00:18:05 · Harrison Chase, Harry Stebbings
[00:18:05] S0: Alex is great. I think a lot of why people use OpenRouter and Vercel and other gateways is largely for the model access. So I think there's also a difference between a router and a gateway. And so I would probably say most of OpenRouter's usage, and Alex obviously knows the numbers a lot better than I do, but if I had to guess, I would guess a lot of it just comes from being a gateway, not a router. So people specify the model they want upfront, and OpenRouter just provides a single API with a single API key for accessing that. In terms of like smart routing, I think that's actually being done like way less than people think. I And think when it's done, it's built into the harnesses by folks like Factory. I think Factory does a tremendous job about this. They've written a number of technical blogs that are very well done in terms of how they do routing. But I think right now, I I pretty strongly believe that, like, routing, you have to be harness aware. You have to know what the task is. You have to know what the harness looks like. So I don't think you'll have routing that works for law and works for finance and works for coding and have it all be the same. It's gonna be very domain and very harness specific. [00:19:05] S1: I'm sorry, when I listened to you, it seemed so unwaveringly obvious that routing should be at the harness layer, given the context and memory that is stored there. When I have such an obvious response, I like to understand why am I stupid? What is the bull case for it being at the gateway layer?
00:19:20 · Harrison Chase
[00:19:20] S0: I think when people talk about it at the gateway layer, I think there's maybe like two things that they're talking about. So one, it makes it way easier to port. So you can take Cloud Code. People love Cloud Code. Right? If you tell people, hey, you can't use Cloud Code. You have to use this other harness. And there are other good harnesses out there, but let's pretend for a second that you love Cloud Code. And you want to let them keep on using that, but you want to do some semblance of kind of, like, routing to maybe, like, a deep seek or or an open source model or something like that, then the only way you can do that is at the gateway layer because you don't control the harness. Side note, this is another argument for open source harnesses that you can control and modify. And then the second thing is when when people talk about kind of like routing at the gateway layer, I think they're oftentimes implicitly assuming that there's a domain. So there's kind of like two general domains that you could maybe argue that you can build general routers for. And I actually think OpenRouter has done this to some extent. You could argue coding. Coding is like the the most popular domain. That's where all the usage is going. Let's build a router that works really good for coding. Maybe we won't be like harness aware, but we'll be like use case aware. So we know different models are good for different things in coding, and we can put it at the gateway. The other thing you could say is like, hey, we've got this, general chatbot. Like, we're not trying to be best in the class at law or best in the class at finance, but we're just gonna be like a general chatbot across the company. And, like, we can know well enough that if you say, hi, how are you doing? You don't need, like, Opus for that. You can use a cheap model. And so I think when people talk about routing at the gateway level, they're implicitly kind of assuming that you're doing either, like, general purpose chat or coding, and you're not doing, like, a domain specific thing. As soon as you get into a domain specific thing, I really think it has to be at the harness layer. You're not gonna use an off the shelf gateway router for that.