The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:11:55 · Harry Stebbings, Harrison Chase
[00:11:55] S1: automation. If value is derived through verification, who wins? The harness? [00:12:02] S0: If value is derived through verification, I think the people who win are the domain experts, the people who are able to kind of, like, quantify what that value means, which is very hard. Like, building evals is a very hard job. And so I think a lot of the value will accrue to whoever can kind of write those verifiers, capture that domain knowledge. I'm massively bullish on vertical agents for different domains for this reason, because I think the domain experts who are best suited to write those verifiers or build those evals, as we kind of talk about it, They will be able to build harnesses and models. And it it won't be just the harness. It won't be just the model. It will be some combination of them. So so very practically, if you're getting started building a vertical agent, I would highly suggest you start at the harness layer. You do context engineering. You kind of, like, tweak the harness. And then when you've got users, then you start tweaking the model in post training. I think it's it's much higher ROI to start at the context or harness layer first. And so I think we'll have all these different verticals. They'll be able to quantify what good looks to them in a very domain specific way, and that'll let them build not only, like, vertical agents, but also vertical agent experiences. Different agents might have different experiences. And so I think, like, a lot of the value massively bullish on vertical agents, and so we want to enable a lot of that.
00:13:13 · Harry Stebbings
[00:13:13] S1: You said if I was building that, I would start at the harness layer. We're seeing a lot, whether it's Harvey or I think it's Ramp, build their own models. To what extent are they actually building their own models? Mike Cannonbrooks at Alasdians, don't know, show we really say that it's largely kind of, respectively, BS, and they're fine tuning open source Chinese models. Is that true? And and how do you think about that?
00:13:35 · Harrison Chase
[00:13:35] S0: I think a lot of it is they are post training open open weight models. Absolutely. I still think that's a valuable thing to do. I don't think there's kind of like anything wrong with that. I think that... I think there's different, like, dimensions that you can go in terms of owning your intelligence. I think the cheapest way is you just control the context. You use an off the shelf harness, off the shelf model. The next level down is you start tweaking the harness, and and that gives you more control than just the context, but it's a it's a little bit more expensive to do. The next level down is is you post train an open weight model. And so you might take an an existing open weight Chinese model and post train it for your domain. And then the very kind of, like, bottom layer is they actually train a model from scratch. I think many folks are training kind of like a model from scratch. I think most are post training. I think if you're training a model from scratch, you're probably at NeoLab. Think those are one of the few that are doing But, yeah, the ramps and the Harveys of the world, they have a ton of talent. They have a ton of data. They have a bunch of usage and are trying to think about really optimizing it, and so they've gotten all the juice they can out of the context, all the juice they can out of the harness, and now they've arrived at basically post training of these models.
00:14:35 · Harry Stebbings
[00:14:35] S1: Can I ask you, the jury's kind of out on the efficacy of these specialized models in the post training realm like we mentioned there? Some reports I've seen say they're massively more efficient and better, and some less so. How do you think this will play out?
00:14:50 · Harrison Chase
[00:14:50] S0: I think what we've seen in regards to post training is that a lot of the post training that practically makes sense right now is probably for, like, simpler tasks that make up a larger harness. So what I mean by that is, like, when you have an agent harness that's doing interesting work, it's actually very rarely just one model running the loop. That might be the core of it, but there's a bunch of other model calls that happen either as like guardrails or routing or sub agents or sub workflows or things like that. And so a lot of the places where we see post training being efficient now are for simpler tasks, like classification or extraction or summarization that might be a tool the agent uses or a guardrail or things like that. I think those types of post training right now make a ton of sense and will continue going forward. I think the question that is more unanswered is for this core model that's driving the loop. Because the the main way an agent works is there's an LLM running in a loop calling tools. And so the question is for this model that's driving that loop, will those be kind of like post trained models, or will those be kind of like frontier models? And I think this is where we see more of like a back and forth, where like every time a new frontier model comes out, the better they are at just running this kind of like core loop. We do see people like Harvey and Ramp and others kind of like experimenting with post training models for this core loop, and so I think Harvey recently in the past month or so announced something exactly in that vein. Question. Think
00:16:09 · Harry Stebbings, Harrison Chase
[00:16:09] S1: If you were building your own vertical application, would you post train your own or not? [00:16:14] S0: Not to start. Absolutely not to start. But like if we're talking like five years in the future, assuming that I'm getting a ton of usage, I think it could be a realistic possibility. [00:16:22] S1: One of the things that harnesses also need to do, I imagine, is model routing, depending on request. How much value is in the model routing layer? [00:16:32] S0: Model routing is so interesting, and I think it's one of the hottest and maybe most divisive topics right now. So a few thoughts. First, there's a question of where the model routing should even be. Should it be at the harness level, or should it be in the gateway level? And this is getting a little bit into the technical weeds. [00:16:45] S1: What the difference? I'm sorry.