00:10:11 · Irene Solaiman, Rebecca Balan
[00:10:11] S2: That that that word is so important, that phrase, good enough. I think this really comes down to a trade off for how much you also wanna spend on token maxing. I shudder to think about token cost without open weight models, and this is not a reflection of the Frontier Labs. It's just incentives in in the marketplace. For what folks want to use open models for, what they're capable for, they can do quite a lot, if not the same agentic tests. I rely entirely on almost exclusively GLM 5.3 these days, sometimes a kidney k three using a Herm... Hermes agent for almost everything. And I think this is the trade off for it might need extra prompting. It doesn't give me refusals in the way that a frontier model would. But I think this really depends where I I would say I use it for everything. [00:11:04] S1: Yeah. And, know, it's not just about, like, you know, there's a good enough model, but so much of it increasingly is the harness. Right? It's it's the it's the structure, the infrastructure that's built around the the model itself. So I mentioned... You've mentioned to me before that there are, like, less capable models with a better harness can actually outperform a frontier model. So how do you think that this is affecting, you know, overall performance?
00:11:04 · Rebecca Balan, Irene Solaiman
[00:11:04] S1: Yeah. And, know, it's not just about, like, you know, there's a good enough model, but so much of it increasingly is the harness. Right? It's it's the it's the structure, the infrastructure that's built around the the model itself. So I mentioned... You've mentioned to me before that there are, like, less capable models with a better harness can actually outperform a frontier model. So how do you think that this is affecting, you know, overall performance? [00:11:27] S2: Spot on. On these more specialized startups or tasks, one thing that we had discussed before, thinking about cybersecurity a lot, there are harnesses that can zoom in to specific cyber defense tasks and leverage models, and it will still use a generally powerful model that can address vulnerability identities, for example. These kinds of harnesses, where just so many, are are popping up more and more, are custom made. This can be a revenue stream in and of itself of folks who are able to design harnesses that our economy needs right now.
00:11:27 · Irene Solaiman, Rebecca Balan
[00:11:27] S2: Spot on. On these more specialized startups or tasks, one thing that we had discussed before, thinking about cybersecurity a lot, there are harnesses that can zoom in to specific cyber defense tasks and leverage models, and it will still use a generally powerful model that can address vulnerability identities, for example. These kinds of harnesses, where just so many, are are popping up more and more, are custom made. This can be a revenue stream in and of itself of folks who are able to design harnesses that our economy needs right now. [00:12:06] S1: Yeah. And we're seeing more and more of that. And it's it's like, you know, being a journalist, my inbox is filled with start ups trying to, like, ride the lightest wave. So it was... You know, at first, it was just, like, app companies built on AI, and then it was agent companies. Those are still pouring in. And now it's like harness companies. I guess agent is a... Like the harness is the agent or, you know, they're they're related in a lot of ways. You know, at Disrupt, you and I talked a lot about model evaluations. There's a lot of work that can be done to make sure model evaluations are are being done in the right way, but there's not... One of the things that appears to be missing is that there's no good evaluation for harnesses and tooling. Yeah. What do you think about that?
00:12:06 · Rebecca Balan, Irene Solaiman
[00:12:06] S1: Yeah. And we're seeing more and more of that. And it's it's like, you know, being a journalist, my inbox is filled with start ups trying to, like, ride the lightest wave. So it was... You know, at first, it was just, like, app companies built on AI, and then it was agent companies. Those are still pouring in. And now it's like harness companies. I guess agent is a... Like the harness is the agent or, you know, they're they're related in a lot of ways. You know, at Disrupt, you and I talked a lot about model evaluations. There's a lot of work that can be done to make sure model evaluations are are being done in the right way, but there's not... One of the things that appears to be missing is that there's no good evaluation for harnesses and tooling. Yeah. What do you think about that? [00:12:44] S2: Yes. Yeah. I loved... I love generally talking with you, but especially in evaluations. It's a nerdy area that has gotten attention for how important it is for deployment, but it is an area of science. We just have a lot more science around natural language processing, the basis field for LLMs. We don't have a lot of background for harnesses as they're deployed now as relatively novel. We don't have explicit evaluation suites for harnesses in ways that are standardized predeployment of harnesses. And the way that... I think a big barrier for this is just how darn costly they are. AgenTeq workloads, anybody who's used them, just cost a lot of money for all of these tools. And so if you're running evaluation, it'll easily go up into the millions.
00:12:44 · Irene Solaiman, Rebecca Balan
[00:12:44] S2: Yes. Yeah. I loved... I love generally talking with you, but especially in evaluations. It's a nerdy area that has gotten attention for how important it is for deployment, but it is an area of science. We just have a lot more science around natural language processing, the basis field for LLMs. We don't have a lot of background for harnesses as they're deployed now as relatively novel. We don't have explicit evaluation suites for harnesses in ways that are standardized predeployment of harnesses. And the way that... I think a big barrier for this is just how darn costly they are. AgenTeq workloads, anybody who's used them, just cost a lot of money for all of these tools. And so if you're running evaluation, it'll easily go up into the millions. [00:13:32] S1: Yeah. Yeah. So how can a, like, a startup manage this? Right? How can they know whether a cheaper model and its surrounding tools are are good enough for its product cheap enough for its product if there's no reliable way to measure, you know, the spend and ROI and just like what you're getting?
00:13:32 · Rebecca Balan, Irene Solaiman
[00:13:32] S1: Yeah. Yeah. So how can a, like, a startup manage this? Right? How can they know whether a cheaper model and its surrounding tools are are good enough for its product cheap enough for its product if there's no reliable way to measure, you know, the spend and ROI and just like what you're getting? [00:13:49] S2: It's gonna be a trade off for the near term and long term. In the near term, it might be easier just to run inference, to call a friend to your lab. If you have, like, high intensity tasks, if you don't have a lot of AI experts on hand, totally understandable to use what's easy, and a lot of the time, that's gonna be closed. If you want something that you wanna scale up a bit, you know you'll use it longer term, and you can... This is why a lot of folks are buying GPUs and not renting. You can set it up on prem. You can optimize that model over time without renting out the cost of compute going up. Your long term costs will likely plateau or decrease, desirable depending on the business that you're running. And for the choices that you're making for high intensity coding tasks, the latest open weight models, Kimi K3 DeepSeek v4.1 Flash and the Qwen 3.8 Max are optimized for coding tasks. So you can look at these types of benchmarks. You can look at the types of coding tasks you're interested in. Or if, again, you have these skills, you can fine tune that model. There are a lot on Hugging Face, a lot of data sets for fine tuning to optimize for certain tasks.