00:07:12 · Irene Solaiman, Rebecca Balan
[00:07:12] S2: It's a tricky decision on if you're releasing an open model that doesn't quite reach or beat Frontier level. A lot of folks love using Frontier. I mean, they're great products. They're easy. Then how do you get those kinds of users interested in deploying? And this is where these customized licenses come in for The US model providers on the open weight side. Some of the more visible ones who have done a great job are, I mean, Meta historically has been an open weight champion, and RC, Reflection have been releasing with permissive licenses, but they also do enterprise contracts. So you'll see with Reflection, they did a enterprise agreement with South Korea and have a partnership to build a sovereign AI cloud. So there are other ways to do these kinds of custom solutions. [00:08:03] S1: Okay. So so almost like a white glove service. Will, like, you know, you wanna build your own sovereign or enterprise solution and you wanna use open source model school, we'll build it for you kind of thing. [00:08:12] S2: Yeah. It's... For how much interest there is in AI, there are very few people that really know how to customize this relatively.
00:08:12 · Irene Solaiman, Rebecca Balan
[00:08:12] S2: Yeah. It's... For how much interest there is in AI, there are very few people that really know how to customize this relatively. [00:08:19] S1: Yeah. I mean, we're seeing that with, know, the whole... I mean, it's kind of side note, but with the whole FTE trend that we're seeing, everyone's... You know, this is becoming the job in AI as a Exactly. For an employed engineer because there's so few people that actually, you know, know enough about business on the one side, and then how to, you know, work with AI tools on the other in order to build anything that's bespoke for enterprises or governments. [00:08:39] S2: Exactly. You [00:08:40] S1: know, a lot of our listeners are founders. So why should a founder care? Like, what do they need to check before trying to build a product on an open model?
00:08:40 · Rebecca Balan, Irene Solaiman
[00:08:40] S1: know, a lot of our listeners are founders. So why should a founder care? Like, what do they need to check before trying to build a product on an open model? [00:08:51] S2: Before building, I mean, this is a question of what are their preferences for the task, and this is where we get into routing. We don't have great model routers to pick and choose the right model for the task. Being able to create a stack where you have a potentially lower cost but effective model for certain tasks, let's just say, like, I'm thinking a lot about cybersecurity, so some level of cyber monitoring for giving summaries for low stakes infrastructure, you don't probably need something that is the the most premier frontier model, but you will need to understand what is the capability of the model for the task that I want, where do I put it throughout my infrastructure, And do I have the capacity on hand to run this model, or do I need to rely on inference, whether that be an open weight, which could be cheaper than a Frontier or a Frontier model itself? [00:09:51] S1: Yeah. Okay. That's interesting. Like, this is something I keep hearing, that not every company needs the most powerful model for every task. Right? So what kinds of work are open models? You know, some of the maybe smaller ones, they increasingly good enough to handle? Like, where? And and then, you know, the second question for that is where do the more advanced models still have a meaningful edge?
00:09:51 · Rebecca Balan, Irene Solaiman
[00:09:51] S1: Yeah. Okay. That's interesting. Like, this is something I keep hearing, that not every company needs the most powerful model for every task. Right? So what kinds of work are open models? You know, some of the maybe smaller ones, they increasingly good enough to handle? Like, where? And and then, you know, the second question for that is where do the more advanced models still have a meaningful edge? [00:10:11] S2: That that that word is so important, that phrase, good enough. I think this really comes down to a trade off for how much you also wanna spend on token maxing. I shudder to think about token cost without open weight models, and this is not a reflection of the Frontier Labs. It's just incentives in in the marketplace. For what folks want to use open models for, what they're capable for, they can do quite a lot, if not the same agentic tests. I rely entirely on almost exclusively GLM 5.3 these days, sometimes a kidney k three using a Herm... Hermes agent for almost everything. And I think this is the trade off for it might need extra prompting. It doesn't give me refusals in the way that a frontier model would. But I think this really depends where I I would say I use it for everything.
00:10:11 · Irene Solaiman, Rebecca Balan
[00:10:11] S2: That that that word is so important, that phrase, good enough. I think this really comes down to a trade off for how much you also wanna spend on token maxing. I shudder to think about token cost without open weight models, and this is not a reflection of the Frontier Labs. It's just incentives in in the marketplace. For what folks want to use open models for, what they're capable for, they can do quite a lot, if not the same agentic tests. I rely entirely on almost exclusively GLM 5.3 these days, sometimes a kidney k three using a Herm... Hermes agent for almost everything. And I think this is the trade off for it might need extra prompting. It doesn't give me refusals in the way that a frontier model would. But I think this really depends where I I would say I use it for everything. [00:11:04] S1: Yeah. And, know, it's not just about, like, you know, there's a good enough model, but so much of it increasingly is the harness. Right? It's it's the it's the structure, the infrastructure that's built around the the model itself. So I mentioned... You've mentioned to me before that there are, like, less capable models with a better harness can actually outperform a frontier model. So how do you think that this is affecting, you know, overall performance?
00:11:04 · Rebecca Balan, Irene Solaiman
[00:11:04] S1: Yeah. And, know, it's not just about, like, you know, there's a good enough model, but so much of it increasingly is the harness. Right? It's it's the it's the structure, the infrastructure that's built around the the model itself. So I mentioned... You've mentioned to me before that there are, like, less capable models with a better harness can actually outperform a frontier model. So how do you think that this is affecting, you know, overall performance? [00:11:27] S2: Spot on. On these more specialized startups or tasks, one thing that we had discussed before, thinking about cybersecurity a lot, there are harnesses that can zoom in to specific cyber defense tasks and leverage models, and it will still use a generally powerful model that can address vulnerability identities, for example. These kinds of harnesses, where just so many, are are popping up more and more, are custom made. This can be a revenue stream in and of itself of folks who are able to design harnesses that our economy needs right now.