00:43:52 · Anika, Maya, Priya
[00:43:52] S2: Yeah. Like one of the reasons they value that is like a lot of really big enterprises don't really wanna be locked in with one provider. Yeah. And [00:44:00] S0: we can go into like what a coding harness is. So basically like you have the model, you can give us an input, which is like a set of tokens or a set of words, and then you get an output, which is also another set of words. And these set of words could be like Python executable, like language and stuff. And so there's so much that you need to build around that to make it work. And a harness is everything surrounding the model. And so it's things like giving the model tools or telling the model, or like basically dynamically putting prompts together. So you can ask the... You can, you know, add the prompt like you're a crazy software engineer. Now your task is to do this, or now you have this tool available to you. And so essentially what I think about a harness is just like ways to... Like if you're an input, like ways to optimize that input to get a better output. [00:44:47] S1: And so... And know your environment really well too. Because like every single enterprise is super different in how they use things and what softwares they use, like in general. And so it's basically like all the scaffolding around like the model itself.
00:44:47 · Priya, Maya, Anika
[00:44:47] S1: And so... And know your environment really well too. Because like every single enterprise is super different in how they use things and what softwares they use, like in general. And so it's basically like all the scaffolding around like the model itself. [00:45:01] S0: Yeah. And so the Claude Code is example of a harness, Cognition has a harness, Cursor is an example of a harness. Even things like open evidences like ask bar, like all these like chat applications, like ChatGPT is also an example of harness. Like, Harvey is an example of a harness. There's a lot of application layer companies that wrap a model in general. [00:45:25] S2: And I feel like one of the reasons people have been talking about harnesses a lot also is as people talk about not just having like a bunch of slop that can, like, affect your security or affect your entire, like, code base. Because one of the examples that gets given is, like, LLMs are not deterministic. And you can actually, like, see this. Like, sometimes if you, like, tell an LOM, like, hey, don't do this. It's not like an actual rule and it'll, like, ignore you, which I [00:45:53] S1: was gonna say. Honestly, he gets really upset about this. [00:45:57] S2: Well, it's just, like, insane. Like, I literally, like, written in my clod in, like, 20 different places, like, just stop using Updashers and, like, it'll keep doing it.
00:46:04 · Priya
[00:46:04] S1: Definitely, as we see more AI code that's generated... Because, like, cogeneration is so cheap now. Like, we all know that, and you can literally have anything do it. Things move to trusting the code that's written, like, verification. Like, it's the entire reason that you're, like, I'm still spending so much time before something can be shipped. Yeah. Because there is so much that goes into it on, like, the trust and verification layer. And I feel like there's been, like, Horizon incidents, like Horizon, like, so many things across, like, every company. Andrej Karpathy also said that vibe coding is passe. I don't know if you guys saw that quote. He was like, vibe coding is passe, and the real value now is in systems that can plan, execute, test, and iterate. And the really interesting thing... I don't know if you guys read it. I sent it in our chat. But MongoDB came out with this paper, and it was basically about, like, agent harnesses and the importance of them. And one of the examples was of Vercel. And so Vercel deleted 80% of their agent tools, and success went from 80 to a 100% in, like, outputting code. And the latency fell from seven hundred seconds to one hundred seconds, and they were way more efficient with the exact same model. And so that shows that, like, if you keep the model the same, like, the infrastructure that you build around it has, like, a significant difference in, like, what actually gets shipped to, like, production code bases, how fast it can be done, and also the cost that you're spending on it. Well, I
00:47:20 · Maya, Priya
[00:47:20] S0: think it all comes down to like literally prompting and context management, which is like prompting is like basically the question that you ask the model. And what the harness is doing is like attaching all of these prompts together. And then you attach too many prompts together or you don't like write the prompt in the most effective way, you get a worse output. Yeah. And so like, that's why there you see actually all these like harness benchmarks and Claude Code is not at the top of the harness benchmarks. Because if you actually spent time really thinking about like, like the minimal set of tools that you should give like a harness, you'll realize that it's not that many. And if you give more, it's like you get actually negative returns rather than like flat amount of returns. And often like in the harness, it's like the simpler that you can be, the better that you can be. But there's like this kind of like middle thing where it's like, okay, obviously having nothing is not great, but then having too much is too much. And so [00:48:09] S1: Figuring you could it out. [00:48:10] S0: Well, it's also like an experiment that you can run like... So there's this thing called like Gemma, I think is what it is. I don't know. Someone correct me probably. But you, can optimize the prompt you're giving. Like, you can actually like run an experiment on a ton of prompts and, like, optimize your prompt, like, deterministically to get the best output. So you can, like, like... Wait. What... [00:48:34] S1: What's the output that drama gives you?
00:48:34 · Priya, Maya, Anika
[00:48:34] S1: What's the output that drama gives you? [00:48:36] S0: Like... Oh, sorry. Gemma is, like, the technique. You can, like, give it, like, essentially, like, thousands of prompts and you can figure out, like, what is, like, the most optimal prompt. And so you could actually apply this to harnesses where you can give like a thousand different descriptions for tools and figure out which is like the best version of the description for the tool so that the tool is used most effectively by the model. [00:48:55] S1: Do you think that Anthropic will like optimize Claude Code to the point where, like, most people will just end up using that? [00:49:00] S0: I mean, I think that they are trying to. Yeah. [00:49:02] S1: Well, yeah. They're definitely trying to. [00:49:03] S2: It seems like a lot of Cognition's competitive advantage right now is how compliant it is with industry certifications Yeah. And like finance and health And I just don't know if that's a long term strategy because it seems like these health insurance companies are partnering... Like, all these really regulated industries are very pro AI and have pretty deep partnerships that they're developing with AI companies. I assume Anthropic included. Yeah. OpenAI included. [00:49:32] S1: I feel like... I actually... Well, I
00:49:32 · Priya, Maya, Anika
[00:49:32] S1: I feel like... I actually... Well, I [00:49:33] S0: was gonna I was gonna draw to some historical context. This one is not from the nineteen hundreds. It's just Google. But Google has always been like the number three player in cloud computing, like, or in in the cloud. Like g g... What is it? GCP. GCP. Has always been like number three. Yeah. But they've brought in their position to 14% of the market and they're they were profitable. And once they became profitable, they quickly shut off in terms of like their margins and stuff. And it took them a long time. Like they launched in 2008 and they only turned profitable in 2023. And now they're like, actually making a ton of money from it. And and the so, way they competed was just by like undercutting AWS and Azure as much as possible. [00:50:12] S1: Mhmm. [00:50:12] S0: And so it's like, yeah, you're gonna probably like, spend time doing things that are not sustainable. Like, maybe like this compliance thing is like the way that Devin is trying to be like, is trying to win the contracts. I'm sure they're also undercutting by some extent, and they're not profitable by some extent. But once you're in, you're in. And once you have like an enterprise... [00:50:31] S1: Once you're in in something like this, it's like so sticky. It's like the most sticky. [00:50:35] S2: Yeah. [00:50:36] S0: So I do think it's like... [00:50:38] S1: I I think that's what [00:50:39] S0: the VCs are betting on honestly. Like, this is the same thing with AMD. AMD has always been number three. But...