00:06:30 · Thariq Shihipar, Aditya Agarwal
[00:06:30] S0: And when was that? I mean, I'm excited. I don't do pre training, you know. So I I I think it's obvious... Obviously, like a good signal in the sense that, like, Karpathy, you know, cares about education a lot, but also is a great researcher. And, like, where do you spend your time, especially, like, automating pre training. Yeah. I mean, like, if he thinks that's the best place to spend his time, I think that's a strong signal. [00:06:53] S1: Yeah. I mean, I think a lot of you have heard, like, a phrase we use here, but, you know, best founders being super talent greedy. And when I saw it, I was like, they're talent greedy. That's amazing. And magnetic to be able to land Andre there. That's awesome. Well, let's get kind of into the sort of little more of the the technical discussion. And something you've tweeted about, but also I know Claude has talked about is sort of, like, avoiding the capability overhang and, like, making sure that you're building sort of on the exponential. Let's spend a couple minutes. Like, why don't you expand on that? But I think what's most interesting maybe to this audience, especially given where you sit, is to the extent that you can sort of bring forward some live examples of of you guys building. It's very meta you guys are building Claude to then allow us to use Claude. But, like, in the ways that you build, how you actively think about those two concepts.
00:07:43 · Thariq Shihipar
[00:07:43] S0: Yeah. So capability overhang is like when we think the models are smarter and more intelligent than maybe the harness or the interaction with the user like allows them to express basically. And I think that this is probably like a forever problem in the models. I I think like if you were to freeze model development right now and like we'd probably still be like figuring out new capability overhang for like another like six or twelve months, you know. The key thing to note, I think, is that the models do get smarter. They are getting smarter with every generation, but they get smarter in these, like, spiky non intuitive ways. I think it's like a really exciting problem to try and figure out how to do it or how to how to, like, utilize that. So for example, like in 2024, the common like thesis was like, oh, the way that models will get really good at coding is that our context windows will get really huge. Right? There'll be like a 100,000,000 token context windows and like you just fit the entire code base in it and then it just solves it. Right? But then, you know, with Claude Code, you're like, oh no, actually, what if the models can build their own context? Right? And if they can build their own context, then they'll be able to solve this problem in a different way. But if you didn't have that realization, like, would have just been waiting for the 100,000,000 context window. Right? The models get better in these, like, hard to explain ways that are like, you just have to grok internally. So one one thing I've been thinking about a lot recently is Plan Mode. Right? And so Plan Mode, when it first came out, was like a kind of an Opus four feature, I'd say, where you're like, hey, I need the model to think very clearly ahead of time about the plan and then execute it. Because like, it's not smart enough if I just asked a question to do it. And now a lot of us on the team have stopped using Plan Mode because the model just like thinks correctly. Right? And it just like knows how to execute something if you give enough detail. But then the problem is still like, okay, well, if the model's that smart, like, is it not perfect? You know, like, what are the reasons people are not able to, like, fully utilize it? And some examples are... Okay. There's a lot of ambiguity.
00:09:56 · Thariq Shihipar
So I I wrote a lot about HTML recently in a way of like trying to understand how how can you understand what they're doing more so that you can be sure that this is what you want. Right? I think another like feature that we released just today is called the workflows. And workflows are an interesting way of addressing the capability overhang because it essentially creates a custom harness on the fly. An example of a custom harness is like deep research for example. Deep research is a task where you need to maybe do a 100 different search results. Right? And before, what we do is we'd manually code that up and like have a have a harness that does it. Now with workflows, Claude makes that harness on the fly and then executes it. And so it's a way of like that... Us utilizing that capability overhang where if we had just prompted Claude Code like, oh, hey, like, spend a lot of time thinking about this research, or like, will not be able to do it within its context window or even with sub agents. But if you create like this custom harness on the fly, it can. Right? And so Does somebody plan for that? Because I... Opus 4.8
00:11:01 · Aditya Agarwal, Thariq Shihipar
[00:11:01] S1: came out today, and I'm a little bit behind. So I actually don't even know what was released there. But if we use that as an example of workflows, how would somebody plan ahead of time in terms of making sure that they're... Whatever they're building, whether or not it's the prompts or the skills or the agents, are gonna be well positioned to take advantage of like the next leap versus then having to go back and sort of like rebuilding it or or overfitting for like what you think the capabilities are today. [00:11:30] S0: Yeah. I mean, I I think this is, just a really hard problem. I think that, like, practically, we rewrite a lot of the Claude Code harness all the time. And I think that, like, in order to stay on the frontier, you probably have to. You know? I think that I... One way of thinking about it is, like, you're trying to consume end, like, outcomes. Right? And so, like, you can sort of plan for the outcomes to get better even though you don't know exactly how Yeah. Like that process will work.
00:11:30 · Thariq Shihipar, Aditya Agarwal
[00:11:30] S0: Yeah. I mean, I I think this is, just a really hard problem. I think that, like, practically, we rewrite a lot of the Claude Code harness all the time. And I think that, like, in order to stay on the frontier, you probably have to. You know? I think that I... One way of thinking about it is, like, you're trying to consume end, like, outcomes. Right? And so, like, you can sort of plan for the outcomes to get better even though you don't know exactly how Yeah. Like that process will work. [00:12:04] S1: Maybe we're jumping ahead because I I did wanna touch on... So early May, you guys had a developer conference. Show of hands, who was who was there? Couple of people. And there are a lot of good videos. You guys actually do a pretty good job with content. But one of the key principles for at least agent design is sort of this like evals for hill climbing. And I'd be curious how recursive Anthropics development is where there's a new model, you're sort of discovering what are the new capabilities. Do you apply that principle to, I built this thing, now let's switch out the model, now let's run a bunch of evals very quickly and then have it recursively figure out how to change itself so it takes advantage of the new capabilities? Or let's maybe touch a little bit on like that concept, but also this broader concept of like how do you just allow the system to continue to improve? Yeah. And specifically how's Anthropic to it?
00:12:55 · Thariq Shihipar
[00:12:55] S0: Yeah. All serious. I think so. I think apart you put out auto research a couple months ago, right, which is like this example of a harness that given a goal, like, reduce the loss function is just like iterates until it does it. And it does like some weird stuff sometimes, but it, like, usually, like, it works. Right? So I I think that's the, like, concept of, like, okay, I think the models have gotten better at this, verifiable result thing. I I think recently, Jarred from Bun rewrote Bun into Rust, and this is another example of, something where if you just give the model a a really intense test suite, for example, it can just like do the task. And so, yeah, I I think that like when you have... Like, I think this is a new model capability. I I think evals, we don't often use them for new feature development. You know what I mean? So, for example, for workflows, we won't we wouldn't like be like, okay, we have this like long horizon task eval and, like, let's, like, build the workflows feature and just, like, figure out if it works. I I think evals are not a zero to one thing. And so, like, for me, a lot of startups maybe should not be making evals and and just sort of, like, be iterating fast and building your intuition for if they're like... If it's working.