00:05:26 · Thariq Shihipar, Aditya Agarwal
[00:05:26] S0: Yeah. I mean, this is a good question. I I think that... Yeah. For me, the moment was Opus four. Not that Opus 4.5 was not better at doing long running tasks, but I just think you can sort of see the progression of it. And I think I was actually surprised by how popular Opus 4.5 was kind of. Like, I I thought it was a jump, but I I don't think I realized how much like, I think there are subtle things where like, if you're good enough at prompting, maybe the previous generation of models can do it. But like, then the next generation just does it automatically, you know? And so I think it it feels a lot like... Doesn't feel to me like a single eureka moment. I think it's like you spend time with the models all the time and you sort of develop these habits. And sometimes you find things that, like, you know, we've sort of, like, been exposed to for a while, and then we release to the world and and peep... Forget how big the jump can be sometimes. Yeah. [00:06:23] S1: So one last, like, quick hit intro. What was the first thought that came to mind when you heard that Karpathy was joining? [00:06:30] S0: And when was that? I mean, I'm excited. I don't do pre training, you know. So I I I think it's obvious... Obviously, like a good signal in the sense that, like, Karpathy, you know, cares about education a lot, but also is a great researcher. And, like, where do you spend your time, especially, like, automating pre training. Yeah. I mean, like, if he thinks that's the best place to spend his time, I think that's a strong signal.
00:06:30 · Thariq Shihipar, Aditya Agarwal
[00:06:30] S0: And when was that? I mean, I'm excited. I don't do pre training, you know. So I I I think it's obvious... Obviously, like a good signal in the sense that, like, Karpathy, you know, cares about education a lot, but also is a great researcher. And, like, where do you spend your time, especially, like, automating pre training. Yeah. I mean, like, if he thinks that's the best place to spend his time, I think that's a strong signal. [00:06:53] S1: Yeah. I mean, I think a lot of you have heard, like, a phrase we use here, but, you know, best founders being super talent greedy. And when I saw it, I was like, they're talent greedy. That's amazing. And magnetic to be able to land Andre there. That's awesome. Well, let's get kind of into the sort of little more of the the technical discussion. And something you've tweeted about, but also I know Claude has talked about is sort of, like, avoiding the capability overhang and, like, making sure that you're building sort of on the exponential. Let's spend a couple minutes. Like, why don't you expand on that? But I think what's most interesting maybe to this audience, especially given where you sit, is to the extent that you can sort of bring forward some live examples of of you guys building. It's very meta you guys are building Claude to then allow us to use Claude. But, like, in the ways that you build, how you actively think about those two concepts.
00:07:43 · Thariq Shihipar
[00:07:43] S0: Yeah. So capability overhang is like when we think the models are smarter and more intelligent than maybe the harness or the interaction with the user like allows them to express basically. And I think that this is probably like a forever problem in the models. I I think like if you were to freeze model development right now and like we'd probably still be like figuring out new capability overhang for like another like six or twelve months, you know. The key thing to note, I think, is that the models do get smarter. They are getting smarter with every generation, but they get smarter in these, like, spiky non intuitive ways. I think it's like a really exciting problem to try and figure out how to do it or how to how to, like, utilize that. So for example, like in 2024, the common like thesis was like, oh, the way that models will get really good at coding is that our context windows will get really huge. Right? There'll be like a 100,000,000 token context windows and like you just fit the entire code base in it and then it just solves it. Right? But then, you know, with Claude Code, you're like, oh no, actually, what if the models can build their own context? Right? And if they can build their own context, then they'll be able to solve this problem in a different way. But if you didn't have that realization, like, would have just been waiting for the 100,000,000 context window. Right? The models get better in these, like, hard to explain ways that are like, you just have to grok internally. So one one thing I've been thinking about a lot recently is Plan Mode. Right? And so Plan Mode, when it first came out, was like a kind of an Opus four feature, I'd say, where you're like, hey, I need the model to think very clearly ahead of time about the plan and then execute it. Because like, it's not smart enough if I just asked a question to do it. And now a lot of us on the team have stopped using Plan Mode because the model just like thinks correctly. Right? And it just like knows how to execute something if you give enough detail. But then the problem is still like, okay, well, if the model's that smart, like, is it not perfect? You know, like, what are the reasons people are not able to, like, fully utilize it? And some examples are... Okay. There's a lot of ambiguity.
00:09:56 · Thariq Shihipar
So I I wrote a lot about HTML recently in a way of like trying to understand how how can you understand what they're doing more so that you can be sure that this is what you want. Right? I think another like feature that we released just today is called the workflows. And workflows are an interesting way of addressing the capability overhang because it essentially creates a custom harness on the fly. An example of a custom harness is like deep research for example. Deep research is a task where you need to maybe do a 100 different search results. Right? And before, what we do is we'd manually code that up and like have a have a harness that does it. Now with workflows, Claude makes that harness on the fly and then executes it. And so it's a way of like that... Us utilizing that capability overhang where if we had just prompted Claude Code like, oh, hey, like, spend a lot of time thinking about this research, or like, will not be able to do it within its context window or even with sub agents. But if you create like this custom harness on the fly, it can. Right? And so Does somebody plan for that? Because I... Opus 4.8
00:11:01 · Aditya Agarwal, Thariq Shihipar
[00:11:01] S1: came out today, and I'm a little bit behind. So I actually don't even know what was released there. But if we use that as an example of workflows, how would somebody plan ahead of time in terms of making sure that they're... Whatever they're building, whether or not it's the prompts or the skills or the agents, are gonna be well positioned to take advantage of like the next leap versus then having to go back and sort of like rebuilding it or or overfitting for like what you think the capabilities are today. [00:11:30] S0: Yeah. I mean, I I think this is, just a really hard problem. I think that, like, practically, we rewrite a lot of the Claude Code harness all the time. And I think that, like, in order to stay on the frontier, you probably have to. You know? I think that I... One way of thinking about it is, like, you're trying to consume end, like, outcomes. Right? And so, like, you can sort of plan for the outcomes to get better even though you don't know exactly how Yeah. Like that process will work.
00:11:30 · Thariq Shihipar, Aditya Agarwal
[00:11:30] S0: Yeah. I mean, I I think this is, just a really hard problem. I think that, like, practically, we rewrite a lot of the Claude Code harness all the time. And I think that, like, in order to stay on the frontier, you probably have to. You know? I think that I... One way of thinking about it is, like, you're trying to consume end, like, outcomes. Right? And so, like, you can sort of plan for the outcomes to get better even though you don't know exactly how Yeah. Like that process will work. [00:12:04] S1: Maybe we're jumping ahead because I I did wanna touch on... So early May, you guys had a developer conference. Show of hands, who was who was there? Couple of people. And there are a lot of good videos. You guys actually do a pretty good job with content. But one of the key principles for at least agent design is sort of this like evals for hill climbing. And I'd be curious how recursive Anthropics development is where there's a new model, you're sort of discovering what are the new capabilities. Do you apply that principle to, I built this thing, now let's switch out the model, now let's run a bunch of evals very quickly and then have it recursively figure out how to change itself so it takes advantage of the new capabilities? Or let's maybe touch a little bit on like that concept, but also this broader concept of like how do you just allow the system to continue to improve? Yeah. And specifically how's Anthropic to it?