00:11:01 · Aditya Agarwal, Thariq Shihipar
[00:11:01] S1: came out today, and I'm a little bit behind. So I actually don't even know what was released there. But if we use that as an example of workflows, how would somebody plan ahead of time in terms of making sure that they're... Whatever they're building, whether or not it's the prompts or the skills or the agents, are gonna be well positioned to take advantage of like the next leap versus then having to go back and sort of like rebuilding it or or overfitting for like what you think the capabilities are today. [00:11:30] S0: Yeah. I mean, I I think this is, just a really hard problem. I think that, like, practically, we rewrite a lot of the Claude Code harness all the time. And I think that, like, in order to stay on the frontier, you probably have to. You know? I think that I... One way of thinking about it is, like, you're trying to consume end, like, outcomes. Right? And so, like, you can sort of plan for the outcomes to get better even though you don't know exactly how Yeah. Like that process will work.
00:11:30 · Thariq Shihipar, Aditya Agarwal
[00:11:30] S0: Yeah. I mean, I I think this is, just a really hard problem. I think that, like, practically, we rewrite a lot of the Claude Code harness all the time. And I think that, like, in order to stay on the frontier, you probably have to. You know? I think that I... One way of thinking about it is, like, you're trying to consume end, like, outcomes. Right? And so, like, you can sort of plan for the outcomes to get better even though you don't know exactly how Yeah. Like that process will work. [00:12:04] S1: Maybe we're jumping ahead because I I did wanna touch on... So early May, you guys had a developer conference. Show of hands, who was who was there? Couple of people. And there are a lot of good videos. You guys actually do a pretty good job with content. But one of the key principles for at least agent design is sort of this like evals for hill climbing. And I'd be curious how recursive Anthropics development is where there's a new model, you're sort of discovering what are the new capabilities. Do you apply that principle to, I built this thing, now let's switch out the model, now let's run a bunch of evals very quickly and then have it recursively figure out how to change itself so it takes advantage of the new capabilities? Or let's maybe touch a little bit on like that concept, but also this broader concept of like how do you just allow the system to continue to improve? Yeah. And specifically how's Anthropic to it?
00:12:55 · Thariq Shihipar
[00:12:55] S0: Yeah. All serious. I think so. I think apart you put out auto research a couple months ago, right, which is like this example of a harness that given a goal, like, reduce the loss function is just like iterates until it does it. And it does like some weird stuff sometimes, but it, like, usually, like, it works. Right? So I I think that's the, like, concept of, like, okay, I think the models have gotten better at this, verifiable result thing. I I think recently, Jarred from Bun rewrote Bun into Rust, and this is another example of, something where if you just give the model a a really intense test suite, for example, it can just like do the task. And so, yeah, I I think that like when you have... Like, I think this is a new model capability. I I think evals, we don't often use them for new feature development. You know what I mean? So, for example, for workflows, we won't we wouldn't like be like, okay, we have this like long horizon task eval and, like, let's, like, build the workflows feature and just, like, figure out if it works. I I think evals are not a zero to one thing. And so, like, for me, a lot of startups maybe should not be making evals and and just sort of, like, be iterating fast and building your intuition for if they're like... If it's working.
00:14:14 · Aditya Agarwal, Thariq Shihipar
[00:14:14] S1: Is it at scale or at at the point where it's like, there's a bunch of stuff that you can now ship that is like largely alpha and you're just experimenting, which you wouldn't take the time to build a bunch of ETHLs for that. But like once it's baked in and you need to maintain, improve, that's probably where you invest in kind of a good [00:14:31] S0: Maybe. I I think that there is a challenge that a lot of founders run into and that like they build the first system and then they need to manage people to build the next system or to continue it. And then evals become a way of legibility into that. I don't think that's necessarily the same as driving a result. Do you know what I mean? Like I I think like, yes. Like if you have an eval, you can manage against that eval. But I I think honestly it's kind of an art quite even quite late into the process, I think. Yeah. Even at Anthropic? For some parts of it.
00:15:03 · Aditya Agarwal, Thariq Shihipar
[00:15:03] S1: Yeah. Yeah. One other concept that I think people are still coming around you and it's like, obviously, this is... We're literally talking about four points that coming out in February. So not a lot of time has elapsed. But people are starting to think about how do you decompose, especially with agents themselves, whether or not you're building them internally or as part of your product for customers. How do you get really smart about structuring agents and decomposing them? And there's been a lot of chatter around at what point do you start to break these up as sub agents, and there are a lot of reasons to do that. Love for you to expand a little bit on like the current thinking Anthropic, but also maybe a couple examples of where you guys are building this stuff internally and you've sort of hit these moments where you either... That was sort of like the foundation for that principle and then obviously sharing with others or where it was really important to do. Because I think a lot of us... And I think we've shared... We're building a lot of this internally at SPC. So we're, like, live having these debates around when is this too bloated? When do I sort of, like, refactor it so it's multiple skills, multiple sub agents? [00:16:11] S0: Yeah. This is really, really hard. I I think I think that the art of like this, like, harness engineering, which is... Can be the harness, it can be skills, it can be how you prompt, is like a really rare skill right now. And that, like, it's hard to know if you're even good at it or not sometimes, you know, because...
00:16:11 · Thariq Shihipar, Aditya Agarwal
[00:16:11] S0: Yeah. This is really, really hard. I I think I think that the art of like this, like, harness engineering, which is... Can be the harness, it can be skills, it can be how you prompt, is like a really rare skill right now. And that, like, it's hard to know if you're even good at it or not sometimes, you know, because... [00:16:29] S1: Is it still like... Is there still... Is this still mostly human taste even with Anthropic? Or are you starting to do a lot of this recursively where you would try to imbue meta agents to help kinda get everybody to do this properly? [00:16:44] S0: Yeah. Well, so I mean, Anthropic is not like a single, like, you know, monolithic entity, and so everyone has different opinions. Right? And I think that this, like, self recursive, like, harness thing is... Some people are into it. I think other people are not. You know what I mean? And I think that, like, it's Of the Claude Code. It's a mix. Okay. Yeah. Yeah. I I think we try and see what works. But I I think like something like workflows, for example, which I think is huge and I think I'm excited for everyone to try it out, is definitely something where we saw it internally just do really well and we're like, oh, shit, this is just sick. And it did help drive some results like the Rusty Wright. But yeah, also we like, you know, we're... We use the models every day and and it was better.