00:23:06 · Thariq Shihipar
[00:23:06] S0: Yeah. Memory is a great one. Like, I I think that memory is extremely extremely high dimensional problem, you know, and, like, how do you do it? Right? Like, some people, like like, do do do in a rag, like, do in file search, like, do do it in training, like, what are your RL environments? Like, all of these things. I I think that, like, it is, I will say, very, very fun to build, for example, like a custom memory system. I think the thing that I see sometimes with, like, startups is, like, the founders build a v one or something for a certain set of capabilities, and they're never able to hire someone to build the v two, you know? And so your memory system is just like whatever you did initially for like Opus four or something. And then like, because harness engineering is so rare, it's like like they can't... And it might even outperform, like, our memory stuff. Right? Because it's your own specific problem, or you might have the specific insight. But then, like, how do you stay on it is really tough. And I I think that, like, generally, I try and, like I think it's good to build scaffolding, but I think, like, deleting stuff is really important and just being like, okay. Like, hey, the model is good at this now. I think OPUS four point seven and four point eight are really good at writing and reading memory via files, and, you know, that's how we sort of expect our memory to to be done. And so I I think like once there's sort of a consensus, I think it's like unusual to, like, keep your, like, bespoke scaffolding in the same way that, like, I think, like, rag, especially for context search, is now, like, maybe an anti pattern, you know, and that you should use, like, grep instead. But if... I know plenty of startups that built a rag indexing thing in, like, 2024. You know? And so, yeah, I think that's one of the difficulties of, like, scaling your startup.
00:24:49 · S4, Thariq Shihipar, Aditya Agarwal
[00:24:49] S4: I have a question around running a long running task versus smaller scope task, running them in parallel. Because if you are running a long running task, then you are dependent on a lot of auto compaction. And then how do you have the right validations for that? So what's your opinion on that? [00:25:04] S0: Yeah. I mean, I... Again, across the Claude Code team, there are different people with different opinions. Some people are very, very long running task build. I am like... I I think that for me, I prefer somewhat shorter task unless there's something clearly ambitious that I need to do that I can like spec out all upfront. I think that it's sort of a preference thing. I I do think that like a lot of times when I see people who are, for example, running into limits on the max 20 x plan, it's sort of like turning things into long running tasks when maybe they don't need to be, you know, or like not like interrogating like, hey, do I know everything about this problem or something? And I I think you can definitely set up a problem where you're like, oh, like my task is I do the prompt and there are five verification agents and then the... Then we fix the problems that the verification agents running. And then it's running for twelve hours, but maybe you didn't need the twelve hours. It was a little bit of [00:25:59] S1: a corollary related question. To what extent and I think this was hinted at maybe in the developer day that you guys did. But to what extent is their vision that Claude Code should become more
00:25:59 · Aditya Agarwal, Thariq Shihipar
[00:25:59] S1: a corollary related question. To what extent and I think this was hinted at maybe in the developer day that you guys did. But to what extent is their vision that Claude Code should become more [00:26:11] S0: of an actor as opposed to a session machine? So essentially one single session continuously compounding or or compacting, like having memory essentially feeling more like tactilely like a like an open claw. Yeah. Like proactiveness. I I think this definitely is something we're working towards. Right? I I think proactiveness is another example of like how how does it unlock in the capability overhang is actually like hard to say. I think that like, yeah, is it one Claude that just like you never worry about comp... Compact or context and it just spins off sub agents as needed? Is it... Is there like an orchestration workflow more... Maybe more like workflows? Or is it like a UX thing or something? So I think there's a lot to figure out there and... But, yeah, we're we're thinking a lot about it.
00:27:00 · S5
[00:27:00] S5: Two questions for you. You mentioned that evals are more of like an art than a science, and it's kind of... Or maybe it's a little bit of both objectively. It can be a science, but over time, it's still an art. Right? Because it's so nondeterministic and and flexible. So I'm just curious how you guys know when something is ready, like what type of evals you guys are running to know something is truly ready. And then the second question I have is that, Anthropic, the pace that you guys are moving is so fast. Right? And the quality is so high. So I'm curious how you would describe the way you guys work, whether it's in product development or even across the organization that's very different than likely how other organizations are working and what your advice might be?
00:27:38 · Thariq Shihipar
[00:27:38] S0: Yeah. I mean, so on the first part, I think harness engineering overall is an art. You know I mean? Like, I think evals are a little bit more of a science because, like, you can, you know, hill climb against them. I I do think, like, designing evals is very, very hard. Like, I don't know if you guys saw the, like, Japanese guy who's, like, annotated, like, millions of images with bounding boxes or something, you know. It's gone kind of viral on Twitter, but it's like this guy who spent three years just like going through millions of images and drawing very specific bounding boxes around like the eyes of people and mouths and like the skeletons and stuff. And his like setup now like outperforms every other like, you know, skeleton recognition and face recognition thing. But he's just like, it's mind numbing work. You know what I mean? And it's even hard to like outsource because, you know, like it's very high skill too. You know? And so I I think evals are one of these things where, yeah, it's actually almost hard to buy them even. Right? Because like you need like very high skill people to do very honestly boring work, know? And so, yeah, you can't have an eval for everything. And and for example, like let's say that someone... People are always like, okay, Claude Code feels dumber. You know what I mean? Like, in order for us to like... If we run Claude Code against SWE-bench or SWE-bench Pro or whatever, that will measure like a software engineering performance. But like maybe what people are saying is that like sometimes Claude stops early. You know what I mean? When I think it should continue. Then we need a stop early eval. Right? But how would we know we need that eval until we see the like the the problem, you know? And so I I think there's like this... Like you can't have an eval for any possible problem that Claude could run into. You know, it's it's hard, I I think. And evals are very, very valuable. I think I think, like, this is one of those problems where I sort of point people at sometimes when they're like, hey, like, I want to get into AI. I want to do something. And I'm like, okay, great. Choose something low status and not like sexy or interesting. And like, evals are like easily it, you know? And then on your second point about stuff that Anthropic does, obviously, like Evan said, like, the talent is very high.
00:29:48 · Thariq Shihipar
I think a particular thing I've appreciated about like, Thariq's approach is that he... I think sometimes you can say reasonable things about trade offs. You know, you can be like, okay, like, here, pick two, like good, fast, cheap, you know, or something. Right? And it's like, that makes sense. Right? Like, it's very reasonable that you would have to pick two out of these three things. But then it's like, well, no, what if we did all of them actually? You know what I mean? And like, I think sometimes you can sort of have this like reasonable product brain where you're like, hey, look, let's just prioritize these things and let's just like do the highly prioritized things. But like, maybe you're wrong. Like, you know, maybe you can do all these things. Right? And I I think that like forcing reality to show you, like, oh, no. These are the real trade offs and how things trade off against each other is sometimes unintuitive. And it's better to be like, maybe like somewhat unreasonable and just be like, hey. Like, yeah, we can do... Like, we can build a safe AI that also makes a lot of revenue and, you know, like brings, you know, a lot of, like, value to the world, and we can race to the top. So, yeah, I think that's something I've really appreciated. Let's spend a little more