00:18:25 · Thariq Shihipar
[00:18:25] S0: Yeah. I I think that like the way I think about it is, okay, when you have a task, you have like no knowns. Like like you know you want this and... But like maybe is it in the prompt or not? Like, did you write it down? You know? Then you have unknown knowns. You know? So you're like, you you think you want it, but like you don't know it until you see it. You know what I mean? And then you have like unknown unknowns, where you just like don't know that this is possible, you know? And you like don't understand the system well enough or something. Right? And then you have like known unknowns where you're like, oh, I I don't know how this works, you know, but I would like to figure it out. Right? And and so I I think for every problem, analyzing it along those dimensions helps me pull out like, okay, this is how I should prompt the model. Right? And so for example, like if I'm just being lazy about prompting, sometimes what I like to ask the model is like, hey, like interview me about this problem. You know? So like here is my idea. Let me like interview me using the ask user question tool, and then I'll answer. So like that's one example. Sometimes if it's like, okay, I'm working on a design and I know what I... I'll I'll know it when I see it, you know? I'll ask it to like do a bunch of explorations. And so I'll be like, explore this design and make a bunch of mock ups in HTML and show it to me. And then of those like whatever four, then I might ask it to do more like exploration there. But like, again, it's like in my head, I like I have some preference, but I I don't know yet. Right? And I think a meta prompting technique there is like, if you're still figuring something out, do you want to do it in a cheap way? So like, you know, if you're doing UI, it's cheaper to do it in HTML because it's fake. You know, like you're not rewriting the state and the like, you know, all this other wiring up stuff. Right? Like if you ask a model to add a button, in HTML it's just a button. But inside of your app, maybe there's a new route. There's a new like reducer, all of this stuff. Right? And so in the same way that like, you can do this for like back end endpoints and testing and things like that too, just like, what's the cheapest way to figure out like, is this the thing that you want? Right? So... Yeah. Jason?
00:20:52 · Jason, Thariq Shihipar, Aditya Agarwal
[00:20:52] S2: Do you have any advice around whether we should say thank you? Because it it does... I'm imagining it probably is a good signal of you did good, you know. It's like, yay. Or is it like just a complete absolute waste of tokens? [00:21:06] S0: This is actually like an interpretability question, which I like, you know, I I learned a lot about a Goodfire. I think that like probably... It it's hard to say. I don't think we run like a thank you eval, you know. But we have done some research recently about like emotions in Claude. And like, you know, if you are mean to it, it does activate this like, person is being mean to it feature, you know. And it does guide results in some way. I I think that like... I don't think we've done like swee bench against like meanness or something, you know? But yeah, I am generally like positive for the models. Think in the same way that you should be like, you know, nice to most things, you know. And and the very least, so that you're not embarrassed to share your transcript with me. Because I definitely get run into bugs sometimes. I'm like, oh, great. Like, I can help you just like send me the transcript. And they're like... That reveals you're a terrible person. Yeah. [00:22:06] S1: But I've literally had people be like, please don't judge [00:22:08] S0: me for this. It was like a really long and tough night. I'm like, okay. I I got it. I got it.
00:22:08 · Thariq Shihipar, S3
[00:22:08] S0: me for this. It was like a really long and tough night. I'm like, okay. I I got it. I got it. [00:22:13] S3: Yeah. I have a quick question about how you think about the design of Claude Code and the harnesses. One of the things that is kind of emerging across the landscape is that we're seeing convergence across things like different skills or approaches. And I'm wondering internally if there is any type of debate around that, whether they're gonna be different architectural approaches across the different platforms or whether you believe that there's gonna be one unified way to do maybe a couple things or even everything. [00:22:41] S0: Like, are you asking across models? Like, if the same skills across like Opus and GPT or like just Well, things like memory, learning, question planning, they they were a [00:22:55] S3: little staggered in terms of rollout in different different platforms and harnesses picked it up. But the thinking behind that is is probably quite similar in terms of what needs to be true to make something work.
00:23:06 · Thariq Shihipar
[00:23:06] S0: Yeah. Memory is a great one. Like, I I think that memory is extremely extremely high dimensional problem, you know, and, like, how do you do it? Right? Like, some people, like like, do do do in a rag, like, do in file search, like, do do it in training, like, what are your RL environments? Like, all of these things. I I think that, like, it is, I will say, very, very fun to build, for example, like a custom memory system. I think the thing that I see sometimes with, like, startups is, like, the founders build a v one or something for a certain set of capabilities, and they're never able to hire someone to build the v two, you know? And so your memory system is just like whatever you did initially for like Opus four or something. And then like, because harness engineering is so rare, it's like like they can't... And it might even outperform, like, our memory stuff. Right? Because it's your own specific problem, or you might have the specific insight. But then, like, how do you stay on it is really tough. And I I think that, like, generally, I try and, like I think it's good to build scaffolding, but I think, like, deleting stuff is really important and just being like, okay. Like, hey, the model is good at this now. I think OPUS four point seven and four point eight are really good at writing and reading memory via files, and, you know, that's how we sort of expect our memory to to be done. And so I I think like once there's sort of a consensus, I think it's like unusual to, like, keep your, like, bespoke scaffolding in the same way that, like, I think, like, rag, especially for context search, is now, like, maybe an anti pattern, you know, and that you should use, like, grep instead. But if... I know plenty of startups that built a rag indexing thing in, like, 2024. You know? And so, yeah, I think that's one of the difficulties of, like, scaling your startup.
00:24:49 · S4, Thariq Shihipar, Aditya Agarwal
[00:24:49] S4: I have a question around running a long running task versus smaller scope task, running them in parallel. Because if you are running a long running task, then you are dependent on a lot of auto compaction. And then how do you have the right validations for that? So what's your opinion on that? [00:25:04] S0: Yeah. I mean, I... Again, across the Claude Code team, there are different people with different opinions. Some people are very, very long running task build. I am like... I I think that for me, I prefer somewhat shorter task unless there's something clearly ambitious that I need to do that I can like spec out all upfront. I think that it's sort of a preference thing. I I do think that like a lot of times when I see people who are, for example, running into limits on the max 20 x plan, it's sort of like turning things into long running tasks when maybe they don't need to be, you know, or like not like interrogating like, hey, do I know everything about this problem or something? And I I think you can definitely set up a problem where you're like, oh, like my task is I do the prompt and there are five verification agents and then the... Then we fix the problems that the verification agents running. And then it's running for twelve hours, but maybe you didn't need the twelve hours. It was a little bit of [00:25:59] S1: a corollary related question. To what extent and I think this was hinted at maybe in the developer day that you guys did. But to what extent is their vision that Claude Code should become more
00:25:59 · Aditya Agarwal, Thariq Shihipar
[00:25:59] S1: a corollary related question. To what extent and I think this was hinted at maybe in the developer day that you guys did. But to what extent is their vision that Claude Code should become more [00:26:11] S0: of an actor as opposed to a session machine? So essentially one single session continuously compounding or or compacting, like having memory essentially feeling more like tactilely like a like an open claw. Yeah. Like proactiveness. I I think this definitely is something we're working towards. Right? I I think proactiveness is another example of like how how does it unlock in the capability overhang is actually like hard to say. I think that like, yeah, is it one Claude that just like you never worry about comp... Compact or context and it just spins off sub agents as needed? Is it... Is there like an orchestration workflow more... Maybe more like workflows? Or is it like a UX thing or something? So I think there's a lot to figure out there and... But, yeah, we're we're thinking a lot about it.