00:14:14 · Aditya Agarwal, Thariq Shihipar
[00:14:14] S1: Is it at scale or at at the point where it's like, there's a bunch of stuff that you can now ship that is like largely alpha and you're just experimenting, which you wouldn't take the time to build a bunch of ETHLs for that. But like once it's baked in and you need to maintain, improve, that's probably where you invest in kind of a good [00:14:31] S0: Maybe. I I think that there is a challenge that a lot of founders run into and that like they build the first system and then they need to manage people to build the next system or to continue it. And then evals become a way of legibility into that. I don't think that's necessarily the same as driving a result. Do you know what I mean? Like I I think like, yes. Like if you have an eval, you can manage against that eval. But I I think honestly it's kind of an art quite even quite late into the process, I think. Yeah. Even at Anthropic? For some parts of it.
00:15:03 · Aditya Agarwal, Thariq Shihipar
[00:15:03] S1: Yeah. Yeah. One other concept that I think people are still coming around you and it's like, obviously, this is... We're literally talking about four points that coming out in February. So not a lot of time has elapsed. But people are starting to think about how do you decompose, especially with agents themselves, whether or not you're building them internally or as part of your product for customers. How do you get really smart about structuring agents and decomposing them? And there's been a lot of chatter around at what point do you start to break these up as sub agents, and there are a lot of reasons to do that. Love for you to expand a little bit on like the current thinking Anthropic, but also maybe a couple examples of where you guys are building this stuff internally and you've sort of hit these moments where you either... That was sort of like the foundation for that principle and then obviously sharing with others or where it was really important to do. Because I think a lot of us... And I think we've shared... We're building a lot of this internally at SPC. So we're, like, live having these debates around when is this too bloated? When do I sort of, like, refactor it so it's multiple skills, multiple sub agents? [00:16:11] S0: Yeah. This is really, really hard. I I think I think that the art of like this, like, harness engineering, which is... Can be the harness, it can be skills, it can be how you prompt, is like a really rare skill right now. And that, like, it's hard to know if you're even good at it or not sometimes, you know, because...
00:16:11 · Thariq Shihipar, Aditya Agarwal
[00:16:11] S0: Yeah. This is really, really hard. I I think I think that the art of like this, like, harness engineering, which is... Can be the harness, it can be skills, it can be how you prompt, is like a really rare skill right now. And that, like, it's hard to know if you're even good at it or not sometimes, you know, because... [00:16:29] S1: Is it still like... Is there still... Is this still mostly human taste even with Anthropic? Or are you starting to do a lot of this recursively where you would try to imbue meta agents to help kinda get everybody to do this properly? [00:16:44] S0: Yeah. Well, so I mean, Anthropic is not like a single, like, you know, monolithic entity, and so everyone has different opinions. Right? And I think that this, like, self recursive, like, harness thing is... Some people are into it. I think other people are not. You know what I mean? And I think that, like, it's Of the Claude Code. It's a mix. Okay. Yeah. Yeah. I I think we try and see what works. But I I think like something like workflows, for example, which I think is huge and I think I'm excited for everyone to try it out, is definitely something where we saw it internally just do really well and we're like, oh, shit, this is just sick. And it did help drive some results like the Rusty Wright. But yeah, also we like, you know, we're... We use the models every day and and it was better.
00:16:44 · Thariq Shihipar, Aditya Agarwal
[00:16:44] S0: Yeah. Well, so I mean, Anthropic is not like a single, like, you know, monolithic entity, and so everyone has different opinions. Right? And I think that this, like, self recursive, like, harness thing is... Some people are into it. I think other people are not. You know what I mean? And I think that, like, it's Of the Claude Code. It's a mix. Okay. Yeah. Yeah. I I think we try and see what works. But I I think like something like workflows, for example, which I think is huge and I think I'm excited for everyone to try it out, is definitely something where we saw it internally just do really well and we're like, oh, shit, this is just sick. And it did help drive some results like the Rusty Wright. But yeah, also we like, you know, we're... We use the models every day and and it was better. [00:17:36] S1: So... Switching gears a little bit. And by the way, as I mentioned, if you have a burning question already, just raise your hand and we'll kinda get you in and I'll continue to ask. I see Jason there, so we'll have you queue up. One quick one while Richard's listening to that is like when you came to startup school, you shared some of your favorite prompts. Like they were kind of meta prompts. Right? The the types of things that you can ask the model to make you better at your building. Can you share some of those again? Because I thought it was helpful or some of the the new ones. And I forget specifically this... The specific one you shared at startup school, but I I I love that line of thinking of just like, are the meta prompts that that are useful?
00:17:36 · Aditya Agarwal, Thariq Shihipar
[00:17:36] S1: So... Switching gears a little bit. And by the way, as I mentioned, if you have a burning question already, just raise your hand and we'll kinda get you in and I'll continue to ask. I see Jason there, so we'll have you queue up. One quick one while Richard's listening to that is like when you came to startup school, you shared some of your favorite prompts. Like they were kind of meta prompts. Right? The the types of things that you can ask the model to make you better at your building. Can you share some of those again? Because I thought it was helpful or some of the the new ones. And I forget specifically this... The specific one you shared at startup school, but I I I love that line of thinking of just like, are the meta prompts that that are useful? [00:18:15] S0: Yeah. I... I'm trying to work on a post about this as well. I I think that like [00:18:20] S1: Follow Thariq on Twitter. He's already... What are you? Like, 250,000 followers now? It's a big deal.
00:18:25 · Thariq Shihipar
[00:18:25] S0: Yeah. I I think that like the way I think about it is, okay, when you have a task, you have like no knowns. Like like you know you want this and... But like maybe is it in the prompt or not? Like, did you write it down? You know? Then you have unknown knowns. You know? So you're like, you you think you want it, but like you don't know it until you see it. You know what I mean? And then you have like unknown unknowns, where you just like don't know that this is possible, you know? And you like don't understand the system well enough or something. Right? And then you have like known unknowns where you're like, oh, I I don't know how this works, you know, but I would like to figure it out. Right? And and so I I think for every problem, analyzing it along those dimensions helps me pull out like, okay, this is how I should prompt the model. Right? And so for example, like if I'm just being lazy about prompting, sometimes what I like to ask the model is like, hey, like interview me about this problem. You know? So like here is my idea. Let me like interview me using the ask user question tool, and then I'll answer. So like that's one example. Sometimes if it's like, okay, I'm working on a design and I know what I... I'll I'll know it when I see it, you know? I'll ask it to like do a bunch of explorations. And so I'll be like, explore this design and make a bunch of mock ups in HTML and show it to me. And then of those like whatever four, then I might ask it to do more like exploration there. But like, again, it's like in my head, I like I have some preference, but I I don't know yet. Right? And I think a meta prompting technique there is like, if you're still figuring something out, do you want to do it in a cheap way? So like, you know, if you're doing UI, it's cheaper to do it in HTML because it's fake. You know, like you're not rewriting the state and the like, you know, all this other wiring up stuff. Right? Like if you ask a model to add a button, in HTML it's just a button. But inside of your app, maybe there's a new route. There's a new like reducer, all of this stuff. Right? And so in the same way that like, you can do this for like back end endpoints and testing and things like that too, just like, what's the cheapest way to figure out like, is this the thing that you want? Right? So... Yeah. Jason?