Latent Space: The AI Engineer Podcast
00:09:58 · Zach Lloyd
[00:09:58] S2: Yeah. So both. So we provide out of the box a set of ones that we think are like generically useful. But then, you know, as we work with companies, you know, they may have specific failure modes for their agents that they're interested in tracking. They may be like, hey, is the agent making a specific type of thing that is, like, change that is, like, breaking our build? They may want something, you know, very specific to their workflow. So, again, this is all defined in code and all customizable. And what you do is, like, when you set up one of these, what's actually happening behind the scenes is that there's some set of runs that you you... When I say runs, I mean, like, historical agent conversations that have been stored that an agent then goes and, like, retroactively grades. And so this is a good example of one where it's grading for redundant tests, which is like a... This is a common failure mode for using coding agents. Like, sometimes they write a bunch of tests that you don't need. They write it in the course of, like, figuring out how to, like, implement something, and then they leave it in the code. And so this, like, bloats your code base, it wastes tokens. And so, you know, say you wanted to check for that, you'd... You know, you basically create one of these evals. And just to show you, like, a little more about what the eval looks like, the eval is... Again, think of it as LLM as a judge. So you're picking a a judge model. You're you're then seeing, okay, like, where do these run? What's the prompt for judging it? And then a sampling rate because you might not... Like, these are agents that are running. You might... You gotta be wary of the cost of them. But when you have this set up, think of it then as, like, you have all these smart agents that are looking across your team as... At how all your your coding sessions are going, and they're saying, okay. Sometimes there are agents that are writing too many tests. Right? And so you get this record and this understanding of where you should look to sort of improve the way that your team is using coding agents. So far so good? Because I wanna show a couple other things just to, like, make it more specific what you can do with this.
00:12:05 · S1, Zach Lloyd
[00:12:05] S1: Overall, good. I'm I'm a little bit wondering if this... If basically there's different levels of of of Warp where there's like... This is the CTO level, what you're showing us. Right? This... You're you're running the company. You've got a bunch of coders. [00:12:20] S2: This is like... The the people who use this, it's partly the engineering manager or the CTO, but it's also like very commonly like the platform team. But it's not your, like, day to day engineer, typically. But, like, when we work with companies, you know, they'll often have an internal team that's been trying to, like... It's like the dev x team or the platform team, and they're trying to figure out how to optimize coding agents. So what I'm showing you here, yeah, is like the tool set that they are using to be like, how do we get the most out of Claude Code at our company or Codex or something like that. [00:12:51] S1: That that part is clear.
00:12:53 · Zach Lloyd
[00:12:53] S2: Yep. Cool. Yeah. And so like a a couple other things that I would just wanna emphasize, not to make this just, like, completely about Warp factories, but, like, the general principle here for anyone who's trying to build their internal factory, whether it's on Warp factories or not, is, like, you wanna get to a spot where everything is sort of stored, everything is observable, everything is measurable because that's when you can actually start to, like, apply engineering rigor to how you make coding agents better. You know, you probably... You see this on Twitter all the time. Everyone is always tweeting about, like, you should check out this new model, you should use this new harness, what's better? Is it pie or Claude Code or, you know, cursor or whatever? And so what our opinion on this is and what the the world sort of needs is, like, out of the the realm of vibes for understanding these coding agents and actually into the world of measurement. And so we think that this is the direction that companies are gonna wanna go. They're they're gonna wanna, like, put in place a system that's like WarpFactories, whether they actually use WarpFactories or they build their own, which collects all this information and provides all this way to see how your coding agents are doing. I guess one other thing I like to kinda take for granted that's part of this system is like you can create all these different automations around it. So like I I mentioned people work in Slack, but people are also setting up a different trigger based ways of putting work into the factory. Once you have this view that I showed earlier of, like, scoring how your factory is doing, You can also set up another kinda loop of agents, and everything is kinda loops these days, where those agents are looking over. They're looking for failures in your scores. Like, they're looking for things that aren't going well and trying to suggest changes to your system to make it go better. And so, like, a really simple example here, we call this self improvement, is like, okay. Our self improvement agents were looking at how well our triage step was doing. So we have a triage step, and they're saying, okay. We have 54 failure cases of this particular form where it would have been better if our triager had been able to spawn a sub agent to do its investigation.
00:15:14 · Zach Lloyd
And so what they're able to do, and like, you could go look at like why it thinks this if you want. But the cool thing is like, because you have your whole factory defined as code, and these are coding agents, they can essentially make a pull request against the factory definition to try to improve the factory itself. And so this is like a like a full learning loop that's built into the system to try to actually self optimize. And like, we don't have this yet, but the next thing that we want to do with this is like, actually, for any suggested change like this, you could go back, replay the runs, and make sure that this actually would have improved them. So the possibilities of, like, how you can engineer this stuff get pretty cool once you have the right primitives in place. And I'll show one last thing, and then we could sort of maybe talk more generally about the whole approach here. But the other thing that is, you know, I think companies should strive for that we find very useful internally is, like, the ability to actually do what we call benchmarking. And so benchmarking in this context, in the factory context, means essentially the ability to, like, take any coding agent that's running your system, you can sort of add it to a benchmark suite, which means, like, you know, you might curate a set of coding agent runs that were related to, like, doing front end coding or something like that. And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so, you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And so, you know, in Warp, we have our own harness, but we actually will let you benchmark against Claude Code or Codex or Pi. You can pick the model, and when you do something like this, what you end up getting is a result that looks very similar to, like, the published results for SWE-bench or TerminalBench, but on your company's own real runs. For instance, we're looking at, like, a sort of cost versus correctness trade off for a certain set of things here. And, like, you wanna be in the upper left hand corner, so you want, like, lowest cost, highest correctness. These are... These models are now slightly older because the world's changing so fast.
00:17:33 · Zach Lloyd
But the idea is basically, you know, we could see, okay, GPT five six SOL here was, like, 95% correct on this task set, but it costs $6 on average, and GLM five three Flash was 93% and cost $2. And then the cool thing about this is you can take this information and feed it into a model routing strategy or harness routing strategy, honestly, that's defined specifically for your own workflows in code base. And so the... You know, if I had again to zoom out, the really big picture thing that I think is so cool and powerful here is, like, you're getting out of the realm of guessing and you're getting into what we call factory engineering, where it's like you are taking a real engineering mindset. You're like, how do I... We have this amazing new primitive of of coding agents. How do I know as a team, as a company, that we're actually deploying them in the right way? And so, you know, we're trying to build this infrastructure and tools for companies to do this. And, like, this approach is really, really resonating with a lot of companies that have been trying to build, you know, their own their own versions of this. So that's like a quick tour of like, what do I mean when when I talk about factory? I can I can sort of stop talking for a sec?
00:18:49 · S1
[00:18:49] S1: I do think that, you know, the the the power power move, I guess, is to sort of thought lead so hard that everyone ends up trying to build Warp even if they don't use Warp. Yeah. So, you know, you have on your landing page something I I like it. In your talks, you've done a few of these as well. Something I really like also is any surface. Right? It really has to be any surface. It has to be factory access code. You have to have the eval loop. Like, what is the whole... Like, people calling this the ADLC. I don't know if you used that term. It looks like you use SDLC still. You know, all all that stuff.