Latent Space: The AI Engineer Podcast
00:06:22 · Zach Lloyd
[00:06:22] S2: Yes. So... Okay. So you have this whole thing set up in code, and then once you have that in place, and so the... Again, metaphor is like someone starts a task in Slack, or in linear, or through an automation, like, you know, there's a server error, it all goes through one system is the kind of key thing. And once you have that working, you basically get a whole bunch of benefits that flow out of it. And so one of the big benefits is that wherever these agents are running, and you're sort of seeing a real time view across Warp's internal team, how are we using our own factories, you get a full trace of what everyone is doing with coding agents across any product that has a factory for it. And this is cool not just for, like, oh, I wanna see what people are doing, like, I wanna see what Safik is doing, which is cool, But it also starts centralizing all of the agent conversations and agent artifacts in all of the work that are... Is being done by coding agents across your team. And so when I talk about, like, having a closed loop system, these agent conversations, these agent traces are, like, the key aspect of how this system improves. And so for instance, just to look at one of these agent traces, Satvik is working on our mobile app here. Satvik was doing this in Slack, so I could pull this up in Slack if I wanted to. But, like, the thing is actually generating it. It has a full record of the conversation, has a full record of all of the these are computer use screenshots that are coming out of building this thing. You can see a full record of exactly what the different pieces of the orchestration agent were doing. And so you have this, but you have it not just for one thing, you have it for every single thing. So on your team today, what we... When I talk to companies, for the most part, they don't have something like this because most of their engineers are, like, individually using Claude Code in their terminal or Codex, the Codex app or something like that in their own local setups, but they wanna get to a world where they have this centralization. When you have the centralization, you can basically get all this interesting data on how your agentic coding is working across your company. And you could think of this data along different dimensions.
00:08:38 · Zach Lloyd, S1
So you could think of this data as like, okay, what's our team's output in the world of using coding agents? And so that's things like, you know, traditional Dora metrics, how long does it take to ship something. It's things like how automated are you, and so we have a sort of metric around, like, for every PR that gets merged, how many times did a human have to do something with it? What's your throughput? It's metrics around cost. And so we're using this, like, constantly. And when we talk to companies, this cost thing is very, very important where you wanna be able to tune your factory with the right models, the right skills to basically bring this cost per PR down by time. You wanna be able to slice the cost by person. You wanna be able to slice it by automation, by by where in the SDLC are you spending money. The final thing that's that's really cool, once you have this sort of closed loop system going, is that you can actually look underneath the hood at how well your coding agents are doing along different dimensions. And so you can think of this as like online evals or we call it scores. And these can go from very general things to, like, are your agents being too verbose? Are they being efficient? Is the code quality good? We have work to do on this. [00:09:54] S1: Do I as a user define this or do you provide this as a platform?
00:09:58 · Zach Lloyd
[00:09:58] S2: Yeah. So both. So we provide out of the box a set of ones that we think are like generically useful. But then, you know, as we work with companies, you know, they may have specific failure modes for their agents that they're interested in tracking. They may be like, hey, is the agent making a specific type of thing that is, like, change that is, like, breaking our build? They may want something, you know, very specific to their workflow. So, again, this is all defined in code and all customizable. And what you do is, like, when you set up one of these, what's actually happening behind the scenes is that there's some set of runs that you you... When I say runs, I mean, like, historical agent conversations that have been stored that an agent then goes and, like, retroactively grades. And so this is a good example of one where it's grading for redundant tests, which is like a... This is a common failure mode for using coding agents. Like, sometimes they write a bunch of tests that you don't need. They write it in the course of, like, figuring out how to, like, implement something, and then they leave it in the code. And so this, like, bloats your code base, it wastes tokens. And so, you know, say you wanted to check for that, you'd... You know, you basically create one of these evals. And just to show you, like, a little more about what the eval looks like, the eval is... Again, think of it as LLM as a judge. So you're picking a a judge model. You're you're then seeing, okay, like, where do these run? What's the prompt for judging it? And then a sampling rate because you might not... Like, these are agents that are running. You might... You gotta be wary of the cost of them. But when you have this set up, think of it then as, like, you have all these smart agents that are looking across your team as... At how all your your coding sessions are going, and they're saying, okay. Sometimes there are agents that are writing too many tests. Right? And so you get this record and this understanding of where you should look to sort of improve the way that your team is using coding agents. So far so good? Because I wanna show a couple other things just to, like, make it more specific what you can do with this.
00:12:05 · S1, Zach Lloyd
[00:12:05] S1: Overall, good. I'm I'm a little bit wondering if this... If basically there's different levels of of of Warp where there's like... This is the CTO level, what you're showing us. Right? This... You're you're running the company. You've got a bunch of coders. [00:12:20] S2: This is like... The the people who use this, it's partly the engineering manager or the CTO, but it's also like very commonly like the platform team. But it's not your, like, day to day engineer, typically. But, like, when we work with companies, you know, they'll often have an internal team that's been trying to, like... It's like the dev x team or the platform team, and they're trying to figure out how to optimize coding agents. So what I'm showing you here, yeah, is like the tool set that they are using to be like, how do we get the most out of Claude Code at our company or Codex or something like that. [00:12:51] S1: That that part is clear.
00:12:53 · Zach Lloyd
[00:12:53] S2: Yep. Cool. Yeah. And so like a a couple other things that I would just wanna emphasize, not to make this just, like, completely about Warp factories, but, like, the general principle here for anyone who's trying to build their internal factory, whether it's on Warp factories or not, is, like, you wanna get to a spot where everything is sort of stored, everything is observable, everything is measurable because that's when you can actually start to, like, apply engineering rigor to how you make coding agents better. You know, you probably... You see this on Twitter all the time. Everyone is always tweeting about, like, you should check out this new model, you should use this new harness, what's better? Is it pie or Claude Code or, you know, cursor or whatever? And so what our opinion on this is and what the the world sort of needs is, like, out of the the realm of vibes for understanding these coding agents and actually into the world of measurement. And so we think that this is the direction that companies are gonna wanna go. They're they're gonna wanna, like, put in place a system that's like WarpFactories, whether they actually use WarpFactories or they build their own, which collects all this information and provides all this way to see how your coding agents are doing. I guess one other thing I like to kinda take for granted that's part of this system is like you can create all these different automations around it. So like I I mentioned people work in Slack, but people are also setting up a different trigger based ways of putting work into the factory. Once you have this view that I showed earlier of, like, scoring how your factory is doing, You can also set up another kinda loop of agents, and everything is kinda loops these days, where those agents are looking over. They're looking for failures in your scores. Like, they're looking for things that aren't going well and trying to suggest changes to your system to make it go better. And so, like, a really simple example here, we call this self improvement, is like, okay. Our self improvement agents were looking at how well our triage step was doing. So we have a triage step, and they're saying, okay. We have 54 failure cases of this particular form where it would have been better if our triager had been able to spawn a sub agent to do its investigation.
00:15:14 · Zach Lloyd
And so what they're able to do, and like, you could go look at like why it thinks this if you want. But the cool thing is like, because you have your whole factory defined as code, and these are coding agents, they can essentially make a pull request against the factory definition to try to improve the factory itself. And so this is like a like a full learning loop that's built into the system to try to actually self optimize. And like, we don't have this yet, but the next thing that we want to do with this is like, actually, for any suggested change like this, you could go back, replay the runs, and make sure that this actually would have improved them. So the possibilities of, like, how you can engineer this stuff get pretty cool once you have the right primitives in place. And I'll show one last thing, and then we could sort of maybe talk more generally about the whole approach here. But the other thing that is, you know, I think companies should strive for that we find very useful internally is, like, the ability to actually do what we call benchmarking. And so benchmarking in this context, in the factory context, means essentially the ability to, like, take any coding agent that's running your system, you can sort of add it to a benchmark suite, which means, like, you know, you might curate a set of coding agent runs that were related to, like, doing front end coding or something like that. And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so, you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And so, you know, in Warp, we have our own harness, but we actually will let you benchmark against Claude Code or Codex or Pi. You can pick the model, and when you do something like this, what you end up getting is a result that looks very similar to, like, the published results for SWE-bench or TerminalBench, but on your company's own real runs. For instance, we're looking at, like, a sort of cost versus correctness trade off for a certain set of things here. And, like, you wanna be in the upper left hand corner, so you want, like, lowest cost, highest correctness. These are... These models are now slightly older because the world's changing so fast.