Latent Space: The AI Engineer Podcast
00:12:05 · S1, Zach Lloyd
[00:12:05] S1: Overall, good. I'm I'm a little bit wondering if this... If basically there's different levels of of of Warp where there's like... This is the CTO level, what you're showing us. Right? This... You're you're running the company. You've got a bunch of coders. [00:12:20] S2: This is like... The the people who use this, it's partly the engineering manager or the CTO, but it's also like very commonly like the platform team. But it's not your, like, day to day engineer, typically. But, like, when we work with companies, you know, they'll often have an internal team that's been trying to, like... It's like the dev x team or the platform team, and they're trying to figure out how to optimize coding agents. So what I'm showing you here, yeah, is like the tool set that they are using to be like, how do we get the most out of Claude Code at our company or Codex or something like that. [00:12:51] S1: That that part is clear.
00:12:53 · Zach Lloyd
[00:12:53] S2: Yep. Cool. Yeah. And so like a a couple other things that I would just wanna emphasize, not to make this just, like, completely about Warp factories, but, like, the general principle here for anyone who's trying to build their internal factory, whether it's on Warp factories or not, is, like, you wanna get to a spot where everything is sort of stored, everything is observable, everything is measurable because that's when you can actually start to, like, apply engineering rigor to how you make coding agents better. You know, you probably... You see this on Twitter all the time. Everyone is always tweeting about, like, you should check out this new model, you should use this new harness, what's better? Is it pie or Claude Code or, you know, cursor or whatever? And so what our opinion on this is and what the the world sort of needs is, like, out of the the realm of vibes for understanding these coding agents and actually into the world of measurement. And so we think that this is the direction that companies are gonna wanna go. They're they're gonna wanna, like, put in place a system that's like WarpFactories, whether they actually use WarpFactories or they build their own, which collects all this information and provides all this way to see how your coding agents are doing. I guess one other thing I like to kinda take for granted that's part of this system is like you can create all these different automations around it. So like I I mentioned people work in Slack, but people are also setting up a different trigger based ways of putting work into the factory. Once you have this view that I showed earlier of, like, scoring how your factory is doing, You can also set up another kinda loop of agents, and everything is kinda loops these days, where those agents are looking over. They're looking for failures in your scores. Like, they're looking for things that aren't going well and trying to suggest changes to your system to make it go better. And so, like, a really simple example here, we call this self improvement, is like, okay. Our self improvement agents were looking at how well our triage step was doing. So we have a triage step, and they're saying, okay. We have 54 failure cases of this particular form where it would have been better if our triager had been able to spawn a sub agent to do its investigation.
00:15:14 · Zach Lloyd
And so what they're able to do, and like, you could go look at like why it thinks this if you want. But the cool thing is like, because you have your whole factory defined as code, and these are coding agents, they can essentially make a pull request against the factory definition to try to improve the factory itself. And so this is like a like a full learning loop that's built into the system to try to actually self optimize. And like, we don't have this yet, but the next thing that we want to do with this is like, actually, for any suggested change like this, you could go back, replay the runs, and make sure that this actually would have improved them. So the possibilities of, like, how you can engineer this stuff get pretty cool once you have the right primitives in place. And I'll show one last thing, and then we could sort of maybe talk more generally about the whole approach here. But the other thing that is, you know, I think companies should strive for that we find very useful internally is, like, the ability to actually do what we call benchmarking. And so benchmarking in this context, in the factory context, means essentially the ability to, like, take any coding agent that's running your system, you can sort of add it to a benchmark suite, which means, like, you know, you might curate a set of coding agent runs that were related to, like, doing front end coding or something like that. And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so, you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And so, you know, in Warp, we have our own harness, but we actually will let you benchmark against Claude Code or Codex or Pi. You can pick the model, and when you do something like this, what you end up getting is a result that looks very similar to, like, the published results for SWE-bench or TerminalBench, but on your company's own real runs. For instance, we're looking at, like, a sort of cost versus correctness trade off for a certain set of things here. And, like, you wanna be in the upper left hand corner, so you want, like, lowest cost, highest correctness. These are... These models are now slightly older because the world's changing so fast.
00:17:33 · Zach Lloyd
But the idea is basically, you know, we could see, okay, GPT five six SOL here was, like, 95% correct on this task set, but it costs $6 on average, and GLM five three Flash was 93% and cost $2. And then the cool thing about this is you can take this information and feed it into a model routing strategy or harness routing strategy, honestly, that's defined specifically for your own workflows in code base. And so the... You know, if I had again to zoom out, the really big picture thing that I think is so cool and powerful here is, like, you're getting out of the realm of guessing and you're getting into what we call factory engineering, where it's like you are taking a real engineering mindset. You're like, how do I... We have this amazing new primitive of of coding agents. How do I know as a team, as a company, that we're actually deploying them in the right way? And so, you know, we're trying to build this infrastructure and tools for companies to do this. And, like, this approach is really, really resonating with a lot of companies that have been trying to build, you know, their own their own versions of this. So that's like a quick tour of like, what do I mean when when I talk about factory? I can I can sort of stop talking for a sec?
00:18:49 · S1
[00:18:49] S1: I do think that, you know, the the the power power move, I guess, is to sort of thought lead so hard that everyone ends up trying to build Warp even if they don't use Warp. Yeah. So, you know, you have on your landing page something I I like it. In your talks, you've done a few of these as well. Something I really like also is any surface. Right? It really has to be any surface. It has to be factory access code. You have to have the eval loop. Like, what is the whole... Like, people calling this the ADLC. I don't know if you used that term. It looks like you use SDLC still. You know, all all that stuff.
00:19:23 · Zach Lloyd
[00:19:23] S2: This is how I would think about it. Again, whether it's Warp or not, if it's... I would obviously... You know, Warp Warp is this is what we're fully focused on. We have an awesome platform for it. But just in general, what I would think about if you are trying to build something in this space is, a, I would view it as like a stack, so as infrastructure. At the bottom of the stack, I think it's really foundational and important that you define your factory declaratively as code. So again, the analogy is like Terraform here, you get the same benefits, but actually more. So the benefits you get from defining it as code is, like, you get a versioned view of, like, what was your weak SDLC, AI DLC, p... PDLC, whatever DLC you want. You get a versioned view of that, and then you get the ability to create branches off it, which you can test. You get the ability if you change it to revert it. So all the obvious things about source control that are great. And then you get the added benefit that's extremely cool, where because it's in code, coding agents can suggest improvements to it. So to me, this is like the foundation of a self improving software factory setup, is define your factory declaratively. The next step up in this stack, as you think about architecting it, is some place to gather all of the information that agents are producing. Because what what you wanna be able to do is examine that information, look for failure modes, or replay those same past tasks with different variables to sort of... I don't know, scientifically is the right word, but from a measurement standpoint, really optimize your system. So that's the mindset. So you need the data to do that. You need the agent conversations, and you need the agent artifacts. You need a record of all of the interactions from all of the tools to get to a result. And so what I mean by that is, like, you might have a a factory conversation that starts in Slack, and then continues into linear, and then continues into GitHub, and there's a code review comment. And that whole interaction is gonna result in some piece of merged code. In fact, it might even result in something that gets shipped, and then has a server crashed. And you want the ability to, like, retroactively go back with an agent, most likely, but you could do it as a person and and diagnose how could that have been better?