Latent Space: The AI Engineer Podcast
00:18:49 · S1
[00:18:49] S1: I do think that, you know, the the the power power move, I guess, is to sort of thought lead so hard that everyone ends up trying to build Warp even if they don't use Warp. Yeah. So, you know, you have on your landing page something I I like it. In your talks, you've done a few of these as well. Something I really like also is any surface. Right? It really has to be any surface. It has to be factory access code. You have to have the eval loop. Like, what is the whole... Like, people calling this the ADLC. I don't know if you used that term. It looks like you use SDLC still. You know, all all that stuff.
00:19:23 · Zach Lloyd
[00:19:23] S2: This is how I would think about it. Again, whether it's Warp or not, if it's... I would obviously... You know, Warp Warp is this is what we're fully focused on. We have an awesome platform for it. But just in general, what I would think about if you are trying to build something in this space is, a, I would view it as like a stack, so as infrastructure. At the bottom of the stack, I think it's really foundational and important that you define your factory declaratively as code. So again, the analogy is like Terraform here, you get the same benefits, but actually more. So the benefits you get from defining it as code is, like, you get a versioned view of, like, what was your weak SDLC, AI DLC, p... PDLC, whatever DLC you want. You get a versioned view of that, and then you get the ability to create branches off it, which you can test. You get the ability if you change it to revert it. So all the obvious things about source control that are great. And then you get the added benefit that's extremely cool, where because it's in code, coding agents can suggest improvements to it. So to me, this is like the foundation of a self improving software factory setup, is define your factory declaratively. The next step up in this stack, as you think about architecting it, is some place to gather all of the information that agents are producing. Because what what you wanna be able to do is examine that information, look for failure modes, or replay those same past tasks with different variables to sort of... I don't know, scientifically is the right word, but from a measurement standpoint, really optimize your system. So that's the mindset. So you need the data to do that. You need the agent conversations, and you need the agent artifacts. You need a record of all of the interactions from all of the tools to get to a result. And so what I mean by that is, like, you might have a a factory conversation that starts in Slack, and then continues into linear, and then continues into GitHub, and there's a code review comment. And that whole interaction is gonna result in some piece of merged code. In fact, it might even result in something that gets shipped, and then has a server crashed. And you want the ability to, like, retroactively go back with an agent, most likely, but you could do it as a person and and diagnose how could that have been better?
00:21:45 · Zach Lloyd, S1
What was the failure in it? You also need... This part will seem fairly obvious, but you need a way of getting data into this system, so you need some way of plugging in MCPs. And then we find that as companies get bigger, they want different access for different parts of their organization. So they want things like what MCPs can the marketing team access versus the sales team versus the engineering team. And so, like, again, you'll you'll find there's a bunch of complexity if you're... Depending on how big your organization is if you're trying to roll something like this out. You really want a compute layer. You should think of that as a separate thing. And so there's there's obviously a lot of agent sandbox providers at this point. So if you're gonna use a third party one, I would make... Just make that pluggable. We use namespace as our primary one. [00:22:37] S1: Oh. They launched it at AI Europe.
00:22:40 · Zach Lloyd
[00:22:40] S2: Namespace is awesome. But this should be pluggable. I think that's really important. So if you wanted to use Daytona or Modal or E2B or whatever, like, I would architect it. So this is Plugable component of it. I would make sure, depending on your requirements, that you can do different platforms. So, again, by default, you know, I think you'll probably wanna use, like, a Linux type sandbox, but for computer use, like, the way that we test, like, iOS apps is you're gonna wanna have the ability to run a Mac VM and actually like have it run the simulator and click around. We have a very open approach to inference, and this is like... This is very much with our ethos of trying to be super duper flexible. So we find that a lot of companies right now will have preexisting commits with Anthropic or OpenAI or Google that they wanna burn down. And so we're... We provide, like, a very big variety of, you can plug in your own inference provider or you can buy inference through us. You can even plug in your own harness, which I think is a pretty big deal. So, you know, you you can orchestrate things like pie, like I mentioned earlier, any ACP compatible harness. And then you want this improvement layer. And so this is some of the stuff that I showed in in the demo where it's like, the real power of this whole system is is the ability to improve it. And so just to give some stats at Warp, when we first started using the factory approach, it was costing us about $80 per PR because we were using, like, Opus where we shouldn't have. Our triage agent was using, like, a really high powered model. You know, we were doing a bunch of redundant work with our... The shape of agents. Like, we were doing too many sub agents relative to just having fewer agents. And so we basically did a whole bunch of internal tuning and testing using these built in features, and we were able to get the cost per PR down to, like, $20 or $15, something like that. Well, we can measurably improve the throughput of our PRs. Over the last month, it's gone up by 40%. We measure, like, the revert rate, and it hasn't gone up. And so I really feel like a lot of confidence in this general approach of, like, observe, measure, tune. Finally, you need some sort of, like, orchestrator. Sometimes people talk about the orchestrator like it's the whole platform. I think it's a piece of it.
00:25:01 · Zach Lloyd
What I mean by orchestrator is, like, you need some system that will take the factory definition code and then actually move work along the different stages, calling out to the right models, loading the right skills, etcetera. You need something that will fire the right automations. And then finally, you need a bunch of access points. And again, our philosophy with this, which you can... I could see people have different philosophies. Our philosophy is to sort of bring the factory into a company's existing knowledge work tools. So think of like, you know, the most obvious one is Slack, and like we're doing like 60% of our work is starting in Slack. But sometimes it makes sense to start in the task tracker. Sometimes it makes sense for for a developer who is more comfortable using, like, Claude Code in their terminal to do a bunch of work there and then submit a spec into the factory. And so, you know, our philosophy is to try to to make this whole thing API driven and MCP driven so that the factory itself is like a piece of kinda headless infrastructure where it has all these different entry points. So I think this this stack based view of it when I've shown it or when we talk to companies, seems to land very well. It's like, okay. Yes. This this makes sense. This is a new thing in the, like, developer infrastructure stack. It kinda sits one level back from, like, where CICD is, but I think most companies will have something like this for software development.
00:26:29 · S1, Zach Lloyd
[00:26:29] S1: Yeah. That has a lot of layers, actually. I've never actually seen it spelled out like like you have. I wonder... Because I think most, you know, Cognition and I've... I... You know, Cursor, what... All these words, They all try to do some bundle of, like, the vert... Like, from bottom to top. And there's a little bit of bundling, unbundling with all software. And so there's a critical question of, like, look, I am trying to do my job here as CTO of a, you know, b to b SaaS. Like, I'm... [00:27:00] S2: Do I need like all these, like, six layers? I think it's like, who's this for? Maybe is is the kind of the question. So when we talk to companies that... Where this resonates the most is it has to be a company that has some some amount of scale. Like, this is... If you're like a few people, honestly, this is this is overkill. Yeah. Don't do this. Use the Claude code sub or the codec sub or whatever, and just... You'll go faster. You probably don't need And You could use it. Yeah. [00:27:30] S1: Same for, like, any, like, larger team stack. Right? Like, you... At some point, you switch from, like, fire, YOLO, whatever, to we actually need a platform and system here to accelerate Exactly.