Latent Space: The AI Engineer Podcast
00:21:45 · Zach Lloyd, S1
What was the failure in it? You also need... This part will seem fairly obvious, but you need a way of getting data into this system, so you need some way of plugging in MCPs. And then we find that as companies get bigger, they want different access for different parts of their organization. So they want things like what MCPs can the marketing team access versus the sales team versus the engineering team. And so, like, again, you'll you'll find there's a bunch of complexity if you're... Depending on how big your organization is if you're trying to roll something like this out. You really want a compute layer. You should think of that as a separate thing. And so there's there's obviously a lot of agent sandbox providers at this point. So if you're gonna use a third party one, I would make... Just make that pluggable. We use namespace as our primary one. [00:22:37] S1: Oh. They launched it at AI Europe.
00:22:40 · Zach Lloyd
[00:22:40] S2: Namespace is awesome. But this should be pluggable. I think that's really important. So if you wanted to use Daytona or Modal or E2B or whatever, like, I would architect it. So this is Plugable component of it. I would make sure, depending on your requirements, that you can do different platforms. So, again, by default, you know, I think you'll probably wanna use, like, a Linux type sandbox, but for computer use, like, the way that we test, like, iOS apps is you're gonna wanna have the ability to run a Mac VM and actually like have it run the simulator and click around. We have a very open approach to inference, and this is like... This is very much with our ethos of trying to be super duper flexible. So we find that a lot of companies right now will have preexisting commits with Anthropic or OpenAI or Google that they wanna burn down. And so we're... We provide, like, a very big variety of, you can plug in your own inference provider or you can buy inference through us. You can even plug in your own harness, which I think is a pretty big deal. So, you know, you you can orchestrate things like pie, like I mentioned earlier, any ACP compatible harness. And then you want this improvement layer. And so this is some of the stuff that I showed in in the demo where it's like, the real power of this whole system is is the ability to improve it. And so just to give some stats at Warp, when we first started using the factory approach, it was costing us about $80 per PR because we were using, like, Opus where we shouldn't have. Our triage agent was using, like, a really high powered model. You know, we were doing a bunch of redundant work with our... The shape of agents. Like, we were doing too many sub agents relative to just having fewer agents. And so we basically did a whole bunch of internal tuning and testing using these built in features, and we were able to get the cost per PR down to, like, $20 or $15, something like that. Well, we can measurably improve the throughput of our PRs. Over the last month, it's gone up by 40%. We measure, like, the revert rate, and it hasn't gone up. And so I really feel like a lot of confidence in this general approach of, like, observe, measure, tune. Finally, you need some sort of, like, orchestrator. Sometimes people talk about the orchestrator like it's the whole platform. I think it's a piece of it.
00:25:01 · Zach Lloyd
What I mean by orchestrator is, like, you need some system that will take the factory definition code and then actually move work along the different stages, calling out to the right models, loading the right skills, etcetera. You need something that will fire the right automations. And then finally, you need a bunch of access points. And again, our philosophy with this, which you can... I could see people have different philosophies. Our philosophy is to sort of bring the factory into a company's existing knowledge work tools. So think of like, you know, the most obvious one is Slack, and like we're doing like 60% of our work is starting in Slack. But sometimes it makes sense to start in the task tracker. Sometimes it makes sense for for a developer who is more comfortable using, like, Claude Code in their terminal to do a bunch of work there and then submit a spec into the factory. And so, you know, our philosophy is to try to to make this whole thing API driven and MCP driven so that the factory itself is like a piece of kinda headless infrastructure where it has all these different entry points. So I think this this stack based view of it when I've shown it or when we talk to companies, seems to land very well. It's like, okay. Yes. This this makes sense. This is a new thing in the, like, developer infrastructure stack. It kinda sits one level back from, like, where CICD is, but I think most companies will have something like this for software development.
00:26:29 · S1, Zach Lloyd
[00:26:29] S1: Yeah. That has a lot of layers, actually. I've never actually seen it spelled out like like you have. I wonder... Because I think most, you know, Cognition and I've... I... You know, Cursor, what... All these words, They all try to do some bundle of, like, the vert... Like, from bottom to top. And there's a little bit of bundling, unbundling with all software. And so there's a critical question of, like, look, I am trying to do my job here as CTO of a, you know, b to b SaaS. Like, I'm... [00:27:00] S2: Do I need like all these, like, six layers? I think it's like, who's this for? Maybe is is the kind of the question. So when we talk to companies that... Where this resonates the most is it has to be a company that has some some amount of scale. Like, this is... If you're like a few people, honestly, this is this is overkill. Yeah. Don't do this. Use the Claude code sub or the codec sub or whatever, and just... You'll go faster. You probably don't need And You could use it. Yeah. [00:27:30] S1: Same for, like, any, like, larger team stack. Right? Like, you... At some point, you switch from, like, fire, YOLO, whatever, to we actually need a platform and system here to accelerate Exactly.
00:27:42 · Zach Lloyd
[00:27:42] S2: Yeah. So you have to be a certain scale. We start seeing that maybe around, like, five to 10 engineers would maybe be the floor where this makes sense. And then all the way up to to very large companies, they want something like this. The bigger the company, the more that this requires, you know, implementation work. Like, I'm sure if you're, you know, you're working with Cognition or or... I don't know if Comment Cursor does, but, you know, to get something stood up like this within a company's own infrastructure, it's not gonna be purely plug and play. There's, like, you know, the sort of complexity of a team's internal developer setup just grows with size. Within, like... But there's a bunch of companies where, you know, maybe, like, it's, like, a 10 to 100 engineers, fast growing, like, growth stage startup, where it is pretty self serve, actually, where it's, like, you just... If they're using, like, the happy stack of, like, Slack, you know, linear Jira... Hang on. Did you call it the happy stack? I called it, like, yeah. I... That's like our... I don't know. No no no shade to to Teams or whatever, like, sure. But, like, there's, like, you know, like, a set of integrations that I find very common at startups where you can plug something like this in and get, like, immediate immediate value without too much setup. I don't know if you... How much you're working like this, but if you're working with, like, Devin or something, you can kinda get some of the same experience where when you're working in a... Some... An environment that's natively multiplayer, so like Slack, for instance, or linear, there's all these benefits that you might not realize just by having, like, this central system of record where you can have, you know, for, like, the Slack integration, for instance, it's like, you can get some flow of information coming in through, like, a server crash channel and some through a user feedback channel. You can just feed that directly into the factory to have it do the work. The factory can loop in multiple people on your team depending on who knows the right part of the code base. And so, you know, I don't think you have to be that big for this for this to make sense, but it definitely doesn't make sense for us...
00:29:48 · Zach Lloyd, S1
For a small team, and it really makes the most sense for a company that's like, I wanna I wanna put in place the right infrastructure so that as models and agents continue to change, we have a system that we're building upon. [00:30:00] S1: There's a lot to go on here. The the the one follow one follow-up question, and then I'm gonna have you plug your AI e talks and just generally where to where to follow-up with you. If you... What is the most steel man criticism of setting up all this? Like, when you encounter criticism or pushback, what should people know as, like, here here is, like, what people are running into? [00:30:24] S2: I'll answer this in, like, there's a certain class of customer where I don't think it makes sense to, like, go on to this particular thing and that's like [00:30:32] S1: Yeah. And beyond the size thing. [00:30:34] S2: Yeah. Where we have challenges right now are companies with very, very bespoke infrastructure or companies with, like, are are kind of not a good fit customer for this, for instance, would be like Uber or Ramp. Like, they're building their own thing. They have very large internal developer teams. But apart from the size thing, I'm thinking like... I mean, there's definitely like the... For some people, they should... It's just gonna be more economical to just use the subs, like, for small... Very small teams, but that comes back to the size Actually...