The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:38:29 · Harrison Chase
[00:38:29] S0: I think we spend probably roughly 30% of our engineering salary on coding tools. It's actually gone down the last two months, while tokens, the number of tokens that we've used, has actually gone up. And so this has... We've done this by three steps. First is basically just visibility. We tracked all of our coding agent usage to Langsmith so we could tell who was spending how much, what they were spending it on. We could do analytics over that. Second is just basic cost controls. We set kind of like a budget. And and if you if you hit the budget, you can talk to our VP of engineering and and he'll almost always, like, increase it, but it's just a safeguard against kind of like agents being used incorrectly or going in a in a in a bad loop. And then third would be we control the harness. So one of the things we've done is we actually built our own in house cloud coding agent. So not not a IDE, not a not a terminal based agent. We we let people use whatever they want for that, but we built our own cloud based agent, Oop... OpenSUI is what we call it. It's actually the only way I code anymore because I'm in Slack all the time, and so I just message it from Slack and it kicks off. But in that harness, we've done a bunch of work to control costs. So things like prompt caching, but also things like routing, we've built into this harness. And I actually think that's like a pretty repeatable three step process for kind of, like, getting coding agent spend under control. Just first, visibility. Second, like, basic controls. Third, like, try to improve the harness itself.
00:39:42 · Harry Stebbings, Harrison Chase
[00:39:42] S1: Totally get that. I do just wanna unpack that. You mentioned that visibility, see who's spending what. Were the people spending the most the most effective? [00:39:51] S0: Sometimes the people spending most are the most effective, and other times they're not. So what I mean by that is we have a few senior engineers who spend a lot, and I a 100% trust that what they're doing is well worth it. No questions there. We've also just had some cases do do more to, like, kind of, like, silly mistakes or kind of, like, improper setup than anything. We've had, like, massive spikes in usage. So so an example of this, we we had one or two tasks where some plug in was being used that basically, like, would would screenshot or take a snapshot of of the DOM of a web page and send that back every turn of the agent because they were doing some front end engineering, and they wanted to take what it looked like and just send it back. It would just run this massive loop. And so I think we spent 40 k in a week on on that one thing. You know, when it gets to that, like, again, like, I think we're pretty kind of, like, liberal as a company and trust people, but I I know that's probably not worth it. So And that's just like an example of like... Again, it's... It wasn't malicious. It wasn't even intended usage really. It was kind of just a mistake. So I just think just like catching that is well worth it. So it depends a bit on the side of the spectrum. [00:40:52] S1: 30% of salary, as you mentioned. [00:40:55] S0: That's guesstimate. Yeah.
00:40:55 · Harrison Chase, Harry Stebbings
[00:40:55] S0: That's guesstimate. Yeah. [00:40:56] S1: A guesstimate. Totally fine. What is that in two years?
00:41:00 · Harrison Chase
[00:41:00] S0: I think the hard part about estimating coding agent spend is that orgs are just changing so rapidly. So what I mean by that is I think there's a future where orgs are much smaller and coding agent spend is, you know, the same as human salary or or or more or an order of magnitude more. And you have a small number of people that are driving and executing incredibly rapidly and incredibly performantly. And I and I think that's an enticing vision in terms of of of what you think about. I don't think many companies, including us, are are probably there. I think we still see that we need a bunch of different people with a bunch of different backgrounds and a bunch of different expertises collaborating on things, and there's a bunch of communication and overhead and review. And, like, yes, it would be ideal to have one person driving a 100 agents and just, like, driving everything that, like, makes up, you know, a large part of the product. But I think we just see that it's still a very human component to it. And so I spend a lot of time thinking about kind of, like, org design. Is it possible to have, like, small teams, one person teams, two person teams that are kind of, like, driving agents? I think it's very tempting. And in that situation, I would say, like, you know, agent spend, maybe three x the cost of... But then also, you know, the engineers who can do that are incredibly valuable. And so maybe it actually will just stay the same because the people who can actually do that and who can drive all those agents, there's fewer people, but you're paying them a lot more. And so the spend actually stays the same. So I think org design is very, very interesting right now. And so because of that, I think it's actually really hard to answer that question.
00:42:30 · Harry Stebbings, Harrison Chase
[00:42:30] S1: To me, that number is so central because it is the fundamental determinant of Anthropics enterprise value. If that number goes to 5%, then the $2,000,000,000,000 IPO might be a little bit overpriced. If that number actually goes to 300% like it is in McCall's case, then it's dramatically underpriced. And so that's why I focus so much on it. Do you agree that number underpins Anthropix value? [00:42:54] S0: I think it's probably right that that number underpins Anthropix value. I think the tricky part is that the only place we've seen agents really have effect is in software engineering so far, though. And there are a ton of other places within an enterprise that I I would argue have yet to kind of be affected by by agents. And so I think it's possible. Again, software engineering, think very difficult to predict. But but I think there's basically two levers here. One is like what happens in software engineering? Is it the same as today? Is it more than? Is it less than? But I think undoubtedly for the other domains, it will be a lot more than it is today. And so I think from that perspective, I think that's very bullish on Anthropic and OpenAI and other model providers.
00:42:54 · Harrison Chase, Harry Stebbings
[00:42:54] S0: I think it's probably right that that number underpins Anthropix value. I think the tricky part is that the only place we've seen agents really have effect is in software engineering so far, though. And there are a ton of other places within an enterprise that I I would argue have yet to kind of be affected by by agents. And so I think it's possible. Again, software engineering, think very difficult to predict. But but I think there's basically two levers here. One is like what happens in software engineering? Is it the same as today? Is it more than? Is it less than? But I think undoubtedly for the other domains, it will be a lot more than it is today. And so I think from that perspective, I think that's very bullish on Anthropic and OpenAI and other model providers. [00:43:34] S1: I think another question that a lot of people are asking is about kind of the proliferation of open models and how that changes the landscape. You work with some very large enterprises. One of my dear friends, Jason Lampkin from SASTA says that I'm not gonna do his accent because I'll butcher it, and I've done it before, he got rather... He said it wasn't very good. Do you find that large enterprises are fearful of working with open source Chinese models, or actually they don't mind? [00:43:56] S0: We see that some enterprises are fearful of working with open source Chinese models. We we we see others that don't care. [00:44:02] S1: Seriously, not fear is justified.