The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:36:00 · Harrison Chase
[00:36:00] S0: Cost is one of the big things that we talk with people about right now. I think there's a few different dimensions to this. One, I think like the frontier models are relatively getting cheaper over time. Astra is, like price per performance, fantastic compared to a lot of things that existed a year ago. Two, there's just way more demand. When you start talking about agents talking to agents or multi agent systems, there are just so many more tokens that are getting burned, and so the costs are going up. We see this in coding primarily. Coding costs have skyrocketed, and I think a lot of organizations, including ours, are now thinking about ways to kind of like pair pair those costs back. How does this end up getting resolved? I mean, if you look at how we're tackling coding agents, I think there's a few hints there. So so when we're tackling coding agents, think of it in three steps. One, we think of basically just, visibility showing kind of, like, how much things will cost, how things have cost to admins. Two is cost control. So you could say something like, hey, you know, go get me a reservation, but don't spend more than $5, like, doing it. And then third is actually impacting the harness itself. So I think primarily a lot of harness engineering has been done around making the agent more efficient because that's the... Or sorry. A lot of the harness engineering has been done to make the agent just like better, more accurate because that's the biggest blocker. I think as cost becomes more concerned, we're seeing people care more about things like prompt caching, which are really good for kind of like reducing costs. We're seeing model providers actually make this easier by... And this getting a little bit in the weeds, but they introduce things like where where you can change the tools that are available to an agent mid flight. Previously, this would have busted the cache and caused like a massive increase in in cost. Now you can just add those dynamically and keep the cost the same. So I think we're seeing more attention to both the model APIs and and harnesses in terms of, like, controlling costs.
00:37:38 · Harry Stebbings, Harrison Chase
[00:37:38] S1: When you talk to large enterprises today, what matters most to them, is it accuracy? Is it cost? Or is it security? [00:37:46] S0: Another thing would be latency, by the way. So so we so we run developer surveys every year where where we ask developers what they care about. For the last two years, the number one response, basically double the next two, has been accuracy. So that's still the the primary thing. The next two are latency latency and and cost, cost, and security is, like, a a little bit after that. Pretty close. When we talk to large enterprises recently, I'd say, like, cost has has risen in terms of concern. I'd still probably say it's below accuracy, but it's probably higher than latency. And then security is... For large enterprises, security is higher, but still probably below accuracy and cost. [00:38:21] S1: What percent of your engineering salary, give or take, not specifically, but do you spend on coding tools?
00:38:29 · Harrison Chase
[00:38:29] S0: I think we spend probably roughly 30% of our engineering salary on coding tools. It's actually gone down the last two months, while tokens, the number of tokens that we've used, has actually gone up. And so this has... We've done this by three steps. First is basically just visibility. We tracked all of our coding agent usage to Langsmith so we could tell who was spending how much, what they were spending it on. We could do analytics over that. Second is just basic cost controls. We set kind of like a budget. And and if you if you hit the budget, you can talk to our VP of engineering and and he'll almost always, like, increase it, but it's just a safeguard against kind of like agents being used incorrectly or going in a in a in a bad loop. And then third would be we control the harness. So one of the things we've done is we actually built our own in house cloud coding agent. So not not a IDE, not a not a terminal based agent. We we let people use whatever they want for that, but we built our own cloud based agent, Oop... OpenSUI is what we call it. It's actually the only way I code anymore because I'm in Slack all the time, and so I just message it from Slack and it kicks off. But in that harness, we've done a bunch of work to control costs. So things like prompt caching, but also things like routing, we've built into this harness. And I actually think that's like a pretty repeatable three step process for kind of, like, getting coding agent spend under control. Just first, visibility. Second, like, basic controls. Third, like, try to improve the harness itself.
00:39:42 · Harry Stebbings, Harrison Chase
[00:39:42] S1: Totally get that. I do just wanna unpack that. You mentioned that visibility, see who's spending what. Were the people spending the most the most effective? [00:39:51] S0: Sometimes the people spending most are the most effective, and other times they're not. So what I mean by that is we have a few senior engineers who spend a lot, and I a 100% trust that what they're doing is well worth it. No questions there. We've also just had some cases do do more to, like, kind of, like, silly mistakes or kind of, like, improper setup than anything. We've had, like, massive spikes in usage. So so an example of this, we we had one or two tasks where some plug in was being used that basically, like, would would screenshot or take a snapshot of of the DOM of a web page and send that back every turn of the agent because they were doing some front end engineering, and they wanted to take what it looked like and just send it back. It would just run this massive loop. And so I think we spent 40 k in a week on on that one thing. You know, when it gets to that, like, again, like, I think we're pretty kind of, like, liberal as a company and trust people, but I I know that's probably not worth it. So And that's just like an example of like... Again, it's... It wasn't malicious. It wasn't even intended usage really. It was kind of just a mistake. So I just think just like catching that is well worth it. So it depends a bit on the side of the spectrum. [00:40:52] S1: 30% of salary, as you mentioned. [00:40:55] S0: That's guesstimate. Yeah.
00:40:55 · Harrison Chase, Harry Stebbings
[00:40:55] S0: That's guesstimate. Yeah. [00:40:56] S1: A guesstimate. Totally fine. What is that in two years?
00:41:00 · Harrison Chase
[00:41:00] S0: I think the hard part about estimating coding agent spend is that orgs are just changing so rapidly. So what I mean by that is I think there's a future where orgs are much smaller and coding agent spend is, you know, the same as human salary or or or more or an order of magnitude more. And you have a small number of people that are driving and executing incredibly rapidly and incredibly performantly. And I and I think that's an enticing vision in terms of of of what you think about. I don't think many companies, including us, are are probably there. I think we still see that we need a bunch of different people with a bunch of different backgrounds and a bunch of different expertises collaborating on things, and there's a bunch of communication and overhead and review. And, like, yes, it would be ideal to have one person driving a 100 agents and just, like, driving everything that, like, makes up, you know, a large part of the product. But I think we just see that it's still a very human component to it. And so I spend a lot of time thinking about kind of, like, org design. Is it possible to have, like, small teams, one person teams, two person teams that are kind of, like, driving agents? I think it's very tempting. And in that situation, I would say, like, you know, agent spend, maybe three x the cost of... But then also, you know, the engineers who can do that are incredibly valuable. And so maybe it actually will just stay the same because the people who can actually do that and who can drive all those agents, there's fewer people, but you're paying them a lot more. And so the spend actually stays the same. So I think org design is very, very interesting right now. And so because of that, I think it's actually really hard to answer that question.