The Frontier with Brendan Foody
00:15:50 · Brendan Foody, Andy Fang
[00:15:50] S0: How do you think about the tri the mix of open source models versus frontier models and your either through the lens of your tokens or your dollars spent on each evolving over the next couple of years? [00:16:03] S1: I actually don't look that much at token spend. I look I look more at dollar spend actually. Um what I've actually heard is that um like from people on the team is that Fable, even though it's more expensive, it's more token efficient. So that actually the spend doesn't end up being as crazy for certain tasks. [00:16:18] S0: For software engineering, especially, I imagine. Yeah. Yeah. [00:16:20] S1: And so to your point, like the token spend is like a little bit of a vanity metric where it's like you really care about the dollar spent to perform the task. That's something that we pay attention to, I'd say. Um, but you know, I think we're trying to figure out how to, you know, outside of code review and coding, like how do we figure out the other opportunities there, basically?
00:16:20 · Andy Fang, Brendan Foody
[00:16:20] S1: And so to your point, like the token spend is like a little bit of a vanity metric where it's like you really care about the dollar spent to perform the task. That's something that we pay attention to, I'd say. Um, but you know, I think we're trying to figure out how to, you know, outside of code review and coding, like how do we figure out the other opportunities there, basically? [00:16:38] S0: It definitely feels like the frontier model demand will keep skyrocketing for a very long time. But I wonder if we'll get to the point where there's a specific amount of tasks that you guys have where there's a sufficiency threshold. Yes. Of like, hey, Kimi or GLM or whatever open source model has hit that sufficiency threshold and then you're able to move over. It's a fascinating thing to imagine what percent of these tasks within a firm have a sufficiency threshold versus what portion don't. Because there's obviously certain kinds of software engineering tasks where there's almost no sufficiency threshold or it's decades out, uh, while there's certain kinds of you know, basic search or tool use, as you were getting at earlier, that have a much lower sufficiency.
00:17:20 · Andy Fang, Brendan Foody
[00:17:20] S1: Like, you know, we have some like uh classification pipelines that for actually even like more than a year ago, we've actually used open weight models on and it's performed really well. I definitely think a lot of the workflows will have that good enough benchmark. But I think the thing that I don't know about is as AI and agents get smarter and as enterprises figure out more and more how to embed them, like, are there gonna be like these new types of workflows that people are gonna want to have agents do that aren't happening yet? Like I think one example I can talk about at DoorDash is you know, our accounting team is like trying to figure out ways to leverage AI to make things better. At the same time, we're seeing like, you know, the the labs are claiming that the tasks that our accounting team are doing are like easily solvable by the models, but then our accounting team is like they're saying the models aren't good enough. So, like for me, it's like a disconnect of like, are we not using the models well or is there like a capability gap? Um, and so I think that's something that's like interesting. [00:18:17] S0: We're launching an accounting benchmark shortly, is hopefully we'll be able to uh in partnership uh with Ramp. Are you able to share how much you guys are spending on inference right now? [00:18:28] S1: I'll say a stat that I think our inference spent has gone up 20x since the beginning of the year. [00:18:34] S0: Wow. So what how much do you expect it to go up over the next 24 months?
00:18:34 · Brendan Foody, Andy Fang
[00:18:34] S0: Wow. So what how much do you expect it to go up over the next 24 months? [00:18:41] S1: Well, I would say even the past like month and a half, it's like plateaued. Like it hasn't gone down, it hasn't shrunk, but it's definitely like flatlined. I think there's definitely like efforts we've had internally to curb the spend. [00:18:52] S0: It makes sense that would coincide with shipping your benchmarks and the open source model is starting to work. [00:18:57] S1: Yeah, it's just like hard to predict, like because maybe like and I was just talking to like Tony, our CEO, about this earlier today. It's like if we're seeing ROI from the spend, wouldn't we want to spend more? Yeah. Right. And so like I think that's the thing that I mean, I think everybody's trying to figure out is like where are those opportunities? Because let's say if you're generating revenue from token spend, you would you would spend that token spend all day. For sure. But like I think a lot of what we're focused on as an industry right now, quite frankly, is on like cost efficiency and all that stuff. But like software engineering is a great example of where like when an engineer could do the work of like, you know, five engineers pre-AI, like you would probably want to spend a lot more there.
00:18:57 · Andy Fang, Brendan Foody
[00:18:57] S1: Yeah, it's just like hard to predict, like because maybe like and I was just talking to like Tony, our CEO, about this earlier today. It's like if we're seeing ROI from the spend, wouldn't we want to spend more? Yeah. Right. And so like I think that's the thing that I mean, I think everybody's trying to figure out is like where are those opportunities? Because let's say if you're generating revenue from token spend, you would you would spend that token spend all day. For sure. But like I think a lot of what we're focused on as an industry right now, quite frankly, is on like cost efficiency and all that stuff. But like software engineering is a great example of where like when an engineer could do the work of like, you know, five engineers pre-AI, like you would probably want to spend a lot more there. [00:19:34] S0: I agree. And I find that a lot of the time employees are not motivated to say this is a low ROI investment. And so really the way that you have to understand what are these sufficiency threshold tasks versus what are these super high leverage tasks you should invest in frontier intelligence is that it comes down to the benchmark of like you need some ground truth of how well are the different models doing and what is the true capability gap insofar as how it frames the marginal cost.
00:19:34 · Brendan Foody, Andy Fang, S2
[00:19:34] S0: I agree. And I find that a lot of the time employees are not motivated to say this is a low ROI investment. And so really the way that you have to understand what are these sufficiency threshold tasks versus what are these super high leverage tasks you should invest in frontier intelligence is that it comes down to the benchmark of like you need some ground truth of how well are the different models doing and what is the true capability gap insofar as how it frames the marginal cost. [00:20:03] S1: Yeah. And I think like some of the stuff you guys are doing with like some of the public benchmarks you're doing, I feel like that's the direction things are gonna go where someone on Twitter was saying, like, you know, some of these like I'm not I'm not gonna like throw any particular benchmark under the bus, but like some of these public benchmarks are like becoming like more noise and signal. [00:20:20] S2: Yeah. [00:20:20] S1: Like for us, like dash bench, the reason why we built it is because we literally wanted to figure out the ROI like for our enterprise. And like I think some of these like public benchmarks, I think don't like cool, like you can publish your model card and like Fable's good at this or whatever, but I'm like for my enterprise, like we gotta try it to find out, you know. And so I think we just gotta every single dimension we're applying AI, we gotta do that.