The Frontier with Brendan Foody
00:12:58 · Brendan Foody, Andy Fang
[00:12:58] S0: Reading between the lines, it's that you guys obviously own all the data on the order history, and you want to make sure that stays secure and private to your users, uh, versus a ChatGPT or Gemini or Claude would own all the chat history, and the chat history data might be less relevant to how someone is interacting with the DoorDash agent in a thoughtful way. And so they're not that combined with it not being as much of a priority for them means that it's less of a fit to enable orders through those interfaces. [00:13:29] S1: We've tried, like we were one of the early partners for Chat GPT's like app, I forgot it was called like app thing. One, it was like really hard for people to like connect DoorDash to Chat GPT, and then two, the user experience was not very good. And so for us, it's like we have a lot of other like priorities and experiments we want to run, and like this is like kind of like the outcome we're gonna get. And there's like you're not guaranteeing like user data privacy and stuff, which I think they want to, to be clear, but I just don't think like it's the I think there's like feasibility issues there, maybe for us. It's like, why don't we just try other mechanisms? Like, I think that that was a reason actually why we were excited about kind of uh the limited beta for our like uh command line interface. You know, it's like let the users control like the connection and then let's just see what the use cases are. And I think, you know, it's early innings there for our beta. That was kind of thinking behind it.
00:14:15 · Brendan Foody
[00:14:15] S0: The other thing that you guys have been posting about a bunch, which I'm really excited about, is all of your benchmarks for uh all of the different capabilities that you're prioritizing with indoor dash. Of course, one thing we think about a lot is that there's this giant disconnect between the academic benchmarks that everyone has on their leaderboards and model scorecards versus the actual capabilities that you all are prioritizing and how you allocate your inference dollars. And so, what are the top benchmarks that you guys have launched and which of those are you most excited about?
00:14:50 · Andy Fang, Brendan Foody
[00:14:50] S1: We need to like catch up to you guys because it's all the benchmarks you guys have, but we didn't publish the benchmark, but we uh publicized our benchmark, I guess, called Dash Bench, which is just our code review benchmark. Essentially, it's like we are able to test various, not only models, but also harnesses to see how well they perform on our internal code review agent. And I think what we found and finding that we published was basically that we were able to get Fable plus Kimi K 2.6 to perform uh just as well as our uh previous, I guess, combination of models and harnesses at a cheaper cost. And so, like, if we can actually get fable level intelligence, but actually like pay less. Yeah, so that's kind of like what our like kind of our first, like I would say, legit benchmark. And I think what we're trying to do is we're trying to expand that to other parts of the business. And you and I have talked about this in some of the meetings we've had, but you know, I think there's a lot of things that agents can be doing for companies and enterprises that would be really good to have bet benchmarks to be able to see like how do you measure the ROI basically of the agent on various tests. [00:15:50] S0: How do you think about the tri the mix of open source models versus frontier models and your either through the lens of your tokens or your dollars spent on each evolving over the next couple of years?
00:15:50 · Brendan Foody, Andy Fang
[00:15:50] S0: How do you think about the tri the mix of open source models versus frontier models and your either through the lens of your tokens or your dollars spent on each evolving over the next couple of years? [00:16:03] S1: I actually don't look that much at token spend. I look I look more at dollar spend actually. Um what I've actually heard is that um like from people on the team is that Fable, even though it's more expensive, it's more token efficient. So that actually the spend doesn't end up being as crazy for certain tasks. [00:16:18] S0: For software engineering, especially, I imagine. Yeah. Yeah. [00:16:20] S1: And so to your point, like the token spend is like a little bit of a vanity metric where it's like you really care about the dollar spent to perform the task. That's something that we pay attention to, I'd say. Um, but you know, I think we're trying to figure out how to, you know, outside of code review and coding, like how do we figure out the other opportunities there, basically?
00:16:20 · Andy Fang, Brendan Foody
[00:16:20] S1: And so to your point, like the token spend is like a little bit of a vanity metric where it's like you really care about the dollar spent to perform the task. That's something that we pay attention to, I'd say. Um, but you know, I think we're trying to figure out how to, you know, outside of code review and coding, like how do we figure out the other opportunities there, basically? [00:16:38] S0: It definitely feels like the frontier model demand will keep skyrocketing for a very long time. But I wonder if we'll get to the point where there's a specific amount of tasks that you guys have where there's a sufficiency threshold. Yes. Of like, hey, Kimi or GLM or whatever open source model has hit that sufficiency threshold and then you're able to move over. It's a fascinating thing to imagine what percent of these tasks within a firm have a sufficiency threshold versus what portion don't. Because there's obviously certain kinds of software engineering tasks where there's almost no sufficiency threshold or it's decades out, uh, while there's certain kinds of you know, basic search or tool use, as you were getting at earlier, that have a much lower sufficiency.
00:17:20 · Andy Fang, Brendan Foody
[00:17:20] S1: Like, you know, we have some like uh classification pipelines that for actually even like more than a year ago, we've actually used open weight models on and it's performed really well. I definitely think a lot of the workflows will have that good enough benchmark. But I think the thing that I don't know about is as AI and agents get smarter and as enterprises figure out more and more how to embed them, like, are there gonna be like these new types of workflows that people are gonna want to have agents do that aren't happening yet? Like I think one example I can talk about at DoorDash is you know, our accounting team is like trying to figure out ways to leverage AI to make things better. At the same time, we're seeing like, you know, the the labs are claiming that the tasks that our accounting team are doing are like easily solvable by the models, but then our accounting team is like they're saying the models aren't good enough. So, like for me, it's like a disconnect of like, are we not using the models well or is there like a capability gap? Um, and so I think that's something that's like interesting. [00:18:17] S0: We're launching an accounting benchmark shortly, is hopefully we'll be able to uh in partnership uh with Ramp. Are you able to share how much you guys are spending on inference right now? [00:18:28] S1: I'll say a stat that I think our inference spent has gone up 20x since the beginning of the year. [00:18:34] S0: Wow. So what how much do you expect it to go up over the next 24 months?