The Frontier with Brendan Foody
00:09:46 · Brendan Foody, Andy Fang
[00:09:46] S0: Especially for like search over a giant catalog is actually a really surprisingly challenging problem. That exactly I agree that mainly the distribution is strong in the uh closed source models, but not yet the open source ones. One thing I've been thinking about a lot is that right now the these agents are mainly reactive. If we go to and we say we want a high protein meal, and then we're able to get that. And over time they'll start to become more proactive, right? Where it might identify what are the things that you might want to buy in the future, paying you around the time that you usually order dinner, all that kind of stuff. Yes. How are you thinking about that future developing? Definitely. [00:10:24] S1: I mean, we're actually working on some uh concepts like that internally right now. But I think like if I were thinking a little bit further out, I think that's kind of where things go. It's like where you have these agents in the background actually doing stuff for you. And I think for us, what we're really excited about is not only are there these agents that kind of do stuff in the cloud, but DoorDash actually has a huge operational network that can actually do stuff in the real world. And so I think there's some interesting ideation that we've had internally about like, hey, what could be possible if these agents can perform things in the background? And so, you know, early innings there, but we'll see where things go. But I think it's gonna be a really interesting time to be disrupting consumer experiences, I'd say, over the next 12 months.
00:10:24 · Andy Fang, Brendan Foody
[00:10:24] S1: I mean, we're actually working on some uh concepts like that internally right now. But I think like if I were thinking a little bit further out, I think that's kind of where things go. It's like where you have these agents in the background actually doing stuff for you. And I think for us, what we're really excited about is not only are there these agents that kind of do stuff in the cloud, but DoorDash actually has a huge operational network that can actually do stuff in the real world. And so I think there's some interesting ideation that we've had internally about like, hey, what could be possible if these agents can perform things in the background? And so, you know, early innings there, but we'll see where things go. But I think it's gonna be a really interesting time to be disrupting consumer experiences, I'd say, over the next 12 months. [00:11:04] S0: I order my dinner at 6 p.m. on DoorDash every day. And so I feel like if it just said via text with here's the new meal or restaurant that I I should try that I haven't yet gone to, I uh it feels like there's so much impact to the user experience that could be possible there. What about allowing people to interact with it via other surfaces like Chat GPT? How are you thinking about that? I mean, I know this is a probably a controversial topic of how much to sort of use the own the end user interaction.
00:11:34 · Andy Fang
[00:11:34] S1: Um I don't think it's that controversial. I mean, my take on it is I think we're happy to explore different ways to get people deliveries. My personal philosophy on it is the more deliveries we add to the network, the more that increases our the scale of our network. The more we drive there, that's net goodness to the ecosystem. I would say maybe there's a question on like, oh, what happens with ads, but I would say like outside of that, like people, even a customer ordering through DoorDash, you need to figure out like what the right support mechanism is. If you want double dash, like you need to go through our ecosystem to get the best, not only the best like logistics experience, but also the best prices. Also, like all the merchant like promos and all that stuff, that's all like flowing through our ecosystem. To me, it's not really, I don't really have a big concern with it. I think the thing we're most uh concerned about, and we've even had conversations with the labs about it, is like, hey, making sure that like we're being protective of like the user data and like being respectful for it. And I don't think it's for a lack of trying that the labs haven't really played ball with us there, but I think it's more in terms of like the technology and like the evolution of it. I think there's just like a lot going on there. And I think quite frankly, with some of these labs, I don't think this is their top priority. I mean, in terms of like the conversations we've had, I don't think they've like had a huge priority in making this experience world class. And so for us to like partner with them on it, I think we'd wanna see like we we want the user experience to be good.
00:12:58 · Brendan Foody, Andy Fang
[00:12:58] S0: Reading between the lines, it's that you guys obviously own all the data on the order history, and you want to make sure that stays secure and private to your users, uh, versus a ChatGPT or Gemini or Claude would own all the chat history, and the chat history data might be less relevant to how someone is interacting with the DoorDash agent in a thoughtful way. And so they're not that combined with it not being as much of a priority for them means that it's less of a fit to enable orders through those interfaces. [00:13:29] S1: We've tried, like we were one of the early partners for Chat GPT's like app, I forgot it was called like app thing. One, it was like really hard for people to like connect DoorDash to Chat GPT, and then two, the user experience was not very good. And so for us, it's like we have a lot of other like priorities and experiments we want to run, and like this is like kind of like the outcome we're gonna get. And there's like you're not guaranteeing like user data privacy and stuff, which I think they want to, to be clear, but I just don't think like it's the I think there's like feasibility issues there, maybe for us. It's like, why don't we just try other mechanisms? Like, I think that that was a reason actually why we were excited about kind of uh the limited beta for our like uh command line interface. You know, it's like let the users control like the connection and then let's just see what the use cases are. And I think, you know, it's early innings there for our beta. That was kind of thinking behind it.
00:14:15 · Brendan Foody
[00:14:15] S0: The other thing that you guys have been posting about a bunch, which I'm really excited about, is all of your benchmarks for uh all of the different capabilities that you're prioritizing with indoor dash. Of course, one thing we think about a lot is that there's this giant disconnect between the academic benchmarks that everyone has on their leaderboards and model scorecards versus the actual capabilities that you all are prioritizing and how you allocate your inference dollars. And so, what are the top benchmarks that you guys have launched and which of those are you most excited about?
00:14:50 · Andy Fang, Brendan Foody
[00:14:50] S1: We need to like catch up to you guys because it's all the benchmarks you guys have, but we didn't publish the benchmark, but we uh publicized our benchmark, I guess, called Dash Bench, which is just our code review benchmark. Essentially, it's like we are able to test various, not only models, but also harnesses to see how well they perform on our internal code review agent. And I think what we found and finding that we published was basically that we were able to get Fable plus Kimi K 2.6 to perform uh just as well as our uh previous, I guess, combination of models and harnesses at a cheaper cost. And so, like, if we can actually get fable level intelligence, but actually like pay less. Yeah, so that's kind of like what our like kind of our first, like I would say, legit benchmark. And I think what we're trying to do is we're trying to expand that to other parts of the business. And you and I have talked about this in some of the meetings we've had, but you know, I think there's a lot of things that agents can be doing for companies and enterprises that would be really good to have bet benchmarks to be able to see like how do you measure the ROI basically of the agent on various tests. [00:15:50] S0: How do you think about the tri the mix of open source models versus frontier models and your either through the lens of your tokens or your dollars spent on each evolving over the next couple of years?