The Frontier with Brendan Foody
00:18:57 · Andy Fang, Brendan Foody
[00:18:57] S1: Yeah, it's just like hard to predict, like because maybe like and I was just talking to like Tony, our CEO, about this earlier today. It's like if we're seeing ROI from the spend, wouldn't we want to spend more? Yeah. Right. And so like I think that's the thing that I mean, I think everybody's trying to figure out is like where are those opportunities? Because let's say if you're generating revenue from token spend, you would you would spend that token spend all day. For sure. But like I think a lot of what we're focused on as an industry right now, quite frankly, is on like cost efficiency and all that stuff. But like software engineering is a great example of where like when an engineer could do the work of like, you know, five engineers pre-AI, like you would probably want to spend a lot more there. [00:19:34] S0: I agree. And I find that a lot of the time employees are not motivated to say this is a low ROI investment. And so really the way that you have to understand what are these sufficiency threshold tasks versus what are these super high leverage tasks you should invest in frontier intelligence is that it comes down to the benchmark of like you need some ground truth of how well are the different models doing and what is the true capability gap insofar as how it frames the marginal cost.
00:19:34 · Brendan Foody, Andy Fang, S2
[00:19:34] S0: I agree. And I find that a lot of the time employees are not motivated to say this is a low ROI investment. And so really the way that you have to understand what are these sufficiency threshold tasks versus what are these super high leverage tasks you should invest in frontier intelligence is that it comes down to the benchmark of like you need some ground truth of how well are the different models doing and what is the true capability gap insofar as how it frames the marginal cost. [00:20:03] S1: Yeah. And I think like some of the stuff you guys are doing with like some of the public benchmarks you're doing, I feel like that's the direction things are gonna go where someone on Twitter was saying, like, you know, some of these like I'm not I'm not gonna like throw any particular benchmark under the bus, but like some of these public benchmarks are like becoming like more noise and signal. [00:20:20] S2: Yeah. [00:20:20] S1: Like for us, like dash bench, the reason why we built it is because we literally wanted to figure out the ROI like for our enterprise. And like I think some of these like public benchmarks, I think don't like cool, like you can publish your model card and like Fable's good at this or whatever, but I'm like for my enterprise, like we gotta try it to find out, you know. And so I think we just gotta every single dimension we're applying AI, we gotta do that.
00:20:20 · Andy Fang, Brendan Foody
[00:20:20] S1: Like for us, like dash bench, the reason why we built it is because we literally wanted to figure out the ROI like for our enterprise. And like I think some of these like public benchmarks, I think don't like cool, like you can publish your model card and like Fable's good at this or whatever, but I'm like for my enterprise, like we gotta try it to find out, you know. And so I think we just gotta every single dimension we're applying AI, we gotta do that. [00:20:42] S0: Well, and there's two other issues there, which is like first of all, all of these public evals are super easy to overfit to because you can look at all the tasks and just do variations on those specific tasks. And so it makes it really challenging to get genuine signal from that. And the second thing is that now the labs are realizing that their benchmark roadmap is functionally their product roadmap. And so they're not going to open source their like whole product roadmap on their model scorecard. And so it makes it difficult to really understand what are the gaps between frontier intelligence and open intelligence unless you have some way of measuring that against the ground truth that you guys care about. Right. Totally. Are there any other benchmarks you guys have launched, or is the engineering benchmark the main one so far? [00:21:28] S1: The engineering one's the main benchmark so far. Hopefully by the time this launches, we'll have more, but we're working on some other ones.
00:21:28 · Andy Fang, Brendan Foody
[00:21:28] S1: The engineering one's the main benchmark so far. Hopefully by the time this launches, we'll have more, but we're working on some other ones. [00:21:34] S0: Nice. Excited. The other thing I was super curious about is all the work that you guys are doing in physical AI, which is of course becoming a giant new trend in the economy as we take these models that are extraordinary in the digital world and apply that to the things that people do in the physical world. And I think it connects in a lot of ways to the mission of DoorDash, to the work that you guys are doing on a day to day basis. And so how do you see the vision for that unfolding over the coming few years?
00:22:04 · Andy Fang
[00:22:04] S1: I'll start with some of the stuff we're already doing. So I mean, my co-founder Stanley, he's been leading up our autonomy lab for eight years now, since 2018. Um, and so I think that's like a big investment we've made in exploring like autonomy. That's going to become a bigger part of our future in terms of how the future of our logistics network. And I think we learned a ton about that uh from that in terms of like how the autonomy space is evolving, but also how the technology problem in autonomy is widely solved, but the operational problem is a totally different beast. And like I think this is also applying to robotics as well, which we can probably get into right after this. But um, a demo is super easy, you know, like getting a demo to work, like driving from A to B, that's like easy now. And the hard thing is like, how do you solve for all the edge cases? How do you do all the fleet management? You know, like what happens if the car battery dies, or like what happens if it's raining, or all that stuff. So I think that's something that we've learned a lot about. And we're like an early, we're actually live already in a couple cities with our DoorDash dots. Robotics, like more generally, I think is a very interesting space. That's something where we're trying to figure out like what the future is gonna be. And it's funny because like, I mean, you know this too. It's like there's not even agreed upon paradigm in terms of what the working model is gonna be. And so I think it's like a very messy space right now, but like one we're trying to navigate and see you know what the opportunities there are.
00:23:23 · Brendan Foody
[00:23:23] S0: Yeah, and I find that physical world AI is so challenging because when we talk about this point of like sufficiency thresholds, the room for error is so much lower. As a software engineer, like once we hit 50% on sweet badge, that's like spectacular, right? And like we'll ship it, and if it doesn't pass our tests or like fails in production, we'll just like redo the code versus it's radically different in the fiscal world where making a mistake, you know, even five or 10% of the time can create huge problems. And so I definitely think that will limit adoption uh for for most types of tasks as these things sort of develop. And then I know you guys are starting to work on uh RLHF data in the physical world, or I saw something about that. Yeah. Of course, we don't do much in terms of physical world or the lower skilled work, but I'm curious how you see the vision for that unfolding. Cause I think I certainly agree that most work over time will converge on training the models. And I admire the incredible talent network that DoorDash has built to set you guys up for doing that.