00:32:27 · Dmitri Dolgov, Ruchi Sanghvi
[00:32:27] S0: Yeah. So I think this gets back to this fundamental question of metrics and evaluation. [00:32:33] S1: I have to ask right after this, how strong is your eval team? We've heard upon eval so much.
00:32:38 · Dmitri Dolgov
[00:32:38] S0: Yeah, we have a very strong eval team, the team that builds the infrastructure, the simulator, all of the metrics, the data science team that supports that. But it's also, like, the eval first mindset, that is spread across the entire company, across definitely all of the engineering functions. And you do... It's not one or two simple metrics that would describe that, what does it mean for an autonomous system to be ready and good enough? And it's not a couple of things that you measure at the system level at the very end of your development. It starts kind of... You know, do, you know, eval and hard data driven evaluation every part of your system. Right? Now it starts with that model level. And, you whatever model you're training, and you don't train it monolithically, and you you kinda train your encoders, you train your decoder, and they have, you know, many mesh. We can talk a little bit about end to end and, you know, structured representation because that is an important aspect of supporting, you know, rigorous training and evaluation. And then, you know, that goes to kind of dense proxy metrics of the system level performance that are maybe not the final things you care about, but they're very dense, you know, predictive signals about the performance that you will see in the field. And then you do rigorous system level evaluation, you know, everything from safety, you know, to quality. And we run billions of miles of simulation. And every release before we put it out on the road goes through this multi week rigorous evaluation and validation cycle that includes testing both in the physical world and in simulation. And then we have a team of well, highly experienced subject matter experts to refuel of the data, and all of that gets fully vetted before we put the software on railroads.
00:34:26 · Aditya Agarwal, Dmitri Dolgov, Ruchi Sanghvi
[00:34:26] S2: To your point earlier, it feels as though that having that pipeline and having that infrastructure to be able to both iterate and then deploy with confidence feels like a real source of competitive advantage. And I suspect that anybody who's deploying any kind of physical, whether robots, drones and so on, at scale will have to have that. But I don't know if how many like there are that many companies that have that today, right? [00:34:52] S0: Yeah. No, I think that's one one of the hardest things to build. And really, again, there's some applications where that might not be required. But there is a big divide. And if you are on the other side of it, where it's safety critical, it does require closed loop training, closed loop evaluation, then you're in a very different world where, you know, those fundamentals matter tremendously. [00:35:16] S1: If you had three exceptional engineers who are starting today, they had Waymo sized ambitions but a startup budget, where would you tell them to start? [00:35:26] S0: What space? Physical AI, anything? [00:35:29] S1: Physical AI. Let's keep to physical AI.
00:35:33 · Dmitri Dolgov
[00:35:33] S0: Oh, man. There's so much I'm super excited about all the things that we could be building in physical AI. If, I think, yeah, if you have Waymo's ambition, but a qualitatively different budget, if I think that puts you on that side of the spectrum where you need to, today, look at applications where you can simplify the problem. Maybe you can control the environment, or maybe you can build a system that helps humans as opposed to being fully autonomous. Again, this, I think, my mind, this is the big divide in our domain. Right? Are you building, you know, a driver assist system or are building a fully autonomous system with superhuman safety that you're gonna deploy and scale? Qualitatively different problems. So I think there's a similar analogous divide in other applications of AI in the physical world. So if you are looking for a relatively quick product you can deploy, or application, a problem you can solve, an application you can deliver, then I look for something where it can be simplified. Again, maybe something in, let's say, warehouse, in you can control the environment, you can maybe put some guardrails in the system. There, I think you can make a lot of progress by taking the ingredients that already exist kind of off the shelf, again, doing some, you know, open loop imitation things and deploy it, and then iterate from a deployed product. Other... If you're on the other side of it, then I think you have to be realistic that you need to invest in the fundamentals, which is less about the product and iterating and, you know, deploying and iterating, but more about focus on the fundamentals, like world models, realistic simulators for whatever agent you're building, and the whole infrastructure and evaluation machinery.
00:37:26 · Ruchi Sanghvi
[00:37:26] S1: So in your case, like, you know, let's say someone was exploring autonomous systems. You mentioned you did a lot of research before you even thought of building a product. The demo only took about eighteen months, but the actual product took about fifteen years. So for everyone in this room, where the demo actually works, what are the signals you need in maybe a year or two that tell you you're on a fifteen year old fifteen year problem solving trajectory or you're working on a system that you could produce a workable product in the next three years? And I get the fully autonomous versus not is a good, like, rubric, but are there other things?
00:38:06 · Dmitri Dolgov
[00:38:06] S0: Let's say you're working on a problem. You said you have three or fifteen years. Let's say you've been working on it maybe for a year or two, and you kind of want to see if you're a year away or ten plus years away. One thing, kind of one pattern that I've seen over and over again is do you have convergence of discovery? Right? Is the problems that you are seeing today, how different are they from the problems you saw six months ago? Do you feel like as you're pushing on the system, does it feel like you have stability and you're just kind of iteratively improving your metrics in the same areas, or does it feel like every six months, the landscape is shifting from under you and the problems of today are very different from what you saw a while ago? So if you don't have that stability, then that's often kind of a good indicator that you've got some ways to go. The second thing I would mention is judging how close you are to a real deployment, a real product is, again, I go back to metrics and evaluation, and do you have a scorecard? If you think you are, you know, six month or a year away from deployment, do you have a scorecard that if it were green today would allow you to deploy with confidence? If you don't have one, you're kind of saving that for later, often that turns out to be more difficult than you expected and there's a long distance to go. But if you already have one and you're seeing stability in your discovery and stability in your metrics and it's just a matter of closing the gap, that is a really good indicator that you are getting pretty close.