00:13:20 · Dmitri Dolgov
[00:13:20] S0: So when we started in 2009, it was not about building a product. Our first order of business was just to understand the problem better. So in service of that, when we started, and honestly, the question at that time was, should we even try? Is the time appropriate, or should we just go back and come back in a decade or so? So in service of understanding the problem, when we started this project in 2009, we created a couple of milestones for ourselves with that goal of exploring the space. And one challenge that we created for ourselves was to drive 100,000 miles autonomously, which at the time was, you know, orders of magnitude more than anyone has, you know, ever done. And the second one, which actually turned out to be more difficult, was to drive 10 routes all over the Bay Area, and each one was a 100 miles long. And we had to do each one from beginning to end with no human intervention. There was a human behind the wheel, but the challenge was to complete one. And in doing that, you end up building every part of the system. Right? So just the crazy, amazing, magical days of an early startup, right? Calibrating and putting new hardware in the car, you're building perception, planning, controlling all of the tools and GUIs. Everybody is doing everything, but you're learning at a crazy rate. So that was, you know, first couple of years we succeeded, and then we said, okay. Now there's, you know, there's, you know, a there there, so let's think about what the product could be. And we kinda went back to building more of the fundamentals and laying the foundation for that what became eventually Waymo and the product you see today.
00:14:58 · Ruchi Sanghvi, Dmitri Dolgov
[00:14:58] S1: So if you were to explain Waymo, like how one would explain the networking stack or something that we all understand, like how would you explain the Waymo end to end system? [00:15:09] S0: So there's a couple of things. End to end has, you know, specific meaning in the field, I can, you know, speak to that. The way I would describe, you know, the, of course, you know, on the car, it's a system that goes, from sensors. We use three main sensing modalities, cameras, sliders, and radars. That information gets processed and kind of encoded into a more compressed representation. And then there's a generative part, which, you know, on the car is the, you know, the the part that makes decisions and generates driving behaviors. You know, that's the the decoder or the planning part. All of that, you know, the the the that's running in the car that we call the Waymo Driver, that's just one part of the system. Really, when we think about the technical challenge that we're solving, there's three main pillars. We call them the driver, the simulator, and the critic. And it starts with us with building kind of large, high capacity off board models. They have kind of three teachers for each one of those tasks. And then they get distilled into their students. And the driver acts, the simulator generates features, often in synthetic environment, and the critic judges and evaluates. Right? So then all of them kinda have to work together in order for this ecosystem to move forward.
00:15:09 · Dmitri Dolgov, Ruchi Sanghvi
[00:15:09] S0: So there's a couple of things. End to end has, you know, specific meaning in the field, I can, you know, speak to that. The way I would describe, you know, the, of course, you know, on the car, it's a system that goes, from sensors. We use three main sensing modalities, cameras, sliders, and radars. That information gets processed and kind of encoded into a more compressed representation. And then there's a generative part, which, you know, on the car is the, you know, the the part that makes decisions and generates driving behaviors. You know, that's the the decoder or the planning part. All of that, you know, the the the that's running in the car that we call the Waymo Driver, that's just one part of the system. Really, when we think about the technical challenge that we're solving, there's three main pillars. We call them the driver, the simulator, and the critic. And it starts with us with building kind of large, high capacity off board models. They have kind of three teachers for each one of those tasks. And then they get distilled into their students. And the driver acts, the simulator generates features, often in synthetic environment, and the critic judges and evaluates. Right? So then all of them kinda have to work together in order for this ecosystem to move forward. [00:16:37] S1: Do they all run on the car itself? Like, does the simulator need to run on the car, or can it run on the cloud?
00:16:37 · Ruchi Sanghvi, Dmitri Dolgov, Aditya Agarwal
[00:16:37] S1: Do they all run on the car itself? Like, does the simulator need to run on the car, or can it run on the cloud? [00:16:43] S0: Oh, no. Absolutely not. So the only thing that runs on the car is the driver, And the other two parts are in service of training, evaluating, and validating the driver. [00:16:52] S2: Do you think that the architecture that you have described is an emergent one based on how you have seen almost the path dependence of the learnings over the last seventeen years of building this? Or would you almost derive this kind of three models, kind of like the simulator, the driver, the critic, does that feel like the right shape of architecture for real world, low latency, high feedback loops for a variety of different kind of devices and so on?
00:17:22 · Dmitri Dolgov
[00:17:22] S0: Yeah, think so. With some caveats. On and I think the I guess I may jump to that. I think the key distinction and the key aspect of the problem, whether you need all three, will be the level of complexity of the task. And the key distinction will be whether you can get by with kind of open loop imitation and open loop evaluation and validation, which for some applications is sufficient, not in our case. And if it's not sufficient, then you do need the entire ecosystem. So the way in our space, I think you'll see the same property merge in other fully autonomous applications of AI in the physical world where you cannot sufficiently control the environment or make the problem simpler. If open loop imitation and open loop evaluation and validation isn't sufficient, that means that closed loop is required for validation, as in the case of our product, and it is incredibly valuable for training. Right? There's some things you can do in closed loop training that you cannot do in open loop. Then that leads you to needing a high fidelity realistic simulator. Right? Number one. And number two, if you are in... Playing in this closed loop training ecosystem, then the separation between actor and a critic, as often happens in kind of reinforcement learning frameworks, is a very natural one. So then if you are in this closed loop as required, then I think you very naturally end up with a trifecta.
00:19:05 · Aditya Agarwal, Dmitri Dolgov
[00:19:05] S2: You've often said that the hard part of driving isn't just the perception. It's the fact that there are multi agents kind of like each of them kind of playing their own part. You're having to essentially model out not just your own behavior but how the world is going to react to some of your own kind of actions. And then you pair that with kind of the closed loop RL which is what gets you to superhuman safety. But how do you think about the closed loop RL in a world where your bad trajectories lead to very bad outcomes? [00:19:34] S0: First of all, you don't do training in the real world. [00:19:37] S2: You have [00:19:38] S0: to say that. You can't learn the real world is for data collection, for grounding. All of the training and all of the closed loop training especially happens in the simulator. But, of course, you cannot build a realistic high fidelity simulator in a vacuum. It's an accelerator, but it still has to be grounded. And this is where the tight loop of building the driver, the simulator, the evaluation machinery, and kind of letting them evolve together and be grounded and validated in real world data with real deployments is what's required.