00:19:38 · Dmitri Dolgov, Aditya Agarwal
[00:19:38] S0: to say that. You can't learn the real world is for data collection, for grounding. All of the training and all of the closed loop training especially happens in the simulator. But, of course, you cannot build a realistic high fidelity simulator in a vacuum. It's an accelerator, but it still has to be grounded. And this is where the tight loop of building the driver, the simulator, the evaluation machinery, and kind of letting them evolve together and be grounded and validated in real world data with real deployments is what's required. [00:20:17] S2: We've talked publicly about how the fifth generation driver was where you kind of made a big jump, right? Like to kind of like being a I guess you describe it as AI as kind of the more of the backbone. What kind of motivated that change? Like it sounds as though that was in some ways a fairly big architectural kind of like difference as you kind of made that jump. Like why what ended up kind of motivating that?
00:20:40 · Dmitri Dolgov
[00:20:40] S0: Well, it was a natural evolution of the state of the art of the tech. I think around that time, I think it became pretty clear that we wanted to kind of bet on this paradigm of building high capacity outboard models and then distill them into the students, Waymo Driver that runs in the car being the student. You just get better scaling loss that way. And it's not specific to autonomous vehicles. It's just kind of a fairly known property that you start with high capacity models, and then you distill. You get better scaling loss than just from working on the small capacity models directly. And so in order to do that, you want the... Your, you know, your your driver to be an effective student. So that means it needs to have kind of maybe, you know, have a strong ML backbone. Otherwise, you just can't, you know, learn from the teacher so well. Right? And also at that time, you know, we've gone through several big waves of innovation in this project. Right? We've, you know, we've seen ConvNets, Then, you know, we've seen transformers. You know, now there's VLMs, VLAs. Right? So it just seemed like a very natural progression. You mentioned end to end as, you know, one of the kind of design dimensions in this whole space. This was also all of that was coming together.
00:22:00 · Ruchi Sanghvi
[00:22:00] S1: You've been through multiple innovations in the course of this project, but it feels like these innovations were so significant that you would have to rewrite large parts of the system to take into account these innovations? I mean, do you do that? Like, how do you rewrite a system, especially when it already works?
00:22:19 · Dmitri Dolgov
[00:22:19] S0: That's a great question. Look, architecture's come and go. I just mentioned a few really big waves of innovation that we've seen. The foundation is kind of all of the machinery that allows you to grab the fruits of the latest state of the art technology at any moment in time. And that's your evaluation, that's your data, that's infrastructure. If you don't have that, if you don't have those strong fundamentals, then you kind of evolve into a tech stack that kind of works and has certain implicit properties that have organically emerged over time. But it's... It can be very, very difficult, if not impossible, to make big changes to it. Right? And then, you know, there's plenty of companies, I think, in the history of technology development that kind of got stuck in those local minima. So what you want if you have... Well, conversely, if you have those strong fundamentals, that allows you to take really big technical swings. Again, whether it's transformers or convnets or what what what have you. And land kind of big disruptive change in technology without the disruption of momentum of your product and without seeing regressions in the field, which, of course, is so important for us because we're operating in a safety critical environment. So, if anything, I would say that's probably the most difficult thing that we've built at Waymo, and probably the most important muscle that we, as a company, develop over time. It's not, again, the technology of the day. It's this ability to bet on the latest and greatest innovation and then take big disruptive steps in technology development without breaking momentum of product development and deployment.
00:24:14 · Ruchi Sanghvi, Dmitri Dolgov, Aditya Agarwal
[00:24:14] S1: So to repeat, the key would be infrastructure, data, and then evaluation. [00:24:20] S0: That's right. [00:24:20] S2: Given the model that you have trained and then you have distilled to kind of run on these cars, if that was the only IP that Waymo had today, right? Like, Okay, you have these models that have been trained, but you didn't actually have kind of like the historical trove of essentially evals and data and so on, would that be an asset in like, would that be a sufficient asset to start kind of like a new AV company?
00:24:43 · Dmitri Dolgov
[00:24:43] S0: Yeah. I can give you the, you know, the model that we have right now. I don't think you would know what to do with it. Like, you wouldn't, you know, be able to validate it if you wanted to go, you know, apply it to a new deployment. I I don't think you would have the machinery to validate it and deploy with confidence, let let alone iterate on it. Right? So it really comes down to the fundamentals of infrastructure, metrics, eval, training recipes. Right? It's not it's not static. Right? It's all about just how, you know, quickly you can iterate and your level of confidence and understanding the properties of their thing they're building. So it's much more about kind of the machine for building the product as opposed to a snapshot of the product you have at any moment in time, Right? And this is an industry where it's always been very easy to get started. Right? In 2009, it was a dozen of us, in a course of about, what is it, twelve to eighteen months, build a system that could drive those 10 routes that were very difficult and with no human invention done each. But it's been seventeen years to get to where we are today. And it's this kind of the natural property of many of these technical curves. Right? They start out steep and then they, you know, flatten out. And with all of every breakthrough that we've seen over the years in technology, whether it's, you know, hardware, computers, off board, onboard, ML, massive, it really has a drastic impact on, you know, the early part of the curve and getting started has, you know, I think been getting easier. Right? So nowadays, can take your, you know, favorite, model, state of the art, some VLM or VLA, yeah, there you go. Pick your favorite one and go from kind of do open loop end to end imitation learning on it. And you will very quickly, with, you know, tiny budgets and a tiny team, you know, in a matter of, you know, weeks, if not days, get something that works. That You