Y Combinator Startup Podcast
00:16:11 · Dmitri Dolgov
However, if you are targeting full autonomy and you're targeting superhuman, strongly superhuman performance, you find that weak sensing just leads to a safety curve that flattens out way too early. So at Waymo, we've taken an approach where we use multiple sensing modalities. We use cameras, LiDARs, and radars, and they all complement each other. Cameras give you high resolution and color, but they're passive and they degrade in darkness and glare. LiDAR gives you a direct measurement of the three d structure of the world around you, and radar is very good at punching through environmental conditions and weather like fog or rain or snow, and it directly can measure velocity using Doppler. LiDAR and radar are active sensors, so that means they see just as well in pitch darkness or, for example, when driving into a blinding sunset. And these different sensing modalities, of course, they're not backups to each other. In our stack, each modality has an encoder and the information from all of those sensors get fused into a single view of the world around us that is much more precise and generally vastly superior to what you get with any one sensor. So let me show you a few examples. Here's the scene where Waymo is driving in a dust storm in Phoenix. So what you see here is what the scene looks like to our fairly advanced high resolution and high dynamic range camera. It's very close to what a can, you know, a human would see in the same conditions, which is not much. And here on the right is what the LIDAR sees for the exact same frame, and you can much more clearly see that there's a pedestrian standing on the side of the road. So if they were to step onto the road, that early detection can make a really big difference in how the situation plays out and the safety of everyone involved. Here's another example. At night, driving along, and there are a couple of pedestrians who are about to jump onto the road over a concrete construction barrier. Again, at the bottom you see the camera, really can't see much, and the lighter view at the top. Again, lighter versus camera. Here's another example. There are a couple of dogs chasing a bull, and a couple of kids chasing the dogs. And big difference, here's what it looks like to the camera.
00:18:48 · Dmitri Dolgov
Here's the LiDAR, and the early detection of the kids is off to the side, and there are no headlights, there are no lamps there, it's complete darkness, so it makes a big difference. Or think about what happens when something physically obstructs the view of your sensors. Yeah, if you don't have redundancy in sensing, you can have, you know, a single leaf land on your sensors and bring your robot to a full stop. Now, so you need redundancy. Redundancy of course does not necessarily mean multiple sensing modalities, but if you need redundancy anyway, you might as well benefit from the complementary physics of the different sensing modalities in the nominal case. So here's a video of one of our cars that picked up a leaf, or actually I think a full branch of a tree that our wipers were unable to shake, and the car detected that and because we have sensing redundancy, it safely was able to get back to the depot for proper cleaning. So specifically, when it comes to hardware, do not anchor to today's components prices. We are on the sixth generation of the Waymo Driver, the Waymo Hardware Suite today, and with every generation, the hardware not only delivered amazing capability, but we're able to drastically simplify and radically reduce the cost of the hardware as well. So betting your company, betting your approach on today's hardware prices is just betting your company on a number that has a fairly short shelf life and is going to expire. So hardware would change. Many components will get commoditized and drop in price, so design for that future and be ready to upgrade. And that brings us to the next lesson, lesson number three, technology moves incredibly fast, especially nowadays. So you need to be ready to ride those tech waves and do that repeatedly. And when you do, as to not only think about the wins in performance and the wins in capability, you have to be very mindful about unification and simplification. Over the years, we've seen a number of major breakthroughs in technology, a lot of them around AI, and with every wave of innovation, we pretty much rebuild the Waymo Driver around that major wave of AI breakthroughs, and we often push the state of the art in those areas forward ourselves. We leveraged ConvNets around 2013 for computer vision and perception.
00:21:27 · Dmitri Dolgov
Then when Transformers came about around 2017, we bet big on them both for perception and for the task of behavior prediction and decision making and planning. Turns out the task of driving is not that dissimilar from the task of modeling language because of the social aspects of driving. You're kinda having a conversation with other dynamic actors in the world, but you're doing that in the space, kinda body language of your agent, your car, as opposed to just the language of words. And you operate in sequences and local continuity matters, but so does global context. And today we're leveraging the latest in VLMs and frontier role models. Now using the latest tech for capability and performance wins, I don't wanna say it's easy, but it can be, you know, reasonably straightforward, doing applied research in isolation or starting a tiger team to, you know, prototype some new technology is not the most difficult part. There's many companies, many teams that are excellent in this. The much harder muscle to build is to carry that bleeding edge research into production and deploy it in a safety critical environment without regressions, and do it without breaking stride on the scaling of your product. And adding capability, again, is not the hardest part, but adding capability while at the same time reducing fragmentation and reducing complexity, that is really important. And finally, the hard muscle to build as a company is to be able to do that repeatedly through multiple waves of technical innovation or technical breakthroughs. So on this front, have two bits of advice. The first one, when the technology, a new technology shows up, yeah, it can be very exciting, very tempting to kick off a new effort, a tiger team to pursue it, and that's great, you should absolutely do that. However, when you do, it's very important that you consider what you would do after. Under a success scenario, let's say that effort succeeds, you should be very clear on what the path of that new innovation is for your company, for your entire product, for your entire system. Oftentimes I've seen a failure mode where a project, a very difficult technical project succeeds, and then there's a dead end. So that can be very wasteful, that can be completely deflating.
00:24:04 · Dmitri Dolgov
The second bit of advice that I have here is when pursuing new tech, again, don't just ask what does this new tech give me in terms of capability and performance, also ask has it simplified my stack and has it led to fragmentation or unification? So set your launch bar to demand both breakthrough performance and at the same time, radical simplification and unification. And this exact philosophy and this muscle that we've built at Waymo over the years is what produced our latest core technology, and the heart of it is the Waymo Foundation Model. Now, Waymo Foundation Model is a multimodal world action language model. It's kind of a mouthful, so let me unpack the ingredients. It's a multimodal model because it is able to process these multimodal sensory inputs, cameras, LiDARs, radar. It's a world model because it inherently understands how the world works, the physics, the dynamics, as well as the social and semantic aspect of it. It's an action model because we are not just passively observing how the world evolves, we're an active participant. So the model needs to understand the effects of the actions of our agent on the world and be able to tell the good ones from bad ones. And finally, it's aligned with language, and that allows us to unlock general world knowledge from visual language models, and that's incredibly useful in the long tail of rare semantic situations. So more specifically, this is what the architecture looks like. It's kind of your typical encoder decoder architecture. The encoder part takes in the multimodal sensing and compresses it or encodes it into an efficient representation that retains all of the relevant data, all of the relevant information for the generative part or the decoder. It's an end to end model, which has a couple of nice properties. It allows us to effectively backpropagate the gradient from the task that we actually care about all the way to the early layers of the model, and it allows the encoder to reach, you know, kind of learn the right rich representations for what the generative part needs to solve the task. It uses the system one, system two, think fast, think slow architecture, and it leverages the general world knowledge of VLMs for efficient learning of semantic tasks. So let's dive deeper. First, the think fast path.
00:26:43 · Dmitri Dolgov
That part fuses the raw data from our cameras, our LiDARs, our radars, and that allows for split second safety critical decisions. So So you can think of it as kind of your driving instincts. This is what allows the car to brake instantly if let's say a pedestrian runs into the road or a cyclist that's nearby swerves into your path. This is like the, if you will, the lizard brain of your agent that deals with a lot of geometric tasks and can react in milliseconds. Second is the slow path. That's the part that's responsible for the more complex semantic and scene level understanding type tasks. And these sort of tasks, these things don't typically change in milliseconds, so there you can afford a bit more latency and you can trade that off for higher capability and higher levels of reasoning. So for example, if the Waymo Driver encounters a situation when there's, you know, a vehicle, let's say it's on fire on the side of the road, the fast path might just see it as a generic obstacle and reason that the path ahead of us is clear. And this is where the slow path comes in, and that path can use deep semantic reasoning to understand the semantics of that object, the car being fire, and the broader scene context, and that allows our driver to decide to take a very different action or a different route entirely, even if geometrically the path ahead of us is clear. And finally, there's the generate component. That's the decoder. That's the component that understands and can produce behavior. It understands how other actors behave, and it allows us to make predictions and plan our own driving decisions. And our Waymo Foundation model powers the Waymo Driver that runs on different generations of hardware and runs on different vehicle platforms. You have our fifth generation and sixth generation, the Jaguar I-PACEs, the Ojai and the Hyundai Ioniq, and in the future, will power different products and different commercial applications like trucking and personally owned vehicles. So by leveraging the strategy of focusing on the high capacity foundation upward model, we're able to move a lot of complexity upstream to that large shared foundation, and that allows us to make that specialization layer that's running on the car are pretty lightweight, and that in turn allows us to speed up the development process.
00:29:25 · Dmitri Dolgov
So the most important muscle in this lesson is for your company to not just leverage the tech of the day, but have the ability and build up muscle to repeatedly ride those tech waves and pull in the results of that innovation into production without regression, without breaking stride in deployment and scaling, and without drowning complexity. So let's move to the next lesson. There is a well known lesson in the AI community that general methods that leverage massive compute and massive data will always beat methods that rely on handcrafted engineered human knowledge. That's the so called bitter lesson that Richard Sutton published and formulated in 2019. And we have lived this and we have seen this in every wave of technical breakthroughs. Each time the bitter lesson holds methods that scale best with compute, with data, they always win out. And by the way, this is one of the reasons why we bet on the approach of building the foundation model. There is a well known property that if you bet on high capacity model and you use your data and your compute on that, you just get better scaling laws, and then you distill into smaller, more efficient models that are running on your agent in real time, you just get better scaling laws as opposed to just focusing on the smaller models directly. So one nuance area where this lesson shows up is the user structure in your models. And depending on how you use your structure, you can end up on either side of the bitter lesson. Essentially structure that fights scale will always lose, and structure that channel scale always wins. In particular this comes up around the discussion of end to end models. As I mentioned, an end to end model has some very nice properties. You back propagate gradient from the final tasks all the way through the model, and it allows the API between the encoder and decoder to use rich learn representations. And you know, those are the easiest models to build and train.