Y Combinator Startup Podcast
00:00:07 · Dmitri Dolgov
[00:00:07] S0: Good afternoon, everyone. It's great to be here. Now, we talk a lot about AI that lives on your screen, lives in the digital world. And today I'd like to talk to you about a different kind of AI that we've been building at Waymo, AI that lives in the real physical world. How many of you, by the way, have been in a Waymo? Just raise your arms. Wow, okay. That is impressive. Especially, I understand many of you are out of town. The folks who are visiting and have not had a chance to check out Waymo, I hope while you're here in the Bay Area, give it a try. So this being a startup school, I structured this presentation as a sequence of lessons, seven lessons that we've learned over the years at Waymo around what it takes to build and safely ship today's most mature application of AI in the physical world, the Waymo Driver. Let me start with a short video. This is a clip from a ride that I recently took in a Waymo with my kids. So as you see here, you know, we're moving forward, we're proceeding through an intersection, and a couple of human drivers just decide to cut in right in front of us. And the Waymo Driver reacted safely, reacted smoothly, in fact, so much so that the kids, my kids were preoccupied in the backseat, they didn't even notice that anything happened. And to me, this was a pretty powerful moment. I've been working on this technology and this product for close to two decades, and you know, it just did something fairly important. It acted safely, it kept my kids safe, it kept everybody safe, and nobody noticed. And that I think will be a bit of a theme in general when it comes to physical AI, that the best AI moments will look like nothing happened, it's just the task got done safely and smoothly. And these sort of moments where the Waymo Driver kept everyone safe are happening daily across our fleet. Today, the Waymo Driver is serving around 500 trips per week and driving over 4,000,000 fully autonomous miles every week in 15 cities across The United States. Just for a comparison, that's over three hundred years every week of an average American driver per year. And the Waymo Driver is accomplishing that with a superhuman safety record. So what what does it take to build and deploy an AI agent in the physical world at scale? Now in Silicon Valley, there's a common mantra to move fast and break things.
00:02:49 · Dmitri Dolgov
However, when you're dealing with atoms instead of bits, breaking things is not really okay. So the thing you have to do is to move fast and ship safely. And that's a much more difficult thing to do. You have to build systems that are robust from day one, you have to build AI models, and you have to build training recipes where safety is the foundation and not an afterthought, not an add on. And by the way, the problem itself of physical AI is different from digital AI. There are four main gaps that you have to contend with if you're building AI for the physical world versus the digital world. First, there is the cost of error gaps. You have a language model or a chatbot or a copilot and it makes a mistake, you know, usually it costs you a retry. In the physical world, the cost of a mistake can be measured in human lives, not tokens. There's simply not an undo and a retry button. Secondly, you have the latency gap. And typically when you're running a VLM or, you know, a digital assistant, it can take many seconds, sometimes minutes to come back with an answer to you. A car traveling at freeway speeds moves about a 100 feet in one second, so their milliseconds really matter, and you have to run all of your inference, make all of your decisions onboard a compute that fits in the trunk of your car. Next there's the data gap. Digital AI had the internet, this wonderful, immense cache of pre labeled human knowledge and human thought that we've ever assembled. There's no digitized version of the internet for the physical world. And lastly there's the validation gap. In digital AI, often you can ship something that's good enough, and then you let your users use your product, they find the edge cases, and that allows you to deploy on day one practically at unlimited scale. And then you can just iterate on hill climb and quality from there. In physical AI, the situation is different. Given the high cost of errors, you need to have a very high level of safety and a very high level of confidence on day one, before you deploy your first robot, before you drive your first autonomous mile. Now at the same time, when you're dealing with physical AI, the actual experience of having your agent in the real world is invaluable and it's irreplaceable.
00:05:26 · Dmitri Dolgov
These systems are not just something that you can build in the lab, you know, get it perfect and then deploy at full scale overnight. So given those two factors, you really need to super clearly and super crisply define the operating conditions and the deployment parameters of your agent, and then build a rigorous framework to guide your deployment so that you can scale in a responsible manner. And this is absolutely critical. This is how you earn trust from your customers, from the communities, from the regulators and yourself. So at Waymo, we see these gaps, of course, in the context of autonomous vehicles, but these gaps will show up in practically any sort of nontrivial physical agent that we will deploy in some shape or form. And driving is just simply the first domain where AI has crossed these four gaps at scale with the public interacting with our product. So let's dive into those lessons that we've learned over the years at Waymo from working on this problem and talk about how we address those gaps. I have seven lessons in this talk. They're all technical. There's a lot more that goes into building a company and building a product, but today I'll just focus on the technical aspects of building AI for the physical world. And each one of those lessons, I think by itself, will not be exactly earth shattering. You know, a lot of it will overlap with likely things you've heard elsewhere, but I hope that the grounding of these lessons in our experience and some of the nuance that I can add about how they showed up in our experience of deploying a physical agent and scaling it safely will be interesting and useful for many of you who are in the space as you build your product, as you build your startup. So let's dive in. The first lesson has to do with this massive, frustrating, sometimes soul crushing difference between a demo and a real product. And a working demo is 1% at best of the work that you have to do. The many of performance, the many nines of reliability that follow, that's where the real work happens. And if you're a founder in the room, chances are you are focused on getting that first prototype, that first demo off the ground. And when you hit that first version of a system that works, that first 90%, when the demo actually works, it feels incredible.
00:08:06 · Dmitri Dolgov
You feel like you solved it, the sky's the limit, you're extrapolating forward, and in our world, we hit that first milestone, that first 90% back around 2010. So when this project started in 2009, before we started building the system, we set a couple of pretty ambitious goals for ourselves. One was to drive a 100 autonomous, a 100,000 miles in autonomous mode. The second goal was to drive 10 routes, each one was a 100 miles long, chosen to cover a variety of conditions across the Bay Area, and we had to do each one from beginning to end without a human intervention. We had at the time a team of about a dozen engineers, and we accomplished both of these goals in about a year and a half. And keep in mind, this was well before any of the AI breakthroughs, before ConvNets, before transformers, before VLMs, before any of the stuff that we talk about today. And yet, you know, we got it done, and kinda by demo standards, autonomous driving was solved in 2010, right? We handled everything. We could drive during the day, during the night, we handled traffic, pedestrians, cyclists, traffic lights, construction zones, on freeways, on surface streets, so we were quote unquote capability complete, and you know, at the time we felt like we're on top of the world, but then we quickly ran, as we started building towards the product, we quickly ran into a brutal reality that there's a massive difference between doing something once or driving 10 routes once and building a scalable service with nobody behind the wheel. It took us about ten more years to begin providing a service, and then five more years to scale to half a million trips per week. So the demo took eighteen months, the product took about fifteen years. But now we're scaling exponentially. To date, we've served well over 20,000,000 fully autonomous trips, and we've driven well over 200,000,000 fully autonomous miles, and we have rider only vehicles operating in 15 cities across The United States, And we're scaling exponentially, it took us fifteen years to get to that first 100,000,000 miles and about seven months to drive the next 100,000,000. It took us about eight years to go from the time when we started our initial writer only operation to the time when we had... When we were serving writers in four cities. Earlier this year, we launched four cities in just one day.
00:10:49 · Dmitri Dolgov
So why does bridging that gap from demo to product take so long? Because there's this harsh engineering reality that you can't really cheat, that reliability and performance lives on this exponential letter of nines. So getting to that first 90% or 99%, that's the easy part. But then every next nine that you wanna add, that takes about 10 times more effort. So you need to know upfront exactly how many nines your product actually needs. So demo might need, you know, one nine, an assist product or, you know, Copilot might need a few, but a fully autonomous AI agent that we're gonna be putting out in the physical world that engages with the public, you know, with kids running around, that needs a whole stack of them. And at scale, the long tail is the problem space, it's your entire problem statement. When you drive millions of miles per week, a rare event that might happen once in a million miles, that just becomes your daily reality. And getting those next nines means doing something different every time. So you don't get to say six nines of performance or reliability by doing the same thing that you did for, you know, to achieve the first two, but longer. You have to do fundamentally different things, it requires a fundamentally different approach. For example, you can take reliability. You can get to the first couple of nines by just doing proper engineering and doing some bug fixes, but to get to the next few, you need to invest in fundamentally different approaches. You need to build fully redundant systems, have tiered fullback architectures and so forth and so on, and the same thing holds for the performance of AI models. So what that actually means is that in this space, it's incredibly easy to get started, but it can be excruciatingly difficult to get to the real product, and that effect is only amplified with every wave of technological breakthroughs, and that naturally leads to hype cycles. So every AI breakthrough from deep learning to cognitive nets to transformers, via lanes, you name it, it makes it that much easier to get started. Your demos, your prototypes, they get a 100 times easier, but the tail, that's where the hard problems are that moves much less. It moves, but the effect is muted, and that's why every hype cycle produces a wave of absolutely spectacular demos and very few real products.
00:13:31 · Dmitri Dolgov
And the recurring mistake of every cycle is spending on the demo what you should be saving for the nines. Now, you know, this being a startup school, the last thing I wanna do is throw too much cold water on the magic and the excitement of those early days. This this time is absolutely magical. It's amazing. Cherish it. Leverage it. But the key is to remain honest about the product that you are building, the number of nines and performance and reliability that that product demands, and not cutting corners to get there. Otherwise, you might be in for a pretty rude awakening later, so count your nines before you count your demo views. And this brings us to the second lesson. Once you know how many nines your product actually needs, it fundamentally dictates the architecture and the core technical approach that you need to pursue. Now every technology has a performance versus effort curve, right? They all tend to start fairly steep and go up, and then they flatten out, And you know, as I just mentioned, every other nine gets an order of magnitude more difficult. So common failure mode is picking the tech that gives you the fastest early ramp, riding that steep curve, feeling like you're winning, projecting that you know, steep slope into the future and feeling like the sky's the limit, and then hitting the plateau and discovering that the technology path that you picked actually flattens out way before the performance that is required by your product. Now you might still choose to be, at least for a while, on that steep curve for a variety of practical reasons. You know, maybe you want to prototype something or demo something or build something in service of learning, but be honest with yourself where you're building for the purpose of a demo, for the purpose of learning, or towards an actual product. So let's take an example from our domain, autonomous vehicle and sensing. There's been a long standing debate about what kind of sensors do you actually need for autonomous driving. Naturally, more sensors means higher performance, but also means higher complexity. So humans of course can drive with just eyes, so there's that proof of existence. Now, and if the goal were to just approximately match human performance or to build an assist product, that's a very reasonable way to go.