Dream Machines: Physical AI, Robotics, Being Human
00:50:28 · Aravind Kandiah, Evan Helda
[00:50:28] S0: Yeah. So most of our customers have raised, like, 1 to $2,000,000,000 already. So, you know, they also have the money to go and build it, and most of also, like, try to build it themselves. The reality is, first of all... And maybe this is a thing that may most of you don't wanna hear, but you're just gonna ship faster and better. Right? That's first of all. And two, just make it super easy to use. But most important of all, I think you need to have a very opinionated take on how it needs to be solved. And our opinionated take is... We call it, like, the autonomous simulator. Right? What we saw previously at almost every simulation company is all the scenarios had to be hand built. Everything had to be done by hand. I have to go in. I have to define what three d objects have to be. I have to define how the scenario plays out. As a result, there's this category called simulation engineering, which is you hire a whole bunch of people to manually program the simulator to do a whole bunch of stuff. And as a result, you're you're hiring, like, 30 to 10 people just to serve, like, some internal teams. And most of the time, the internal team can barely even keep up and serve the entire organization. Mhmm. Especially with, like, these larger organizations who are, like, you know, scaling... Hyperscaling, there's, like, inside, there's, like, 15 different platform teams, and this one team cannot provide the data to help these, like, 15 teams. [00:52:00] S1: Mhmm.
00:52:00 · Evan Helda, Aravind Kandiah, Charles Wong
[00:52:00] S1: Mhmm. [00:52:00] S0: So we come in and say that, hey, guys. You shouldn't be building scenarios by hand, first of all. Your data, your real world data, should drive your simulator. So it's a data driven simulator. It should inform what scenarios you build. It should inform that rails to sim gap for the photorealism. Your simulator should be driven by real world data, and that's how you also close the SIM gap, but also how you do less work. Right? And you can produce higher quality data. So this concept of the autonomous simulator is really an opinion thing. It's like, how high conviction are you on this direction? And you're willing to commit all these engineering resources to build the thing. And most teams don't have that high conviction. They just do the simple thing which say, let's just build a simulator from scratch. There's procedural generation. Use this. Use that. And it's all the... On paper seems like the right thing to do, but they're not taking the high risk back, which is like, oh my god. Okay. How do we build real world data pipelines that can define... Auto define all these things that, like, generate these scenarios at scale? [00:53:05] S2: How does that [00:53:06] S1: how does that data flow work when you are trying to let real world data define the simulator and define the synthetic data, like, from, you know, if if you're talking to an engineer that's trying to build this, like, can can you break that down for us a little bit?
00:53:06 · Evan Helda, Aravind Kandiah
[00:53:06] S1: how does that data flow work when you are trying to let real world data define the simulator and define the synthetic data, like, from, you know, if if you're talking to an engineer that's trying to build this, like, can can you break that down for us a little bit? [00:53:24] S0: Yeah. So the way you can think about it is, let's say the the data is coming from your camera. It actually has so much compressed information about the type of objects, the type of scenarios, the types of layouts [00:53:41] S1: Mhmm. [00:53:41] S0: The time of day, the weather. There's all this, like, metadata. Yep. Essentially, are just looking at this and pulling out all this metadata and using it to inform what you generate. And we help automate that setup process. [00:53:58] S1: Got it. How is the automation happening? So let's take an example. Maybe take one people can relate to, a humanoid in the house. [00:54:04] S0: I [00:54:06] S1: see a bunch of packages on the ground. There's gaps that I'm identifying in my space, like, those pixels come in, and then you guys have, like, some kind of world model you're using to produce variations of that? Are you guys using an LLM or VLM to identify that? And then you have humans sit there and go, like, recreate three d assets that aren't in the room that should be like... What what does that technically automation look like and how does it feed into the the final simulation or final experience that you guys enable?
00:54:06 · Evan Helda, Aravind Kandiah
[00:54:06] S1: see a bunch of packages on the ground. There's gaps that I'm identifying in my space, like, those pixels come in, and then you guys have, like, some kind of world model you're using to produce variations of that? Are you guys using an LLM or VLM to identify that? And then you have humans sit there and go, like, recreate three d assets that aren't in the room that should be like... What what does that technically automation look like and how does it feed into the the final simulation or final experience that you guys enable? [00:54:38] S0: Yeah. You're you're actually halfway there already. So it's like, the the image comes in. Right? Mhmm. And you have both BLMs, which are very good at, like, general purpose understanding of seeing context and imagery. And you can actually have a very, like, structured prompts to pull out all this metadata. So it's [00:54:54] S1: like Got it.
00:54:54 · Evan Helda, Aravind Kandiah
[00:54:54] S1: like Got it. [00:54:55] S0: Hey. I want you to tell me time of day. I want you to me type of scenario. I want you to draw bounding boxes across all the different types of things. This is actually, like, 90 to 80%. You can... It is good enough. And then you layer on top a fine tune, like, segmentation network that can look at all the objects and extract up all the objects. That goes into a different pipeline that from a two d image, it can reconstruct the three d object itself. Right? So what you're doing is essentially you're looking at an image. You're taking out the parts, and then you're feeding those parts into different pipelines to recreate those objects and things. Right? And you're also extracting the metadata of the... Oh, what's the lighting look like? What does the scene layout look like? [00:55:38] S1: Yeah. Yeah. [00:55:38] S0: Right? And then you are recombining that to a physics simulation. So you go from two d image to that three d sim. Right? And the part is you want to come to a point where it is an autonomous simulator, where your real world data is being piped in, It's informing your simulator. It's defining the test. It's doing all these things, but you could go in and you can expand your coverage. Right? Because now you have full control of your simulator. You can just say that, hey, you know, my real world data has these, like, 10 types of boxes and tables, but let's make it more crazy. Let's, like, increase the coverage.
00:55:38 · Aravind Kandiah, Evan Helda
[00:55:38] S0: Right? And then you are recombining that to a physics simulation. So you go from two d image to that three d sim. Right? And the part is you want to come to a point where it is an autonomous simulator, where your real world data is being piped in, It's informing your simulator. It's defining the test. It's doing all these things, but you could go in and you can expand your coverage. Right? Because now you have full control of your simulator. You can just say that, hey, you know, my real world data has these, like, 10 types of boxes and tables, but let's make it more crazy. Let's, like, increase the coverage. [00:56:14] S1: Right. Where are we in on that journey to just a fully autonomous simulation loop where agents are able to go and do all of the... There's a term you guys use called data patching, which is a... This idea of like treating data like a bug fix. How far are we from a world where like you don't need a human in the loop to do really any of the stuff you're talking about? [00:56:35] S0: I... So there's there's two ways of thinking about it. One is the the safety side of me is, like, there should always be a human in the loop verifying the patches, which is like, hey. It identified the thing. You should fix it. And there's other sites like accelerationism, which like, oh my god. Just just automate the whole thing. It's like a non safety thing. [00:56:57] S1: Mhmm. [00:56:57] S0: I think it's possible to automate the thing entirely, but I think it's unsafe. Yep. Simply because these are mission critical systems.