Dream Machines: Physical AI, Robotics, Being Human
00:53:06 · Evan Helda, Aravind Kandiah
[00:53:06] S1: how does that data flow work when you are trying to let real world data define the simulator and define the synthetic data, like, from, you know, if if you're talking to an engineer that's trying to build this, like, can can you break that down for us a little bit? [00:53:24] S0: Yeah. So the way you can think about it is, let's say the the data is coming from your camera. It actually has so much compressed information about the type of objects, the type of scenarios, the types of layouts [00:53:41] S1: Mhmm. [00:53:41] S0: The time of day, the weather. There's all this, like, metadata. Yep. Essentially, are just looking at this and pulling out all this metadata and using it to inform what you generate. And we help automate that setup process. [00:53:58] S1: Got it. How is the automation happening? So let's take an example. Maybe take one people can relate to, a humanoid in the house. [00:54:04] S0: I [00:54:06] S1: see a bunch of packages on the ground. There's gaps that I'm identifying in my space, like, those pixels come in, and then you guys have, like, some kind of world model you're using to produce variations of that? Are you guys using an LLM or VLM to identify that? And then you have humans sit there and go, like, recreate three d assets that aren't in the room that should be like... What what does that technically automation look like and how does it feed into the the final simulation or final experience that you guys enable?
00:54:06 · Evan Helda, Aravind Kandiah
[00:54:06] S1: see a bunch of packages on the ground. There's gaps that I'm identifying in my space, like, those pixels come in, and then you guys have, like, some kind of world model you're using to produce variations of that? Are you guys using an LLM or VLM to identify that? And then you have humans sit there and go, like, recreate three d assets that aren't in the room that should be like... What what does that technically automation look like and how does it feed into the the final simulation or final experience that you guys enable? [00:54:38] S0: Yeah. You're you're actually halfway there already. So it's like, the the image comes in. Right? Mhmm. And you have both BLMs, which are very good at, like, general purpose understanding of seeing context and imagery. And you can actually have a very, like, structured prompts to pull out all this metadata. So it's [00:54:54] S1: like Got it.
00:54:54 · Evan Helda, Aravind Kandiah
[00:54:54] S1: like Got it. [00:54:55] S0: Hey. I want you to tell me time of day. I want you to me type of scenario. I want you to draw bounding boxes across all the different types of things. This is actually, like, 90 to 80%. You can... It is good enough. And then you layer on top a fine tune, like, segmentation network that can look at all the objects and extract up all the objects. That goes into a different pipeline that from a two d image, it can reconstruct the three d object itself. Right? So what you're doing is essentially you're looking at an image. You're taking out the parts, and then you're feeding those parts into different pipelines to recreate those objects and things. Right? And you're also extracting the metadata of the... Oh, what's the lighting look like? What does the scene layout look like? [00:55:38] S1: Yeah. Yeah. [00:55:38] S0: Right? And then you are recombining that to a physics simulation. So you go from two d image to that three d sim. Right? And the part is you want to come to a point where it is an autonomous simulator, where your real world data is being piped in, It's informing your simulator. It's defining the test. It's doing all these things, but you could go in and you can expand your coverage. Right? Because now you have full control of your simulator. You can just say that, hey, you know, my real world data has these, like, 10 types of boxes and tables, but let's make it more crazy. Let's, like, increase the coverage.
00:55:38 · Aravind Kandiah, Evan Helda
[00:55:38] S0: Right? And then you are recombining that to a physics simulation. So you go from two d image to that three d sim. Right? And the part is you want to come to a point where it is an autonomous simulator, where your real world data is being piped in, It's informing your simulator. It's defining the test. It's doing all these things, but you could go in and you can expand your coverage. Right? Because now you have full control of your simulator. You can just say that, hey, you know, my real world data has these, like, 10 types of boxes and tables, but let's make it more crazy. Let's, like, increase the coverage. [00:56:14] S1: Right. Where are we in on that journey to just a fully autonomous simulation loop where agents are able to go and do all of the... There's a term you guys use called data patching, which is a... This idea of like treating data like a bug fix. How far are we from a world where like you don't need a human in the loop to do really any of the stuff you're talking about? [00:56:35] S0: I... So there's there's two ways of thinking about it. One is the the safety side of me is, like, there should always be a human in the loop verifying the patches, which is like, hey. It identified the thing. You should fix it. And there's other sites like accelerationism, which like, oh my god. Just just automate the whole thing. It's like a non safety thing. [00:56:57] S1: Mhmm. [00:56:57] S0: I think it's possible to automate the thing entirely, but I think it's unsafe. Yep. Simply because these are mission critical systems.
00:57:06 · Charles Wong
[00:57:06] S2: Yeah. Yeah. I think I think that, you know, another useful way to look at this is sort of like, you know, fortunately, unfortunately, the way AI teams are set up in most companies today, they're being asked to build, like, their software engineering teams, when in reality, they are r and d teams. Right? And there's a difference. Right? R and d teams are supposed to go out and do experiments, you know, come back with a thesis, and you're not you're not... You shouldn't be measured by the outcome. You should be measured by the experiments you're putting into the into the work, the inputs. Right? Mhmm. But software engineering is like, okay. It's it's a fairly structured thing. You have a spec. You build it. You build the feature well. You get a thumbs up. Right? And somehow, we're taking this very complex AI r and d process... AI development process and being asked to deliver outcomes that look like software engineering. Right? And that's often the engineering and... Versus management mismatch, where management is like, oh, you know, I'd love for... You know, we need to ship by December. Before we ship, we need the model to achieve 95% accuracy. Right? Firstly, where do they pull that number from? It's just something that, you know, it's pretty arbitrary. And then now the engineers, the r and d engineers have to go and figure out how we're gonna get there. Right? But a r and d isn't so so so deterministic. Right? So it's this relentless crawl. Right? Mhmm. Even if you gave these, you know, developers, you know, all the right ingredients. Right? The best data, the best pipelines, all the GPUs they could dream of. If you don't give them a recipe as well, the cake is gonna taste like shit.
00:58:37 · Evan Helda, Charles Wong
[00:58:37] S1: Mhmm. [00:58:38] S2: Right? So that's really what we're trying to think about when we build, like, this data patching a loop. Right? How do we take the principles of software engineering, like unit testing, like iterative patches, and iterative workflow, and apply that to AI so that developers have an easier time of stepping their way up to something that actually works? Right? That's how we that's how we think about it. And that's... It's it's a continuous journey. Right? Yeah. One day AI development will be just a a step of the finger, but we're not there yet, and we hope that Bifrost is the is the thing that gets us there. Yeah. [00:59:09] S1: That's that's a really thoughtful approach for sure. Go to Aravind.