Dream Machines: Physical AI, Robotics, Being Human
00:49:03 · Evan Helda, Charles Wong, Aravind Kandiah
[00:49:03] S1: And [00:49:05] S2: and really... And, you know, this is almost less of a robotic specific conversation or AI specific conversation, just like product specific... Product... More broadly product principles. Right? Mhmm. [00:49:16] S0: And I [00:49:16] S2: think one thing that we really try to hold ourselves to is how do we make their lives easier? Right? Not just the robotics developer, the user, but also, you know, the economic buyer, the procurement guy, the director of autonomous, all the way up the chain. Right? How do we make this a no brainer for them compared to whatever they're about to spend on or whatever whatever they're about to do? Yep. And so which is why, like, when we compare ourselves against some, you know, simulation tooling that's out there, we we... Like, one of our positions is that you don't need to hire, you know, 30 people in your sim team to run this and build this capability for you. What Bifrost, above all, purports is that you just need to provide one AI developer, one user, and you can start generating data within five minutes. Right? And just the ease of onboarding, the ease of use from a non three d, non SIM expert is usually more than enough to get our foot in the door. Right? Once your foot is in the door, then the game begins. Right? Because now you're being attacked from all sides. There's internal politics. There's, like, new new tools coming out, and that's where it gets kind of interesting, I guess. Sure. Aravind?
00:50:28 · Aravind Kandiah, Evan Helda
[00:50:28] S0: Yeah. So most of our customers have raised, like, 1 to $2,000,000,000 already. So, you know, they also have the money to go and build it, and most of also, like, try to build it themselves. The reality is, first of all... And maybe this is a thing that may most of you don't wanna hear, but you're just gonna ship faster and better. Right? That's first of all. And two, just make it super easy to use. But most important of all, I think you need to have a very opinionated take on how it needs to be solved. And our opinionated take is... We call it, like, the autonomous simulator. Right? What we saw previously at almost every simulation company is all the scenarios had to be hand built. Everything had to be done by hand. I have to go in. I have to define what three d objects have to be. I have to define how the scenario plays out. As a result, there's this category called simulation engineering, which is you hire a whole bunch of people to manually program the simulator to do a whole bunch of stuff. And as a result, you're you're hiring, like, 30 to 10 people just to serve, like, some internal teams. And most of the time, the internal team can barely even keep up and serve the entire organization. Mhmm. Especially with, like, these larger organizations who are, like, you know, scaling... Hyperscaling, there's, like, inside, there's, like, 15 different platform teams, and this one team cannot provide the data to help these, like, 15 teams. [00:52:00] S1: Mhmm.
00:52:00 · Evan Helda, Aravind Kandiah, Charles Wong
[00:52:00] S1: Mhmm. [00:52:00] S0: So we come in and say that, hey, guys. You shouldn't be building scenarios by hand, first of all. Your data, your real world data, should drive your simulator. So it's a data driven simulator. It should inform what scenarios you build. It should inform that rails to sim gap for the photorealism. Your simulator should be driven by real world data, and that's how you also close the SIM gap, but also how you do less work. Right? And you can produce higher quality data. So this concept of the autonomous simulator is really an opinion thing. It's like, how high conviction are you on this direction? And you're willing to commit all these engineering resources to build the thing. And most teams don't have that high conviction. They just do the simple thing which say, let's just build a simulator from scratch. There's procedural generation. Use this. Use that. And it's all the... On paper seems like the right thing to do, but they're not taking the high risk back, which is like, oh my god. Okay. How do we build real world data pipelines that can define... Auto define all these things that, like, generate these scenarios at scale? [00:53:05] S2: How does that [00:53:06] S1: how does that data flow work when you are trying to let real world data define the simulator and define the synthetic data, like, from, you know, if if you're talking to an engineer that's trying to build this, like, can can you break that down for us a little bit?
00:53:06 · Evan Helda, Aravind Kandiah
[00:53:06] S1: how does that data flow work when you are trying to let real world data define the simulator and define the synthetic data, like, from, you know, if if you're talking to an engineer that's trying to build this, like, can can you break that down for us a little bit? [00:53:24] S0: Yeah. So the way you can think about it is, let's say the the data is coming from your camera. It actually has so much compressed information about the type of objects, the type of scenarios, the types of layouts [00:53:41] S1: Mhmm. [00:53:41] S0: The time of day, the weather. There's all this, like, metadata. Yep. Essentially, are just looking at this and pulling out all this metadata and using it to inform what you generate. And we help automate that setup process. [00:53:58] S1: Got it. How is the automation happening? So let's take an example. Maybe take one people can relate to, a humanoid in the house. [00:54:04] S0: I [00:54:06] S1: see a bunch of packages on the ground. There's gaps that I'm identifying in my space, like, those pixels come in, and then you guys have, like, some kind of world model you're using to produce variations of that? Are you guys using an LLM or VLM to identify that? And then you have humans sit there and go, like, recreate three d assets that aren't in the room that should be like... What what does that technically automation look like and how does it feed into the the final simulation or final experience that you guys enable?
00:54:06 · Evan Helda, Aravind Kandiah
[00:54:06] S1: see a bunch of packages on the ground. There's gaps that I'm identifying in my space, like, those pixels come in, and then you guys have, like, some kind of world model you're using to produce variations of that? Are you guys using an LLM or VLM to identify that? And then you have humans sit there and go, like, recreate three d assets that aren't in the room that should be like... What what does that technically automation look like and how does it feed into the the final simulation or final experience that you guys enable? [00:54:38] S0: Yeah. You're you're actually halfway there already. So it's like, the the image comes in. Right? Mhmm. And you have both BLMs, which are very good at, like, general purpose understanding of seeing context and imagery. And you can actually have a very, like, structured prompts to pull out all this metadata. So it's [00:54:54] S1: like Got it.
00:54:54 · Evan Helda, Aravind Kandiah
[00:54:54] S1: like Got it. [00:54:55] S0: Hey. I want you to tell me time of day. I want you to me type of scenario. I want you to draw bounding boxes across all the different types of things. This is actually, like, 90 to 80%. You can... It is good enough. And then you layer on top a fine tune, like, segmentation network that can look at all the objects and extract up all the objects. That goes into a different pipeline that from a two d image, it can reconstruct the three d object itself. Right? So what you're doing is essentially you're looking at an image. You're taking out the parts, and then you're feeding those parts into different pipelines to recreate those objects and things. Right? And you're also extracting the metadata of the... Oh, what's the lighting look like? What does the scene layout look like? [00:55:38] S1: Yeah. Yeah. [00:55:38] S0: Right? And then you are recombining that to a physics simulation. So you go from two d image to that three d sim. Right? And the part is you want to come to a point where it is an autonomous simulator, where your real world data is being piped in, It's informing your simulator. It's defining the test. It's doing all these things, but you could go in and you can expand your coverage. Right? Because now you have full control of your simulator. You can just say that, hey, you know, my real world data has these, like, 10 types of boxes and tables, but let's make it more crazy. Let's, like, increase the coverage.