Dream Machines: Physical AI, Robotics, Being Human
00:26:27 · Aravind Kandiah, Charles Wong
[00:26:27] S0: Right? [00:26:27] S2: So a lot of our work was how do we abstract that away, present it in a way that an AI developer can understand. So today, our product is a web app. Right? It's a it's a platform called Stardust, where on one side is it's like a three d preview of what you're building being streamed into you on your browser, so you don't need compute. And on the right is a Python notebook. Right? Where you can... That's how you spec your scenarios and build your generations and all that whole. Right? So the idea here is that if you're a company and you wanna adopt simulation in your company, you're... Instead of spinning up a 30, 50 man simulation team, you know, with, like, three d artists, technical artists, three d developers, just use Bifrost. It's just one person, one seat, one experience as an AI developer, and you don't need all that additional support. Right? We are currently working on a new product that's going up to market soon. Nice. Product is specifically focused on robotics evaluation. Got Where the previous product start us was more around, like, focused on perception, general autonomy. This one is hyper focused on solving what we feel now is the hardest problem, which is evaluation. Right? How as a robotics developer do I go from my initial policy next to, like, a parameter sweep across all the things that I care about, doing it in a scalable way, doing it in an easy way that I don't have to, you know, again, wade into documentation, figure out Docker containers, and stuff like that.
00:27:56 · Evan Helda, Aravind Kandiah, Charles Wong
[00:27:56] S1: Wow. I have a lot of questions about that product. I'm seeing a lot of people go off that same space and interesting innovation happening there. We can get into that. But before, first one comment that you'll find fun. I was at AWS for six years, and my whole mission was to build a solution for three d content creation, three d content management, three d asset pipelines, three d asset configuration, trying to automate all that. We did launch it. It launched as a solution called spatial data management, or Amazon Spatial Data Management. It's a much more primitive version of what I was trying to get to before before leaving, but the code name was Nebula. And so you've mentioned Stardust. I'm like, I think you have similar ideas of, like, three d data, spatial data simulation type worlds. And Great minds. Great minds. Exactly. I I... I'm determined to make Nebula be a prog name some... At some point in my career because I think it's a great a great name. [00:28:54] S0: Very close to name he is as well. [00:28:56] S2: This is [00:28:56] S1: Yeah. Exactly. Maybe it's destiny. [00:28:59] S0: Who knows?
00:28:59 · Aravind Kandiah, Evan Helda, Charles Wong
[00:28:59] S0: Who knows? [00:29:01] S1: You mentioned earlier this idea of making surmountable. And people will toss Cindereal around, it's become a buzzword. People oftentimes will use that excuse and even try because it is such a daunting thing. And there are some use cases where it's very, very real, and you're not gonna overcome it anytime soon. The most common example is like deformation or interaction with certain types of cloth or liquids or things like this where like the physics is just hard. How do you guys convince people that sim to real is surmountable? And then where is your focus and where are you guys nailing it and make... And and it truly is surmountable for you guys? It's repeatable and you're... That you're not even having to struggle with it versus, like, where is that sort of middle ground where you're close? And then where is that edge where you're just... It's in the future and you you know it, you admit it, you're not gonna get there yet, but it's on your mind. I guess we'll start with giving confidence and explain to them why it's even worth trying to tackle in the first place. Yeah. [00:30:07] S0: So maybe you can, like, break it down into what does, like, what is it... What does simp to real actually mean? Right? And Mhmm. Simp to real, all it really means is just, you know, if you train something in simulation, does it actually translate to actual performance in the real world? [00:30:23] S1: Mhmm. [00:30:24] S2: And
00:30:24 · Aravind Kandiah
[00:30:24] S0: so most of the time, that's act... Actually where we start, which is like, can we show you proof that it just works in the real works in the real world? Right? And the the means by which we get there, you know, there's like a whole bunch of things that we can do. Right? So it always starts with, like, bringing in the end user and, like, trying to convince them. And most of the time, they are not even, like, AI engineers. They may be a c suite at, a a large MNC or it may be an end user that is just trying to automate some critical task in a in a factory. Right? So then it's like, okay. I have these 10 tasks that I need to do. Can you show me that these 10 tasks can be done, trained entirely in simulation? So it just becomes a train in sim, show in real life, or if I test in SIM and I test in real, are the performance equivalent? So that's like the the proof is in the pudding part of things. Right? Mhmm. It's like how do we showcase it to customers? It's just by doing it for them. The second part where your question is like, okay. How do you break down this whole symptom real thing? And I think you hit it exactly in the head, which is different domains have different thresholds of symptom real affordances, if you will. Right? And symterial is, like, these four big parts. There's, like, the perception symterial, which like, how real does it look? There's the dynamics, symterial, which is, you know, if I move a robot arm in simulation, is it the same movement in reality? And then there's what I call, like, the object to robot interaction, which is like contact forces. And the last one is, like, all your crazy deformable cloths and all that. Perception, is something that we have solved, like that symptom real gap. So, like, does it look realistic? Right? And you can think, like, you know, you can use world models. You can use diffusion models. You can learn from the actual sensor imagery and transfer that onto your simulation. The second one, which is the robot dynamics, there's a lot of new solvers coming out like Newton and all that that... I wouldn't say they are one to one, but they are good enough approximation that customers have found it a meaningful test and meaningful training data already. Right? So MolmoAct, like this paper that came out as well, showed the real to sim performance deltas are actually pretty good as well.
00:32:41 · Aravind Kandiah, Evan Helda
And the most interesting thing I think I've seen so far is what I call the sim to sim to sim to real, which is you train on one type of physics engine, you show that it generalizes to another physics engine, and then you show that it can jump to another physics engine, and therefore, it improves your generalizability when you actually jump to reality as well. Right? Because reality is just another type of physics engine. Right? So you're training a policy that can generalize across these types of physics engines. [00:33:11] S1: Interesting. [00:33:13] S0: Right. [00:33:13] S1: It's cool. It's... You you put it in different physical realities along the way before it gets to, like, the ultimate one. And let it deal with, the the the variety and the the unknowns, the uncertainties, etcetera. Right?
00:33:13 · Evan Helda, Aravind Kandiah, Charles Wong
[00:33:13] S1: It's cool. It's... You you put it in different physical realities along the way before it gets to, like, the ultimate one. And let it deal with, the the the variety and the the unknowns, the uncertainties, etcetera. Right? [00:33:25] S0: Right. So it's like you... The thing is, what are the things that we can do at the speed of software? Right? You remove as much real world bottlenecks as possible Mhmm. To do things as fast as you can at the speed of software. You just improve iteration time. And all of these are approximations, but if you can get the system to be a very, good approximator of, like, the physics, then you can also approximate, the real world physics and just generalize pretty well as well. Right? So that's kind of, how we've been able to overcome all those gaps. But, you know, coming back to the first part that I mentioned, that is majority. It's just like showing customers it actually works. Mhmm. And that's something that we have we have gotten to be very repeatable where data comes in, we can calibrate the system, and then put it out again. [00:34:13] S2: Yep. Well, I will add as well, like, know, and Aravind is hinting at it a little bit. Like, there's a technology when it comes to Centro. There's also the trust aspect. [00:34:20] S0: Mhmm.