The Robotics Stack
00:08:31 · Leo, Varun Nair
[00:08:31] S1: I see. That also bridges the gap with something interesting you said earlier. You've been into data, you've been into deployment, and lately you've been interested in SIM simulation. Usually, people are into simulation before doing deployment. So how did you, like, get into real world stuff before getting into into SIM? [00:08:50] S2: I mean, you could go either way. The thing is you do deployments or you just, like, evaluate in the real world and you see that policies don't work very well and you're like, hey, there might be a better way to scale this up. There might be better ways to run a thousand different trajectories rather than resetting one manually again and again. And that's how I found my way. It's not like I hadn't dabbled in SIEM before, I have, but it's interesting because you have data on one side, you have deployment on one side and you have SIEM sitting in this middle where learning in SIEM is smart, but there is a real gap that exists between transferring a policy that exists in SIEM versus in actual deployments. But it's the other way too. When something fails in deployments, it needs to be rerouted back to SIEM as well to make the simulation even better. And that's a piece that's missing. So it's not just that SIEM informs deployment, but it's also that deployments will inform how to make better SIEMs. I'm actually putting something out there about this later today or tomorrow. So hopefully it'll give you better context about it as well. Yeah.
00:08:50 · Varun Nair, Leo
[00:08:50] S2: I mean, you could go either way. The thing is you do deployments or you just, like, evaluate in the real world and you see that policies don't work very well and you're like, hey, there might be a better way to scale this up. There might be better ways to run a thousand different trajectories rather than resetting one manually again and again. And that's how I found my way. It's not like I hadn't dabbled in SIEM before, I have, but it's interesting because you have data on one side, you have deployment on one side and you have SIEM sitting in this middle where learning in SIEM is smart, but there is a real gap that exists between transferring a policy that exists in SIEM versus in actual deployments. But it's the other way too. When something fails in deployments, it needs to be rerouted back to SIEM as well to make the simulation even better. And that's a piece that's missing. So it's not just that SIEM informs deployment, but it's also that deployments will inform how to make better SIEMs. I'm actually putting something out there about this later today or tomorrow. So hopefully it'll give you better context about it as well. Yeah. [00:09:48] S1: Yeah. We recently discussed with an engineer saying that simulation was still struggling to accurately represent actuators, for example, since robots are pretty much a bunch of actuators. It tells you a lot about how accurate sim can be. Right?
00:09:48 · Leo, Varun Nair
[00:09:48] S1: Yeah. We recently discussed with an engineer saying that simulation was still struggling to accurately represent actuators, for example, since robots are pretty much a bunch of actuators. It tells you a lot about how accurate sim can be. Right? [00:10:02] S2: Yeah. That's really fair. And the frontier of sim is getting better every other month. Nvidia, they swapped out their PhysX engine for Newton that is supposed to be doing deformables, fiction and contact dynamics better, so it will definitely improve in the next coming times. We have companies like V sim, the creators of Isaac Sim are now building their own simulation in The UK. It's a frontier that's definitely getting worked on. And it might also eat up the data business, to be very honest. If simulations work and are calibrated well enough, think about it this way. If I can model the real world in sim, why can't I just use sim as data? Like, it is totally possible. Like, these are, like, components that interact with each other but can also eat into each other very much so. [00:10:43] S1: Do you include world models such as World Labs Atlas into your simulation category?
00:10:43 · Leo, Varun Nair, Oskar
[00:10:43] S1: Do you include world models such as World Labs Atlas into your simulation category? [00:10:50] S2: That's a good question. And I might not be the best expert to talk about this, though I have spoken to some of them, but it's a spectrum. So you have things like Isaac Sim, MuJoCo on one side, which is like simulating every single object, and then you have world models on the other side. Both have their pros and cons. Have world models can like generate fluid, like fluidity better, so rather deformable objects better because it's pixel generation. So it is actually helpful to have these three d rendering like Atlas exist in world models and that is part of simulation. But when you want to do rigid contact dynamics and physics and understand the physics of it, world models aren't the best in understanding physics because it's, again, it's learned from pixels. It's not learned from action data. It hasn't learned the inherent physics behind it. So yeah, they are two separate things, but a good simulation needs to leverage both of them rather than using just one of them. [00:11:43] S0: I see. Maybe just to understand your role a bit better right now, you said you invested in a friend's company. Are you like creating a company yourself? Are you kind of stealthing around? If you have some practical experiences or some recent learnings that you can share from actual deployments, that will be super interesting as well.
00:12:01 · Varun Nair
[00:12:01] S2: For sure. So I'm mostly focused on just helping out the ecosystem as a whole, working at a company that's stealth currently, so can't tell you much about it at the moment. And just breaking down my learnings so far, it's that look at the data, guys. I think that's one of the biggest learnings I have realized in the last year and a half. It's that I created this open source project for this as well. There is no good definition of what good data looks like. Maybe in pre training you don't care about it because it's like, hey, just scale it up and throw it at a model to learn from. And that's good. It'll get you up to a certain point. But it's not enough. Pre training is one part of it. You want to be able to learn from multiple aspects. Let me take a step back. You guys have heard about Astra. Astra is making the rounds about how it's going disrupt the robotics field. I eval ed Astra to find out, and not just me and a lot of people in the industry did, it has great spatial and geometric understanding, but it has really bad sense of depth. So it cannot capture something really well. It can't touch upon let's say a glass, it'll touch right above it. It's because it has not been trained on depth data. It's as simple as that, you add depth into the precleaning mix Astra would perform wonderfully and it probably is something OpenAI, I'm sure, is trying to do at this point. It's about adding more modalities within data and actually learning from it. We can go deeper into these things as well. There are some data that are missing, people are not looking at quality of data, hence its models aren't doing that well. There are a couple of takes I have there. But yeah, broadly speaking, I think look at the data and understand where the failure modes of this data are and how you can sort of fix that is where a lot of this lies. You will see the same thing for post training as well. Deployment data will become post training data going forward.
00:13:44 · Oskar
[00:13:44] S0: So yeah, I would like to get some sense, especially for the beginners watching this, of the full data life cycle. So maybe you can just like go through the average life cycle from capturing towards deployment. How would that approximately look like?