The Robotics Stack
00:09:48 · Leo, Varun Nair
[00:09:48] S1: Yeah. We recently discussed with an engineer saying that simulation was still struggling to accurately represent actuators, for example, since robots are pretty much a bunch of actuators. It tells you a lot about how accurate sim can be. Right? [00:10:02] S2: Yeah. That's really fair. And the frontier of sim is getting better every other month. Nvidia, they swapped out their PhysX engine for Newton that is supposed to be doing deformables, fiction and contact dynamics better, so it will definitely improve in the next coming times. We have companies like V sim, the creators of Isaac Sim are now building their own simulation in The UK. It's a frontier that's definitely getting worked on. And it might also eat up the data business, to be very honest. If simulations work and are calibrated well enough, think about it this way. If I can model the real world in sim, why can't I just use sim as data? Like, it is totally possible. Like, these are, like, components that interact with each other but can also eat into each other very much so. [00:10:43] S1: Do you include world models such as World Labs Atlas into your simulation category?
00:10:43 · Leo, Varun Nair, Oskar
[00:10:43] S1: Do you include world models such as World Labs Atlas into your simulation category? [00:10:50] S2: That's a good question. And I might not be the best expert to talk about this, though I have spoken to some of them, but it's a spectrum. So you have things like Isaac Sim, MuJoCo on one side, which is like simulating every single object, and then you have world models on the other side. Both have their pros and cons. Have world models can like generate fluid, like fluidity better, so rather deformable objects better because it's pixel generation. So it is actually helpful to have these three d rendering like Atlas exist in world models and that is part of simulation. But when you want to do rigid contact dynamics and physics and understand the physics of it, world models aren't the best in understanding physics because it's, again, it's learned from pixels. It's not learned from action data. It hasn't learned the inherent physics behind it. So yeah, they are two separate things, but a good simulation needs to leverage both of them rather than using just one of them. [00:11:43] S0: I see. Maybe just to understand your role a bit better right now, you said you invested in a friend's company. Are you like creating a company yourself? Are you kind of stealthing around? If you have some practical experiences or some recent learnings that you can share from actual deployments, that will be super interesting as well.
00:12:01 · Varun Nair
[00:12:01] S2: For sure. So I'm mostly focused on just helping out the ecosystem as a whole, working at a company that's stealth currently, so can't tell you much about it at the moment. And just breaking down my learnings so far, it's that look at the data, guys. I think that's one of the biggest learnings I have realized in the last year and a half. It's that I created this open source project for this as well. There is no good definition of what good data looks like. Maybe in pre training you don't care about it because it's like, hey, just scale it up and throw it at a model to learn from. And that's good. It'll get you up to a certain point. But it's not enough. Pre training is one part of it. You want to be able to learn from multiple aspects. Let me take a step back. You guys have heard about Astra. Astra is making the rounds about how it's going disrupt the robotics field. I eval ed Astra to find out, and not just me and a lot of people in the industry did, it has great spatial and geometric understanding, but it has really bad sense of depth. So it cannot capture something really well. It can't touch upon let's say a glass, it'll touch right above it. It's because it has not been trained on depth data. It's as simple as that, you add depth into the precleaning mix Astra would perform wonderfully and it probably is something OpenAI, I'm sure, is trying to do at this point. It's about adding more modalities within data and actually learning from it. We can go deeper into these things as well. There are some data that are missing, people are not looking at quality of data, hence its models aren't doing that well. There are a couple of takes I have there. But yeah, broadly speaking, I think look at the data and understand where the failure modes of this data are and how you can sort of fix that is where a lot of this lies. You will see the same thing for post training as well. Deployment data will become post training data going forward.
00:13:44 · Oskar
[00:13:44] S0: So yeah, I would like to get some sense, especially for the beginners watching this, of the full data life cycle. So maybe you can just like go through the average life cycle from capturing towards deployment. How would that approximately look like?
00:13:59 · Varun Nair
[00:13:59] S2: Yeah, let's let's do that. The life cycle of what this data looks like from start to finish is you deploy cameras or any amount of sensors on people's heads, people's hands, and you go ahead and ask them to record a certain task, or ask them to go in the wild and just do some level of tasks, then these footages, whatever's collected, is then sort of like taken up by whether through an SD card or through other mechanisms depending on what the camera is, if it's streaming directly to some level of storage. And it's not trivial. You have to ingest all of this data and assume like petabytes and petabytes of data, if you're a good data vendor, is being ingested at some local facility somewhere. And all of that data then needs to go through its first QA process because it's not as simple as like, Hey, I recorded video and now I can sell it. It's very much that people have wasted time on it. People are fiddling with it, looking at the camera itself. People are like not doing tasks. There's no hands in frame. At a certain level, it's unless you're really tight with quality, it's useless at the very first step. So you have to have a very like tight quality check. Once you have that QA and you've ingested all of this data, it lives on some sort of a cloud storage or your local storage, however you're sort of storing it. It will then go through a level of annotations. Now these annotations could be a bunch of different things. And I want you to also like, not just like me telling you what it is, but sort of understand why we are doing this, whoever's watching this. If a robot is trying to learn how to move its hands, how to move its body, pixels are not enough. Pixels are just color values to it. It means nothing. You want it to see the body. You want it to see the hands. Those are the labels it's learning from. So it's important to have your 21 key hand points. It's important to have a full body pose so that the model can understand that is the movement. It's trying to understand the physics, it's trying to understand the grasp and beyond, like, let's say pose, full body pose, hand pose, have some action to text, which is like describing what's happening in the scene. There are a bunch of annotations we can go down under, but basically, like broadly, this is what would be done. And this is then will be used later for the model's input.
00:16:10 · Varun Nair, Oskar
Once you have this, these also need to go through a QA check. Like a model is coming up with a hand annotation, a body annotation, how do know it actually works? That goes through its own QA check. Some use human reviewers, some use models. And finally, you package all of this up. Foxglove has MCAP file formats, which is great. It can like store videos, JSON files, a bunch of different things, and you sort of deliver that into your client's SDK. That's the entire loop from like collection to ingestion, to annotation, to QA ing that, to finally making that delivery. And then they use that to feed into the model and learn from it. And if I may add really quickly, the biggest metric here is not just scale. It is scale, but it's also diversity because the model needs to see all the different kinds of actions that have happened for it to actually learn from it. So diversity is really important. [00:17:01] S0: Yeah. Would you say there's like a single kind of modality that has, from your experience, a very substantial impact? If I would have limited resources, I would rather focus on this than this, for example.