The Robotics Stack
00:12:01 · Varun Nair
[00:12:01] S2: For sure. So I'm mostly focused on just helping out the ecosystem as a whole, working at a company that's stealth currently, so can't tell you much about it at the moment. And just breaking down my learnings so far, it's that look at the data, guys. I think that's one of the biggest learnings I have realized in the last year and a half. It's that I created this open source project for this as well. There is no good definition of what good data looks like. Maybe in pre training you don't care about it because it's like, hey, just scale it up and throw it at a model to learn from. And that's good. It'll get you up to a certain point. But it's not enough. Pre training is one part of it. You want to be able to learn from multiple aspects. Let me take a step back. You guys have heard about Astra. Astra is making the rounds about how it's going disrupt the robotics field. I eval ed Astra to find out, and not just me and a lot of people in the industry did, it has great spatial and geometric understanding, but it has really bad sense of depth. So it cannot capture something really well. It can't touch upon let's say a glass, it'll touch right above it. It's because it has not been trained on depth data. It's as simple as that, you add depth into the precleaning mix Astra would perform wonderfully and it probably is something OpenAI, I'm sure, is trying to do at this point. It's about adding more modalities within data and actually learning from it. We can go deeper into these things as well. There are some data that are missing, people are not looking at quality of data, hence its models aren't doing that well. There are a couple of takes I have there. But yeah, broadly speaking, I think look at the data and understand where the failure modes of this data are and how you can sort of fix that is where a lot of this lies. You will see the same thing for post training as well. Deployment data will become post training data going forward.
00:13:44 · Oskar
[00:13:44] S0: So yeah, I would like to get some sense, especially for the beginners watching this, of the full data life cycle. So maybe you can just like go through the average life cycle from capturing towards deployment. How would that approximately look like?
00:13:59 · Varun Nair
[00:13:59] S2: Yeah, let's let's do that. The life cycle of what this data looks like from start to finish is you deploy cameras or any amount of sensors on people's heads, people's hands, and you go ahead and ask them to record a certain task, or ask them to go in the wild and just do some level of tasks, then these footages, whatever's collected, is then sort of like taken up by whether through an SD card or through other mechanisms depending on what the camera is, if it's streaming directly to some level of storage. And it's not trivial. You have to ingest all of this data and assume like petabytes and petabytes of data, if you're a good data vendor, is being ingested at some local facility somewhere. And all of that data then needs to go through its first QA process because it's not as simple as like, Hey, I recorded video and now I can sell it. It's very much that people have wasted time on it. People are fiddling with it, looking at the camera itself. People are like not doing tasks. There's no hands in frame. At a certain level, it's unless you're really tight with quality, it's useless at the very first step. So you have to have a very like tight quality check. Once you have that QA and you've ingested all of this data, it lives on some sort of a cloud storage or your local storage, however you're sort of storing it. It will then go through a level of annotations. Now these annotations could be a bunch of different things. And I want you to also like, not just like me telling you what it is, but sort of understand why we are doing this, whoever's watching this. If a robot is trying to learn how to move its hands, how to move its body, pixels are not enough. Pixels are just color values to it. It means nothing. You want it to see the body. You want it to see the hands. Those are the labels it's learning from. So it's important to have your 21 key hand points. It's important to have a full body pose so that the model can understand that is the movement. It's trying to understand the physics, it's trying to understand the grasp and beyond, like, let's say pose, full body pose, hand pose, have some action to text, which is like describing what's happening in the scene. There are a bunch of annotations we can go down under, but basically, like broadly, this is what would be done. And this is then will be used later for the model's input.
00:16:10 · Varun Nair, Oskar
Once you have this, these also need to go through a QA check. Like a model is coming up with a hand annotation, a body annotation, how do know it actually works? That goes through its own QA check. Some use human reviewers, some use models. And finally, you package all of this up. Foxglove has MCAP file formats, which is great. It can like store videos, JSON files, a bunch of different things, and you sort of deliver that into your client's SDK. That's the entire loop from like collection to ingestion, to annotation, to QA ing that, to finally making that delivery. And then they use that to feed into the model and learn from it. And if I may add really quickly, the biggest metric here is not just scale. It is scale, but it's also diversity because the model needs to see all the different kinds of actions that have happened for it to actually learn from it. So diversity is really important. [00:17:01] S0: Yeah. Would you say there's like a single kind of modality that has, from your experience, a very substantial impact? If I would have limited resources, I would rather focus on this than this, for example.
00:17:14 · Varun Nair
[00:17:14] S2: So, I mean, obviously, like visual video is like the main modality for all of this. I would say the next best would be depth in current ways to make sure that you have precision a little better. Depth would be good. Stereo cameras are definitely great. I will point you out to one of my articles about where's the signal in robotics data. What it broadly basically touches upon is you can collect a lot without the modalities, and you can inject in the new modalities and find out. Like, let's say I collected hundred thousand hours without tactile data, I can still assign it tactile data very simply and OpenAI approved this in 2022. They had their VPT paper, video pre training paper, where they had seventy thousand hours of Minecraft gameplay that's unlabeled, so it did not have any annotation or any label. They asked people to come in and start gaming. Two thousand hours of labeled supervised data. What they did with this labeled supervised data was they trained a model that knows how to annotate now. The model isn't here to do robotics or play Minecraft. It knows that left keystroke and moving left is the same thing. Now it understands it from two thousand hours, and it was able to pseudo label seventy thousand hours of data with its actual gameplay labels it did not have. So it's pseudo labels, it's like proxy labels. You can do the same thing if you have a golden dataset for tactile, for depth, for intent, for motion capture, just collect a ten thousand hour dataset. Train a smaller model that's good at doing annotations, and then take all your open source data sets and just give it tactile. Give it give it mocap. That, in my opinion, is the right way to get modalities. You can't go and collect ten thousand, hundred thousand or a million hours of tactile data. You can. It's just so expensive. You're better off training a model. And that's how we did it with depth. In the past, we used to have like depth estimate. We used to have depth estimating models that would then look at monocular, like camera video and estimate depth. Now we have stereo cameras where we can get depth directly, but you can do the same with tactile and other camera, and we should be doing that in my opinion. So do I think there's an important modality? No. I think you can just add on modalities later on, but you need to, like, spend on getting that golden dataset.
00:19:27 · Varun Nair, Oskar, Leo
And I do think the best one's gonna be depth, tactile, and motion capture. [00:19:31] S0: Yeah. That was super interesting that they already did this, like, in 2020. [00:19:34] S2: They did. They just did it with Minecraft and not robotics. So I'm just... I'm like, guys, it's in your it's in your face. [00:19:39] S1: I have a question on that. You mentioned depth is the most important modality to acquire right now. But to me, it feels like every other data acquisition device or data acquisition company is already acquiring depth. Right? [00:19:53] S2: Yes. I meant depth is probably borrowed for Astra. Like, sorry, that's a caveat. Depth for a LLM is important because it doesn't understand what depth is. But you're right. Most data companies do have stereo cameras. I would say tactile is the next frontier for them. And I'm sure a lot of them are already working on tactile. [00:20:10] S1: Yeah. Recently I had an episode with Matt Wolf from Sixth Sense, who is like actively doing tactile data acquisition. [00:20:18] S2: Nice. I've heard some great things about Sixth Sense. Yeah. [00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data?