The Robotics Stack
00:13:44 · Oskar
[00:13:44] S0: So yeah, I would like to get some sense, especially for the beginners watching this, of the full data life cycle. So maybe you can just like go through the average life cycle from capturing towards deployment. How would that approximately look like?
00:13:59 · Varun Nair
[00:13:59] S2: Yeah, let's let's do that. The life cycle of what this data looks like from start to finish is you deploy cameras or any amount of sensors on people's heads, people's hands, and you go ahead and ask them to record a certain task, or ask them to go in the wild and just do some level of tasks, then these footages, whatever's collected, is then sort of like taken up by whether through an SD card or through other mechanisms depending on what the camera is, if it's streaming directly to some level of storage. And it's not trivial. You have to ingest all of this data and assume like petabytes and petabytes of data, if you're a good data vendor, is being ingested at some local facility somewhere. And all of that data then needs to go through its first QA process because it's not as simple as like, Hey, I recorded video and now I can sell it. It's very much that people have wasted time on it. People are fiddling with it, looking at the camera itself. People are like not doing tasks. There's no hands in frame. At a certain level, it's unless you're really tight with quality, it's useless at the very first step. So you have to have a very like tight quality check. Once you have that QA and you've ingested all of this data, it lives on some sort of a cloud storage or your local storage, however you're sort of storing it. It will then go through a level of annotations. Now these annotations could be a bunch of different things. And I want you to also like, not just like me telling you what it is, but sort of understand why we are doing this, whoever's watching this. If a robot is trying to learn how to move its hands, how to move its body, pixels are not enough. Pixels are just color values to it. It means nothing. You want it to see the body. You want it to see the hands. Those are the labels it's learning from. So it's important to have your 21 key hand points. It's important to have a full body pose so that the model can understand that is the movement. It's trying to understand the physics, it's trying to understand the grasp and beyond, like, let's say pose, full body pose, hand pose, have some action to text, which is like describing what's happening in the scene. There are a bunch of annotations we can go down under, but basically, like broadly, this is what would be done. And this is then will be used later for the model's input.
00:16:10 · Varun Nair, Oskar
Once you have this, these also need to go through a QA check. Like a model is coming up with a hand annotation, a body annotation, how do know it actually works? That goes through its own QA check. Some use human reviewers, some use models. And finally, you package all of this up. Foxglove has MCAP file formats, which is great. It can like store videos, JSON files, a bunch of different things, and you sort of deliver that into your client's SDK. That's the entire loop from like collection to ingestion, to annotation, to QA ing that, to finally making that delivery. And then they use that to feed into the model and learn from it. And if I may add really quickly, the biggest metric here is not just scale. It is scale, but it's also diversity because the model needs to see all the different kinds of actions that have happened for it to actually learn from it. So diversity is really important. [00:17:01] S0: Yeah. Would you say there's like a single kind of modality that has, from your experience, a very substantial impact? If I would have limited resources, I would rather focus on this than this, for example.
00:17:14 · Varun Nair
[00:17:14] S2: So, I mean, obviously, like visual video is like the main modality for all of this. I would say the next best would be depth in current ways to make sure that you have precision a little better. Depth would be good. Stereo cameras are definitely great. I will point you out to one of my articles about where's the signal in robotics data. What it broadly basically touches upon is you can collect a lot without the modalities, and you can inject in the new modalities and find out. Like, let's say I collected hundred thousand hours without tactile data, I can still assign it tactile data very simply and OpenAI approved this in 2022. They had their VPT paper, video pre training paper, where they had seventy thousand hours of Minecraft gameplay that's unlabeled, so it did not have any annotation or any label. They asked people to come in and start gaming. Two thousand hours of labeled supervised data. What they did with this labeled supervised data was they trained a model that knows how to annotate now. The model isn't here to do robotics or play Minecraft. It knows that left keystroke and moving left is the same thing. Now it understands it from two thousand hours, and it was able to pseudo label seventy thousand hours of data with its actual gameplay labels it did not have. So it's pseudo labels, it's like proxy labels. You can do the same thing if you have a golden dataset for tactile, for depth, for intent, for motion capture, just collect a ten thousand hour dataset. Train a smaller model that's good at doing annotations, and then take all your open source data sets and just give it tactile. Give it give it mocap. That, in my opinion, is the right way to get modalities. You can't go and collect ten thousand, hundred thousand or a million hours of tactile data. You can. It's just so expensive. You're better off training a model. And that's how we did it with depth. In the past, we used to have like depth estimate. We used to have depth estimating models that would then look at monocular, like camera video and estimate depth. Now we have stereo cameras where we can get depth directly, but you can do the same with tactile and other camera, and we should be doing that in my opinion. So do I think there's an important modality? No. I think you can just add on modalities later on, but you need to, like, spend on getting that golden dataset.
00:19:27 · Varun Nair, Oskar, Leo
And I do think the best one's gonna be depth, tactile, and motion capture. [00:19:31] S0: Yeah. That was super interesting that they already did this, like, in 2020. [00:19:34] S2: They did. They just did it with Minecraft and not robotics. So I'm just... I'm like, guys, it's in your it's in your face. [00:19:39] S1: I have a question on that. You mentioned depth is the most important modality to acquire right now. But to me, it feels like every other data acquisition device or data acquisition company is already acquiring depth. Right? [00:19:53] S2: Yes. I meant depth is probably borrowed for Astra. Like, sorry, that's a caveat. Depth for a LLM is important because it doesn't understand what depth is. But you're right. Most data companies do have stereo cameras. I would say tactile is the next frontier for them. And I'm sure a lot of them are already working on tactile. [00:20:10] S1: Yeah. Recently I had an episode with Matt Wolf from Sixth Sense, who is like actively doing tactile data acquisition. [00:20:18] S2: Nice. I've heard some great things about Sixth Sense. Yeah. [00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data?
00:20:19 · Oskar, Varun Nair
[00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data? [00:20:33] S2: I mean, it's a black box, to be honest. It's not like I'm in the labs myself. Everything I'm saying is actually my views and my guesses. That being said, they're probably using it for pre training right now. It makes the most sense. We're not at a level where we can post train, Post training data looks very different from ego data in my opinion. Once you have your raw footage and you've annotated it, they're probably just spending it on pre training it on, seeing where it fails, what issues it has. I wouldn't be very helpful in understanding what labs are doing with it because it's just like you throw a bunch of compute, you throw a bunch of data, and you have your algo and you're like, all right, let's come up with something. I think a more interesting question is what happens once the labs have a model that works 80% of the times? And I think we move towards doing deployments, not great deployments, but some sort of deployments. And that is going to be a far more exciting and economical time for robotics as a whole industry. And it's going to just push us forward to be doing bigger and better things. So I do think deployment and deployment data is going to be far more valuable than ego data a year or two from now.