The Robotics Stack
00:16:10 · Varun Nair, Oskar
Once you have this, these also need to go through a QA check. Like a model is coming up with a hand annotation, a body annotation, how do know it actually works? That goes through its own QA check. Some use human reviewers, some use models. And finally, you package all of this up. Foxglove has MCAP file formats, which is great. It can like store videos, JSON files, a bunch of different things, and you sort of deliver that into your client's SDK. That's the entire loop from like collection to ingestion, to annotation, to QA ing that, to finally making that delivery. And then they use that to feed into the model and learn from it. And if I may add really quickly, the biggest metric here is not just scale. It is scale, but it's also diversity because the model needs to see all the different kinds of actions that have happened for it to actually learn from it. So diversity is really important. [00:17:01] S0: Yeah. Would you say there's like a single kind of modality that has, from your experience, a very substantial impact? If I would have limited resources, I would rather focus on this than this, for example.
00:17:14 · Varun Nair
[00:17:14] S2: So, I mean, obviously, like visual video is like the main modality for all of this. I would say the next best would be depth in current ways to make sure that you have precision a little better. Depth would be good. Stereo cameras are definitely great. I will point you out to one of my articles about where's the signal in robotics data. What it broadly basically touches upon is you can collect a lot without the modalities, and you can inject in the new modalities and find out. Like, let's say I collected hundred thousand hours without tactile data, I can still assign it tactile data very simply and OpenAI approved this in 2022. They had their VPT paper, video pre training paper, where they had seventy thousand hours of Minecraft gameplay that's unlabeled, so it did not have any annotation or any label. They asked people to come in and start gaming. Two thousand hours of labeled supervised data. What they did with this labeled supervised data was they trained a model that knows how to annotate now. The model isn't here to do robotics or play Minecraft. It knows that left keystroke and moving left is the same thing. Now it understands it from two thousand hours, and it was able to pseudo label seventy thousand hours of data with its actual gameplay labels it did not have. So it's pseudo labels, it's like proxy labels. You can do the same thing if you have a golden dataset for tactile, for depth, for intent, for motion capture, just collect a ten thousand hour dataset. Train a smaller model that's good at doing annotations, and then take all your open source data sets and just give it tactile. Give it give it mocap. That, in my opinion, is the right way to get modalities. You can't go and collect ten thousand, hundred thousand or a million hours of tactile data. You can. It's just so expensive. You're better off training a model. And that's how we did it with depth. In the past, we used to have like depth estimate. We used to have depth estimating models that would then look at monocular, like camera video and estimate depth. Now we have stereo cameras where we can get depth directly, but you can do the same with tactile and other camera, and we should be doing that in my opinion. So do I think there's an important modality? No. I think you can just add on modalities later on, but you need to, like, spend on getting that golden dataset.
00:19:27 · Varun Nair, Oskar, Leo
And I do think the best one's gonna be depth, tactile, and motion capture. [00:19:31] S0: Yeah. That was super interesting that they already did this, like, in 2020. [00:19:34] S2: They did. They just did it with Minecraft and not robotics. So I'm just... I'm like, guys, it's in your it's in your face. [00:19:39] S1: I have a question on that. You mentioned depth is the most important modality to acquire right now. But to me, it feels like every other data acquisition device or data acquisition company is already acquiring depth. Right? [00:19:53] S2: Yes. I meant depth is probably borrowed for Astra. Like, sorry, that's a caveat. Depth for a LLM is important because it doesn't understand what depth is. But you're right. Most data companies do have stereo cameras. I would say tactile is the next frontier for them. And I'm sure a lot of them are already working on tactile. [00:20:10] S1: Yeah. Recently I had an episode with Matt Wolf from Sixth Sense, who is like actively doing tactile data acquisition. [00:20:18] S2: Nice. I've heard some great things about Sixth Sense. Yeah. [00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data?
00:20:19 · Oskar, Varun Nair
[00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data? [00:20:33] S2: I mean, it's a black box, to be honest. It's not like I'm in the labs myself. Everything I'm saying is actually my views and my guesses. That being said, they're probably using it for pre training right now. It makes the most sense. We're not at a level where we can post train, Post training data looks very different from ego data in my opinion. Once you have your raw footage and you've annotated it, they're probably just spending it on pre training it on, seeing where it fails, what issues it has. I wouldn't be very helpful in understanding what labs are doing with it because it's just like you throw a bunch of compute, you throw a bunch of data, and you have your algo and you're like, all right, let's come up with something. I think a more interesting question is what happens once the labs have a model that works 80% of the times? And I think we move towards doing deployments, not great deployments, but some sort of deployments. And that is going to be a far more exciting and economical time for robotics as a whole industry. And it's going to just push us forward to be doing bigger and better things. So I do think deployment and deployment data is going to be far more valuable than ego data a year or two from now.
00:20:33 · Varun Nair, Leo
[00:20:33] S2: I mean, it's a black box, to be honest. It's not like I'm in the labs myself. Everything I'm saying is actually my views and my guesses. That being said, they're probably using it for pre training right now. It makes the most sense. We're not at a level where we can post train, Post training data looks very different from ego data in my opinion. Once you have your raw footage and you've annotated it, they're probably just spending it on pre training it on, seeing where it fails, what issues it has. I wouldn't be very helpful in understanding what labs are doing with it because it's just like you throw a bunch of compute, you throw a bunch of data, and you have your algo and you're like, all right, let's come up with something. I think a more interesting question is what happens once the labs have a model that works 80% of the times? And I think we move towards doing deployments, not great deployments, but some sort of deployments. And that is going to be a far more exciting and economical time for robotics as a whole industry. And it's going to just push us forward to be doing bigger and better things. So I do think deployment and deployment data is going to be far more valuable than ego data a year or two from now. [00:21:36] S1: I see. And deployment data is pretty much post training data you were mentioning a couple of seconds ago? [00:21:42] S2: Yeah, exactly. Deployment data is post training data.
00:21:42 · Varun Nair, Leo, Oskar
[00:21:42] S2: Yeah, exactly. Deployment data is post training data. [00:21:45] S1: Okay. So post training data basically right now is very lab and use case specific in the end, while pre training is very general. Everybody uses the same X billion hours of egocentric data, right? [00:21:57] S2: Sure. I'm sure some people put more or less signal. Some people do different kinds of supervision. Some have their own different dense annotation. Everyone has their own recipe that they think is going to work, that's why we have researchers at different companies doing different kinds of approaches. But yeah, I can't comment on what exactly they're building. I'm curious, what have you guys seen having had so many conversations with people around the data business that might actually disagree with me or even in general, any hot ticks that you've seen around the data space? I'm just curious. [00:22:26] S0: Yeah. I mean, I'm the Tillerop Maxer. I'm always saying, like, Tillerop is the absolute goaded way to go, and some people might say it's not scalable, you know. Leo is also, I think, in another camp, but that's something that I would want to see, especially if I look at new hardware that is coming out there. I think there's like a new UMI adaptation, which is much more specific, so not like a gripper, but more like a entire hand that goes into this direction. So I could see these type of hardware pieces actually helping make teleop more scalable. But that's just like my personal take.