The Robotics Stack
00:17:14 · Varun Nair
[00:17:14] S2: So, I mean, obviously, like visual video is like the main modality for all of this. I would say the next best would be depth in current ways to make sure that you have precision a little better. Depth would be good. Stereo cameras are definitely great. I will point you out to one of my articles about where's the signal in robotics data. What it broadly basically touches upon is you can collect a lot without the modalities, and you can inject in the new modalities and find out. Like, let's say I collected hundred thousand hours without tactile data, I can still assign it tactile data very simply and OpenAI approved this in 2022. They had their VPT paper, video pre training paper, where they had seventy thousand hours of Minecraft gameplay that's unlabeled, so it did not have any annotation or any label. They asked people to come in and start gaming. Two thousand hours of labeled supervised data. What they did with this labeled supervised data was they trained a model that knows how to annotate now. The model isn't here to do robotics or play Minecraft. It knows that left keystroke and moving left is the same thing. Now it understands it from two thousand hours, and it was able to pseudo label seventy thousand hours of data with its actual gameplay labels it did not have. So it's pseudo labels, it's like proxy labels. You can do the same thing if you have a golden dataset for tactile, for depth, for intent, for motion capture, just collect a ten thousand hour dataset. Train a smaller model that's good at doing annotations, and then take all your open source data sets and just give it tactile. Give it give it mocap. That, in my opinion, is the right way to get modalities. You can't go and collect ten thousand, hundred thousand or a million hours of tactile data. You can. It's just so expensive. You're better off training a model. And that's how we did it with depth. In the past, we used to have like depth estimate. We used to have depth estimating models that would then look at monocular, like camera video and estimate depth. Now we have stereo cameras where we can get depth directly, but you can do the same with tactile and other camera, and we should be doing that in my opinion. So do I think there's an important modality? No. I think you can just add on modalities later on, but you need to, like, spend on getting that golden dataset.
00:19:27 · Varun Nair, Oskar, Leo
And I do think the best one's gonna be depth, tactile, and motion capture. [00:19:31] S0: Yeah. That was super interesting that they already did this, like, in 2020. [00:19:34] S2: They did. They just did it with Minecraft and not robotics. So I'm just... I'm like, guys, it's in your it's in your face. [00:19:39] S1: I have a question on that. You mentioned depth is the most important modality to acquire right now. But to me, it feels like every other data acquisition device or data acquisition company is already acquiring depth. Right? [00:19:53] S2: Yes. I meant depth is probably borrowed for Astra. Like, sorry, that's a caveat. Depth for a LLM is important because it doesn't understand what depth is. But you're right. Most data companies do have stereo cameras. I would say tactile is the next frontier for them. And I'm sure a lot of them are already working on tactile. [00:20:10] S1: Yeah. Recently I had an episode with Matt Wolf from Sixth Sense, who is like actively doing tactile data acquisition. [00:20:18] S2: Nice. I've heard some great things about Sixth Sense. Yeah. [00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data?
00:20:19 · Oskar, Varun Nair
[00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data? [00:20:33] S2: I mean, it's a black box, to be honest. It's not like I'm in the labs myself. Everything I'm saying is actually my views and my guesses. That being said, they're probably using it for pre training right now. It makes the most sense. We're not at a level where we can post train, Post training data looks very different from ego data in my opinion. Once you have your raw footage and you've annotated it, they're probably just spending it on pre training it on, seeing where it fails, what issues it has. I wouldn't be very helpful in understanding what labs are doing with it because it's just like you throw a bunch of compute, you throw a bunch of data, and you have your algo and you're like, all right, let's come up with something. I think a more interesting question is what happens once the labs have a model that works 80% of the times? And I think we move towards doing deployments, not great deployments, but some sort of deployments. And that is going to be a far more exciting and economical time for robotics as a whole industry. And it's going to just push us forward to be doing bigger and better things. So I do think deployment and deployment data is going to be far more valuable than ego data a year or two from now.
00:20:33 · Varun Nair, Leo
[00:20:33] S2: I mean, it's a black box, to be honest. It's not like I'm in the labs myself. Everything I'm saying is actually my views and my guesses. That being said, they're probably using it for pre training right now. It makes the most sense. We're not at a level where we can post train, Post training data looks very different from ego data in my opinion. Once you have your raw footage and you've annotated it, they're probably just spending it on pre training it on, seeing where it fails, what issues it has. I wouldn't be very helpful in understanding what labs are doing with it because it's just like you throw a bunch of compute, you throw a bunch of data, and you have your algo and you're like, all right, let's come up with something. I think a more interesting question is what happens once the labs have a model that works 80% of the times? And I think we move towards doing deployments, not great deployments, but some sort of deployments. And that is going to be a far more exciting and economical time for robotics as a whole industry. And it's going to just push us forward to be doing bigger and better things. So I do think deployment and deployment data is going to be far more valuable than ego data a year or two from now. [00:21:36] S1: I see. And deployment data is pretty much post training data you were mentioning a couple of seconds ago? [00:21:42] S2: Yeah, exactly. Deployment data is post training data.
00:21:42 · Varun Nair, Leo, Oskar
[00:21:42] S2: Yeah, exactly. Deployment data is post training data. [00:21:45] S1: Okay. So post training data basically right now is very lab and use case specific in the end, while pre training is very general. Everybody uses the same X billion hours of egocentric data, right? [00:21:57] S2: Sure. I'm sure some people put more or less signal. Some people do different kinds of supervision. Some have their own different dense annotation. Everyone has their own recipe that they think is going to work, that's why we have researchers at different companies doing different kinds of approaches. But yeah, I can't comment on what exactly they're building. I'm curious, what have you guys seen having had so many conversations with people around the data business that might actually disagree with me or even in general, any hot ticks that you've seen around the data space? I'm just curious. [00:22:26] S0: Yeah. I mean, I'm the Tillerop Maxer. I'm always saying, like, Tillerop is the absolute goaded way to go, and some people might say it's not scalable, you know. Leo is also, I think, in another camp, but that's something that I would want to see, especially if I look at new hardware that is coming out there. I think there's like a new UMI adaptation, which is much more specific, so not like a gripper, but more like a entire hand that goes into this direction. So I could see these type of hardware pieces actually helping make teleop more scalable. But that's just like my personal take.
00:22:26 · Oskar, Varun Nair
[00:22:26] S0: Yeah. I mean, I'm the Tillerop Maxer. I'm always saying, like, Tillerop is the absolute goaded way to go, and some people might say it's not scalable, you know. Leo is also, I think, in another camp, but that's something that I would want to see, especially if I look at new hardware that is coming out there. I think there's like a new UMI adaptation, which is much more specific, so not like a gripper, but more like a entire hand that goes into this direction. So I could see these type of hardware pieces actually helping make teleop more scalable. But that's just like my personal take. [00:23:00] S2: I actually agree with the take that like initial deployments are going to be teleoped. And I don't think it should be something to be ashamed of. I think people should embrace it. I think you should teleop your initial deployments. That's the smartest way to do it. Like, it'll help you set up a lot more infrastructure. So, I mean, your take about teleop is not unfounded and people always love to be like, it's not going to work. Yeah, but the whole point of this is that can you do things that don't scale? Can you actually make it work for the time being? And if you're able to, it'll be immensely helpful for learnings that'll help push the entire frontier forward. So I do think the right way is teleop, and I'm sure a lot of deployment companies would be doing teleop at this point. Yeah. [00:23:40] S0: Yeah. It's just a bit clunky. Right? But I think as technology evolves, I think it will get more smooth. It will be more easy.