The Robotics Stack
00:19:27 · Varun Nair, Oskar, Leo
And I do think the best one's gonna be depth, tactile, and motion capture. [00:19:31] S0: Yeah. That was super interesting that they already did this, like, in 2020. [00:19:34] S2: They did. They just did it with Minecraft and not robotics. So I'm just... I'm like, guys, it's in your it's in your face. [00:19:39] S1: I have a question on that. You mentioned depth is the most important modality to acquire right now. But to me, it feels like every other data acquisition device or data acquisition company is already acquiring depth. Right? [00:19:53] S2: Yes. I meant depth is probably borrowed for Astra. Like, sorry, that's a caveat. Depth for a LLM is important because it doesn't understand what depth is. But you're right. Most data companies do have stereo cameras. I would say tactile is the next frontier for them. And I'm sure a lot of them are already working on tactile. [00:20:10] S1: Yeah. Recently I had an episode with Matt Wolf from Sixth Sense, who is like actively doing tactile data acquisition. [00:20:18] S2: Nice. I've heard some great things about Sixth Sense. Yeah. [00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data?
00:20:19 · Oskar, Varun Nair
[00:20:19] S0: Could we expand this life cycle of data a little bit more towards the lab side, what they're doing with it? Are we in bubble territory there? They are acquiring a lot of data right now. How do you see that? How are they working with the data? [00:20:33] S2: I mean, it's a black box, to be honest. It's not like I'm in the labs myself. Everything I'm saying is actually my views and my guesses. That being said, they're probably using it for pre training right now. It makes the most sense. We're not at a level where we can post train, Post training data looks very different from ego data in my opinion. Once you have your raw footage and you've annotated it, they're probably just spending it on pre training it on, seeing where it fails, what issues it has. I wouldn't be very helpful in understanding what labs are doing with it because it's just like you throw a bunch of compute, you throw a bunch of data, and you have your algo and you're like, all right, let's come up with something. I think a more interesting question is what happens once the labs have a model that works 80% of the times? And I think we move towards doing deployments, not great deployments, but some sort of deployments. And that is going to be a far more exciting and economical time for robotics as a whole industry. And it's going to just push us forward to be doing bigger and better things. So I do think deployment and deployment data is going to be far more valuable than ego data a year or two from now.
00:20:33 · Varun Nair, Leo
[00:20:33] S2: I mean, it's a black box, to be honest. It's not like I'm in the labs myself. Everything I'm saying is actually my views and my guesses. That being said, they're probably using it for pre training right now. It makes the most sense. We're not at a level where we can post train, Post training data looks very different from ego data in my opinion. Once you have your raw footage and you've annotated it, they're probably just spending it on pre training it on, seeing where it fails, what issues it has. I wouldn't be very helpful in understanding what labs are doing with it because it's just like you throw a bunch of compute, you throw a bunch of data, and you have your algo and you're like, all right, let's come up with something. I think a more interesting question is what happens once the labs have a model that works 80% of the times? And I think we move towards doing deployments, not great deployments, but some sort of deployments. And that is going to be a far more exciting and economical time for robotics as a whole industry. And it's going to just push us forward to be doing bigger and better things. So I do think deployment and deployment data is going to be far more valuable than ego data a year or two from now. [00:21:36] S1: I see. And deployment data is pretty much post training data you were mentioning a couple of seconds ago? [00:21:42] S2: Yeah, exactly. Deployment data is post training data.
00:21:42 · Varun Nair, Leo, Oskar
[00:21:42] S2: Yeah, exactly. Deployment data is post training data. [00:21:45] S1: Okay. So post training data basically right now is very lab and use case specific in the end, while pre training is very general. Everybody uses the same X billion hours of egocentric data, right? [00:21:57] S2: Sure. I'm sure some people put more or less signal. Some people do different kinds of supervision. Some have their own different dense annotation. Everyone has their own recipe that they think is going to work, that's why we have researchers at different companies doing different kinds of approaches. But yeah, I can't comment on what exactly they're building. I'm curious, what have you guys seen having had so many conversations with people around the data business that might actually disagree with me or even in general, any hot ticks that you've seen around the data space? I'm just curious. [00:22:26] S0: Yeah. I mean, I'm the Tillerop Maxer. I'm always saying, like, Tillerop is the absolute goaded way to go, and some people might say it's not scalable, you know. Leo is also, I think, in another camp, but that's something that I would want to see, especially if I look at new hardware that is coming out there. I think there's like a new UMI adaptation, which is much more specific, so not like a gripper, but more like a entire hand that goes into this direction. So I could see these type of hardware pieces actually helping make teleop more scalable. But that's just like my personal take.
00:22:26 · Oskar, Varun Nair
[00:22:26] S0: Yeah. I mean, I'm the Tillerop Maxer. I'm always saying, like, Tillerop is the absolute goaded way to go, and some people might say it's not scalable, you know. Leo is also, I think, in another camp, but that's something that I would want to see, especially if I look at new hardware that is coming out there. I think there's like a new UMI adaptation, which is much more specific, so not like a gripper, but more like a entire hand that goes into this direction. So I could see these type of hardware pieces actually helping make teleop more scalable. But that's just like my personal take. [00:23:00] S2: I actually agree with the take that like initial deployments are going to be teleoped. And I don't think it should be something to be ashamed of. I think people should embrace it. I think you should teleop your initial deployments. That's the smartest way to do it. Like, it'll help you set up a lot more infrastructure. So, I mean, your take about teleop is not unfounded and people always love to be like, it's not going to work. Yeah, but the whole point of this is that can you do things that don't scale? Can you actually make it work for the time being? And if you're able to, it'll be immensely helpful for learnings that'll help push the entire frontier forward. So I do think the right way is teleop, and I'm sure a lot of deployment companies would be doing teleop at this point. Yeah. [00:23:40] S0: Yeah. It's just a bit clunky. Right? But I think as technology evolves, I think it will get more smooth. It will be more easy.
00:23:40 · Oskar, Varun Nair
[00:23:40] S0: Yeah. It's just a bit clunky. Right? But I think as technology evolves, I think it will get more smooth. It will be more easy. [00:23:47] S2: The world of Atoms is just... It's not easy as software. It's not as scalable. It breaks for random reasons, and it's weird. And people need to realize that the frontier is like we had nice exponential scale up and it's going to take like time. There are so many companies that were doing hard work that had to iterate a bazillion times before it like worked. There's an investor I follow on Twitter who mentioned this and it's like there's patient capital and there's impatient capital and I feel a lot of impatient capital is following robotics and it's usually the same SaaS software investors who are impatient with their capital looking for quick returns and I don't think robotics is something that's going to be kind to the impatient capital. It will be extremely, extremely helpful for the patient capital because you have to see what works in the long run. And we all know that physically AI and robotics will be the next step function of mankind as a whole. So it's a matter of time rather than a matter of like whether or not it will happen. [00:24:43] S0: We see a lot more benchmarks now with robotics that have been kind of difficult to do, of course, because it's not just an LLM. How do you see these ideas from AI move over towards robotics? Like benchmarks, for example.