00:39:41 · Michael Cho, Vincent Liu
[00:39:41] S0: I see. [00:39:41] S1: And I guess, you know, you have like your KD, whatever distortion coefficients, and those default to zero, which in the fisheye, you have high distortion. So we actually undistort the image, but even if you undistort these fisheye images, they don't look very, like, pinhole y. You know? They still look pretty warped. And so I feel like this might have been an issue. This might have been, like, a reason why, you know, when we try, you know, using, like, a binocular depth estimation model and we we ground it in, like, a ArUco tags in the scene or even master slam here, maybe these are reasons why, like, those methods don't work so well because the camera is, like, very not... It's, like, very distorted. Right? But but also, like, I guess, another aspect of the fisheye lens is that it's actually very nice because it minimizes occlusion. So Mhmm. You know, like, as I'm grabbing the fruit here, you're pretty... It's pretty much not occluded until you actually cover it with your hand. And, again, I mentioned, like, Ademina had tried setup close... Similar to DexWild where we used the iPhone, a chest mounted iPhone, and also because it's lower, but also because it's a linear kind of pinhole. One of the reasons we think it didn't work well is because as soon as you reached out to it, you would occlude the entire scene. And so, like, the fisheye is kinda nice in this way, but also maybe it lends itself to, like, a difficult time extracting depth.
00:40:56 · Michael Cho, Vincent Liu
[00:40:56] S0: Yeah. I'm I'm trying to think what what what do you think will be the end state of scaling something like this? Like, practically. Like, give a bunch of these kind of glasses to a bunch of people and just ask them to do, like, just do stuff in the wild. Basically, do stuff in the They don't even, like, because if it's it's meant to be so natural anyway. So even if the the... Like, there's dirtier data that come back up because you you probably get so much data, you should be able to get, like, you know, good good signal data from. Because I do agree. I mean, having text while and all that. So we actually printed text while... I I haven't seen it, but I haven't tried myself, but my team may have tried. It's... Yeah. Just fitting it on is not very natural. I I think the the the s hand version, like, you feel like a robot. Right? And and definitely, there's stuff that you can't do. Way worse than a menace glove, I would say. [00:41:57] S1: So the question was, like [00:42:00] S0: Yeah. I mean, how how do you think that this? Yeah. Yeah. How do you think this will scale? Right? I mean, do you think it's it's just as simple as giving, you know, a bunch of these glasses out to, you know, literally random folks and just record their daily life? Right. And you you take the slam info that you can get from the glasses, and that could be a much better way to to scale this compared to TeleOp and compared to, like, DexWild, you know, DexUMI kind of thing.
00:42:29 · Ademi Adeniji
[00:42:29] S2: Yeah. I mean, I'm happy to take a stab at that. So I think there's an interesting margin or boundary between what the capabilities of our learning systems are. And then like, what are the sort of real world incentive structures and sort of like extrinsic or externals that would lend themselves to generating these datasets? Right. So that could be like, for instance, researchers come together and say, we're going to do an open source initiative. We're going to give people lots of these classes. We're going to have like this pool dataset, sort of like what we saw with TeleOp. And then we're going to basically like establish a benchmark so that people can train models. And then the best model is sort of like, distributed around and like people try and improve this over time. Like, I think that that would be great. Right. And I think like folks are already working on this today. There's folks at Meta working on this. There's folks like, I'm in academia that are actually working on these sorts of things. I think like there's also interesting sort of commercial legends that you can take on this where, you know, the, the ask for someone to wear these glasses is like very low, right? Compared to maybe like more traditional approaches or even like the last generation of like sort of robots in the world where they had to be pre programmed by someone. Someone had to know how to program a robot to get them to be useful. Now it's, Hey, do your normal job or do whatever you were doing before, but what are these classes? And then everything else after that is sort of like a data driven system. So I think if you can tie data collection to some application, some incentive where people are maybe going to be able to just do what they normally do and have that data feedback and improve aspects of what they're already doing via robots that are operating in amongst them and learning from them. I think that's also a virtuous sort of data flywheel that you can build. Right. And your incentives are there. They are aligned now. I think there's a lot of different ways you can come at it. But I think our job as researchers here was to paint this picture of what you can do at the limit of ergonomics. And we're hoping to create more buy in from the rest of the research community around, like, this particular form factor for for collecting.
00:44:51 · Michael Cho, Vincent Liu
[00:44:51] S0: Got it. I'm curious, like... I mean, I've never tried the Aria glasses. It may... Does look like there's a big box here. I'm sure the future versions are gonna improve, but how many hours... Can you wear it the whole day? Is it is it, like, similar to the Ray Ban kind of glasses, or they're still kind of heavy? [00:45:05] S1: I wanna say they ran out of battery after recording after maybe a couple hours. [00:45:10] S0: Oh, so so that's the limitation to the battery. I see. [00:45:13] S1: Yeah. I mean, we recorded, like, sections of data much much shorter than that, but I think it was something... It wasn't like a whole day for sure. [00:45:19] S0: Got it. But it's... At least from... In terms of ergonomics, it was never a bottleneck. Right? You you never feel like... It's not a wearing FIAL glasses like a Quest, you know? After a hundred hours, you need to take it out. Okay. Okay. Cool. [00:45:30] S1: They actually feel like just normal glasses. They're like little... The frames are thicker, but they just feel like normal glasses. That's awesome. Cool. [00:45:37] S0: Yeah. I was
00:45:37 · Michael Cho, Vincent Liu
[00:45:37] S0: Yeah. I was [00:45:37] S1: also gonna add maybe to a Demi's thing, like, emphasis on the alignment of incentives. But I think like a cool, like one way to do that in a, I guess, in between both options that Demi mentioned would be maybe if you had, if there was a world in which glasses were like the intermediate embodiment of AI and you know, you could wear them and you have like your it's like the Ray Ban thing where you have like assistants in there and stuff. And so while people are wearing them and recording things, like they would also get utility out of them. I think that'd be a pretty difficult, probably cultural trend that you'd have to crack. But then everyone be wearing them. Right?
00:46:11 · Michael Cho
[00:46:11] S0: Yeah. I mean, you know, I I live in Singapore, so so Ray Ban... The metal Ray Ban is not so yet, but I I have a two metal Ray Ban because I I love the idea of so much, but they actually haven't turned on the Meta AI in Singapore yet. But when I'm US, I I will use it. Right? Like, it's it's... I find it just so useful to have a assistant Mhmm. And then, you know, have a camera can see what I'm saying. And so I I think that's... Mean, I whether it's meta and all this, they will all have some kind of form factor that's basically cameras on the head. So it's it's just so easy. Like, I find that taking off the phone is just too difficult. It's too too troublesome. Maybe it's first world problem, the fact that I have a decent phone. But, I I mean, the ergonomics is just so easy. I just speak and then... And I ask, and I think have they... Having the utility just will will get a lot more people to wear these kind of smart glasses. And once society accept, okay, you are just by default turning on the record anyway to let your AI see, so must must have just press record and then contribute to it, to a new dataset. So I can totally see that kind of future. Yeah. Then maybe you don't even need to incentivize, like, pay people to do it. Like, you know people would just want to have a device on their head for utility, you know, reasons.