00:57:01 · Lawrence Zhu, Chris Paxton, Michael Cho
[00:57:01] S1: one to three. Wait. Wait. You mean battery. Right? [00:57:03] S0: Yeah. Battery. Like, mean, if you [00:57:05] S1: press record, how long does it go for? How long does it go? So continuous rec... We don't we don't do continuous recording because that will make the data has to like, super large. Right? When you process it, it will go, like, out of RAM... Memory. So we do, like, three... We usually do, like, three, four minutes per episode. And for EMMA, we did, like, think, five minute episode per demo, and then we'll just rerecord. But the data will last around... Not the data. Sorry. Like, the the glasses battery will last around one hour for, like, continuous data recording. And we recently invented something called data stations. Basically, we have a power station and a chair, and then we have some... We have, like, a power cable plugged into the glasses so you can collect the infinite data. [00:57:45] S0: Yeah. Yeah. This is this is how humans become cyborg. I I tell you. You can see those smart smart glasses, man. Let's plug in more. [00:57:52] S3: You know? You know? Yeah. I mean, I felt like the 10 glasses thing, though. That's like another part of the data angle. Right? Like, you said it was three times as fast to collect human data, but you have 10 times as many pairs of glasses as your robot. So it's about 30 times as fast. It's the [00:58:07] S0: I'm... Because I was gonna say that one to three is is very impressive. You guys are very good teleoperator, man. [00:58:12] S3: Yeah. That's a lot of robot data. Yep.
00:58:12 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:58:12] S3: Yeah. That's a lot of robot data. Yep. [00:58:15] S1: That's a lot of pretty fast robot data collection. So [00:58:19] S3: Yep. [00:58:20] S0: Yeah. I guess maybe, you know, just to wrap things up, like, you know, and any any other... We... I guess we we talk a fair bit about, like, you know, like, it's like a extension of work. Like, what what is possible? But what do you think are some limitations that, you know, that you just wanna highlight with with this work or maybe even going forward? I think, you know, Pranav, you already said that we will probably continue to need, like, some kind of robot data to augment all this huge amount of, like, smart glasses data. But any other limitations that you guys wanna highlight? [00:58:54] S1: I would say the key assumption would be, like, during a rollout, you have to be, like, looking at the direction of, like, your moving direction approximately. So it will be like a... This is, like, the first assumption we made at the beginning of this talk would be, like, a strict subset assumption. [00:59:12] S3: I had one other question that maybe is... Sorry. [00:59:15] S1: Mhmm. Go go for it. Yeah. No. Please go, Michael. Go for [00:59:17] S3: This is a slightly unfair question, I think. But it's like, how much of this work would have been made easier by you guys having an actual humanoid robot? Like, I noticed a lot of the things like the sidestepping, all of that kind of stuff. Yeah.
00:59:17 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:59:17] S3: This is a slightly unfair question, I think. But it's like, how much of this work would have been made easier by you guys having an actual humanoid robot? Like, I noticed a lot of the things like the sidestepping, all of that kind of stuff. Yeah. [00:59:30] S1: I mean, it... MS actually, fully capable of extending to, for example, unit three g one, for example. But, like, I would assume still some level of retargeting is needed, especially I don't think having full body control is making this task any easier because I will say having a willed bimanual manipulator is pretty much having... It's not... It's somehow limiting our problem, but I don't think it's necessarily a bad thing because we can't abstract out certain level of task complexity, and we only care about, like, future reappoints. We only care about, like, joint actions. I would say it's a, yeah, interesting direction to have, like, full humanoids, but I would say transferring human knowledge to humanoid, it will be a huge problem for future [01:00:17] S3: Yeah. Causes a lot of... Yeah. I could... Makes things a lot harder in other ways. I could see even Yeah. It'll be really close. [01:00:23] S1: Trains their, like, actual full body policy. Yeah. [01:00:27] S3: I I guess Imagine an HD wearing a pair of Aria glasses is the... That's the That would be great. So go ahead. Yeah. [01:00:34] S0: Maybe maybe a final question from me. It's... I I think I'm I'm I'm curious, like like, have you guys actually considered using the eye gaze, the eyeball tracking data? Do you think it's useful? I don't know.
01:00:34 · Chris Paxton, Lawrence Zhu
[01:00:34] S0: Maybe maybe a final question from me. It's... I I think I'm I'm I'm curious, like like, have you guys actually considered using the eye gaze, the eyeball tracking data? Do you think it's useful? I don't know. [01:00:46] S1: I I don't think anyone I know from the robotic community is using it. I think it's more like a IoT community's interest. I would say for affordance prediction, that might be something that's useful, but I think what ARIA does is like it tells you where your eyeball is so looking at at a pixel it will just be the UV coordinate on that image, and I think it's like pretty good, not super accurate because, of course, like, for everybody, eye gaze is, like, noisy. You have to, like, calibrate from user to user. But, Pranav, do you wanna talk about, like, how we can leverage eye gaze on Aria glasses?
01:00:46 · Lawrence Zhu, Pranav
[01:00:46] S1: I I don't think anyone I know from the robotic community is using it. I think it's more like a IoT community's interest. I would say for affordance prediction, that might be something that's useful, but I think what ARIA does is like it tells you where your eyeball is so looking at at a pixel it will just be the UV coordinate on that image, and I think it's like pretty good, not super accurate because, of course, like, for everybody, eye gaze is, like, noisy. You have to, like, calibrate from user to user. But, Pranav, do you wanna talk about, like, how we can leverage eye gaze on Aria glasses? [01:01:23] S2: Yeah. Yeah. Yeah. So eye gaze is something that I'm thinking of using for a future project. And I think the value in eye gaze comes from intent. Right? Because when you are trying to do a task, you typically look at what you're doing. So you can kind of, like, see that as, like, almost like an insight to how a person reasons through a task or, like, an... Like, almost like a... Yeah. Essentially, like, a way to kind of, like, get some insight because, like, we don't have, like, notes coming out of our brain, like, telling us what we're thinking. Right? But you can see where we're looking. Like, if I'm crossing a street, like, look left, right, I look and see if a car... If there's a car there, I stop. Or if there's not, I walk. So, like, I think eye gaze could be potentially useful in, like, that kind of situation.
01:01:23 · Pranav, Lawrence Zhu, Michael Cho
[01:01:23] S2: Yeah. Yeah. Yeah. So eye gaze is something that I'm thinking of using for a future project. And I think the value in eye gaze comes from intent. Right? Because when you are trying to do a task, you typically look at what you're doing. So you can kind of, like, see that as, like, almost like an insight to how a person reasons through a task or, like, an... Like, almost like a... Yeah. Essentially, like, a way to kind of, like, get some insight because, like, we don't have, like, notes coming out of our brain, like, telling us what we're thinking. Right? But you can see where we're looking. Like, if I'm crossing a street, like, look left, right, I look and see if a car... If there's a car there, I stop. Or if there's not, I walk. So, like, I think eye gaze could be potentially useful in, like, that kind of situation. [01:02:02] S1: But we're also, like, looking into, like, how can we come up with a task that differentiates eye gaze... The help of eye gaze or, like, why is this task needs eye gaze rather than, like, we can have some sort of, like, image even, action now behavior cloning methods. But, like, I will say, having extra sensing modality is definitely helpful. [01:02:21] S3: With, like, a really high resolution camera or something that you can't [01:02:24] S1: practice Yeah. To Probably. DRAM. [01:02:26] S3: I don't know. It's a... Like, it's like our... That's... That is the thing with our eyes. Right? Like, we're very high Yeah. At very high resolution. No. That's cool. Mhmm. It's really interesting stuff, though.