00:59:17 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:59:17] S3: This is a slightly unfair question, I think. But it's like, how much of this work would have been made easier by you guys having an actual humanoid robot? Like, I noticed a lot of the things like the sidestepping, all of that kind of stuff. Yeah. [00:59:30] S1: I mean, it... MS actually, fully capable of extending to, for example, unit three g one, for example. But, like, I would assume still some level of retargeting is needed, especially I don't think having full body control is making this task any easier because I will say having a willed bimanual manipulator is pretty much having... It's not... It's somehow limiting our problem, but I don't think it's necessarily a bad thing because we can't abstract out certain level of task complexity, and we only care about, like, future reappoints. We only care about, like, joint actions. I would say it's a, yeah, interesting direction to have, like, full humanoids, but I would say transferring human knowledge to humanoid, it will be a huge problem for future [01:00:17] S3: Yeah. Causes a lot of... Yeah. I could... Makes things a lot harder in other ways. I could see even Yeah. It'll be really close. [01:00:23] S1: Trains their, like, actual full body policy. Yeah. [01:00:27] S3: I I guess Imagine an HD wearing a pair of Aria glasses is the... That's the That would be great. So go ahead. Yeah. [01:00:34] S0: Maybe maybe a final question from me. It's... I I think I'm I'm I'm curious, like like, have you guys actually considered using the eye gaze, the eyeball tracking data? Do you think it's useful? I don't know.
01:00:34 · Chris Paxton, Lawrence Zhu
[01:00:34] S0: Maybe maybe a final question from me. It's... I I think I'm I'm I'm curious, like like, have you guys actually considered using the eye gaze, the eyeball tracking data? Do you think it's useful? I don't know. [01:00:46] S1: I I don't think anyone I know from the robotic community is using it. I think it's more like a IoT community's interest. I would say for affordance prediction, that might be something that's useful, but I think what ARIA does is like it tells you where your eyeball is so looking at at a pixel it will just be the UV coordinate on that image, and I think it's like pretty good, not super accurate because, of course, like, for everybody, eye gaze is, like, noisy. You have to, like, calibrate from user to user. But, Pranav, do you wanna talk about, like, how we can leverage eye gaze on Aria glasses?
01:00:46 · Lawrence Zhu, Pranav
[01:00:46] S1: I I don't think anyone I know from the robotic community is using it. I think it's more like a IoT community's interest. I would say for affordance prediction, that might be something that's useful, but I think what ARIA does is like it tells you where your eyeball is so looking at at a pixel it will just be the UV coordinate on that image, and I think it's like pretty good, not super accurate because, of course, like, for everybody, eye gaze is, like, noisy. You have to, like, calibrate from user to user. But, Pranav, do you wanna talk about, like, how we can leverage eye gaze on Aria glasses? [01:01:23] S2: Yeah. Yeah. Yeah. So eye gaze is something that I'm thinking of using for a future project. And I think the value in eye gaze comes from intent. Right? Because when you are trying to do a task, you typically look at what you're doing. So you can kind of, like, see that as, like, almost like an insight to how a person reasons through a task or, like, an... Like, almost like a... Yeah. Essentially, like, a way to kind of, like, get some insight because, like, we don't have, like, notes coming out of our brain, like, telling us what we're thinking. Right? But you can see where we're looking. Like, if I'm crossing a street, like, look left, right, I look and see if a car... If there's a car there, I stop. Or if there's not, I walk. So, like, I think eye gaze could be potentially useful in, like, that kind of situation.
01:01:23 · Pranav, Lawrence Zhu, Michael Cho
[01:01:23] S2: Yeah. Yeah. Yeah. So eye gaze is something that I'm thinking of using for a future project. And I think the value in eye gaze comes from intent. Right? Because when you are trying to do a task, you typically look at what you're doing. So you can kind of, like, see that as, like, almost like an insight to how a person reasons through a task or, like, an... Like, almost like a... Yeah. Essentially, like, a way to kind of, like, get some insight because, like, we don't have, like, notes coming out of our brain, like, telling us what we're thinking. Right? But you can see where we're looking. Like, if I'm crossing a street, like, look left, right, I look and see if a car... If there's a car there, I stop. Or if there's not, I walk. So, like, I think eye gaze could be potentially useful in, like, that kind of situation. [01:02:02] S1: But we're also, like, looking into, like, how can we come up with a task that differentiates eye gaze... The help of eye gaze or, like, why is this task needs eye gaze rather than, like, we can have some sort of, like, image even, action now behavior cloning methods. But, like, I will say, having extra sensing modality is definitely helpful. [01:02:21] S3: With, like, a really high resolution camera or something that you can't [01:02:24] S1: practice Yeah. To Probably. DRAM. [01:02:26] S3: I don't know. It's a... Like, it's like our... That's... That is the thing with our eyes. Right? Like, we're very high Yeah. At very high resolution. No. That's cool. Mhmm. It's really interesting stuff, though.
01:02:26 · Michael Cho, Chris Paxton, Lawrence Zhu
[01:02:26] S3: I don't know. It's a... Like, it's like our... That's... That is the thing with our eyes. Right? Like, we're very high Yeah. At very high resolution. No. That's cool. Mhmm. It's really interesting stuff, though. [01:02:35] S0: Yeah. Yeah. Cool. Well... So... Yeah. I guess, thank you so much. I I I think that that's a that's a good place to to wrap up. Maybe one one final question we always ask. Like, is is there any cool paper that you guys came across recently you wanna give a shout out [01:02:48] S1: to? Here we go. Okay. We actually advertise one of our own papers because Yeah. [01:02:52] S0: Of course. Of course, man. Yeah. Yeah. Yeah. Yeah. You should you should flock your own thing. [01:02:56] S1: Yeah. It's actually a... We have two concurrent projects. And if you look at the author list, they're actually flipped. Because... Yeah. So, I'm gonna recommend this paper called, this paper called ego bridge domain adaptation for generalizable imitation from, egocentric human data. So same team, author list flipped, basically. And in that paper, we pretty much tried to, leverage, the... We tried to leverage dynamic time warping and optimal transport to solve direct transferring human knowledge into robots so that we achieve scene generalization zero shot and motion generalization zero shot. It's a really interesting paper. Yep.
01:02:56 · Lawrence Zhu, Pranav, Chris Paxton, Michael Cho
[01:02:56] S1: Yeah. It's actually a... We have two concurrent projects. And if you look at the author list, they're actually flipped. Because... Yeah. So, I'm gonna recommend this paper called, this paper called ego bridge domain adaptation for generalizable imitation from, egocentric human data. So same team, author list flipped, basically. And in that paper, we pretty much tried to, leverage, the... We tried to leverage dynamic time warping and optimal transport to solve direct transferring human knowledge into robots so that we achieve scene generalization zero shot and motion generalization zero shot. It's a really interesting paper. Yep. [01:03:42] S2: Yeah. So, like, if you think about it, like, inside of the latent space of, like, a model, you have like the human distribution and the robot distribution. It's like, how can we align those two distributions? How can we bring them together so that in the model you kind of see it very similarly, and then you can learn how to learn different behaviors like motions and tasks in a much better way. [01:04:02] S0: Got it. [01:04:03] S2: Yeah, I don't think we should spoil too much, Lawrence. So I think, I think, you can visit us in NeurIPS or, you can check out the paper. [01:04:10] S0: Cool. Cool. Cool. Yeah. Definitely this one. Alright. Cool. I guess, yeah, that... That's a wrap. Thank you so much for making the time, guys. I really appreciate it. Really fascinating one. You. [01:04:18] S1: Thanks so much for having us. [01:04:20] S2: Covering EMMA. [01:04:21] S3: Yeah. Thank you, guys. [01:04:22] S1: Cool. Alright.