00:54:31 · Michael Cho, Lawrence Zhu
[00:54:31] S3: It's included. Yeah. [00:54:33] S1: Also, if we talk about one extension of, like, because we said, like, one of our acknowledged limitation is, like, by directly transferring, we can only do, like, manipulation, navigation switches. That's because, like, we don't have robot data during the navigation phase, right? So especially when we use ground control, you don't have such data, then how would you kind of deploy your commands? So in this case, I think recently there have been works for... There are works in data engine. For example, there's Phantom, there's Masquerade. And so what they're trying to do is they try to kind of augment human data to look like robots, right? I would say that's a very interesting direction to explore as well. So to, like, make, synthetic robot data from human demonstrations to, like, further mitigate this, visual gap. And, ideally, can... Like, we can actually also, like, solve inverse kinematics or, like, just buy the data engine solution. We can actually get the join commands of, like, performing the same, human motion or, like, the same object trajectory from, any robot environment. I'll say that would be some pretty cool direction to go as well. [00:55:50] S3: You think... Does that just add another thing into your co training mix? And now I have real data, robot data, synthetic data... Or, sorry, human data, [00:55:56] S1: which is synthetic data. Okay. From beta, it's always the better the... Yeah. Yeah. Yeah. It's always the more, the better. Yeah.
00:55:56 · Lawrence Zhu, Chris Paxton, Pranav, Michael Cho
[00:55:56] S1: which is synthetic data. Okay. From beta, it's always the better the... Yeah. Yeah. Yeah. It's always the more, the better. Yeah. [00:56:01] S0: Yeah. I I... Well, I mean, just just the three, and I I think partially, like, the thing with all these smart glasses is, like, the the the test data is so much. You you can probably build a reasonable world model, like, gen thing, and then you can also now have a world model. Then you, yeah, you can do that as well. [00:56:18] S1: Yeah. [00:56:19] S2: Yeah. So... Yeah. It's funny. Like, all of these things you're mentioning is, like, a lot of the active extensions we're, like, investigating right now with our team. So, like, like, as you're saying, like, with the world model is something we're also looking into. [00:56:31] S0: Yeah. It makes sense. I mean, you have the data. It it Mhmm. Probably should just build a world model on top of this. It's quite easy to to get a lot of it. Just want a pair of smart glasses. [00:56:40] S3: How many pairs of smart glasses do you have? Oh, yeah. Sorry. It's okay. Can can you just, like, all wear them around? Like, how big of dataset can you guys collect at Georgia Tech? [00:56:48] S1: I think we are we are, like, the top four in our lab. [00:56:52] S3: So I think so. [00:56:54] S1: We do... I think I think we have, like, 10 of these. [00:56:57] S0: Yeah. 10. [00:56:58] S1: Oh, yeah. How long do [00:56:59] S0: they go for? [00:57:00] S1: Like, I You can do way [00:57:00] S3: better than [00:57:01] S1: one to three. Wait. Wait. You mean battery. Right?
00:57:01 · Lawrence Zhu, Chris Paxton, Michael Cho
[00:57:01] S1: one to three. Wait. Wait. You mean battery. Right? [00:57:03] S0: Yeah. Battery. Like, mean, if you [00:57:05] S1: press record, how long does it go for? How long does it go? So continuous rec... We don't we don't do continuous recording because that will make the data has to like, super large. Right? When you process it, it will go, like, out of RAM... Memory. So we do, like, three... We usually do, like, three, four minutes per episode. And for EMMA, we did, like, think, five minute episode per demo, and then we'll just rerecord. But the data will last around... Not the data. Sorry. Like, the the glasses battery will last around one hour for, like, continuous data recording. And we recently invented something called data stations. Basically, we have a power station and a chair, and then we have some... We have, like, a power cable plugged into the glasses so you can collect the infinite data. [00:57:45] S0: Yeah. Yeah. This is this is how humans become cyborg. I I tell you. You can see those smart smart glasses, man. Let's plug in more. [00:57:52] S3: You know? You know? Yeah. I mean, I felt like the 10 glasses thing, though. That's like another part of the data angle. Right? Like, you said it was three times as fast to collect human data, but you have 10 times as many pairs of glasses as your robot. So it's about 30 times as fast. It's the [00:58:07] S0: I'm... Because I was gonna say that one to three is is very impressive. You guys are very good teleoperator, man. [00:58:12] S3: Yeah. That's a lot of robot data. Yep.
00:58:12 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:58:12] S3: Yeah. That's a lot of robot data. Yep. [00:58:15] S1: That's a lot of pretty fast robot data collection. So [00:58:19] S3: Yep. [00:58:20] S0: Yeah. I guess maybe, you know, just to wrap things up, like, you know, and any any other... We... I guess we we talk a fair bit about, like, you know, like, it's like a extension of work. Like, what what is possible? But what do you think are some limitations that, you know, that you just wanna highlight with with this work or maybe even going forward? I think, you know, Pranav, you already said that we will probably continue to need, like, some kind of robot data to augment all this huge amount of, like, smart glasses data. But any other limitations that you guys wanna highlight? [00:58:54] S1: I would say the key assumption would be, like, during a rollout, you have to be, like, looking at the direction of, like, your moving direction approximately. So it will be like a... This is, like, the first assumption we made at the beginning of this talk would be, like, a strict subset assumption. [00:59:12] S3: I had one other question that maybe is... Sorry. [00:59:15] S1: Mhmm. Go go for it. Yeah. No. Please go, Michael. Go for [00:59:17] S3: This is a slightly unfair question, I think. But it's like, how much of this work would have been made easier by you guys having an actual humanoid robot? Like, I noticed a lot of the things like the sidestepping, all of that kind of stuff. Yeah.
00:59:17 · Michael Cho, Lawrence Zhu, Chris Paxton
[00:59:17] S3: This is a slightly unfair question, I think. But it's like, how much of this work would have been made easier by you guys having an actual humanoid robot? Like, I noticed a lot of the things like the sidestepping, all of that kind of stuff. Yeah. [00:59:30] S1: I mean, it... MS actually, fully capable of extending to, for example, unit three g one, for example. But, like, I would assume still some level of retargeting is needed, especially I don't think having full body control is making this task any easier because I will say having a willed bimanual manipulator is pretty much having... It's not... It's somehow limiting our problem, but I don't think it's necessarily a bad thing because we can't abstract out certain level of task complexity, and we only care about, like, future reappoints. We only care about, like, joint actions. I would say it's a, yeah, interesting direction to have, like, full humanoids, but I would say transferring human knowledge to humanoid, it will be a huge problem for future [01:00:17] S3: Yeah. Causes a lot of... Yeah. I could... Makes things a lot harder in other ways. I could see even Yeah. It'll be really close. [01:00:23] S1: Trains their, like, actual full body policy. Yeah. [01:00:27] S3: I I guess Imagine an HD wearing a pair of Aria glasses is the... That's the That would be great. So go ahead. Yeah. [01:00:34] S0: Maybe maybe a final question from me. It's... I I think I'm I'm I'm curious, like like, have you guys actually considered using the eye gaze, the eyeball tracking data? Do you think it's useful? I don't know.
01:00:34 · Chris Paxton, Lawrence Zhu
[01:00:34] S0: Maybe maybe a final question from me. It's... I I think I'm I'm I'm curious, like like, have you guys actually considered using the eye gaze, the eyeball tracking data? Do you think it's useful? I don't know. [01:00:46] S1: I I don't think anyone I know from the robotic community is using it. I think it's more like a IoT community's interest. I would say for affordance prediction, that might be something that's useful, but I think what ARIA does is like it tells you where your eyeball is so looking at at a pixel it will just be the UV coordinate on that image, and I think it's like pretty good, not super accurate because, of course, like, for everybody, eye gaze is, like, noisy. You have to, like, calibrate from user to user. But, Pranav, do you wanna talk about, like, how we can leverage eye gaze on Aria glasses?