00:00:00 · Michael Cho, Vincent Liu, Ademi Adeniji
[00:00:00] S0: Everyone, welcome to another episode of RoboPapers. Today, we're super happy to have Vincent Liu and Adami Adanigi. Did I pronounce it? Adami? Yeah. Authors from this paper called EgoZero, Robot Learning from Smart Glasses are here to share about their work. So thanks for coming on the pod. Maybe as a start, Vincent and and Ademi, you guys can share a little bit about yourself first. [00:00:21] S1: Yes. Yeah. Maybe I'd go first and then Ademi. Yeah. I'm Vincent. I received my bachelor's in math and master's in computer science from Stanford, and I'm not actually affiliated with an academic institution until last summer when Ademi was visiting NYU, and we sort of talked about some cool ideas in robot learning, like scaling up data from human to robots, sort of started going in. With the support of Lerrel, we were exploring this direction for the last year. [00:00:47] S2: Yeah, I'm a Demi. I'm a fourth year PhD student at UC Berkeley studying with Pieter Abbeel. But as Vincent mentioned, starting around this time last year, I was visiting NYU studying with Lerrel, and we were able to explore sort of a number of directions in human to robot learning, scaling up data. We think came to some pretty cool results. So happy to talk more about those today.
00:01:10 · Vincent Liu
[00:01:10] S1: So yeah, we're going to talk about EgoZero, Robot Learning from Smart Glasses. Ademi and I are both co authors on this paper along with a stellar NYU undergrad. And we recently actually presented this at an RSS workshop. So yeah, let's dive right in. So first, let's talk about like, you know, current robot capabilities and how they don't scale. So teleoperation doesn't scale. It's a high friction to deploy robots in diverse environments, and it's bottled by requiring a very trained teleoperator to collect data. And on the other end, Sim2Real exploits the sit and breaks in the real world. And in between, you sort of have, like, these handheld UMI style demonstrations, which are, you know, still unnatural, slow, and kind of, like, in the local optima between the two trade offs. And so, basically, what we need is higher quality ground truth data at scale specifically, and we sort of, like, felt that neither of these approaches really sufficed. And we think the answer is sort of all around us already. There's already 8,000,000,000 people on the planet doing work all the time. You know, the analogy is like to the digital Internet, where this would be the greatest corpus of training data for embodied intelligence ever, and it is constantly renew... Renewing. And so, you know, we have, like... We sort of like to say that human data, like the real in the wild from egocentric point of view, is, like, the renewable energy or, like, the most scalable source of robot data if we can figure out how to learn from it well. And for people to collect data in their natural embodiments and environments, the data collection setup must be both minimal and ergonomic. So for example, the most popular teleoperation approaches in the top row, they don't scale because they're not really ergonomic and they're not natural, and teleoperators are doing this not in their own embodiment, which requires, like, some scale gap and some training in order to actually collect data well. And in the bottom row, there are some, like, recent work of human to robot learning, which also sort of... Each one of these comes with sort of an asterisk. Right? Like, have DexWild where you need to wear gloves, AR tags, wrist cameras, or you're assuming that you have stationary cameras, or even in the smart...
00:03:21 · Vincent Liu
Like, the previous smart glasses work, EgoMimic, they need to co train with Teleopt data. And so, you know, in all these previous works, you need to make assumptions about your data collection setup. And our goal was to sort of like, how can we actually design the system in a way where you can make basically no assumptions about your environment? And so really conceptually, EgoZero builds on point policy, which is a work that come that came out of the same group by Siddhant and Lerrel. And basically, in this paper, they rep... Sort of defined this unified representation space so that you can represent humans and robot morphologies in a single space. This way, if you define it well, in which case it's three d points, you're able to collect data in one, represent it in this space, and have it transferred to the other morphology. So this is exactly what they do. The process is they label some two d points on an expert image, map these points onto each demo, and track them throughout the demo. And then in point policy, he uses two cameras to triangulate these points into three d. And this triangulation is used also for detecting these hand key points. So he uses, like, some basic hand model, and you're able to triangulate these into three d. And so before going into, like, I guess, the egocentric setting, we should talk about why these three d points are good and, like, the benefits of these three d points. First is that they are morphology agnostic. So you're basically encoding, like, very sparse geometry and correspondence of the object and also, like, the location of your end effector. And, like, these concepts are applicable in both human data and, like, robot robot inference, I guess. And then also, because now your policy learning problem is a three d to three d mapping, it is very easy to learn and very easy to apply data augmentation in this very linear space. And because there's correspondence, the object pose is encoded explicitly where your points are ordered. And... Yeah. So so, like like, in the, like, the vision based, like, VLA approaches, you would kinda have to learn these this information through data. And, also, like, the three d to three d space avoids this difficult two d to three d learning problem, which is kind of inherent to all these vision based methods. Yeah.
00:05:33 · Vincent Liu, Chris Paxton
And so the main contribution of EgoZero is building on this unified three d representation space. We can show that it works in a zero shot setting from Ego centric in the wild data. And so these are some videos that you can find on our website as well. [00:05:48] S3: So question before I move on. So, like, I I love the point policy kind of, like, representation space. I think it's really cool. But, like, one issue with those kinds of things is always, like, what if the morphology of the robot is very different from the human? Now what do you what do you think about that? So, like, for example, like your Franka there. Right? Like, I have five fingers. The Franka only has two. How do you handle that? [00:06:10] S1: Right. I think, like, that issue sort of is also present in Teleop where you're... But, like, you're you're handling that in a different point in the stack. Right? And so... Yeah. So, basically, to to, like, retarget anything to, like, a gripper setting, you know, when you're teleoping by, like, a band manual gripper, you're also constraining... You're making assumptions about the tasks you can solve and how you're collecting the data. So in this work, it's pretty... You know, we just retarget, like, the the midpoint of the index and thumb to, the midpoint of the gripper, Just more so because we just wanted to demonstrate that, like, you know, this concept works in the egocentric setting. But you're right, like going forward, you know, maybe we can extract end effector positions of all five fingers then we target that to a dextrous hand.
00:06:10 · Vincent Liu, Ademi Adeniji, Michael Cho
[00:06:10] S1: Right. I think, like, that issue sort of is also present in Teleop where you're... But, like, you're you're handling that in a different point in the stack. Right? And so... Yeah. So, basically, to to, like, retarget anything to, like, a gripper setting, you know, when you're teleoping by, like, a band manual gripper, you're also constraining... You're making assumptions about the tasks you can solve and how you're collecting the data. So in this work, it's pretty... You know, we just retarget, like, the the midpoint of the index and thumb to, the midpoint of the gripper, Just more so because we just wanted to demonstrate that, like, you know, this concept works in the egocentric setting. But you're right, like going forward, you know, maybe we can extract end effector positions of all five fingers then we target that to a dextrous hand. [00:06:59] S2: Yeah. It makes sense. [00:07:00] S0: Yeah. I can see in your videos that... At least from those that... On your website that I can see, like, if I'm not wrong, you you ask the the the person to kind of, like, pretend his hand is kinda like a cripple. Right? So it's maybe a lot easier to just tell the guy what to do, like, you know, imitate a little bit like the the end. Is is that intentional? I I assume it's intentional.
00:07:00 · Michael Cho, Vincent Liu
[00:07:00] S0: Yeah. I can see in your videos that... At least from those that... On your website that I can see, like, if I'm not wrong, you you ask the the the person to kind of, like, pretend his hand is kinda like a cripple. Right? So it's maybe a lot easier to just tell the guy what to do, like, you know, imitate a little bit like the the end. Is is that intentional? I I assume it's intentional. [00:07:21] S1: Yeah. That was definitely slightly intentional for this just mainly because, like, again, we're sort of proving the concept of this. And in our work, we basic... I mean, there's many open directions of improvement for this. Right? Like, one is that, you know, we also don't, as a result, learn the rotation of the end effector of the Franka. And so therefore, you know, like, if if you go the extra step, you know, perhaps you can also extract the rotation of your hand based on the hand model. But we sort of like, there's already a lot going on to sort of get this proof of concept to work, then we just wanted to show that basically. Got it. Also, Adame, if you have any things you'd like to add, just feel free to jump in as well.