Going Direct Conversations
00:25:11 · Jordan Wolfe, Chris Paxton
[00:25:11] S2: So when you say a different type of video, are you talking about like, are you talking about, let's use an example, right? So let's use the example of like bartending, for instance, right? So are you talking about like a dedicated video to like slicing a lime or preparing a cocktail and like having very intentional data to like... [00:25:34] S0: Usually, yeah, that's right. But also, usually, depending on how you're doing it a lot of the time, that means that you need to be able to estimate where the hands are in the video, which means that it's not just a video of a bartender doing a lime. It's like a video from the bartender's point of view of their hands slicing a lime or whatever. So that means that because to be able to know what the robot should do, I need to know where this finger is a lot of the time. Otherwise, because that's what the robot's going to have to do itself later. So a lot of learning from human video stuff ends up getting very complicated like this. Now, of course, that's not all of it. There's quite a lot of different ways that people have learned from human video. But yeah.
00:26:12 · Michael Cho
[00:26:12] S1: Yeah, I should also just add actually video alone is not going to get us there also to human performance because like you can imagine if you take an egocentric video, there's a lot of occlusion. Let's say when I pick up a box, you cannot see the rest of my big fingers because the box is covering the fingers, but I'm actually touching the box, right? So we humans take it for granted. We have very, very good touch sensors all over our body, right? Especially the fingers, dips and all that. So we actually are organizing this robotic origami competition. We announced it a couple of weeks ago. And we actually are co-organizing that competition together with some folks from the Japan Origami Association. So these are like the real human experts doing origami. So we actually invited like a grandmaster of the origami association to this robotic conference. And then next to that, the old man doing origami is is a robot doing origami and the robot absolutely like it's very impressive but nothing compared to the old man but what i want to highlight though is i was like standing in front of this like grandmaster for like 10 minutes just observing what he's doing half the time he's talking to people around him he's not even looking it's all touch Yeah. Yeah. Like, you know, making his paper crane on that. So, so I think, so, so there's also a very big hardware thing that we, we just don't have good tough senses, right? Anywhere near as good as ours. Sure. In terms of like, you know, like fatality at the cost that it's, that makes sense. So that's also another one. And I think you're understanding a little bit how good the robot part was.
00:27:46 · Chris Paxton, Jordan Wolfe
[00:27:46] S0: The robot guy, I walked by this booth, obviously, at this conference with lots of top researchers. And a lot of them were saying, this is one of the coolest robot manipulation things I've seen. And he's just getting blown away by the old man next door. Right. So like, that's just like, I want to just be clear. It was actually very good. The robot part was very good. It was just, yeah. [00:28:08] S2: Yeah. Yeah. We take it for granted how like, how unbelievable our hands are, right? Like just the dexterity and like what you can manipulate and how you can manipulate. It's pretty, it's pretty remarkable. All right, let's go to kind of one last one, which is sim to reel or reel to sim. So Chris, maybe kind of break that down for us, what that means and kind of your perspective on the level of use, like usability in training and training.
00:28:40 · Chris Paxton
[00:28:40] S0: Yeah, so this is actually something that I worked on for a very long time, like six plus years, I think. But so robotics. So yeah, SynthReal means that we train the robot how to do something in a simulation. And then it also has to work in the real world, right? So I take the robot, and maybe I create a digital twin of the task that I want it to perform. If it's an industrial task or an industrial kidding task, maybe I would just spawn all the objects that I need to place on a table and have a copy of the robot and have it sit and have and then maybe I create a whole bunch like a million of these robots in simulation I have them all practice picking and placing the object and into into the target and whatever that right and I use that data like I simulate cameras or whatever sensors that I may have access to and then I train a policy, train a neural network on that simulated data. And the sim-to-real part, that's the sim part. The sim-to-real part is then I have to set that up in such a way that I can then take that policy and use it in the real world. So this is the way that a lot of things work right now, particularly when it comes to all the dancing robot videos that you see are sim-to-real. all of them are trained this way. But there's a lot of other stuff like central manipulation is another, is an active research area, so.
00:30:04 · Jordan Wolfe, Michael Cho, Chris Paxton
[00:30:04] S2: Let's go over to RoboPapers. Thank you for that. So actually, before we do that, I wanna, I guess if we were to step back, when you think about all these different training methods, and obviously several folks working on a variety of both open source and their own closed source robot foundational models and multimodal models and world foundational models, whatever you wanna call it, I want to get each of your perspectives on, on, on, on how do we, how do we ultimately solve this training gap? Like what is the thing that we can do today to move the field forward in the most effective way, I guess, uh, around the training gap. If there's one thing that you pick, you pick, we could do. Gosh, [00:30:46] S1: uh, I think you need, yeah, it's very hard to pick one thing because like, it's a, it's very unclear which one is actually going to scale. Like all, all these, uh, [00:30:55] S0: I mean, if it's like wave your magic wand, infinite money. Sorry. Go ahead. I have an answer.
00:30:55 · Chris Paxton, Michael Cho, Jordan Wolfe
[00:30:55] S0: I mean, if it's like wave your magic wand, infinite money. Sorry. Go ahead. I have an answer. [00:31:01] S1: I have talked to so many different researchers now. Different researchers have different, like, kind of, let's say, I don't know, like, inclination of biases. Some people think, you know, TeddyWarp is really the way. Some people are really betting on, like, role model. I mean, I should say something about simulation, right? So to me, Simulation data. Simulation have kind of two types now. You have the traditional simulation engine, and then you have the wall model, which is the new stuff. Because the wall model is, in a way, a simulator as well. It just behaves very, very differently and trains very differently. I think simulation has got to... Okay, if I were to bet, I feel that simulation alone can get us 80%, 90% there. It's my guess. Okay. But because you need 99.9999% so it's not enough therefore. But it can probably move us all near to 90 something percent. The issue with simulation, I don't know whether Chris you agree with me, at least with the classical simulation engine, is that I believe contrary to popular belief, we're not bottlenecked by compute actually, I think. I think that in a way, the its capabilities is also its curse, right? Because with a very good simulation engine, you can literally simulate anything you want. And that's a big problem because the search space or the exploration space is nearly infinite. [00:32:26] S2: It's too wide, yeah.