The Robotics Stack
00:26:23 · Oskar, Varun Nair, Leo
[00:26:23] S0: 100%. [00:26:24] S2: Yeah. I mean, Leo, I never got a take from you. If there's a take around data or deployment or sim or anything in physical AI that you have that is your take that might be a hot take or anything like that. Yeah. [00:26:35] S1: So, there is one thing I said before, which is I believe in UMI very strongly. I believe that in the future, we might see a mid training step where UMI is the bulk of the data used there. So you still have pre training using ego video data, post training using robot specific data through Teleop and UMI as a middle ground because UMI, if it's normalized correctly, you can scale it and you can transfer it between embodiments, for example. Yeah. I could see it as a middle ground. [00:27:07] S2: It is... You're you're right. It is used in mid training today. So you're not wrong. UMI is used in mid training. [00:27:13] S1: It is already the case. Wow. Okay. [00:27:15] S2: It's for better action retargeting for certain embodiments and midtrading. It is used for that. So I mean, guys... It seems like 20 episodes of Call is calling you far. You guys... Whatever takes you're giving are, like, pretty much definitely on track in the right way that data is evolving and robotics as a space is evolving. Yeah. [00:27:30] S1: Nice to hear. I have a question for you. So there has been a lot of debate around Astra lately. And one of the main question was, has GPT Astra been trained on robot data or ego data on three d motion capture data? Does it have, like, a VLA in its sets of tools in its harness? What's your take on that?
00:27:30 · Leo, Varun Nair
[00:27:30] S1: Nice to hear. I have a question for you. So there has been a lot of debate around Astra lately. And one of the main question was, has GPT Astra been trained on robot data or ego data on three d motion capture data? Does it have, like, a VLA in its sets of tools in its harness? What's your take on that? [00:27:50] S2: I don't think it's been trained on robotics data. I think it's been trained on a lot of computer use data, just general computers. And I'll tell you why that works. Computer uses a two d screen, right? So with enough data it understands spatial and geometric reasoning. It understands that in a 13 by 11 grid what a browser looks like, what moving something from here to here looks like. And you can see that when it looks at it in the pixel world, it sees a glass, it approaches it, and it tries to hold on to it. But where it fails is the three d. It is that it hasn't been trained on three d or depth data because if it had, it would understand what a close and open is. Even if it had done on a little bit of depth data, it clearly did not get enough of depth or three d data. So I actually think Astra is not trained on depth at all. That being said, it's not that far fetched for the next Astra or OpenAI's next robotics model to be trained on depth. It's not like it can't be done and it's probably being done right now. [00:28:48] S1: So, yeah. What examples specifically that you might have seen make you say Astra has not been trained on depth?
00:28:48 · Leo, Varun Nair
[00:28:48] S1: So, yeah. What examples specifically that you might have seen make you say Astra has not been trained on depth? [00:28:55] S2: I evaluated this myself and it's not just me. It's Jay from Robocurve. It's the RoboDojo paper I'm talking about. In their report, they also said the exact same thing that it has geometric and spatial understanding, but it lacks depth understanding. I can share that paper with you as well. I mean, I guess it's on my side of, like, robotics data eval Twitter where I, like, see this a lot, but, yeah, it was kinda consensus. [00:29:19] S1: That that totally makes sense. I'm just trying to find a simple example where the model has geometric understanding but not depth understanding, for example. What kind of what kind of task would it fail having geometric but not depth? [00:29:32] S2: Pick pick up picking up something. Let's say there's a camera I have up here. It looks at it and it can see this remote. It captures here. From its perspective, it looks like it's from its perspective, it's like, oh, I must be capturing it. I must be capturing it. But it clearly is not capturing it. So it's it's trying geometrically. It's right in between this, so it must be a grasp. But if it understood depth, it would realize that it has to go further in. That's the that's the example. And it's not just them. It's like I evaluated and saw the exact same issue happen as well.
00:29:32 · Varun Nair, Oskar, Leo
[00:29:32] S2: Pick pick up picking up something. Let's say there's a camera I have up here. It looks at it and it can see this remote. It captures here. From its perspective, it looks like it's from its perspective, it's like, oh, I must be capturing it. I must be capturing it. But it clearly is not capturing it. So it's it's trying geometrically. It's right in between this, so it must be a grasp. But if it understood depth, it would realize that it has to go further in. That's the that's the example. And it's not just them. It's like I evaluated and saw the exact same issue happen as well. [00:30:03] S0: It's interesting that you draw the comparison to the computer use because that's something that I also looked into, specifically because Astra got way faster and way better at computer use, and way more accurate, and they apparently changed the way that they do computer use as well. They started making code, basically, based off the screenshot that moves the cursor. But super interesting because there was this poll that asked people if they believe Astra was trained on robotics data, and the majority actually said yes. [00:30:30] S2: Yeah. I don't think it was. I think it would actually perform way better if it was trained on robotics data, which is why it's scary that even not having been trained on robotics data, it has such emergent properties. [00:30:42] S1: And considering the recent numerous hires from OpenAI in the robotics space and the opening of a new robotics lab, it will definitely happen in future models.
00:30:42 · Leo, Varun Nair, Oskar
[00:30:42] S1: And considering the recent numerous hires from OpenAI in the robotics space and the opening of a new robotics lab, it will definitely happen in future models. [00:30:50] S2: 100%. OpenAI is not shy about it. They're definitely working on robotics. [00:30:54] S0: Speaking of embodiments before, what is your take on this? Do you believe in humanoids? 100%. Do you think? Depends on the case. [00:31:02] S2: Depends on the case. I don't think initial deployments happen with humanoids, maybe semi humanoids, maybe just bimanual arm setups, maybe all of the above. I am not very convinced that humanoids are the right way to go right away. I do think in the future they will be, but I think there are a lot of intermediary steps before we actually hit humanoids. And I might be wrong because it's not like I'm a hardware expert here, but I do have done deployments and like from that perspective, it's not very useful to have a humanoid. It's much more useful to just have like a bimanual arm setup with a policy running on it. [00:31:35] S0: What are the hardest type of deployments that you see? If you could categorize them a little bit?
00:31:35 · Oskar, Varun Nair
[00:31:35] S0: What are the hardest type of deployments that you see? If you could categorize them a little bit? [00:31:40] S2: I think there's a company called Persona. It's one of the very latest age companies that does robotics welding. I think that's pretty hard for a regular policy to do. I'm not sure how they do it, but welding is definitely on the harder side of things. Anything that's very precise and dexterous is just usually very hard. The more margin of error or what we'd say is the tolerance of a certain task is easier to do. So if I just have to put a box in a package and the package opening is very big and I can just put it anywhere and it just goes in and I just flap it open, that's easy to do. But if I have to put a very precise part, like a screw, into a very small hole and put that in, it's much harder to do because precision is just more important there. And on that spectrum, a lot of tasks sit. The lower the tolerance, the more precision is required, the harder the task is, usually speaking. Or if there is a lot of tool use, that's another place where a lot of models fail today, they're not good at using tools, they're good at using their end effector. They're good at using their hands or their end effector paddle jaw grippers. They're not good at using other tools to make something happen, and I feel like that is definitely a big part of what the frontier will look like when it starts using tools. [00:32:49] S0: Interesting. And simulation specifically, maybe something that changed your mind recently?