The Robotics Stack
00:27:30 · Leo, Varun Nair
[00:27:30] S1: Nice to hear. I have a question for you. So there has been a lot of debate around Astra lately. And one of the main question was, has GPT Astra been trained on robot data or ego data on three d motion capture data? Does it have, like, a VLA in its sets of tools in its harness? What's your take on that? [00:27:50] S2: I don't think it's been trained on robotics data. I think it's been trained on a lot of computer use data, just general computers. And I'll tell you why that works. Computer uses a two d screen, right? So with enough data it understands spatial and geometric reasoning. It understands that in a 13 by 11 grid what a browser looks like, what moving something from here to here looks like. And you can see that when it looks at it in the pixel world, it sees a glass, it approaches it, and it tries to hold on to it. But where it fails is the three d. It is that it hasn't been trained on three d or depth data because if it had, it would understand what a close and open is. Even if it had done on a little bit of depth data, it clearly did not get enough of depth or three d data. So I actually think Astra is not trained on depth at all. That being said, it's not that far fetched for the next Astra or OpenAI's next robotics model to be trained on depth. It's not like it can't be done and it's probably being done right now. [00:28:48] S1: So, yeah. What examples specifically that you might have seen make you say Astra has not been trained on depth?
00:28:48 · Leo, Varun Nair
[00:28:48] S1: So, yeah. What examples specifically that you might have seen make you say Astra has not been trained on depth? [00:28:55] S2: I evaluated this myself and it's not just me. It's Jay from Robocurve. It's the RoboDojo paper I'm talking about. In their report, they also said the exact same thing that it has geometric and spatial understanding, but it lacks depth understanding. I can share that paper with you as well. I mean, I guess it's on my side of, like, robotics data eval Twitter where I, like, see this a lot, but, yeah, it was kinda consensus. [00:29:19] S1: That that totally makes sense. I'm just trying to find a simple example where the model has geometric understanding but not depth understanding, for example. What kind of what kind of task would it fail having geometric but not depth? [00:29:32] S2: Pick pick up picking up something. Let's say there's a camera I have up here. It looks at it and it can see this remote. It captures here. From its perspective, it looks like it's from its perspective, it's like, oh, I must be capturing it. I must be capturing it. But it clearly is not capturing it. So it's it's trying geometrically. It's right in between this, so it must be a grasp. But if it understood depth, it would realize that it has to go further in. That's the that's the example. And it's not just them. It's like I evaluated and saw the exact same issue happen as well.
00:29:32 · Varun Nair, Oskar, Leo
[00:29:32] S2: Pick pick up picking up something. Let's say there's a camera I have up here. It looks at it and it can see this remote. It captures here. From its perspective, it looks like it's from its perspective, it's like, oh, I must be capturing it. I must be capturing it. But it clearly is not capturing it. So it's it's trying geometrically. It's right in between this, so it must be a grasp. But if it understood depth, it would realize that it has to go further in. That's the that's the example. And it's not just them. It's like I evaluated and saw the exact same issue happen as well. [00:30:03] S0: It's interesting that you draw the comparison to the computer use because that's something that I also looked into, specifically because Astra got way faster and way better at computer use, and way more accurate, and they apparently changed the way that they do computer use as well. They started making code, basically, based off the screenshot that moves the cursor. But super interesting because there was this poll that asked people if they believe Astra was trained on robotics data, and the majority actually said yes. [00:30:30] S2: Yeah. I don't think it was. I think it would actually perform way better if it was trained on robotics data, which is why it's scary that even not having been trained on robotics data, it has such emergent properties. [00:30:42] S1: And considering the recent numerous hires from OpenAI in the robotics space and the opening of a new robotics lab, it will definitely happen in future models.
00:30:42 · Leo, Varun Nair, Oskar
[00:30:42] S1: And considering the recent numerous hires from OpenAI in the robotics space and the opening of a new robotics lab, it will definitely happen in future models. [00:30:50] S2: 100%. OpenAI is not shy about it. They're definitely working on robotics. [00:30:54] S0: Speaking of embodiments before, what is your take on this? Do you believe in humanoids? 100%. Do you think? Depends on the case. [00:31:02] S2: Depends on the case. I don't think initial deployments happen with humanoids, maybe semi humanoids, maybe just bimanual arm setups, maybe all of the above. I am not very convinced that humanoids are the right way to go right away. I do think in the future they will be, but I think there are a lot of intermediary steps before we actually hit humanoids. And I might be wrong because it's not like I'm a hardware expert here, but I do have done deployments and like from that perspective, it's not very useful to have a humanoid. It's much more useful to just have like a bimanual arm setup with a policy running on it. [00:31:35] S0: What are the hardest type of deployments that you see? If you could categorize them a little bit?
00:31:35 · Oskar, Varun Nair
[00:31:35] S0: What are the hardest type of deployments that you see? If you could categorize them a little bit? [00:31:40] S2: I think there's a company called Persona. It's one of the very latest age companies that does robotics welding. I think that's pretty hard for a regular policy to do. I'm not sure how they do it, but welding is definitely on the harder side of things. Anything that's very precise and dexterous is just usually very hard. The more margin of error or what we'd say is the tolerance of a certain task is easier to do. So if I just have to put a box in a package and the package opening is very big and I can just put it anywhere and it just goes in and I just flap it open, that's easy to do. But if I have to put a very precise part, like a screw, into a very small hole and put that in, it's much harder to do because precision is just more important there. And on that spectrum, a lot of tasks sit. The lower the tolerance, the more precision is required, the harder the task is, usually speaking. Or if there is a lot of tool use, that's another place where a lot of models fail today, they're not good at using tools, they're good at using their end effector. They're good at using their hands or their end effector paddle jaw grippers. They're not good at using other tools to make something happen, and I feel like that is definitely a big part of what the frontier will look like when it starts using tools. [00:32:49] S0: Interesting. And simulation specifically, maybe something that changed your mind recently?
00:32:49 · Oskar, Varun Nair
[00:32:49] S0: Interesting. And simulation specifically, maybe something that changed your mind recently? [00:32:53] S2: I think with simulation, the biggest thing is it's not like I've changed my mind on this. It's just a new... Decent learning is you wanna just keep recalibrating your sim. And part of that is recalibrating with new objects, new scenes. But a big part of it also, like, if you can, changing up the physics engine, is really hard, but if you're using Isaac Sim, obviously you have no say in that, but having the recalibration needs to come from scene and asset generation, which is easy to do for anybody using these softwares, but you cannot recalibrate a whole engine. So if there is a way to recalibrate the physics engine, in my opinion, that would be game changer, but I don't know how that is done. So I need to do a little bit more digging and understanding that. There are open source engines that must exist, but SIEM is just such an annoying software to use. It's not very intuitive and it requires a lot of compute and it takes a while to set up. That's just 10% of the work. That's not even the actual work. It's just set up just like getting it to like work well. Imagine changing up the entire physics engine, like I can't imagine that's easy for even a team of people to do. [00:33:57] S0: And did you take a look at what Niantic is doing with their geospatial model? [00:34:01] S2: No. Do you want to tell me?