The Robotics Stack
00:23:40 · Oskar, Varun Nair
[00:23:40] S0: Yeah. It's just a bit clunky. Right? But I think as technology evolves, I think it will get more smooth. It will be more easy. [00:23:47] S2: The world of Atoms is just... It's not easy as software. It's not as scalable. It breaks for random reasons, and it's weird. And people need to realize that the frontier is like we had nice exponential scale up and it's going to take like time. There are so many companies that were doing hard work that had to iterate a bazillion times before it like worked. There's an investor I follow on Twitter who mentioned this and it's like there's patient capital and there's impatient capital and I feel a lot of impatient capital is following robotics and it's usually the same SaaS software investors who are impatient with their capital looking for quick returns and I don't think robotics is something that's going to be kind to the impatient capital. It will be extremely, extremely helpful for the patient capital because you have to see what works in the long run. And we all know that physically AI and robotics will be the next step function of mankind as a whole. So it's a matter of time rather than a matter of like whether or not it will happen. [00:24:43] S0: We see a lot more benchmarks now with robotics that have been kind of difficult to do, of course, because it's not just an LLM. How do you see these ideas from AI move over towards robotics? Like benchmarks, for example.
00:24:58 · Varun Nair, Oskar
[00:24:58] S2: Yeah. Jay from Robocurve is doing a great job at this, actually. He started a company called Robocurve. That's a public benefit corporation doing evals and benchmark for robotics. And I think it's just about transparency. It's seeing whether these models do really well or not on certain tasks and how do you actually benchmark them openly is important. The actual process of benchmarking depends on the task, it depends on the environment, it depends on a bunch of different factors. So it's not as easy and you can do it in sim or in real. There's a lot of different factors affecting it. Another really comprehensive benchmark is RoboDojo. It's a combination of a bunch of different labs and universities working together on that And it's the most comprehensive framework for sim and real benchmarks. I think in general, there is no consensus on what is the best eval to be run on robotics. It's kind of done internally for the most part at most companies, which is why I think public facing benchmarks and eval, either companies or projects or research initiatives are in the best interest of the industry as a whole. So I am very big on making sure companies like that exist and are helping the audience interested understand where these robotics models are today. So yeah, it's not very transferable from LLMs, you're right. It's not the same at all, but it should exist and it should exist out in the public so that people can actually read up on it. [00:26:23] S0: 100%.
00:26:23 · Oskar, Varun Nair, Leo
[00:26:23] S0: 100%. [00:26:24] S2: Yeah. I mean, Leo, I never got a take from you. If there's a take around data or deployment or sim or anything in physical AI that you have that is your take that might be a hot take or anything like that. Yeah. [00:26:35] S1: So, there is one thing I said before, which is I believe in UMI very strongly. I believe that in the future, we might see a mid training step where UMI is the bulk of the data used there. So you still have pre training using ego video data, post training using robot specific data through Teleop and UMI as a middle ground because UMI, if it's normalized correctly, you can scale it and you can transfer it between embodiments, for example. Yeah. I could see it as a middle ground. [00:27:07] S2: It is... You're you're right. It is used in mid training today. So you're not wrong. UMI is used in mid training. [00:27:13] S1: It is already the case. Wow. Okay. [00:27:15] S2: It's for better action retargeting for certain embodiments and midtrading. It is used for that. So I mean, guys... It seems like 20 episodes of Call is calling you far. You guys... Whatever takes you're giving are, like, pretty much definitely on track in the right way that data is evolving and robotics as a space is evolving. Yeah. [00:27:30] S1: Nice to hear. I have a question for you. So there has been a lot of debate around Astra lately. And one of the main question was, has GPT Astra been trained on robot data or ego data on three d motion capture data? Does it have, like, a VLA in its sets of tools in its harness? What's your take on that?
00:27:30 · Leo, Varun Nair
[00:27:30] S1: Nice to hear. I have a question for you. So there has been a lot of debate around Astra lately. And one of the main question was, has GPT Astra been trained on robot data or ego data on three d motion capture data? Does it have, like, a VLA in its sets of tools in its harness? What's your take on that? [00:27:50] S2: I don't think it's been trained on robotics data. I think it's been trained on a lot of computer use data, just general computers. And I'll tell you why that works. Computer uses a two d screen, right? So with enough data it understands spatial and geometric reasoning. It understands that in a 13 by 11 grid what a browser looks like, what moving something from here to here looks like. And you can see that when it looks at it in the pixel world, it sees a glass, it approaches it, and it tries to hold on to it. But where it fails is the three d. It is that it hasn't been trained on three d or depth data because if it had, it would understand what a close and open is. Even if it had done on a little bit of depth data, it clearly did not get enough of depth or three d data. So I actually think Astra is not trained on depth at all. That being said, it's not that far fetched for the next Astra or OpenAI's next robotics model to be trained on depth. It's not like it can't be done and it's probably being done right now. [00:28:48] S1: So, yeah. What examples specifically that you might have seen make you say Astra has not been trained on depth?
00:28:48 · Leo, Varun Nair
[00:28:48] S1: So, yeah. What examples specifically that you might have seen make you say Astra has not been trained on depth? [00:28:55] S2: I evaluated this myself and it's not just me. It's Jay from Robocurve. It's the RoboDojo paper I'm talking about. In their report, they also said the exact same thing that it has geometric and spatial understanding, but it lacks depth understanding. I can share that paper with you as well. I mean, I guess it's on my side of, like, robotics data eval Twitter where I, like, see this a lot, but, yeah, it was kinda consensus. [00:29:19] S1: That that totally makes sense. I'm just trying to find a simple example where the model has geometric understanding but not depth understanding, for example. What kind of what kind of task would it fail having geometric but not depth? [00:29:32] S2: Pick pick up picking up something. Let's say there's a camera I have up here. It looks at it and it can see this remote. It captures here. From its perspective, it looks like it's from its perspective, it's like, oh, I must be capturing it. I must be capturing it. But it clearly is not capturing it. So it's it's trying geometrically. It's right in between this, so it must be a grasp. But if it understood depth, it would realize that it has to go further in. That's the that's the example. And it's not just them. It's like I evaluated and saw the exact same issue happen as well.
00:29:32 · Varun Nair, Oskar, Leo
[00:29:32] S2: Pick pick up picking up something. Let's say there's a camera I have up here. It looks at it and it can see this remote. It captures here. From its perspective, it looks like it's from its perspective, it's like, oh, I must be capturing it. I must be capturing it. But it clearly is not capturing it. So it's it's trying geometrically. It's right in between this, so it must be a grasp. But if it understood depth, it would realize that it has to go further in. That's the that's the example. And it's not just them. It's like I evaluated and saw the exact same issue happen as well. [00:30:03] S0: It's interesting that you draw the comparison to the computer use because that's something that I also looked into, specifically because Astra got way faster and way better at computer use, and way more accurate, and they apparently changed the way that they do computer use as well. They started making code, basically, based off the screenshot that moves the cursor. But super interesting because there was this poll that asked people if they believe Astra was trained on robotics data, and the majority actually said yes. [00:30:30] S2: Yeah. I don't think it was. I think it would actually perform way better if it was trained on robotics data, which is why it's scary that even not having been trained on robotics data, it has such emergent properties. [00:30:42] S1: And considering the recent numerous hires from OpenAI in the robotics space and the opening of a new robotics lab, it will definitely happen in future models.