The Robotics Stack
00:06:35 · Varun Nair
[00:06:35] S2: I think they are different. AI so far has been on our screens. It's been in the world of bits. And physical AI is all it's saying is that, hey, can I make intelligent decisions in the real world? I think Waymo, Zoox, Robotaxi are fantastic examples of physical AI, and they're even embodied in a sense, right, they're a core. But I wouldn't call them robotics, even though they're similar. I think it's important to make the distinction that there is robotics, which is a big part of the narrative of physical AI today, but physical AI in general is just any intelligent decisions being made in the real world. We have solved rigid body dynamics through simulations, so that's why things like Waymo's can exist, quadrupeds like Spot can do navigation and locomotion very well because again, bodies have been sort of figured out. But dextrous manipulation is still on the frontier of things because in robotics, that stuff is not fully solved yet. But if you've heard of companies like Spectre, they are not in the robotics field at all. They are an intelligent system for facilities. It plugs into your cameras, your other sensors, everything in like, let's say a factory or a warehouse or a building. It's like Palantir for the building. Like it understands everything. Can say like if there is a robber outside, there's a car that's parked that shouldn't be there, it can like intelligently pick up on those things. Assume a building has just come alive and it is sentient and intelligent because of all the sensor data it has. I think that's physical AI. Like in the Venn diagram, it's clearly physical AI, it's clearly outside of robotics. And that's also a big problem that's getting solved. So largely speaking, physical AI is the entire domain where the world of atoms makes smart decisions, but robotics in specific is the frontier because it's still unsolved for in a lot of ways. And that would be embodied AI, brilliant companies doing some cutting edge work around that. And I would break that down into data, simulation, deployments and foundation models, obviously, that I like to learn from these, broadly speaking. Yeah.
00:08:31 · Leo, Varun Nair
[00:08:31] S1: I see. That also bridges the gap with something interesting you said earlier. You've been into data, you've been into deployment, and lately you've been interested in SIM simulation. Usually, people are into simulation before doing deployment. So how did you, like, get into real world stuff before getting into into SIM? [00:08:50] S2: I mean, you could go either way. The thing is you do deployments or you just, like, evaluate in the real world and you see that policies don't work very well and you're like, hey, there might be a better way to scale this up. There might be better ways to run a thousand different trajectories rather than resetting one manually again and again. And that's how I found my way. It's not like I hadn't dabbled in SIEM before, I have, but it's interesting because you have data on one side, you have deployment on one side and you have SIEM sitting in this middle where learning in SIEM is smart, but there is a real gap that exists between transferring a policy that exists in SIEM versus in actual deployments. But it's the other way too. When something fails in deployments, it needs to be rerouted back to SIEM as well to make the simulation even better. And that's a piece that's missing. So it's not just that SIEM informs deployment, but it's also that deployments will inform how to make better SIEMs. I'm actually putting something out there about this later today or tomorrow. So hopefully it'll give you better context about it as well. Yeah.
00:08:50 · Varun Nair, Leo
[00:08:50] S2: I mean, you could go either way. The thing is you do deployments or you just, like, evaluate in the real world and you see that policies don't work very well and you're like, hey, there might be a better way to scale this up. There might be better ways to run a thousand different trajectories rather than resetting one manually again and again. And that's how I found my way. It's not like I hadn't dabbled in SIEM before, I have, but it's interesting because you have data on one side, you have deployment on one side and you have SIEM sitting in this middle where learning in SIEM is smart, but there is a real gap that exists between transferring a policy that exists in SIEM versus in actual deployments. But it's the other way too. When something fails in deployments, it needs to be rerouted back to SIEM as well to make the simulation even better. And that's a piece that's missing. So it's not just that SIEM informs deployment, but it's also that deployments will inform how to make better SIEMs. I'm actually putting something out there about this later today or tomorrow. So hopefully it'll give you better context about it as well. Yeah. [00:09:48] S1: Yeah. We recently discussed with an engineer saying that simulation was still struggling to accurately represent actuators, for example, since robots are pretty much a bunch of actuators. It tells you a lot about how accurate sim can be. Right?
00:09:48 · Leo, Varun Nair
[00:09:48] S1: Yeah. We recently discussed with an engineer saying that simulation was still struggling to accurately represent actuators, for example, since robots are pretty much a bunch of actuators. It tells you a lot about how accurate sim can be. Right? [00:10:02] S2: Yeah. That's really fair. And the frontier of sim is getting better every other month. Nvidia, they swapped out their PhysX engine for Newton that is supposed to be doing deformables, fiction and contact dynamics better, so it will definitely improve in the next coming times. We have companies like V sim, the creators of Isaac Sim are now building their own simulation in The UK. It's a frontier that's definitely getting worked on. And it might also eat up the data business, to be very honest. If simulations work and are calibrated well enough, think about it this way. If I can model the real world in sim, why can't I just use sim as data? Like, it is totally possible. Like, these are, like, components that interact with each other but can also eat into each other very much so. [00:10:43] S1: Do you include world models such as World Labs Atlas into your simulation category?
00:10:43 · Leo, Varun Nair, Oskar
[00:10:43] S1: Do you include world models such as World Labs Atlas into your simulation category? [00:10:50] S2: That's a good question. And I might not be the best expert to talk about this, though I have spoken to some of them, but it's a spectrum. So you have things like Isaac Sim, MuJoCo on one side, which is like simulating every single object, and then you have world models on the other side. Both have their pros and cons. Have world models can like generate fluid, like fluidity better, so rather deformable objects better because it's pixel generation. So it is actually helpful to have these three d rendering like Atlas exist in world models and that is part of simulation. But when you want to do rigid contact dynamics and physics and understand the physics of it, world models aren't the best in understanding physics because it's, again, it's learned from pixels. It's not learned from action data. It hasn't learned the inherent physics behind it. So yeah, they are two separate things, but a good simulation needs to leverage both of them rather than using just one of them. [00:11:43] S0: I see. Maybe just to understand your role a bit better right now, you said you invested in a friend's company. Are you like creating a company yourself? Are you kind of stealthing around? If you have some practical experiences or some recent learnings that you can share from actual deployments, that will be super interesting as well.
00:12:01 · Varun Nair
[00:12:01] S2: For sure. So I'm mostly focused on just helping out the ecosystem as a whole, working at a company that's stealth currently, so can't tell you much about it at the moment. And just breaking down my learnings so far, it's that look at the data, guys. I think that's one of the biggest learnings I have realized in the last year and a half. It's that I created this open source project for this as well. There is no good definition of what good data looks like. Maybe in pre training you don't care about it because it's like, hey, just scale it up and throw it at a model to learn from. And that's good. It'll get you up to a certain point. But it's not enough. Pre training is one part of it. You want to be able to learn from multiple aspects. Let me take a step back. You guys have heard about Astra. Astra is making the rounds about how it's going disrupt the robotics field. I eval ed Astra to find out, and not just me and a lot of people in the industry did, it has great spatial and geometric understanding, but it has really bad sense of depth. So it cannot capture something really well. It can't touch upon let's say a glass, it'll touch right above it. It's because it has not been trained on depth data. It's as simple as that, you add depth into the precleaning mix Astra would perform wonderfully and it probably is something OpenAI, I'm sure, is trying to do at this point. It's about adding more modalities within data and actually learning from it. We can go deeper into these things as well. There are some data that are missing, people are not looking at quality of data, hence its models aren't doing that well. There are a couple of takes I have there. But yeah, broadly speaking, I think look at the data and understand where the failure modes of this data are and how you can sort of fix that is where a lot of this lies. You will see the same thing for post training as well. Deployment data will become post training data going forward.