Going Direct Conversations
00:07:04 · Michael Cho
[00:07:04] S1: Yeah. So how I started in this space was I literally was just with my younger brother, Sam, who's my co-founder. So during the pandemic, we were just like... On the weekend, we'll... So he's in China, I'm in Singapore at the time. And during the pandemic, I want to say it. So it's the end of the world. We don't know what we should do, but we want to have a reason to keep in contact. So every weekend, we'll come together, build our own sidewall robot. It honestly just started out as a weekend hacking project to keep in touch with my brother. And then we decided to start a company because our observation was that a lot of sidewalk robots that were built by these bigger startups and the Amazons of the world, their robots are pretty expensive, like $10,000 or more. Because at the time, we were assembling our own. I have a pretty good handle on the build of materials. And so we thought that if you take the business worldview, maybe we already have the recipe, all this offline RL invitation learning stuff, which I think four years ago is starting to show some signs of life. And if you look at these robots as real-world robotic data collection devices, like $10,000 robot will just not scale, like won't get you the data. So we thought, okay, very naively, to be honest, at that time, me and my brother have never done robotics as a real, we are not roboticists to begin with. We thought that maybe someone should build a toy that's like $500, that's like a science. sidewall robot toy basically and make we thought it can be built at that price by just making a dumb robot so yank out the big gpu very cheap cameras basically and just have it 100 like human control so no ai so just a dumb robot this is during the pandemic right so the other of the fact that we made was that There was this huge investment that was brought forward, let's say, a decade so that we can have video calls like this. Right. So because before that, the Internet wasn't really geared towards like real time communication, but because of that. So we thought, OK, maybe it's possible for someone in New York to control a robot toy somewhere else. And that could be a really interesting gamification. So honestly, it all started off like that. Everything along the way, you know, it's a stumble into it after the other.
00:09:25 · Michael Cho, Jordan Wolfe
I'll say why I got a lot more into this robotic research is from this robotic game. So think of it almost like a Pokemon Go where you have human gamers that drive around, like pick up virtual items on a map. We started gathering data from very early on, and it turns out that data was actually really useful for some of the folks that are doing urban navigation robotic research. And that's how we started working with some of these really top researchers. Like, for example, we co-organized for a couple of times now this competition called Earth Rover Challenge, where we pit the top human gamers in our game. versus AI models that are built by different labs, right? So, UT Austin, Seoul National University, National University of Singapore. And that one is actually co-organized together with many folks from DeepMind. And so, yeah, I mean, I thought that there's something there because I was such a small, tiny little project. to, to get the attention of this top researcher to spend time with me to organize competition like this. I thought there is something there. And so, yeah. So anyway, yeah. [00:10:32] S2: So essentially just like to summarize then, so that kind of the idea was game game, the gamification of basically tele-operation of, of robots to collect data to help in the training process. Is that kind of the best way to describe it?
00:10:32 · Jordan Wolfe, Michael Cho
[00:10:32] S2: So essentially just like to summarize then, so that kind of the idea was game game, the gamification of basically tele-operation of, of robots to collect data to help in the training process. Is that kind of the best way to describe it? [00:10:48] S1: Yeah, I, yeah, pretty much it. Yeah. I would say that's not the only thing that we do now, but that's definitely, we're still, it's, it's a big thesis of mine. I, I, I kind of feel that we're still a long way from having robots that actually work in homes, which is what, you know, Chris's dream. It's my dream too, but I think, you know, there's too many bottlenecks, but, uh, I think these robots are funny before they are useful. So I think like the current product market fit is actually doing entertainment with robots. Whereas we've seen, I actually also have a separate project that do this like humanoid fighting thing. Yeah. So those couple of things that you may have seen in San Francisco, like, uh, you know, I was kind of behind it for a number of these events. So yeah. Um, Yeah.
00:11:33 · Jordan Wolfe, Chris Paxton
[00:11:33] S2: So let's talk about those bottlenecks, right? So obviously, it's common knowledge that obviously there's a training gap. And obviously, it's like you need the most diverse, large data sets and to deal with edge cases and with larger context windows. We can go on and on and on. Can we drill, I want to get each of your perspectives on really where the main gap lies and the main bottleneck lies because it's so hard to determine between what, especially folks that are building robots put out there from a marketing standpoint and how far away we are to actually make them useful. So Chris, maybe I'll go over to you first. If you were to boil down if there's one or two things that really stand out to you of... the bottleneck around the current bottleneck around training gaps and, you know, where are we really at? What is the current state of the industry? [00:12:26] S0: I mean, I think that the hardest things to do are to get robots out there into the real world and make them do stuff, which, I mean, it's kind of circular, right? But, like, getting robots and deploying them and making them work reliably and robustly and survive. It's like, can you make the robot actually work for a long period of time? It's like, I mean... We need data, but that's secondary to getting robots out there. I think a lot of the time, making them work. Once they work, the data comes in naturally. They're always producing it. [00:13:05] S2: Michael, what's your perspective?
00:13:09 · Michael Cho
[00:13:09] S1: Yeah, I think broadly speaking, I see two huge bottlenecks in robotics. One is obviously lack of robotics data. I think that, so if you look at the internet, like I think that the whole internet is just the output of humans' conscious thoughts, right? So we're able to record those things down. That's why those are great training models for LLM, VLMs and whatnot. But the way humans like, you know, move in a physical world, like the way I move my water bottle, I can only show you. but I can't record it at, let's say, 30 hertz, you know, the exact angle, the velocity and all that. So these action labels, because we can't record it, so that's why the data is totally missing. And except, I guess, maybe Tesla, they put a human in a robot on wheels and then they call it what the human driver is doing. So those action labels. So I think the robotic data is very much needed and there are three types of robotics data, like tablet operation and video data and also like synthetic or simulation data. So that's one big bottleneck, just data and three types of data. The other huge bottleneck I see is real-world evaluation. And I think this is something that is very unique to physical AI. In the digital AI run, like if you, let's say, if DeepMind or OpenAI, if they drop another reasoning model, typically, you know, obviously they have a bunch of benchmarks. And if you look at all those benchmarks, right, like for example, they will take, let the AI do, let's say, a math Olympian test. That test can be done like just like that, right? So even though there's no perfect benchmarks, getting the benchmarks is never a bottleneck in the iteration cycle. Whereas the physical AI, like you can test all you want in simulation, but ultimately, like literally rubber hits the road, right? You know, you need to see how they fail. And because you can't speed up time, Like, how do you know, for example, whether you build a grid model unless you measure how long does it take before it fails? Like, this is the only way to do it. And so I think real world eval would be a huge bottleneck. Literally, if you look at robotics, I mean, Chris and I talk about this all the time, there's no benchmarks. You know, there's no benchmarking because it is just nearly impossible to have benchmarks in robotics. Whereas everybody's robots difference.
00:15:25 · Chris Paxton, Michael Cho, Jordan Wolfe
[00:15:25] S0: Everybody's house is different. [00:15:27] S1: Yeah, exactly. In that paper, they have this success rate. You just have to take it that, okay, they did their work and it is what they say, but no one can verify it. [00:15:38] S0: Sure. It's getting better, but yeah, definitely not. Oh, that's so hard. Yeah. [00:15:42] S2: All right. Well, it's an interesting question, right? I guess this one's for you, Chris, because obviously you're working on or agility is working on the humanoid form factor. And it's kind of like when the world of robotics is like, just show me. It's very different than some abstract or benchmarking process to benchmark LLMs. [00:16:07] S0: Is that okay? Yeah. The thing is that you can't just get a good feel for how a robot works from a video because you can always get a good video. Honestly, the other thing is that I think people have seen lots of really good robot videos for a long time because they've seen Star Wars or whatever. You've seen robots that can do things that they can't even do now. I think you have to be able to show robots in the real world and see it for yourself to be able to see what it can do. I think a lot of videos can be misleading. I wanted to drill into the different types of training, methods for training.