Dream Machines: Physical AI, Robotics, Being Human
00:08:39 · Evan Helda, Aravind Kandiah
[00:08:39] S1: That's really... That's a useful insight. So when you guys talk to these customers, and this kinda kicks off the first question I wanna open with, how is it that you guys are helping them avoid these failure modes? Like, when you go in to talk to these customers, what is your story? Perhaps within this answer, you can kind of give us the thesis of the business as well, we can kind of pack a couple answers into one here. But when you go talk to a customer, these guys probably look at you and they think, what do these guys know that I don't? And they also probably have a bias towards real world data oftentimes and not synthetic data. And they look at you and they say, these guys haven't built robots. What are they? Again, same kind of idea. How do you quickly earn the credibility and trust and then convince them to avoid those pitfalls? And then from there, what is the general story and thesis of Bifrost that you try to land with them in that first interaction? [00:09:30] S0: Yeah. Maybe I can answer the first part, then Charles can answer the the second part and the thesis. [00:09:35] S1: Sure. [00:09:36] S0: So the interesting part is when folks actually come to us, a lot of the times, they're just like, hey. We just need more data of x, like a specific scenario. So they aren't even going into the part of, like, hey. Have you guys built robots or or not? And fun fact is, when me and Charles first started the company, we actually built robot. [00:09:56] S1: Perfect.
00:09:57 · Aravind Kandiah
[00:09:57] S0: So we actually built what looked like physical intelligence, like, first product. And it's, like, a whole bunch of, like, stuff that we posted online, but we used, like, a UR arm. And synthetic data was just the way we trained it. We tried to sell it to factories, but all the factories were like, you know, if you want to sell us this whole thing, you have to build an entire production line, and then we realized it wasn't like a venture scalable business at that time. Right? So coming back, when teams comes to us come to us, their first thing is like, okay. You know, we're building this autonomy system. The performance is not good enough. We are failing in a whole bunch of scenarios. We just need data for x y z scenarios. So that's usually, like, the opening point. Like, they'll come to us and they're like, oh, can you just give me more data for x? But then we tell them, okay. First of all, like, let's, like, slow down. We can give you guys the data, the tool to generate all that data, but let's first look at your evaluation stack. Right? Like, how are you actually determining where the models are failing? And this is actually the most critical part where a lot of teams fail. It's because the evaluation set is either a, super biased, or it's just very, very low coverage. For example, let's say you're building an autonomous boat. What most teams do is they go... They bring the autonomous boat to a specific port in the world and then just collect all the data from that specific port. Right? So their test data is only of that port of specific weather conditions.
00:11:26 · Charles Wong, Aravind Kandiah, Evan Helda
[00:11:26] S2: Mhmm. [00:11:26] S0: Same for an indoor robotics company. They go to a few types of indoor households. They collect a whole bunch of data, and that is their small test set. And whatever metrics you're getting from that is actually like a false perception of performance because it's just on a small subset. So what we do is we come in and we say, hey, guys. Let's speak to your end users and find out what are the scenarios they care most about, and let's map out all the possible scenarios that your system needs to be able to handle. Right? So this is like a scenario grid of all the different permutations that the robot needs to be able to handle. And then once you can map that all out, now you have a truly diverse high coverage tester. And then what we do is we map your real data to that, and then you can then see that, oh, actually, I only have a 30% coverage of all my test sets that I... All my scenarios that I need to cover. And then that's where we begin the journey. Right? So now you have your benchmark, which is high coverage. [00:12:24] S1: Mhmm. [00:12:25] S0: We start testing against it, and then we hill climb that benchmark. And then over time, the end user comes in, and they add in new requirements and new operate scenarios. [00:12:34] S1: Mhmm. [00:12:34] S0: And then your benchmark constantly grows as well. [00:12:36] S1: That's interesting. Question on that 30%. So how exactly do you guys determine that number and and quantify that? And and how do they... How does the customer or you understand what a 100% should be?
00:12:36 · Evan Helda, Aravind Kandiah
[00:12:36] S1: That's interesting. Question on that 30%. So how exactly do you guys determine that number and and quantify that? And and how do they... How does the customer or you understand what a 100% should be? [00:12:50] S0: Yeah. So I think the interesting part is we'll never be able to hit a 100% because of the thing, like, you can you can never, like, get all the long tail scenarios. [00:13:02] S1: Yep. [00:13:02] S0: Right? So the way we think about it is there's two types of, like, coverage we call... The first one is... It's quite straightforward. It's just, like, operational conditions. So the way you can think about this is let's take, like, the the boat example. Right? Let's say you're deploying an autonomous boat and you're putting it in a port. There's actually a few key conditions that you care about. So maybe it's sea state, types of boats, different types of weather conditions, different types of, collision approach... Like, approach angles. All of these parameters, you can actually ask any sea captain or any, like, domain expert, and they'll able to list them all out for you. We list them all out, and then this forms like a permutation grid. Right? So you can think like you're doing parameter sweeps, but you're doing parameter sweeps across scenarios. [00:13:54] S1: Mhmm.
00:13:54 · Evan Helda, Aravind Kandiah
[00:13:54] S1: Mhmm. [00:13:55] S0: Right? And this forms, like, this elaborate grid where it covers every single permutation. Right? So that's, like, bucket one. Bucket two is what I call these anomalous long tail scenarios, where it's like a giant whale comes into the scene. A plastic bag is covering a a sea lion, and it's flailing about. All these very weird things. [00:14:20] S1: Mhmm. [00:14:20] S0: These things actually, like, AI agents are really, really good at coming up with these long tail scenarios. And humans, as they see these things come up, they add them to the the long tail bucket of stuff as well. Mhmm. And this is the same approach that Tesla has taken as well. And it's just the thing that you constantly grow this, like, a long tail scenario library. Right? And the interesting thing that we have seen... Sorry. Go ahead. [00:14:44] S1: I was saying that's a really helpful way to break it down. [00:14:47] S0: Yeah. Because this one, the first bucket is really easy to quantify. But the second bucket, which is like all these anomalous long tail things, you can't really quantify because it's an infinite number of scenarios that can happen. Mhmm. Right. [00:15:01] S1: So you used a fancy word earlier that I doubt most of my audience knows, which is parameter sweeps. And since I'm on the mission to help educate people on the technical elements of the space, can you quickly define what parameter sweeps are and why they matter?
00:15:14 · Aravind Kandiah
[00:15:14] S0: Yeah. So the way you can think about parameter sweeps are... Let's just take a super simple scene. Right? Let's say you are outside and you want to be able to just detect a chair. What a parameter sweep would look like is, let's say I want a parameter sweep lighting conditions. It means I want to move the sun angle to all the different parts of the... Like, from sunset to sunrise through the... All the different angles so you see the chair at all the different lighting angles. So that's what it means, like, parameter sweep through the lighting angles. So you could also do a parameter sweep through different types of chairs. Right? So modern mid century chair, an old antique chair, chrome metal chair, and there's, like, parameters sweeping through the object types. And there's also parameters sweeping through the angle in which you're viewing the chair. So you can see how these, like, compound really, really quickly. And the goal really is just, like, can you provide these teams with these high coverage datasets so that they can test and see where their model is failing even before they deploy it? So a lot of the times, they would generate these tests, these parameter sweeps, and then they're like, oh, actually, it fails on a Chrome chair at this specific angle and this specific lighting condition. We didn't know that this would happen, so they'll go in and they'll try to fix it.