Dream Machines: Physical AI, Robotics, Being Human
00:57:06 · Charles Wong
[00:57:06] S2: Yeah. Yeah. I think I think that, you know, another useful way to look at this is sort of like, you know, fortunately, unfortunately, the way AI teams are set up in most companies today, they're being asked to build, like, their software engineering teams, when in reality, they are r and d teams. Right? And there's a difference. Right? R and d teams are supposed to go out and do experiments, you know, come back with a thesis, and you're not you're not... You shouldn't be measured by the outcome. You should be measured by the experiments you're putting into the into the work, the inputs. Right? Mhmm. But software engineering is like, okay. It's it's a fairly structured thing. You have a spec. You build it. You build the feature well. You get a thumbs up. Right? And somehow, we're taking this very complex AI r and d process... AI development process and being asked to deliver outcomes that look like software engineering. Right? And that's often the engineering and... Versus management mismatch, where management is like, oh, you know, I'd love for... You know, we need to ship by December. Before we ship, we need the model to achieve 95% accuracy. Right? Firstly, where do they pull that number from? It's just something that, you know, it's pretty arbitrary. And then now the engineers, the r and d engineers have to go and figure out how we're gonna get there. Right? But a r and d isn't so so so deterministic. Right? So it's this relentless crawl. Right? Mhmm. Even if you gave these, you know, developers, you know, all the right ingredients. Right? The best data, the best pipelines, all the GPUs they could dream of. If you don't give them a recipe as well, the cake is gonna taste like shit.
00:58:37 · Evan Helda, Charles Wong
[00:58:37] S1: Mhmm. [00:58:38] S2: Right? So that's really what we're trying to think about when we build, like, this data patching a loop. Right? How do we take the principles of software engineering, like unit testing, like iterative patches, and iterative workflow, and apply that to AI so that developers have an easier time of stepping their way up to something that actually works? Right? That's how we that's how we think about it. And that's... It's it's a continuous journey. Right? Yeah. One day AI development will be just a a step of the finger, but we're not there yet, and we hope that Bifrost is the is the thing that gets us there. Yeah. [00:59:09] S1: That's that's a really thoughtful approach for sure. Go to Aravind.
00:59:09 · Evan Helda, Aravind Kandiah, Charles Wong
[00:59:09] S1: That's that's a really thoughtful approach for sure. Go to Aravind. [00:59:13] S0: To to Charles' point, synthetic data is just like a tool. Like, it's like a small part. But the part where we see a lot of people, like, make mistakes is actually the how you use synthetic data. Right? Because it's like, a lot of people actually, like, they may have, like, you know, maybe it's somewhat good enough synthetic data, but how they use the synthetic data, like, the way they use it to train their workflow, the approach, There's so many mistakes in their training and testing phase that actually sets up the project for failure. Right? So for us, it was like, okay. How do you build a more systematic workflow that is not just about how you generate the data, but it's like, how is that data used to train? How do you detect the failures? How do you pipe that back into the simulator and generate the right kind of data? I think the most important word is the... Like, are you generating the right kind of data, and are you training it in the right way? And I think that's the mark that most people miss. They just try to make, like, pretty images essentially. This is like, oh, you know, a general image, it looks like photorealistic [01:00:16] S2: Mhmm. [01:00:17] S0: Syntactic data is done. But they miss that that two parts. Like, are you generating the right kind, and are you using it in the right way?
01:00:17 · Aravind Kandiah, Evan Helda, Charles Wong
[01:00:17] S0: Syntactic data is done. But they miss that that two parts. Like, are you generating the right kind, and are you using it in the right way? [01:00:23] S1: Yeah. Yeah. This whole idea of finding... You know, ensuring or measuring signal versus noise and understanding if if a dataset, a synthetic one or even a real one, is statistically meaningful and, like, moving the needle with model improvement. And it sounds like Stardust, eval tool, is maybe the way to measure that. But, like, maybe break down a little bit for for us. How do you ensure you're avoiding, like, empty calories with the data? And how do you get the signal and the noise in a sort of real time way? Do you have to wait until you just get the policy in an eval and see how performs, or is there something you can do before that to understand how the needle is being moved on the model's capability or the or the policy's capability? [01:01:10] S0: Yeah. Charles, do wanna take this? I can take this. [01:01:14] S2: Yeah. Go ahead. So I [01:01:16] S0: think it just starts first with, like, the end user and what the thing that they're trying to achieve is, setting up a really, really good scenario grid on the evaluations. That's like step one. [01:01:27] S1: Mhmm.
01:01:27 · Evan Helda, Aravind Kandiah
[01:01:27] S1: Mhmm. [01:01:27] S0: And then step two is that, okay, once you have that really, really good scenario grid, can you map your real data to that synthetic grid? So you now have a real test and a synthetic test, but they both have different levels of coverage. The synthetic test definitely has a higher coverage and your real test has a lower coverage. And then you're testing against the two, and number one, you're seeing equivalence. So it's like, if the model is failing on the real, in some cases, and passing on the real, is that happening with the synthetic examples as well? So that is also a confidence building exercise where it's like, hey. Actually, the results you're getting on your real are reflected in the results in SIM. Once you have those two, you can start to see the distributions. And your goal is essentially just to align the distributions so that over time, as you collect more and more data, you're just whacking the the the performance scenarios down over time. Right? Just
01:02:22 · Charles Wong, Evan Helda
[01:02:22] S2: to add to that, the under... The the key to this is, you know, it's... It it starts before you even generate the first piece of data. Right? It's understanding what data you should be generating. Right? So it behooves you, and we have tools for this that we provide to our customers. We... It behooves you to, like, look at your current model, your failures, and figure out where it's failing. Right? Do some clustering, do some embedding space, you know, like filtering and stuff, and figure out, okay. I have achieved only 70% accuracy on this test set. Right? That 30%. Where is it failing? Oh, if I cluster it up, I can visually see... Can use VLM for this as well. I can visually see that, you know, it's failing when there's a white box against the white table, you know, or a, you know, blue box against a green background. And suddenly you say, like, this is precisely the thing I need to fix. And, you know, you can even then make assumptions like, oh, if I fix this cluster of problems, I can expect my model to jump by 10% accuracy because these are 10% of my failures. Right? [01:03:20] S1: Got it. [01:03:20] S2: And then you feed that directly into your model, to your your data generator, you go from there. Right? That's Got you know, hinting back to earlier about that systematic loop. How do we get people motivated to have this success?