Dream Machines: Physical AI, Robotics, Being Human
00:59:09 · Evan Helda, Aravind Kandiah, Charles Wong
[00:59:09] S1: That's that's a really thoughtful approach for sure. Go to Aravind. [00:59:13] S0: To to Charles' point, synthetic data is just like a tool. Like, it's like a small part. But the part where we see a lot of people, like, make mistakes is actually the how you use synthetic data. Right? Because it's like, a lot of people actually, like, they may have, like, you know, maybe it's somewhat good enough synthetic data, but how they use the synthetic data, like, the way they use it to train their workflow, the approach, There's so many mistakes in their training and testing phase that actually sets up the project for failure. Right? So for us, it was like, okay. How do you build a more systematic workflow that is not just about how you generate the data, but it's like, how is that data used to train? How do you detect the failures? How do you pipe that back into the simulator and generate the right kind of data? I think the most important word is the... Like, are you generating the right kind of data, and are you training it in the right way? And I think that's the mark that most people miss. They just try to make, like, pretty images essentially. This is like, oh, you know, a general image, it looks like photorealistic [01:00:16] S2: Mhmm. [01:00:17] S0: Syntactic data is done. But they miss that that two parts. Like, are you generating the right kind, and are you using it in the right way?
01:00:17 · Aravind Kandiah, Evan Helda, Charles Wong
[01:00:17] S0: Syntactic data is done. But they miss that that two parts. Like, are you generating the right kind, and are you using it in the right way? [01:00:23] S1: Yeah. Yeah. This whole idea of finding... You know, ensuring or measuring signal versus noise and understanding if if a dataset, a synthetic one or even a real one, is statistically meaningful and, like, moving the needle with model improvement. And it sounds like Stardust, eval tool, is maybe the way to measure that. But, like, maybe break down a little bit for for us. How do you ensure you're avoiding, like, empty calories with the data? And how do you get the signal and the noise in a sort of real time way? Do you have to wait until you just get the policy in an eval and see how performs, or is there something you can do before that to understand how the needle is being moved on the model's capability or the or the policy's capability? [01:01:10] S0: Yeah. Charles, do wanna take this? I can take this. [01:01:14] S2: Yeah. Go ahead. So I [01:01:16] S0: think it just starts first with, like, the end user and what the thing that they're trying to achieve is, setting up a really, really good scenario grid on the evaluations. That's like step one. [01:01:27] S1: Mhmm.
01:01:27 · Evan Helda, Aravind Kandiah
[01:01:27] S1: Mhmm. [01:01:27] S0: And then step two is that, okay, once you have that really, really good scenario grid, can you map your real data to that synthetic grid? So you now have a real test and a synthetic test, but they both have different levels of coverage. The synthetic test definitely has a higher coverage and your real test has a lower coverage. And then you're testing against the two, and number one, you're seeing equivalence. So it's like, if the model is failing on the real, in some cases, and passing on the real, is that happening with the synthetic examples as well? So that is also a confidence building exercise where it's like, hey. Actually, the results you're getting on your real are reflected in the results in SIM. Once you have those two, you can start to see the distributions. And your goal is essentially just to align the distributions so that over time, as you collect more and more data, you're just whacking the the the performance scenarios down over time. Right? Just
01:02:22 · Charles Wong, Evan Helda
[01:02:22] S2: to add to that, the under... The the key to this is, you know, it's... It it starts before you even generate the first piece of data. Right? It's understanding what data you should be generating. Right? So it behooves you, and we have tools for this that we provide to our customers. We... It behooves you to, like, look at your current model, your failures, and figure out where it's failing. Right? Do some clustering, do some embedding space, you know, like filtering and stuff, and figure out, okay. I have achieved only 70% accuracy on this test set. Right? That 30%. Where is it failing? Oh, if I cluster it up, I can visually see... Can use VLM for this as well. I can visually see that, you know, it's failing when there's a white box against the white table, you know, or a, you know, blue box against a green background. And suddenly you say, like, this is precisely the thing I need to fix. And, you know, you can even then make assumptions like, oh, if I fix this cluster of problems, I can expect my model to jump by 10% accuracy because these are 10% of my failures. Right? [01:03:20] S1: Got it. [01:03:20] S2: And then you feed that directly into your model, to your your data generator, you go from there. Right? That's Got you know, hinting back to earlier about that systematic loop. How do we get people motivated to have this success?
01:03:20 · Charles Wong, Evan Helda
[01:03:20] S2: And then you feed that directly into your model, to your your data generator, you go from there. Right? That's Got you know, hinting back to earlier about that systematic loop. How do we get people motivated to have this success? [01:03:30] S1: That's that's also helpful. And I think, at least for me, so many people... It's like simulation is is a sexy topic, but so few people see into the black box of exactly how to set this up for success. So I appreciate that breakdown. Let's let's look into the future as we as we wind things down here. I use this term, data diet between synthetic versus real and, like, what that what that right balance is today versus tomorrow. What do you think... If we look... Let's say, I know it's so hard to predict the future because the future is here and it's evolving so fast. But if you had to if you had to guess five years in the future, what is that data diet? And do you think that we're gonna get to a point where it's all simulation, it's all synthetic, and you don't even need to touch real world data? Or do think real world data is gonna always have a place even in even in five years when it's incredibly easy to create everything and edge cases.
01:03:30 · Evan Helda, Charles Wong
[01:03:30] S1: That's that's also helpful. And I think, at least for me, so many people... It's like simulation is is a sexy topic, but so few people see into the black box of exactly how to set this up for success. So I appreciate that breakdown. Let's let's look into the future as we as we wind things down here. I use this term, data diet between synthetic versus real and, like, what that what that right balance is today versus tomorrow. What do you think... If we look... Let's say, I know it's so hard to predict the future because the future is here and it's evolving so fast. But if you had to if you had to guess five years in the future, what is that data diet? And do you think that we're gonna get to a point where it's all simulation, it's all synthetic, and you don't even need to touch real world data? Or do think real world data is gonna always have a place even in even in five years when it's incredibly easy to create everything and edge cases. [01:04:29] S2: I have a I have a bit of a left field answer here. Maybe this is less technical, but this is more like... It's just we're talking five years out. Let's talk vision a little bit. Right? Mhmm. When we think about real versus synthetic data, we're trying to think about what do we need to, like, dream up or hallucinate for robots to learn from rather than just experiencing the real world for themselves. [01:04:48] S1: Mhmm.