Synthetic Data, Evaluation, and the Future of Simulation for Physical AI w/ Charles Wong + Aravind Kandiah

The benchmark that decides whether a robot is ready

Bifrost sells simulation and evaluation to robotics teams, and its founders argue the industry's real bottleneck is testing. Teams treat one platform's real data as a test set or babysit rollouts, then report 80% success as readiness. Bifrost maps every scenario a robot must handle into a benchmark that grows with each new customer requirement.

Takeaways

• Robotics teams fail at testing. The biggest failure mode is treating real data from one platform as the test set, or rolling out policies with a human babysitting, getting 80% success and reporting readiness. Most teams test only 30% of the scenarios they need.
• Bifrost builds a coverage benchmark. The team interviews end users, maps every permutation a robot must handle, overlays the customer's real data, then hill climbs the high-coverage benchmark. It keeps growing as users add requirements.
• Long-tail anomalies cannot be enumerated. Coverage splits into operational conditions such as sea state and approach angles, which domain experts can list, and anomalies such as a whale or a plastic bag over a sea lion, which cannot be quantified. Tesla grows its library the same way.
• Sim-to-real is also a trust problem. Perception is largely solved by world models and diffusion, and new solvers such as Newton approximate dynamics well enough for testing. Contact forces and deformable cloth stay hard. Charles says trust is transitive, so skeptics still refuse to deploy.
• Real-world data should drive the simulator. Bifrost's autonomous simulator thesis pipes real data in to define which scenarios get built and how photorealistic they are, closing the sim gap. Teams can then expand coverage past what the real data contains.
• Maritime and aerial replaced urban autonomy. Charles says warehouse robotics and arms were too fragmented to be interesting. Maritime and aerial customers were already shipping drones and autonomous vessels, had urgency, and preferred buying a data stack to reinventing it.
• Evaluation outlives training. Aravind expects training may disappear while rigorous domain-specific evaluation remains. Most customers already use Bifrost for evaluation more than training, the eval product launches in a month, and an edge-compute imagination engine is the moonshot.

How Bifrost reached NASA

1. Charles and Aravind met in freshman year. Charles worked at NuTonomy, making Ethernet cables and driving the company's cars dangerously for scenario testing.
2. Aravind did pre-ChatGPT AI research and built a smartphone eye-screening app for diabetic blindness that hit 99% accuracy in clinical trials.