Jonathan Frankle, Databricks' chief AI scientist, says evals are the biggest bottleneck to enterprise AI adoption, since models have been good enough for many complex tasks since Claude Sonnet 3.5.
Frankle recommends prototyping with a foundation model and prompt engineering, then moving to custom models once a use case is committed, which he says can cut costs five times or more.
Frankle argues AI measurement needs the rigor applied to Waymo cars, and says no one has yet solved how to measure agent permissions and governance.
Enterprise AI stalls on evals, not model capability
Enterprise AI is held back less by model capability than by the ability to define and measure good work. Frankle argues that evals built on a company's own data let it trust models, move to cheaper custom ones, and eventually govern agents. Public benchmarks cannot supply this, because they reflect academic tasks and coding rather than messy, large-scale enterprise work.
Takeaways
Evals are the main bottleneck to enterprise AI. Models have handled many complex enterprise tasks since Claude Sonnet 3.5. The hard part is describing what good work looks like and checking it. One Databricks customer annotated 12,000 examples and calibrated 70-something judges.
Prototype on foundation models, then go custom. Start with a favorite foundation model and prompt engineering, and build a custom model only once a use case is committed. Custom models can cut costs by five times or more, as Cursor and Harvey have discussed.
Enterprise AI needs Waymo-level testing rigor. Frankle compares the field to the 1960s of software. He wants AI measured like Waymo cars, which must pass tests before reaching public roads. Enterprises resist sharing evals with model labs, so Databricks plays Switzerland.
Public benchmarks miss how enterprises work. They lean toward academic skills and coding, though software engineering is about 1% of the economy. Enterprise environments of tens of petabytes cannot fit in a Docker container, so Databricks builds in-house benchmarks like Office QA on Treasury documents back to 1789.
Agent governance remains unsolved. Nobody has figured out how to measure permissions, and agents can surface documents people forgot they could access. Cost and governance top customer worries. Frankle puts AI's share of GitHub issues near 5%, well below an Anthropic friend's 50% forecast.
Work moves up a level instead of vanishing. Frankle expects a Goldilocks outcome where work grows more complex and humans handle higher-level and boundary tasks, such as managing agent pipelines. Models automate only single-digit percentages of enterprise tasks. Cheap eval creation is the missing breakthrough.
Measurement can take researchers further than theory. His 2018 lottery ticket paper was an empirical study of small CIFAR-10 models on two K80 GPUs. One reviewer objected that it had no proof. Frankle calls empiricism principled and its lasting lesson is that measurement goes beyond theory.
How Mosaic became a custom LLM company
Mosaic began as a training-efficiency company that changed training algorithms to reach the same eval quality, which was controversial at the time.
Tap a name to go there · tap a line to see why they are linked
On this episode 2
Show 1
The team first trained custom vision models, then a colleague pushed it toward large language models.
It built a multi-node training stack in summer 2022.
The pivot to custom LLMs produced an open source Llama reproduction, a 7bn parameter model trained on 1tn tokens.
Databricks then acquired Mosaic, since custom training and data go together.