Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor)

Mercor's rise shows expert-written evals now gate AI progress

Mercor is built on the view that AI progress is now limited by measurement. Labs cannot improve a model without a way to score it, and scoring real professional work takes real professionals. Mercor supplies those experts to the top labs, which explains its speed.

Takeaways

• Evals are now the labs' main bottleneck. Foody says if the model is the product, the eval is the product requirement document. Labs struggle most with measuring what success looks like, so evals steer training and also work as sales collateral showing what a model can do.
• Evals now test real work tasks. Older academic benchmarks such as GPQA and Humanity's Last Exam are giving way to practical tests like building software or doing investment banking analysis. A lawyer, for example, writes a rubric for a good redline contract that models are scored against.
• Training is moving toward AI feedback. Foody says the industry is shifting toward reinforcement learning from AI feedback rather than human feedback, and Rachitsky notes an Anthropic cofounder told him the same. Foody argues evals are evergreen, so labs will always need experts to create them.
• Expert labor pays far more than crowdsourcing. Mercor has tens of thousands of experts working at any time, at a median of about $95 an hour and up to roughly $500, against about $30 for crowdsourced workers. Expert networks like GLG mostly arranged one-off calls, while Mercor hires for longer projects.
• Growth has been exceptionally fast. Foody says run rate went from about $1m to $400m in sixteen months. Rachitsky adds a $100m raise at a $2bn valuation, net retention above 1600 percent and no churn. Mercor works with six of the Magnificent Seven.
• Elastic-demand jobs will be most valuable. Foody says software is the most elastic field, so ten times more productivity should mean more gets built, while accounting demand is limited. He expects AI to automate most knowledge work within a decade, driven by more post-training data, and calls superintelligence fears probably overrated.
• Mercor runs on high goals, not long hours. Foody says Mercor set ambitious targets, such as a $50m run rate it promised Benchmark, hires to a high bar, and brought in former Uber executive Sandeep Jain as president. It never mandated long hours, and it scaled from 10 to 100 people too slowly.

How Mercor found its market

1. Founded in January 2023 by cofounders who met at 14 and started the company at 19.
2. It first hired people internationally and used language models to automate matching.
3. It bootstrapped to a $1m run rate before the founders dropped out of college.