Adarsh Hiremath, co-CEO of Mercor, says enterprise agents need tribal knowledge that is often not stored in a form agents can use, so evals must capture what good looks like.
Hiremath says output metrics can mislead, so evals should grade the whole trajectory from inputs to reasoning to output, not just the final answer.
Hiremath says most first-round agent updates should happen in context rather than model parameters, because that is more compute efficient than fine-tuning.
Evals decide whether enterprise agents actually work
Mercor argues that evals are the core infrastructure for enterprise agents. The knowledge that makes work valuable is tribal and undocumented, so agents must be calibrated to a company's own standards and judged on their whole trajectory, not just their output. Representative eval sets also drive the learning loop that fixes failures and keeps them from recurring.
Takeaways
Enterprise agents need specification and evaluation. Hiremath says the work splits into telling an agent what to do and judging how well it does it. The needed knowledge is tribal and rarely digital, so a general model fails without calibration to a company's own standards.
Evals must cover the whole trajectory. Output metrics can mislead, since an agent with a high resolution rate may still communicate badly. Evals should cover the full path from inputs to output, with rubrics that break vibes-based judgments, like consulting slide decks, into gradable items.
Most learning should happen in context, not weights. First-round updates are more compute efficient in context, and fine-tuning is reserved for open source models when warranted. Mercor's platform discovers company context and deploys agents in tools like Slack, then improves them by finding failures and suggesting diffs to a skills file.
Silent failures and security shape rollouts. Failures behind systems with no chat interface are more dangerous than visible ones. Regulated customers, with financial services leading, want sandboxes before production. Mercor's agents already stop automatically when they cannot cite company context.
Apex benchmarks target whole-system work. Apex SWE tests observability and integration tasks across systems rather than small patches, and grades code beyond correctness. Frontier labs examine its data and hill climb on it. The next gap is benchmarks for agents working with each other.
Value will shift from tokens to outcomes. As agents take employee-like roles, Hiremath expects pricing to move toward outcome-based value. He favors applications that raise net productivity over pure cost cutting, and calls enabling agents with enterprise context a multi-trillion-dollar problem.
Mercor wants to be the system of record. It aims to be the system of record for specifying agent behavior. Inference spend may grow a thousandfold, which makes representative eval sets more important for choosing models. Each customer's context stays separate through tenants and staff walls.
Predictions
Agents that take actions needing gating will outnumber humans within a couple of years.
Tap a name to go there · tap a line to see why they are linked
On this episode 3
Show 1
Within twelve to eighteen months, agents will collaborate with, orchestrate and oversee other agents.
One host expects agents to become thin orchestrators over a trajectory-driven system, with 2026 as the year of the model.