Brendan Foody says AI is now close to superhuman at evaluating anything a human can judge over text, from interview transcripts to resume signals.
Mercor automates the steps a human recruiter takes, from resume parsing to ranking, with separate evals for each step, and says the end state may be an API that gives a confidence interval on how a person will perform.
Foody argues crowdsourced data labeling has broken down as models improved, predicting churn for companies stuck in high-volume work while the human data market grows by orders of magnitude.
AI now judges talent, and Mercor is betting on it
Mercor is building the evaluation layer for work. AI can now judge people over text as well as humans can, which began with hiring and moved into expert data for frontier labs, where crowdsourced labeling no longer works. Foody sees the same capability extending to contractors and full-time roles.
Takeaways
AI now judges text-based work near superhuman. Foody says models can assess anything a human can judge over text, including interviews, written assessments and resumes. Mercor's first GPT-4 interviewer hallucinated every two or three questions, but reasoning models fixed context handling. Multimodal evaluation and vibe checks on passion still lag.
Crowdsourced data labeling is breaking down. Once models got strong, labs needed high-quality experts working directly with researchers, not pipelines of hundreds of thousands of crowdsourced SFT or RLHF items. Foody expects churn at high-volume vendors, while human data and evals grow by orders of magnitude.
Mercor automates hiring end to end. Resume parsing, scoring, interview questions and ranking each have their own evals. Off-the-shelf models handle basics, while post-training on customer outcomes drives the final prediction. Foody sees the end state as an API giving a confidence interval on performance.
Chase real pain points, not forced fit. A customer introduced Mercor to xAI's cofounders in August 2023, and frontier lab work began about six months later. Foody advises finding strong pain points where rich customers will pay. His earlier bootstrapped business reached a $1m run rate.
Productivity gains may increase hiring. Foody says demand to build 100 or 1,000 times more software is elastic. The most useful skills are fast learning and daily work with AI. He calls what humans do in five or ten years the biggest open question and faults regulators for ignoring economic disruption.
Mercor kept dilution low to fund incentives. Preemptive Series A and B rounds held dilution to about 5%, building a war chest for referral incentives and post-training data. Mercor recently raised $100m.
OpenAI stays a product company. Foody thinks a general API cannot support many players. Trading firms will eventually strike large partnerships with frontier labs despite being less advanced in post-training. He calls evals underhyped and SFT data overhyped, and says SFT and RLHF budgets are falling.
Mercor's expansion path
Use human data and evals work for top AI labs as the wedge.
Move to contractor hiring for big tech, competing with firms such as Accenture, and then to full-time hiring.
Tap a name to go there · tap a line to see why they are linked
On this episode 2
Show 1
Foody's predictions
An AI software engineer writing pull requests with a higher hit rate than humans arrives later this year or in the first half of next year.
Within two or three years the public will worry that models beat people at their jobs, and custom agents will multiply.