Ep. 042 - GDPx: The Benchmark for Frontier AI

SemiAnalysis launches GDPx to grade frontier AI on real work

GDPx is SemiAnalysis's bid to own the frontier AI scoreboard. Existing benchmarks are full of unfair tasks, so the team commissioned 14 data partners across 11 domains and graded models against real work, including a private equity deal room. The first leaderboard shows models failing binary-graded finance and legal tasks while safety classifiers suppress cybersecurity and biology scores. The team reads that as evidence day-to-day knowledge work is close to solved, though it splits on what that means for jobs.

Takeaways

• GDPx grades frontier AI across 11 domains. SemiAnalysis built it with 14 specialized data partners and ran QA on every task. Domains run from software engineering and chip design to finance, legal, biology, cybersecurity, quant trading and material science.
• Benchmark tasks are unfair rather than hard. Max Kan says many benchmark makers lack eval expertise, so tasks turn unfair rather than genuinely difficult, such as a verifier scoped to one of two equally valid fixes. QA found roughly 60% of Terminal-Bench 4 tasks broken.
• Binary grading exposes failure on real work. On finance, models scored 0% under binary grading except Carbon, which solved one task. Partial credit lifted finance into the 50s and left legal in the thirties or forties, while most other domains score continuously.
• Safety classifiers suppress scores in sensitive domains. Astra refused every cybersecurity task and scored 0%. The headline view excludes the domains where classifiers sandbagged scores, and Carbon still trails Astra and Opus on that apples-to-apples ranking.
• Finance tasks came from a live deal. The finance partner used a real private equity deal, a firm buying a family-owned snack company for just over $800m for 60%, with about 4,000 files across the deal rooms of the PE firm, bank, accountant and company.
• The team splits on AI and jobs. Max Kan says AI plus human coworking has kept good engineers more valuable than ever and expects people to uplevel themselves. Andrew Megalaa says AI has raised the hiring bar and disagrees with that optimism.
• Google sits third behind Anthropic and OpenAI. Max Kan says the gap is likely still wide because both leaders have new flagships coming, and the internal best models across OpenAI, Anthropic, Google, Meta and xAI differ a lot.

How GDPx shipped in a week

1. Tasks were finalized on Thursday night and the benchmark launched on Sunday, a self-imposed deadline set to avoid the following Monday, when a rival flagship was expected.
2. Andrew Megalaa switched to polyphasic sleep for the week and set Claude to wake him when infrastructure broke.