Ep 97: Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It

Jev creator says AI valuations misunderstand the technology

Diogo argues the AI industry is miscalibrated. Expectations baked into valuations, especially Anthropic's, rest on misunderstandings of what the technology can do, and benchmarks no longer measure anything useful. The real constraint on progress is architectural, like the KV cache in coding agents, and builders should regain agency rather than trust lab narratives.

Takeaways

• AI valuations bake in misunderstandings. Diogo says expectations for Anthropic and the sector rest on misreading what AI can do. He predicts miscalibration because some value 'building god' as nearly infinite, comparing 'let them cure cancer' to Marie Antoinette.
• Distribution is not the answer. Diogo says companies that went all in on distribution got burned. He asks early hires and investors what share of LLM calls are meant for machine rather than human consumption, and even 'string-pilled' people put it at many nines of percent.
• TypeSafe launched Jev to find weaknesses. Diogo says the team was proud of the model and wanted the field to co-evolve use cases, despite warnings that launching a capabilities product during the 'pacing the frontier' debate was a bad vibe. He defends calling the model 'system one'.
• The roadmap ignores real-world feedback. Diogo says almost none of TypeSafe's roadmap comes from user feedback and people should complain much more. He says the field suffers Stockholm syndrome, like the people in chairs in Wall-E, and wants intelligence to be reliable and provider-agnostic like SQL.
• Benchmarks have lost their meaning. Diogo is anti-benchmarks and says benchmark fatigue set in faster than expected. RLVR is incredible but still human-in-the-loop, and treating intelligence as a single benchmark-measured axis is wrong. He criticizes peers who ship new shapes that overpromise.
• The KV cache constrains coding agent design. Diogo says the KV cache dictates almost every design decision in coding agents, prevents effective routing and subagents, and makes state append-only like an IPython notebook. He floats coding agents free of the KV cache and multiple agents sharing context.

Where Diogo says the labs missed

• Jev could have been built five years ago. GPT-3.5-era models, possibly GPT-3, already met OpenAI's definition of AGI at the time. Diogo wrote a document at OpenAI titled 'We could have had AGI a year ago.'
• GPQA was saturated around 2023. Diogo says he is basically certain there are tons of use cases for LLMs at every intelligence cutoff since GPT-4.
• The labs' greatest strength is financial engineering. Diogo says they structure deals, especially compute deals, to benefit themselves and disadvantage those taking the risk.