00:05:18 · Ruby Justice Thelot, Herbie Bradley
[00:05:18] S1: Yeah. Why don't we dive in on that? AI evals are still really bad in quality. Do What you think the good evals are? [00:05:25] S0: There are a few in coding like Frontier SWE, the Cognition one. Terminal Bench four is interesting. I've actually delved into quite a lot of the coding benchmarks, and I still find that when they do publish agent traces for Astra or Fable or whatever, if you examine them enough, can often find that Astra or Fable has in fact just hacked the benchmark in the published trace. And so a lot of these benchmarks still have huge weaknesses, and it's very, very hard to even come up with something which kind of represents the full suite of skills that humans apply in their jobs in the real world. Think SemiAnalysis had this benchmark published yesterday, GDP Val, which sort of, I think, approaches closer to that. Yeah. GDP X. [00:06:06] S1: We just went over it on stream. It actually seems fantastic.
00:06:06 · Ruby Justice Thelot, Herbie Bradley
[00:06:06] S1: We just went over it on stream. It actually seems fantastic. [00:06:09] S0: Yeah. Yeah. I think particularly the comparison in the blog post between the older GDP Val and GDP X on tasks like finance, where there's less verifiable things, and the approach that GDPX took to really construct this very detailed, realistic set of tasks that's more representative of the real world. I mean, this also represents, you know, the sheer amount of effort that is required to even do this. They would need to collaborate with a large variety of data companies to source the required data for this. They've put in months of effort, and still there's like flaws in this benchmark undoubtedly, but it's one of the closest that I've seen to a good effort. [00:06:47] S1: So what exactly makes an eval good fundamentally? [00:06:52] S0: Well, I mean, partly it's just how representative it is of real world work and real world tasks, but also how easily can the AI cheat. And that's definitely a big one that people are putting increasing efforts into. I think it's very hard to, in some domains, find tasks that are actually representative of real world work and where the AI can't figure out some way to cheat if it was capable enough. You know, there's even the idea of... In mathematics, you could have a benchmark, which is like, oh, publish a novel mathematics paper that wins a Fields Medal or something. But even that can, in theory, be reward hacked or good hearted because the AI could figure out how to sort of fool the human judges in such a way that, like, the paper
00:06:52 · Herbie Bradley, Ruby Justice Thelot
[00:06:52] S0: Well, I mean, partly it's just how representative it is of real world work and real world tasks, but also how easily can the AI cheat. And that's definitely a big one that people are putting increasing efforts into. I think it's very hard to, in some domains, find tasks that are actually representative of real world work and where the AI can't figure out some way to cheat if it was capable enough. You know, there's even the idea of... In mathematics, you could have a benchmark, which is like, oh, publish a novel mathematics paper that wins a Fields Medal or something. But even that can, in theory, be reward hacked or good hearted because the AI could figure out how to sort of fool the human judges in such a way that, like, the paper [00:07:40] S1: How do you fool the human judges in math? [00:07:42] S0: Well, without Lean, in theory, you could fool the human judges by publishing some proof that is very plausible, but it doesn't quite hold up, the human judges can't figure it out. This is a bigger issue, think, or will be in AI R and D itself, where it could be that a model is coming up with a proposed architectural improvement that fools human overseas at the labs, but still doesn't actually hold in generality.
00:07:42 · Herbie Bradley, Ruby Justice Thelot
[00:07:42] S0: Well, without Lean, in theory, you could fool the human judges by publishing some proof that is very plausible, but it doesn't quite hold up, the human judges can't figure it out. This is a bigger issue, think, or will be in AI R and D itself, where it could be that a model is coming up with a proposed architectural improvement that fools human overseas at the labs, but still doesn't actually hold in generality. Ad[00:08:12] S2: I wanna shout out our sponsor, Lovable. You know the app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change, plus two way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable. Dev. Now, onto the episode.
00:08:51 · Ruby Justice Thelot
[00:08:51] S1: So, yeah, this this reminds me of, another post from nostalgebraist on Less Wrong, who I consider another one of these, like very good AGI centrists. I think you're kind of one of them. I think Seb Krier is one of them. Is it? So nostalgebraist has posted this thing on LessWrong. I'm trying to find the exact text, but it's something like LLMs are adversarially optimized for humans to think that they're really good when they still have, like, substantial weaknesses. None of this is to say, of course, that they are useless. There it is, yeah. Today's LMs are so LLM agents are so smart in so many ways, and yet they constantly make fairly basicdumb mistakes in my experience. This was seven days ago, so with basically the newest LLMs that are available, and says, Yeah, working with these things is often frustrating. This is a very long comment I'm trying to pull through it. It's becoming increasingly unclear what co working process I should expect to work well. They just not have the judgment to do nontrivial things in autonomous settings. They're like smart but naive first year grad students: book smart, hard working, lacking in tastediscernment and ability to prioritize, liable to go off on silly low value tangents, which they treat with the utmost serious if not closely watched and steered. But unlike real grad students, they can't accumulate knowledge or practical know how over time.
00:10:26 · Herbie Bradley
[00:10:26] S0: Yeah, mean, this reminds me of a lot of the things in software engineering where the actual quality of code or quality of decision making particularly and architectural decision making is below what an experienced senior engineer might feel happy with. And yet, because it's a highly verifiable domain, many people at the frontier of using LLMs for coding are just kind of like yellowing, the agent's form on your task, don't worry too much about code quality, just keep iterating, and you'll get to some output that is plausible or reasonable. So far, I think it has, in low stakes domains, been that you can in fact do this, but nonetheless, there have been many examples of some Vibe created failures and people's Vibe created apps for security holes as a result of this, and I think this is actually going to make it harder for the broader economy to adopt AI at scale, especially at the huge agents form level, if they are required to sort of go all in on this kind of ignore quality to some extent or sacrifice some level of some dimension of decision making that senior engineers or others would be happy with in order to just token max because it would improve overall productivity.