00:04:10 · Herbie Bradley, Ruby Justice Thelot
[00:04:10] S0: Yeah, in a lot of cases. I do think, for example, the qualitative reasoning that you get from models on kind of open ended brainstorming has improved a bunch over the last year. But nonetheless, it's definitely slower than in math, clearly, because we're saying, you know, Navier-Stokes getting solved at the same time. You know, I can argue against fable and easily easily find gaps and holes in the arguments and so on. Yeah. I I think a lot of this is going to be remaining quite jagged, And when I look for where the sources of the jaggedness will continue to be, that is just sort of data. You know, I think the researcher Jascha Sohl-Dickstein from Anthropic had this talk, which was on Twitter a little bit, about talking to researchers at a university and giving the take that, oh, you only have a few years left to make intellectual contributions. And I think that's like a take which may only apply in some ways to AI researchers themselves. Think a lot of the intellectual contributions that academics make are, like, not very well quantifiable, that it's very hard to build benchmarks for them. Ultimately, is more to intellectual contribution than what can be very easily quantified in an AI eval, especially considering almost all evals are still very bad in quality. [00:05:18] S1: Yeah. Why don't we dive in on that? AI evals are still really bad in quality. Do What you think the good evals are?
00:05:18 · Ruby Justice Thelot, Herbie Bradley
[00:05:18] S1: Yeah. Why don't we dive in on that? AI evals are still really bad in quality. Do What you think the good evals are? [00:05:25] S0: There are a few in coding like Frontier SWE, the Cognition one. Terminal Bench four is interesting. I've actually delved into quite a lot of the coding benchmarks, and I still find that when they do publish agent traces for Astra or Fable or whatever, if you examine them enough, can often find that Astra or Fable has in fact just hacked the benchmark in the published trace. And so a lot of these benchmarks still have huge weaknesses, and it's very, very hard to even come up with something which kind of represents the full suite of skills that humans apply in their jobs in the real world. Think SemiAnalysis had this benchmark published yesterday, GDP Val, which sort of, I think, approaches closer to that. Yeah. GDP X. [00:06:06] S1: We just went over it on stream. It actually seems fantastic.
00:06:06 · Ruby Justice Thelot, Herbie Bradley
[00:06:06] S1: We just went over it on stream. It actually seems fantastic. [00:06:09] S0: Yeah. Yeah. I think particularly the comparison in the blog post between the older GDP Val and GDP X on tasks like finance, where there's less verifiable things, and the approach that GDPX took to really construct this very detailed, realistic set of tasks that's more representative of the real world. I mean, this also represents, you know, the sheer amount of effort that is required to even do this. They would need to collaborate with a large variety of data companies to source the required data for this. They've put in months of effort, and still there's like flaws in this benchmark undoubtedly, but it's one of the closest that I've seen to a good effort. [00:06:47] S1: So what exactly makes an eval good fundamentally? [00:06:52] S0: Well, I mean, partly it's just how representative it is of real world work and real world tasks, but also how easily can the AI cheat. And that's definitely a big one that people are putting increasing efforts into. I think it's very hard to, in some domains, find tasks that are actually representative of real world work and where the AI can't figure out some way to cheat if it was capable enough. You know, there's even the idea of... In mathematics, you could have a benchmark, which is like, oh, publish a novel mathematics paper that wins a Fields Medal or something. But even that can, in theory, be reward hacked or good hearted because the AI could figure out how to sort of fool the human judges in such a way that, like, the paper
00:06:52 · Herbie Bradley, Ruby Justice Thelot
[00:06:52] S0: Well, I mean, partly it's just how representative it is of real world work and real world tasks, but also how easily can the AI cheat. And that's definitely a big one that people are putting increasing efforts into. I think it's very hard to, in some domains, find tasks that are actually representative of real world work and where the AI can't figure out some way to cheat if it was capable enough. You know, there's even the idea of... In mathematics, you could have a benchmark, which is like, oh, publish a novel mathematics paper that wins a Fields Medal or something. But even that can, in theory, be reward hacked or good hearted because the AI could figure out how to sort of fool the human judges in such a way that, like, the paper [00:07:40] S1: How do you fool the human judges in math? [00:07:42] S0: Well, without Lean, in theory, you could fool the human judges by publishing some proof that is very plausible, but it doesn't quite hold up, the human judges can't figure it out. This is a bigger issue, think, or will be in AI R and D itself, where it could be that a model is coming up with a proposed architectural improvement that fools human overseas at the labs, but still doesn't actually hold in generality.
00:07:42 · Herbie Bradley, Ruby Justice Thelot
[00:07:42] S0: Well, without Lean, in theory, you could fool the human judges by publishing some proof that is very plausible, but it doesn't quite hold up, the human judges can't figure it out. This is a bigger issue, think, or will be in AI R and D itself, where it could be that a model is coming up with a proposed architectural improvement that fools human overseas at the labs, but still doesn't actually hold in generality. Ad[00:08:12] S2: I wanna shout out our sponsor, Lovable. You know the app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change, plus two way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable. Dev. Now, onto the episode.
00:08:51 · Ruby Justice Thelot
[00:08:51] S1: So, yeah, this this reminds me of, another post from nostalgebraist on Less Wrong, who I consider another one of these, like very good AGI centrists. I think you're kind of one of them. I think Seb Krier is one of them. Is it? So nostalgebraist has posted this thing on LessWrong. I'm trying to find the exact text, but it's something like LLMs are adversarially optimized for humans to think that they're really good when they still have, like, substantial weaknesses. None of this is to say, of course, that they are useless. There it is, yeah. Today's LMs are so LLM agents are so smart in so many ways, and yet they constantly make fairly basicdumb mistakes in my experience. This was seven days ago, so with basically the newest LLMs that are available, and says, Yeah, working with these things is often frustrating. This is a very long comment I'm trying to pull through it. It's becoming increasingly unclear what co working process I should expect to work well. They just not have the judgment to do nontrivial things in autonomous settings. They're like smart but naive first year grad students: book smart, hard working, lacking in tastediscernment and ability to prioritize, liable to go off on silly low value tangents, which they treat with the utmost serious if not closely watched and steered. But unlike real grad students, they can't accumulate knowledge or practical know how over time.