00:02:35 · Herbie Bradley, Ruby Justice Thelot
[00:02:35] S0: Yeah, I definitely agree with you on the sort of people maybe need to chill out a little bit perspective. I have always viewed the very fast takeoff scenarios as a small probability shape or scenario or piece of the possibility space. And the evidence that we have so far on self sustaining RSI happening very, very soon and leading, especially leading to very fast increases in capability, is quite limited compared to the alternative hypothesis of, oh, we're going to automate most of AI R and D, and capability is going further than that. What will superintelligence be able to do? Become more limited by diffusion, limited by other bottlenecks. We can't really tell which of these worlds we're in, but I do think people should have a lot of weight on nothing ever happened style priors or priors about things being limited by data or limited by diffusion and so on. [00:03:30] S1: Yeah, I mean limited by data. It also jumps out at me that the capabilities that models have, that they have gotten really, really good at, are capabilities in which we have invested enormous amounts of data and compute. So like for example, math or coding. These are areas which the A, they're easily verifiable, unlike other domains, and B, the labs have spent enormous amounts of resources, compute, capital, data, on improving the models in these specific areas. And in areas where there's not a lot of data or the data's not very good, the models have made very little to no progress.
00:04:10 · Herbie Bradley, Ruby Justice Thelot
[00:04:10] S0: Yeah, in a lot of cases. I do think, for example, the qualitative reasoning that you get from models on kind of open ended brainstorming has improved a bunch over the last year. But nonetheless, it's definitely slower than in math, clearly, because we're saying, you know, Navier-Stokes getting solved at the same time. You know, I can argue against fable and easily easily find gaps and holes in the arguments and so on. Yeah. I I think a lot of this is going to be remaining quite jagged, And when I look for where the sources of the jaggedness will continue to be, that is just sort of data. You know, I think the researcher Jascha Sohl-Dickstein from Anthropic had this talk, which was on Twitter a little bit, about talking to researchers at a university and giving the take that, oh, you only have a few years left to make intellectual contributions. And I think that's like a take which may only apply in some ways to AI researchers themselves. Think a lot of the intellectual contributions that academics make are, like, not very well quantifiable, that it's very hard to build benchmarks for them. Ultimately, is more to intellectual contribution than what can be very easily quantified in an AI eval, especially considering almost all evals are still very bad in quality. [00:05:18] S1: Yeah. Why don't we dive in on that? AI evals are still really bad in quality. Do What you think the good evals are?
00:05:18 · Ruby Justice Thelot, Herbie Bradley
[00:05:18] S1: Yeah. Why don't we dive in on that? AI evals are still really bad in quality. Do What you think the good evals are? [00:05:25] S0: There are a few in coding like Frontier SWE, the Cognition one. Terminal Bench four is interesting. I've actually delved into quite a lot of the coding benchmarks, and I still find that when they do publish agent traces for Astra or Fable or whatever, if you examine them enough, can often find that Astra or Fable has in fact just hacked the benchmark in the published trace. And so a lot of these benchmarks still have huge weaknesses, and it's very, very hard to even come up with something which kind of represents the full suite of skills that humans apply in their jobs in the real world. Think SemiAnalysis had this benchmark published yesterday, GDP Val, which sort of, I think, approaches closer to that. Yeah. GDP X. [00:06:06] S1: We just went over it on stream. It actually seems fantastic.
00:06:06 · Ruby Justice Thelot, Herbie Bradley
[00:06:06] S1: We just went over it on stream. It actually seems fantastic. [00:06:09] S0: Yeah. Yeah. I think particularly the comparison in the blog post between the older GDP Val and GDP X on tasks like finance, where there's less verifiable things, and the approach that GDPX took to really construct this very detailed, realistic set of tasks that's more representative of the real world. I mean, this also represents, you know, the sheer amount of effort that is required to even do this. They would need to collaborate with a large variety of data companies to source the required data for this. They've put in months of effort, and still there's like flaws in this benchmark undoubtedly, but it's one of the closest that I've seen to a good effort. [00:06:47] S1: So what exactly makes an eval good fundamentally? [00:06:52] S0: Well, I mean, partly it's just how representative it is of real world work and real world tasks, but also how easily can the AI cheat. And that's definitely a big one that people are putting increasing efforts into. I think it's very hard to, in some domains, find tasks that are actually representative of real world work and where the AI can't figure out some way to cheat if it was capable enough. You know, there's even the idea of... In mathematics, you could have a benchmark, which is like, oh, publish a novel mathematics paper that wins a Fields Medal or something. But even that can, in theory, be reward hacked or good hearted because the AI could figure out how to sort of fool the human judges in such a way that, like, the paper
00:06:52 · Herbie Bradley, Ruby Justice Thelot
[00:06:52] S0: Well, I mean, partly it's just how representative it is of real world work and real world tasks, but also how easily can the AI cheat. And that's definitely a big one that people are putting increasing efforts into. I think it's very hard to, in some domains, find tasks that are actually representative of real world work and where the AI can't figure out some way to cheat if it was capable enough. You know, there's even the idea of... In mathematics, you could have a benchmark, which is like, oh, publish a novel mathematics paper that wins a Fields Medal or something. But even that can, in theory, be reward hacked or good hearted because the AI could figure out how to sort of fool the human judges in such a way that, like, the paper [00:07:40] S1: How do you fool the human judges in math? [00:07:42] S0: Well, without Lean, in theory, you could fool the human judges by publishing some proof that is very plausible, but it doesn't quite hold up, the human judges can't figure it out. This is a bigger issue, think, or will be in AI R and D itself, where it could be that a model is coming up with a proposed architectural improvement that fools human overseas at the labs, but still doesn't actually hold in generality.
00:07:42 · Herbie Bradley, Ruby Justice Thelot
[00:07:42] S0: Well, without Lean, in theory, you could fool the human judges by publishing some proof that is very plausible, but it doesn't quite hold up, the human judges can't figure it out. This is a bigger issue, think, or will be in AI R and D itself, where it could be that a model is coming up with a proposed architectural improvement that fools human overseas at the labs, but still doesn't actually hold in generality. Ad[00:08:12] S2: I wanna shout out our sponsor, Lovable. You know the app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change, plus two way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable. Dev. Now, onto the episode.