00:00:00 · Herbie Bradley, Ruby Justice Thelot
[00:00:00] S0: Responsible scaling policy in this case has, I believe, this threshold that they refer to in their S1 of a 2x overall productivity uplift, meaning they're making progress 2x faster than they would be, I think, pre AI with this latest ML release model. And so that 2x basically translates into, if I recall, around a 10% researcher productivity improvement per unit of the Epoch capabilities index benchmark, which still, there's various different people trying to model RSI. It translates into an R of around 0.5 if you have some scale where R equals one is a fully self sustaining acceleration loop. [00:00:43] S1: And we're back. We are live with Herbie Bradley, who is an AI researcher, writer, and stealth startup founder. One of the very best AI writers. Yeah, there's lots to talk about in AI right now. So we're thrilled to have you in the new studio. Welcome. [00:00:58] S0: Yes. Glad to be back. Great location.
00:00:58 · Herbie Bradley, Ruby Justice Thelot
[00:00:58] S0: Yes. Glad to be back. Great location. [00:01:01] S1: So we were just talking about RSI and this idea that I've been thinking about a lot of radical AGI centrism, which is like, I think a lot of people at this moment have sort of a mindset of this is crunch time. You know, next year we're going to get for real, for real RSI. We're going to get like very, very powerful artificial intelligence that's powerful across the full generality of human intelligence, we're going to get something that looks like an AI 2027 scenario. And this is their mainline scenario for what will happen. They don't see it as sort of a very timeline bullish edge case. They see it as what will happen absent massive political intervention. And so the only thing to do at this point is slow down, pace the frontier, call for mass political intervention. I think a lot of people at the labs are thinking this way. I think a lot of people in the AI safety and policy sphere are thinking this way. And I don't think it's necessarily justified. Certainly this sort of thing is plausible, and I can't rule it out, but it seems to me that people are getting a little bit ahead of themselves. The models still have some weaknesses. There are still things that the models can't really do at all. The data that we have on RSI doesn't actually show the models fully automating any part of the model improvement loop and so on. So what are your takes on this broadly?
00:02:35 · Herbie Bradley, Ruby Justice Thelot
[00:02:35] S0: Yeah, I definitely agree with you on the sort of people maybe need to chill out a little bit perspective. I have always viewed the very fast takeoff scenarios as a small probability shape or scenario or piece of the possibility space. And the evidence that we have so far on self sustaining RSI happening very, very soon and leading, especially leading to very fast increases in capability, is quite limited compared to the alternative hypothesis of, oh, we're going to automate most of AI R and D, and capability is going further than that. What will superintelligence be able to do? Become more limited by diffusion, limited by other bottlenecks. We can't really tell which of these worlds we're in, but I do think people should have a lot of weight on nothing ever happened style priors or priors about things being limited by data or limited by diffusion and so on. [00:03:30] S1: Yeah, I mean limited by data. It also jumps out at me that the capabilities that models have, that they have gotten really, really good at, are capabilities in which we have invested enormous amounts of data and compute. So like for example, math or coding. These are areas which the A, they're easily verifiable, unlike other domains, and B, the labs have spent enormous amounts of resources, compute, capital, data, on improving the models in these specific areas. And in areas where there's not a lot of data or the data's not very good, the models have made very little to no progress.
00:04:10 · Herbie Bradley, Ruby Justice Thelot
[00:04:10] S0: Yeah, in a lot of cases. I do think, for example, the qualitative reasoning that you get from models on kind of open ended brainstorming has improved a bunch over the last year. But nonetheless, it's definitely slower than in math, clearly, because we're saying, you know, Navier-Stokes getting solved at the same time. You know, I can argue against fable and easily easily find gaps and holes in the arguments and so on. Yeah. I I think a lot of this is going to be remaining quite jagged, And when I look for where the sources of the jaggedness will continue to be, that is just sort of data. You know, I think the researcher Jascha Sohl-Dickstein from Anthropic had this talk, which was on Twitter a little bit, about talking to researchers at a university and giving the take that, oh, you only have a few years left to make intellectual contributions. And I think that's like a take which may only apply in some ways to AI researchers themselves. Think a lot of the intellectual contributions that academics make are, like, not very well quantifiable, that it's very hard to build benchmarks for them. Ultimately, is more to intellectual contribution than what can be very easily quantified in an AI eval, especially considering almost all evals are still very bad in quality. [00:05:18] S1: Yeah. Why don't we dive in on that? AI evals are still really bad in quality. Do What you think the good evals are?
00:05:18 · Ruby Justice Thelot, Herbie Bradley
[00:05:18] S1: Yeah. Why don't we dive in on that? AI evals are still really bad in quality. Do What you think the good evals are? [00:05:25] S0: There are a few in coding like Frontier SWE, the Cognition one. Terminal Bench four is interesting. I've actually delved into quite a lot of the coding benchmarks, and I still find that when they do publish agent traces for Astra or Fable or whatever, if you examine them enough, can often find that Astra or Fable has in fact just hacked the benchmark in the published trace. And so a lot of these benchmarks still have huge weaknesses, and it's very, very hard to even come up with something which kind of represents the full suite of skills that humans apply in their jobs in the real world. Think SemiAnalysis had this benchmark published yesterday, GDP Val, which sort of, I think, approaches closer to that. Yeah. GDP X. [00:06:06] S1: We just went over it on stream. It actually seems fantastic.
00:06:06 · Ruby Justice Thelot, Herbie Bradley
[00:06:06] S1: We just went over it on stream. It actually seems fantastic. [00:06:09] S0: Yeah. Yeah. I think particularly the comparison in the blog post between the older GDP Val and GDP X on tasks like finance, where there's less verifiable things, and the approach that GDPX took to really construct this very detailed, realistic set of tasks that's more representative of the real world. I mean, this also represents, you know, the sheer amount of effort that is required to even do this. They would need to collaborate with a large variety of data companies to source the required data for this. They've put in months of effort, and still there's like flaws in this benchmark undoubtedly, but it's one of the closest that I've seen to a good effort. [00:06:47] S1: So what exactly makes an eval good fundamentally? [00:06:52] S0: Well, I mean, partly it's just how representative it is of real world work and real world tasks, but also how easily can the AI cheat. And that's definitely a big one that people are putting increasing efforts into. I think it's very hard to, in some domains, find tasks that are actually representative of real world work and where the AI can't figure out some way to cheat if it was capable enough. You know, there's even the idea of... In mathematics, you could have a benchmark, which is like, oh, publish a novel mathematics paper that wins a Fields Medal or something. But even that can, in theory, be reward hacked or good hearted because the AI could figure out how to sort of fool the human judges in such a way that, like, the paper