00:03:37 · Jeffrey Ladish, Ruby Justice Thelot
I think that Anthropic should release this data, unredact this data, so that whole world can assess this for themselves. Because I'm like, man, this has huge strategic implications for for national security, for US China stuff, for, like, you know, for all of this stuff. And, yeah, it it seems like we should know. You know? We should release the data. [00:03:57] S1: Look. I definitely don't believe METR is bullshitting, but, like, one thing one thing that stood out to me is that, like, the the main piece of evidence that people were looking at to confirm this, like, large speed up in AI capabilities was the benchmark chart, which shows a... You know, two different trend lines, the trend line from before October 2025 and then the trend line from after October 2025. [00:04:19] S0: Yep. [00:04:19] S1: But it seems to me that, like, the trend line of public models on this chart is not that different from the trend line of internal models. So why was today's report such a big update when we should have already seen the trend line of public models? It looks like the same line to me.
00:04:38 · Jeffrey Ladish
[00:04:38] S0: I mean, I I think... Yeah. You're you're saying, like, this is, how the overall capabilities seemed... Seems to have changed. Yeah. That's a good question. I'm looking at that tweet. Yeah. I don't don't have a very good answer to that. I'm like... My... I I feel like it's hard to look at your subjective sense of exactly how good the models are. I don't know. I think for me, the biggest the biggest jump was like, the difference between pre Fable and post Fable to me was huge. Like, I just went from, like, using agents sometimes to using agents every single day. And, like, that was, like, the really big jump. And now it's a little bit like... I don't know. I think I'm like, I really want Anthropic to release, us to Anthropic. Show us what is it like being an Anthropic engineer? How many agents are they using every day? What does that actually look like? And then concretely, what does that lead to? In some sense, I'm like, yeah, we can look at these benchmarks. That is, I think, best thing we have for measuring AI capabilities. But I remember back in the day when the way we measured AI capabilities was mostly METR bringing in software engineers and having them do these long time horizon tasks. And then it was not like, you have to look at these 40 benchmarks. It was like, look at just like, can the AIs autonomously do the work? That is the thing I most care about. And so I'm like, yeah, I don't know. I think this is a fair criticism of like, well, these benchmarks aren't very good, but I'm like, how do we get back to the world where the benchmark is mostly like, let's take real economically valuable work and see if the AIs can do it autonomously? I think that's what I would prefer to be the measure.
00:06:17 · Ruby Justice Thelot, Jeffrey Ladish, Ruby Justice Thelot
[00:06:17] S2: Yeah. [00:06:18] S0: But but I'm, like, a little bit, like, I'm not sure what METR is supposed to do. Right? Like, it's, like, a tricky it's a tricky problem. [00:06:25] S1: I mean, plausibly, like, if if METR really believed that they had access to information that was, like, world historically important that could prevent humanity's extinction or disempowerment Yeah. Presumably, they would outweigh that in their consideration, or or or rather that in their consideration of whether to release that information would outweigh, like, Anthropic's IP concerns or whatever. And potentially, whistleblowers within Anthropic would make the same calculus. [00:06:54] S0: Well well, I do I do not think that METR should release information they have agreed with Anthropic Not's release. Like, I think that, like, the trust there is really important. I'm like, the fact that they can play the role that they're currently playing really relies on trust. Now I think there are potentially cases for Anthropic whistleblowers to come forward there. My hope is that anthropic leadership themselves will be like, hey. Or like, if I was at anthropic, the first thing I would do is I'd go to leadership and I'd say, this is of world historic importance. You guys agree that this could, like, result in human extinction. We need to be much more transparent with this information. This is really decision relevant information that the world should know.
00:06:54 · Jeffrey Ladish, S3
[00:06:54] S0: Well well, I do I do not think that METR should release information they have agreed with Anthropic Not's release. Like, I think that, like, the trust there is really important. I'm like, the fact that they can play the role that they're currently playing really relies on trust. Now I think there are potentially cases for Anthropic whistleblowers to come forward there. My hope is that anthropic leadership themselves will be like, hey. Or like, if I was at anthropic, the first thing I would do is I'd go to leadership and I'd say, this is of world historic importance. You guys agree that this could, like, result in human extinction. We need to be much more transparent with this information. This is really decision relevant information that the world should know. Ad[00:07:31] S3: I wanna shout out our sponsor, Lovable. You know the app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable codebase you can inspect and change, plus two way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable, lovable. Dev. Now onto the episode.
00:07:31 · S3, Ruby Justice Thelot
Ad[00:07:31] S3: I wanna shout out our sponsor, Lovable. You know the app you've been meaning to build? The internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable codebase you can inspect and change, plus two way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more so you don't have to wire it all up yourself. And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable, lovable. Dev. Now onto the episode. [00:08:10] S2: Well, I guess to Theo's point earlier, like the trend line doesn't look that different from internal models to public models. So wouldn't it be like a much scarier outcome if the internal models... Like, let's say that they halted the release of, like, some models because they couldn't do the proper cyber, guardrails or whatever. But Yeah. Wouldn't it be, like, a much worse outcome if internal... The, like, the internal curve was just, like, significantly higher or steeper than the public curve? Like, this doesn't even seem that... This seems to me, like, actually less concerning because of this.
00:08:44 · Jeffrey Ladish
[00:08:44] S0: Yeah. I mean, I think overall, having a lower, like, deployment gap is better. Right? I would much prefer there to be not... Like, the smaller the gap, the better. I think that's right. Yeah. Mean, another way to sort of rephrase what you guys are saying is like, yep, it's still really, really fast. And the speed up is sort of maybe should already have been priced in. And I'm like, yeah, maybe. But can we look at how insanely fast it is? To me, if you look over the last six months, some people already had this priced into their models. I do think that AI futures projects should get a lot of credit for being insanely bullish on just how fast this would go and Anthropic as well. Sorry, credit here is a little weird for Anthropic because I'm like, they're the ones causing it to happen. And I think imposing massive risks on the rest of the world and society, which I wish they wouldn't. Yeah, mean, so to me, I'm like, wow, it seems like we might be only a year away from AI agents, which could take over all of the digital infrastructure in the world. I don't know. I don't know if you want to talk about threat models here, because I'm basically like, yeah, yeah, so there's a whole bunch of implications for AI development more broadly, but to me, the most important implications are how far are we away from AI agents that we would lose control of entirely? And I think to me, I'm like looking at the last few months, including this recent report, but also like you said, I'm like, yeah, yeah, I think I did have a lot of this priced in. How far are we away from AI agents, which if they wanted to, could take over at least the digital world? And then there's a question of whether they could take over the physical world as well. But yeah, I think we just might not be that far from that, if this is correct, if Anthropic is correct about the capability of their models.