SemiAnalysis Weekly
00:18:22 · Andrew Megalaa, Jordan Nanos
[00:18:22] S2: I think we got, like, a really good tweet. Someone asked about whether, like, you know, GDPval had human baselines for how long tasks take, you know, to like put things in And this was really funny because like I responded to this and I was like, actually, like it doesn't even make sense to have human baselines anymore. Like in the material science tasks, like Astra solved a subtask of a task better than like a published state of the art reference. Like when the models are like essentially past expert level, how do you even put a human baseline to that? And And some of these chip design tasks are just so long horizon and would take teams at companies to even do... And those, there's no human baseline there. How do you even put a number on it? [00:19:07] S0: It's a lot of money. Whatever the hourly equivalent is in human work hours is a lot of money. And I just threw up on screen the the cost per task for the material science works where the biggest ones, like was at 40 or $50 per task, and many of the models are solving these tasks, some of them in $10, which is I think, a pretty unbelievable statement. Maybe you can talk about reading some of the rollouts in more detail. What were some of the pieces of model behavior that surprised you the most?
00:19:07 · Jordan Nanos, Andrew Megalaa
[00:19:07] S0: It's a lot of money. Whatever the hourly equivalent is in human work hours is a lot of money. And I just threw up on screen the the cost per task for the material science works where the biggest ones, like was at 40 or $50 per task, and many of the models are solving these tasks, some of them in $10, which is I think, a pretty unbelievable statement. Maybe you can talk about reading some of the rollouts in more detail. What were some of the pieces of model behavior that surprised you the most? [00:19:45] S2: Yeah, I mean, the models find very creative ways to solve the tasks. And this is again, going back to the whole unfair verifier thing. People don't expect the way that models solve tasks because models are not human. They just solve things in very weird ways. Like, you know, maybe use module B instead of module A, for example. But sometimes just like have completely different taste in like how things are done. And sometimes it's correct and sometimes it's not. And that's like why we have to like go through this transcript. And it's, like, hard to even go through them sometimes because, you know, we're not human experts. But luckily, you can benefit a lot from chatting with AI about certain rollouts to, like, figure out whether this stuff is, like, correct or not. The the model's actually so smart, like, on their own that with the... With with human probing, you can figure out whether something is, like, fair or not fair. Yeah. It's a very interesting experience for sure.
00:20:44 · Jordan Nanos
[00:20:44] S0: Maybe we can talk about the results. I'm getting... I'm realizing that I haven't even shared the, like, flagship leaderboard screenshot that's at the top of the article yet. Maybe you guys can talk about, like, what you learned about the models themselves. I think particularly we've been going back and forth and trying to explain the difference between the flagship leaderboard across all domains and then compare it to the next one where you've got the leaderboard where you filter out biology, cybersecurity, and alignment where safety classifiers are causing models that have better performance in the weights somewhere to not actually demonstrate it on the tasks in the benchmark?
00:21:28 · Max Kan, Andrew Megalaa
[00:21:28] S1: Yeah. Yeah. I think that's a really good point worth emphasizing because we kinda went back and forth between which of those screenshots was the one that should have been the headline result. The one where you include all the domains versus the one where you exclude cyber, biology, and alignment because those were the three domains where sometimes safety classifiers kicked in and therefore sort of sandbag the score for some models. Like, if you look at Astra on cyber, for example, the score is, like, 0% because it just refused to do any the cyber task. And, ultimately, we decided that the one including all the domains is better. But I think it actually is sort of fair to penalize overly strict safety classifiers to some degree. I mean, I know you've experienced this yourself, Jordan, sort of just doing, you know, your day to day work at SemiAnalysis, then you're you're hit with a cyber safety classifier from, Opus Ration. You're like, well, fuck. I guess I gotta use, like, GLM 5.3 now or whatever. [00:22:32] S2: No. Jordan Jordan was... Jordan had a very interesting insight. He actually figured out that Muse Wonder is, like, a top tier cyber model independently. And you were right. Like, you were totally right. Like, Muse Wonder and Muse Wonderful were just top tier for, like, what you can get out of the box for a cyber model. [00:22:50] S1: Yeah. Yeah. Yeah. That... Which was, again, a very surprising result [00:22:53] S2: Yeah.
00:22:53 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:22:53] S2: Yeah. [00:22:53] S1: For us because, I guess, Jordan already knew, but we know we didn't have the Muse, Wonder, and Wonderful number until two hours before launch. So that was that was pretty cool. [00:23:01] S2: And we were like, yeah. Wow. Muse, Wonderful just crushes it. [00:23:04] S0: Yeah, you guys gave away all my alpha there. Look, using the models yourself for something that you think you're pushing the frontier on absolutely gives you insight into which models are good and which models are bad, but you have to actually have that work to do. I had that work to do in one of the 11 domains we're talking about here. Yeah, I could figure that out. And I do think probably the biggest benefit of this benchmark is the ability to filter the results as you see fit and interpret them as it is relevant to you. People with Daybreak access or with CVP access from Anthropic, which we have sometimes. [00:23:48] S1: Well, there's like three or four tiers, and then we only have, like, you know, the bottom or maybe, like, the second to, like, worst tier. And actually, that tier, I think, is more or less useless because you still get flagged doing basically everything. And it's just... I think they both have, like, very confusing sort of access programs for the malls today, which is hoping something they can, like, you know, get better at. [00:24:10] S2: Yeah. Yeah. The crazy thing is, like, you know, if Muse Wonderful does, like, that well in cyber, like, I wonder, like, what Red Access on, like, Astra or or But no no one can [00:24:19] S1: use it.
00:24:19 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:24:19] S1: use it. [00:24:21] S2: Yeah. Unless you work [00:24:22] S1: at, like, all out the network, I guess. Mhmm. [00:24:24] S0: We'll get there. Yeah. But, okay, maybe the the better question about this is just like, maybe guys can emphasize again the fact that this benchmark is meant to measure the performance of the models at the frontiers rather than on day to day tasks. [00:24:40] S1: Yeah. Yeah. For sure. Because again, our our core thesis here is that day to day tasks just don't actually matter. Right? [00:24:47] S2: They're more or less solved. [00:24:50] S1: Yeah. Like like, I think I think most of them, the majority of them are like more or less solved. I think heading into '27, you're not gonna need frontier models for most of these. And we... Like, not only do we think there are sort of poor tasks to use to measure frontier model capabilities, we also think it's probably gonna sort of not be the majority of revenue for OpenAI and Anthropic and the other frontier model companies. And sort of because the thing that we're most interested in tracking with this benchmark is sort of a, where frontier model companies... Sorry, where frontier model capabilities are going, and then b, can you sort of, like, do any like TAM or actually revenue analysis based off of that? We sort of very explicitly focus on just the frontier domains.