SemiAnalysis Weekly
00:21:28 · Max Kan, Andrew Megalaa
[00:21:28] S1: Yeah. Yeah. I think that's a really good point worth emphasizing because we kinda went back and forth between which of those screenshots was the one that should have been the headline result. The one where you include all the domains versus the one where you exclude cyber, biology, and alignment because those were the three domains where sometimes safety classifiers kicked in and therefore sort of sandbag the score for some models. Like, if you look at Astra on cyber, for example, the score is, like, 0% because it just refused to do any the cyber task. And, ultimately, we decided that the one including all the domains is better. But I think it actually is sort of fair to penalize overly strict safety classifiers to some degree. I mean, I know you've experienced this yourself, Jordan, sort of just doing, you know, your day to day work at SemiAnalysis, then you're you're hit with a cyber safety classifier from, Opus Ration. You're like, well, fuck. I guess I gotta use, like, GLM 5.3 now or whatever. [00:22:32] S2: No. Jordan Jordan was... Jordan had a very interesting insight. He actually figured out that Muse Wonder is, like, a top tier cyber model independently. And you were right. Like, you were totally right. Like, Muse Wonder and Muse Wonderful were just top tier for, like, what you can get out of the box for a cyber model. [00:22:50] S1: Yeah. Yeah. Yeah. That... Which was, again, a very surprising result [00:22:53] S2: Yeah.
00:22:53 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:22:53] S2: Yeah. [00:22:53] S1: For us because, I guess, Jordan already knew, but we know we didn't have the Muse, Wonder, and Wonderful number until two hours before launch. So that was that was pretty cool. [00:23:01] S2: And we were like, yeah. Wow. Muse, Wonderful just crushes it. [00:23:04] S0: Yeah, you guys gave away all my alpha there. Look, using the models yourself for something that you think you're pushing the frontier on absolutely gives you insight into which models are good and which models are bad, but you have to actually have that work to do. I had that work to do in one of the 11 domains we're talking about here. Yeah, I could figure that out. And I do think probably the biggest benefit of this benchmark is the ability to filter the results as you see fit and interpret them as it is relevant to you. People with Daybreak access or with CVP access from Anthropic, which we have sometimes. [00:23:48] S1: Well, there's like three or four tiers, and then we only have, like, you know, the bottom or maybe, like, the second to, like, worst tier. And actually, that tier, I think, is more or less useless because you still get flagged doing basically everything. And it's just... I think they both have, like, very confusing sort of access programs for the malls today, which is hoping something they can, like, you know, get better at. [00:24:10] S2: Yeah. Yeah. The crazy thing is, like, you know, if Muse Wonderful does, like, that well in cyber, like, I wonder, like, what Red Access on, like, Astra or or But no no one can [00:24:19] S1: use it.
00:24:19 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:24:19] S1: use it. [00:24:21] S2: Yeah. Unless you work [00:24:22] S1: at, like, all out the network, I guess. Mhmm. [00:24:24] S0: We'll get there. Yeah. But, okay, maybe the the better question about this is just like, maybe guys can emphasize again the fact that this benchmark is meant to measure the performance of the models at the frontiers rather than on day to day tasks. [00:24:40] S1: Yeah. Yeah. For sure. Because again, our our core thesis here is that day to day tasks just don't actually matter. Right? [00:24:47] S2: They're more or less solved. [00:24:50] S1: Yeah. Like like, I think I think most of them, the majority of them are like more or less solved. I think heading into '27, you're not gonna need frontier models for most of these. And we... Like, not only do we think there are sort of poor tasks to use to measure frontier model capabilities, we also think it's probably gonna sort of not be the majority of revenue for OpenAI and Anthropic and the other frontier model companies. And sort of because the thing that we're most interested in tracking with this benchmark is sort of a, where frontier model companies... Sorry, where frontier model capabilities are going, and then b, can you sort of, like, do any like TAM or actually revenue analysis based off of that? We sort of very explicitly focus on just the frontier domains.
00:24:50 · Max Kan, Andrew Megalaa
[00:24:50] S1: Yeah. Like like, I think I think most of them, the majority of them are like more or less solved. I think heading into '27, you're not gonna need frontier models for most of these. And we... Like, not only do we think there are sort of poor tasks to use to measure frontier model capabilities, we also think it's probably gonna sort of not be the majority of revenue for OpenAI and Anthropic and the other frontier model companies. And sort of because the thing that we're most interested in tracking with this benchmark is sort of a, where frontier model companies... Sorry, where frontier model capabilities are going, and then b, can you sort of, like, do any like TAM or actually revenue analysis based off of that? We sort of very explicitly focus on just the frontier domains. [00:25:46] S2: Yeah. And like when we say like day to day tasks, like that actually doesn't even exclude like some of the domains people want to use like these models for. Like it was surprising to see how bad the models are in like finance and legal for how much people talk about like, they will automate finance and legal. Because, like, when we looked at some of the tasks that we got from, like, living context and chakra, like, we went through those rollouts ourselves because we were confused. I I actually thought the models were better at [00:26:11] S1: these Yeah. Andrew Andrew is, like, a tech bro supremacist, and he was like, dude, the model's definitely saturated finance already. That was his prior coming into this.
00:26:11 · Max Kan, Andrew Megalaa
[00:26:11] S1: these Yeah. Andrew Andrew is, like, a tech bro supremacist, and he was like, dude, the model's definitely saturated finance already. That was his prior coming into this. [00:26:19] S2: Yeah. And we actually had like some of our finance friends look at some of these tasks. And like when we were like choosing partners for these, you know, we had different rollouts and tasks. And, like, some of them were like, yeah, these tasks are, like, pretty bad. [00:26:34] S1: Yeah. Like, there there are some tasks from some potential finance partners that we were like, woah. This is, like, not realistic of sort of actual finance work. Like, and the only reason why it's sort of sufficiently difficult for, you know, Opus and Astra is because it's, like, lacks realism and it's also unfair in these ways. And then after you correct with the unfairness, like, oh, look. Opus is, like, saturated. [00:26:56] S2: And so that would make you think that these tasks are solved. But then we got, like, the really high quality tasks. Yeah. [00:27:01] S1: Yeah. [00:27:02] S2: Yeah. And then, like, we had, like, our finance friends look at them, and they're like, yeah. These are just totally realistic of, like, a real thing I would have done in, like, banking and PE. And the model is just, like, failed us.
00:27:13 · Max Kan
[00:27:13] S1: Yeah. And, actually, I I wanna talk more about the finance task in particular because I think it's probably the most understandable kind of domain to the average listener. And I think this will be useful in giving people a sense of, like, why GDPx is such a high quality benchmark. Sort of how people have historically made finance tasks in the past, if you look at something like GDPval from OpenAI, is they would go to, say, a company like Mercor, which has this vast, like, expert network of finance bros and be like, you finance bros, sit down for two hours and try to, like, come up with a task that's, like, representative of something that you would actually do in your job. And then also give me sort of the golden solution for that task. And then, you know, that's gonna be the task that I'm gonna give you about. And, obviously, you sort of have a ton of sort of biases when you're telling someone like, hey. You know, make a task for my for my benchmark and just please try your best to make it realistic, man. What, like, our finance partner did is they took a real private equity transaction where, like, some private equity firm bought a, like, family owned snack company. I think they they paid, like, a little over 800,000,000 for, like, 60 percent of the company. They... Again, they they sort of had this actual transaction, like, all 4,000 files sort of corresponding to it separated into the deal rooms for the private equity firm that was involved, the investment bank that was involved, the accounting firm that was involved, and then also sort of the snack companies sort of own VDR because they know they were selling themselves.