SemiAnalysis Weekly
00:00:20 · Jordan Nanos, Max Kan, Andrew Megalaa
[00:00:20] S0: Yes. You're pleasantly surprised that the benchmark matches your subjective impression of the model. [00:00:25] S1: Yeah. Because because to be clear, right, we actually had no idea what the final leaderboard was gonna look like until about two hours before publishing, maybe four hours. Right? Because what we did is, like, we had all the partners difficult to calibrate on four models. Like, they ran Astra. They ran Opus. They ran 3.8 Flash and then some Carbon because we were kind of working on the benchmark sort of before argon came out. And then also GLM 5.3. And so we only had results for those four models up until, like, three, four hours before the benchmark actually launched. And then Andrew, you know, switched to polyphasic sleep over the past week so he could build all the infra. You know, he set up a thing so that Claude would wake him up if there's, like, something that broke and he needed to, like, you know, provide input. But then he, like, set up all that infra, started running on the rollouts. You know, these are incredibly long, like, rollouts. Right? Sort of doing the task plus grading could easily take, like, twenty four hours. And so we didn't have all the final results until literally, you know, three or four hours before we published. And he saw his leaderboard, we're like, yo. I think he might explode, dude. [00:01:39] S2: Some of [00:01:40] S1: these models, like,
00:01:40 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:01:40] S1: these models, like, [00:01:41] S2: some of these models, like, we haven't used, like, or even seen, like, out in the wild. Like, Carbon, like... We got access to Carbon on, like, Thursday night or something. Like, we haven't used it for anything. And it [00:01:51] S1: was just, like, [00:01:51] S2: the first time we're trying it out. It did well. It was it was surprising. And same with Muse Wonderful. Like, Muse Wonderful is not available in API. What did you We got access to it. It it was... It did great on cyber. [00:02:05] S0: Yeah. When did you fix the tasks? When when did those get finalized? [00:02:11] S1: Like, Thursday night. So we launched on Sunday. The tasks were finalized, like, Thursday night because we were doing a lot of sort of back and forth iteration with our partners where, like, our queue would flag something and be like, hey. Take a look at this. And then they'd be like, oh, like, these two are false positives. Actually, this one's too real, so we got updated, blah blah blah. And it was really, really last minute. Like, I'm on... I'm I'm very happy that we got it out in time. It was very hard, though. I'm also glad to be taking a break now. But, yeah, it's a crazy process. [00:02:46] S2: I think [00:02:46] S0: the work starts now, man. I don't know. [00:02:49] S1: The work? Okay. I'm gonna take, like, I'm gonna take a couple of chill days, you know. Yeah. And and then we can go through the process of, yeah, I guess, kind of reaping the rewards of our effort.
00:02:49 · Max Kan, Jordan Nanos
[00:02:49] S1: The work? Okay. I'm gonna take, like, I'm gonna take a couple of chill days, you know. Yeah. And and then we can go through the process of, yeah, I guess, kind of reaping the rewards of our effort. [00:03:02] S0: Maintaining the dashboard as new models gets released. Did you did you impose this deadline on yourself? What what was the motivation for that? [00:03:10] S1: It was mostly self imposed. I mean, we were we're decently high confidence that, you know, a new OpenAI and Anthropic flagship are probably both coming out this week. And you obviously cannot launch your benchmark on the same day as, like, Opus 5.5. And Especially if your benchmark doesn't have to. Yeah. And I think we were, like, I think we were decently confident that, like, Monday would not be the release date for these models. Maybe it's, like, Tuesday or Thursday or something. But if there is, like, any risk of the model coming out on Monday, we were like, fuck it. You got it. We we're just gonna publish Sunday instead. And then there's some fear that, like, okay, Sunday night, you know, maybe people aren't as online, so you're not gonna sort of have the morality component. But I think we were very happy with how the launch turned out. Yeah, was generally generally happy. [00:04:02] S0: Take a step back. Explain GDPX from scratch for people who maybe haven't seen the article or just hearing about it for the first time or [00:04:09] S1: didn't read it [00:04:09] S0: as many people who were commenting about it. But, yeah.
00:04:09 · Jordan Nanos, Max Kan, Andrew Megalaa
[00:04:09] S0: as many people who were commenting about it. But, yeah. [00:04:14] S1: Yeah. I think I'm gonna be going... Taking like sort of a step all the way back. I think it's important to understand that a lot of the really popular benchmarks out there are kinda trash. This is largely because they're made by people who don't have expertise and, like, making evals for all environments or sort of trained data in general. And it's, like, very easy for you to make a task that kind of seems good on a surface level, but it's actually unfair in some way such that it's a poor test of model capabilities. [00:04:56] S2: Yeah. A lot of people make their tasks harder by making it unfair rather than actually making the task, like, difficult. And they... And it's, like, very hard to figure this out unless you run, like, 10 rollouts, look at the transcripts, and see why the model failed, and determine, like, manually, like, was this actually, like, a legitimate failure? [00:05:15] S0: Yeah. Can you explain the distinction between difficult and impossible here? Like, you did this in the article. It has to do with both the prompt and the grader.
00:05:25 · Max Kan
[00:05:25] S1: Yeah. I think maybe impossible is not quite the right word. It's more like difficult versus unfair. I think one classic example of a task that'd be unfair is if you have, like, a coding problem that asks them all to fix some bug, maybe in some open source repository. And then you would have sort of an expert human software engineer look at it, and they would say, like, oh, there are sort of two ways to fix this bug. You can either use, like, module a or module b. And I, using my human expertise, recognize that both are kind of equally fair, equally valid. You you feel to pick whichever one you want. But when we're making these benchmarks, you also need to include a verifier for each task so you can check if I'm all succeeded or not. It's And totally possible that some might make a verifier that's sort of scoped to just the module a solution. And then when sort of AI comes and tries the problem, maybe it'll sort of rightfully pick module b instead of module a some of the time, And the verifier would mark that as a failure and sort of, you know, the the AI would perform worse on the task. I think this is an example of the task being unfair because it's like the model did technically solve the problem. It's just that your grader is not faithful to the prompt. And it turns out there are actually lots... Like most popular benchmarks out there, like a TerminalBench four or a SWE-Bench Pro or a DeepSuit or stuff like that, And they actually have tons of issues of this type that people have kind of, funny enough, started really noticing over the past, like, month or so. There've been a couple kind of big blog posts on this because it actually just... It's actually a lot of data companies who were sort of just have agents look at all the transcripts, look at all their prompts, sort of run the task through their own QA process, and they see, like, oh, actually, 60% of the Terminal Bench four tasks are broken or whatever.
00:07:14 · Andrew Megalaa
[00:07:14] S2: The funny thing is, like, I feel like everybody knew it. Like, all the labs definitely know it. Anyone who looks at data, like, definitely knew that the benchmarks are broken. It's just like no one really, like, spends the time to, like, write about it. Because, again, like, to, like, most people in the industry, it's, like, not... It's not new information. It's, like, unsurprising.