SemiAnalysis Weekly
00:01:40 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:01:40] S1: these models, like, [00:01:41] S2: some of these models, like, we haven't used, like, or even seen, like, out in the wild. Like, Carbon, like... We got access to Carbon on, like, Thursday night or something. Like, we haven't used it for anything. And it [00:01:51] S1: was just, like, [00:01:51] S2: the first time we're trying it out. It did well. It was it was surprising. And same with Muse Wonderful. Like, Muse Wonderful is not available in API. What did you We got access to it. It it was... It did great on cyber. [00:02:05] S0: Yeah. When did you fix the tasks? When when did those get finalized? [00:02:11] S1: Like, Thursday night. So we launched on Sunday. The tasks were finalized, like, Thursday night because we were doing a lot of sort of back and forth iteration with our partners where, like, our queue would flag something and be like, hey. Take a look at this. And then they'd be like, oh, like, these two are false positives. Actually, this one's too real, so we got updated, blah blah blah. And it was really, really last minute. Like, I'm on... I'm I'm very happy that we got it out in time. It was very hard, though. I'm also glad to be taking a break now. But, yeah, it's a crazy process. [00:02:46] S2: I think [00:02:46] S0: the work starts now, man. I don't know. [00:02:49] S1: The work? Okay. I'm gonna take, like, I'm gonna take a couple of chill days, you know. Yeah. And and then we can go through the process of, yeah, I guess, kind of reaping the rewards of our effort.
00:02:49 · Max Kan, Jordan Nanos
[00:02:49] S1: The work? Okay. I'm gonna take, like, I'm gonna take a couple of chill days, you know. Yeah. And and then we can go through the process of, yeah, I guess, kind of reaping the rewards of our effort. [00:03:02] S0: Maintaining the dashboard as new models gets released. Did you did you impose this deadline on yourself? What what was the motivation for that? [00:03:10] S1: It was mostly self imposed. I mean, we were we're decently high confidence that, you know, a new OpenAI and Anthropic flagship are probably both coming out this week. And you obviously cannot launch your benchmark on the same day as, like, Opus 5.5. And Especially if your benchmark doesn't have to. Yeah. And I think we were, like, I think we were decently confident that, like, Monday would not be the release date for these models. Maybe it's, like, Tuesday or Thursday or something. But if there is, like, any risk of the model coming out on Monday, we were like, fuck it. You got it. We we're just gonna publish Sunday instead. And then there's some fear that, like, okay, Sunday night, you know, maybe people aren't as online, so you're not gonna sort of have the morality component. But I think we were very happy with how the launch turned out. Yeah, was generally generally happy. [00:04:02] S0: Take a step back. Explain GDPX from scratch for people who maybe haven't seen the article or just hearing about it for the first time or [00:04:09] S1: didn't read it [00:04:09] S0: as many people who were commenting about it. But, yeah.
00:04:09 · Jordan Nanos, Max Kan, Andrew Megalaa
[00:04:09] S0: as many people who were commenting about it. But, yeah. [00:04:14] S1: Yeah. I think I'm gonna be going... Taking like sort of a step all the way back. I think it's important to understand that a lot of the really popular benchmarks out there are kinda trash. This is largely because they're made by people who don't have expertise and, like, making evals for all environments or sort of trained data in general. And it's, like, very easy for you to make a task that kind of seems good on a surface level, but it's actually unfair in some way such that it's a poor test of model capabilities. [00:04:56] S2: Yeah. A lot of people make their tasks harder by making it unfair rather than actually making the task, like, difficult. And they... And it's, like, very hard to figure this out unless you run, like, 10 rollouts, look at the transcripts, and see why the model failed, and determine, like, manually, like, was this actually, like, a legitimate failure? [00:05:15] S0: Yeah. Can you explain the distinction between difficult and impossible here? Like, you did this in the article. It has to do with both the prompt and the grader.
00:05:25 · Max Kan
[00:05:25] S1: Yeah. I think maybe impossible is not quite the right word. It's more like difficult versus unfair. I think one classic example of a task that'd be unfair is if you have, like, a coding problem that asks them all to fix some bug, maybe in some open source repository. And then you would have sort of an expert human software engineer look at it, and they would say, like, oh, there are sort of two ways to fix this bug. You can either use, like, module a or module b. And I, using my human expertise, recognize that both are kind of equally fair, equally valid. You you feel to pick whichever one you want. But when we're making these benchmarks, you also need to include a verifier for each task so you can check if I'm all succeeded or not. It's And totally possible that some might make a verifier that's sort of scoped to just the module a solution. And then when sort of AI comes and tries the problem, maybe it'll sort of rightfully pick module b instead of module a some of the time, And the verifier would mark that as a failure and sort of, you know, the the AI would perform worse on the task. I think this is an example of the task being unfair because it's like the model did technically solve the problem. It's just that your grader is not faithful to the prompt. And it turns out there are actually lots... Like most popular benchmarks out there, like a TerminalBench four or a SWE-Bench Pro or a DeepSuit or stuff like that, And they actually have tons of issues of this type that people have kind of, funny enough, started really noticing over the past, like, month or so. There've been a couple kind of big blog posts on this because it actually just... It's actually a lot of data companies who were sort of just have agents look at all the transcripts, look at all their prompts, sort of run the task through their own QA process, and they see, like, oh, actually, 60% of the Terminal Bench four tasks are broken or whatever.
00:07:14 · Andrew Megalaa
[00:07:14] S2: The funny thing is, like, I feel like everybody knew it. Like, all the labs definitely know it. Anyone who looks at data, like, definitely knew that the benchmarks are broken. It's just like no one really, like, spends the time to, like, write about it. Because, again, like, to, like, most people in the industry, it's, like, not... It's not new information. It's, like, unsurprising.
00:07:29 · Max Kan
[00:07:29] S1: Yeah. So both benchmarks are bad. That's sort of one thing to know. And then we had SemiAnalysis. We knew we wanted to make a benchmark ourselves because it's kind of the only way you can consistently say interesting things about new model releases because the team is not gonna sort of spend meaningful time and effort trying out, I don't know, the new minimax model when it comes out to see if it's actually good or not. No no disrespect to minimax, but, like, you know, we're just not gonna do that. And so and so you really want an eval. And then we're not gonna do, like... Like, no one fucking tested, you know, GT 6.1 Terra either. Right? But if you have sort of a really high quality eval that you normally do models on, it can go a long way in sort of tracking the performance there. But at the same time, like, we also recognized that a lot of public benchmarks are bad sort of not because of lack of effort on people's parts. It is actually also just really hard to make, like, a task that is fair, but also sufficiently difficult and also interesting or reflective of, like, real work. And it would have been sort of very naive of us to think that, like, oh, SemiAnalysis is gonna be able to come up with, like, 200 really high quality hacks ourselves. And so the strategy we took is we partnered with 14 different sort of specialized data companies who use, like, a lot of their expertise to actually supply the task for each domain. But we were responsible for sort of actually QA ing all of them. We actually... We, like, dropped more data providers than we actually included in the final benchmark because a lot of them just failed QA. And, also, we are sort of very heavily involved in... With, like, task iteration for the ones who passed QA as well, as we kinda mentioned at the the start of the episode. But that was sort of, like, I guess, the philosophy, methodology for how we actually made the benchmark. In terms of what the benchmark covers, this is obviously sort of another kind of big question we need to figure out because you can... I mean, you can you can kinda pick any domain. Right? AI is affecting everything these days.