SemiAnalysis Weekly
00:46:05 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:46:05] S2: And, you know, there's, like, the thing, like, you know, OpenAI and Anthropic are obviously not releasing their best models. So it's it's just a comparison between the public models. Well, like, Gemini and, like, Meta, like, they are releasing, like, the best that they have so far. Yeah. Anthropic could just be on, like, mythos four at this point. Who knows? Yeah. True. [00:46:24] S1: We're we're gonna get, like, Fable plus one because I I think Anthropic is still kind of rolling with Fable one level models essentially or, like, Mythos one. I guess it's called Fable five. Internally, they're probably, like, at least two generations above that. And then we're finally gonna get, you know, one generation above that hopefully this week. So I think the the gap between sort of the actual best model each lab has including internal versus well, actually, that gap is just sort of very wide across OpenAI, Anthropic, Google, Meta, SpaceX. [00:46:59] S0: Yeah. Well, maybe the biggest takeaway for me there is just how relentless the pace of model releases is. And, I mean, the trend is for it to get faster and faster, where they improve post training and improve the data mixes. [00:47:14] S1: RSI, bro. It's RSI.
00:47:14 · Max Kan, Jordan Nanos
[00:47:14] S1: RSI, bro. It's RSI. [00:47:17] S0: Yeah. Can you talk about next steps? Like, where do you go from here? You think it's possible... Actually, one thing we didn't talk about at all is the concept of releasing one task and one rollout for the category, for each of the categories or, well, for each of the data vendors, so multiple from some of the categories. You talk about that approach that you guys took there of of the mix of kind of partially public, partially private, and how other benchmarks approach it comparison?
00:47:49 · Max Kan
[00:47:49] S1: So I would say the trade off that you have to make for any benchmark is the more public you make it, you know, obviously, the more experts are able to QA everything themselves and kinda verify that your stuff is actually good. At the same time, if you make it public, it just becomes, like, way too easy to hoe climb on a benchmark. And oftentimes, like, your benchmark no longer becomes a measure of, like, model capability and then more becomes a measure of, like, which lab bought more data that looks just like the task and my benchmark. And, again, I'm not gonna name any particular names, but I think there are a lot of public benchmarks that have already sort of, like, fit into that category. It's really just a measure of, like, okay. You know, which which lab spent the most effort to buy the most data of this shape? And priority number one for us is that we don't want people to yield the hill climber benchmark. We sort of want our benchmark to be an accurate measure of sort of real model capabilities. By the same time, we we obviously know that establishing credibility is important. And so releasing, like, the rollouts for one task per partner, we thought was a good trade off there where kind of people can get some visibility into each partner and have confidence of tasks are good. At the same time, sort of us going pretty in-depth in the actual methodology for, like, each partner and explaining how, like, you know, our finance guys actually use this, like, real deal room. Our our cyber guys, like, found their own zero days and further into the task. I think sort of buying all those details there is the other way that we can sort of instill confidence in the quality of the benchmark. Also, just like giving details in our QA process where we say, like, yo. You know, we ran our own agent to QA. And if you take Terminal-Bench four, literally flags, like, 50% of the task is broken. But then we ran out of every partner and we made sure that, like, if you couldn't pass this bar, you just weren't included in the benchmark. I think stuff like that is what allows us to sort of give people confidence in the quality of the bench without making all the tasks public. As for what's next, we...
00:49:57 · Max Kan, Jordan Nanos, Andrew Megalaa
Well, we really hope we can get sort of version one on some major new model release cards when they come out because we do think it's sort of, like, the best benchmark for measuring printer capabilities in the world today. At the same time, we're definitely, like, just getting started, and we plan to do, like, future versions as models become increasingly capable. They saturate the task in g p x v one. There are sort of more domains that get unlocked that can absorb lots of, like, frontier tokens and, like, be super valuable or in higher ROI. And so we definitely wanna release, like, future versions with more, you know, data partners, always using sort of the highest quality data partners at any given point in time, always picking the most important domains at any given point in time. Hopefully, we can do things like have more partners and have more tasks per domain and also run all the models at every reasoning level and stuff like that now that we have, like, you know, more clout and more traction with the benchmark. [00:50:59] S0: Awesome, man. Okay. I think you answered the final question about where we go from here. Andrew, any, like, closing thoughts from you about the future next steps, initial reaction from this stuff? Pretty fun week. [00:51:09] S2: Yeah. I mean, definitely agree with Max. Like, I do want this to be like the benchmark everyone thinks about one day. Think about like what model has which capabilities and where they stand. And this is only the first iteration. Like, we're gonna keep working on it. We're gonna get
00:51:09 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:51:09] S2: Yeah. I mean, definitely agree with Max. Like, I do want this to be like the benchmark everyone thinks about one day. Think about like what model has which capabilities and where they stand. And this is only the first iteration. Like, we're gonna keep working on it. We're gonna get [00:51:25] S1: hopefully, more data partners. [00:51:27] S2: And if anyone listening to this is interested in evals and, you know, maintaining this, please reach out. We are hiring on this team. Of course, just email gdpx@semianalysis.com. Yeah. [00:51:40] S0: Awesome. Thanks for the to end off. Thanks, guys, for joining. Appreciate it. [00:51:44] S1: Yeah. Thank you, Jordan. [00:51:45] S2: Thank you, Jordan.