SemiAnalysis Weekly
00:43:55 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:43:55] S1: Yeah. That was sick. Shout out Sundar. I mean, [00:43:59] S2: we're excited about it too. Like, hey. More Frontier models is better for all of us. We definitely want, you know, better models from everybody. It only benefits the consumer as there are more competition. [00:44:10] S1: Yeah. I do I do however think that the right sort of apples to apples comparison for where the Frontier labs currently sit is if you exclude cyber and bio and alignment and look at that GDPX score. Maybe even switch to partial scoring too because that kinda gives you some more signal on legal and finance where no one's smart enough to do binary grading. So at least get reward for binary grading. And if you do that sort of ranking, you you see that Carbon is still kind of behind Astra and Opus, etcetera. But that being said, they... It is certainly sort of an inflection in the trend for where Google's model releases were going. Because I think at one point, they kinda felt that they were gonna fall farther and farther behind even, like, Meta in particular, potentially even XAI, and potentially even some of the Chinese open source guys. But they've, like, kind of really reestablished themselves as the third place player with Carbon. [00:45:15] S0: I'll come to this as a timing thing then. Like, sorry to interrupt, but Fable five five, you guys are teasing that might come out soon. Obviously, Opus, Sonnet are on the list
00:45:15 · Jordan Nanos, Max Kan, Andrew Megalaa
[00:45:15] S0: I'll come to this as a timing thing then. Like, sorry to interrupt, but Fable five five, you guys are teasing that might come out soon. Obviously, Opus, Sonnet are on the list [00:45:28] S1: of Yeah. So I I think I think the gap between Anthropic slash OpenAI versus Google today is likely still very big because, like, we're gonna have new flagship model releases with them quite soon. But, like, Muse Wonderful came out pretty recently. Right? And so I think you're sort of evaluating Google's position versus Meta and x AI. I think there's a pretty strong urban to me that actually they're, like, the top of that kind of tier two cohort. But I definitely still believe that there's, like, a pretty meaningful gap between open eye slash anthropic versus urban. [00:46:05] S2: And, you know, there's, like, the thing, like, you know, OpenAI and Anthropic are obviously not releasing their best models. So it's it's just a comparison between the public models. Well, like, Gemini and, like, Meta, like, they are releasing, like, the best that they have so far. Yeah. Anthropic could just be on, like, mythos four at this point. Who knows? Yeah. True.
00:46:05 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:46:05] S2: And, you know, there's, like, the thing, like, you know, OpenAI and Anthropic are obviously not releasing their best models. So it's it's just a comparison between the public models. Well, like, Gemini and, like, Meta, like, they are releasing, like, the best that they have so far. Yeah. Anthropic could just be on, like, mythos four at this point. Who knows? Yeah. True. [00:46:24] S1: We're we're gonna get, like, Fable plus one because I I think Anthropic is still kind of rolling with Fable one level models essentially or, like, Mythos one. I guess it's called Fable five. Internally, they're probably, like, at least two generations above that. And then we're finally gonna get, you know, one generation above that hopefully this week. So I think the the gap between sort of the actual best model each lab has including internal versus well, actually, that gap is just sort of very wide across OpenAI, Anthropic, Google, Meta, SpaceX. [00:46:59] S0: Yeah. Well, maybe the biggest takeaway for me there is just how relentless the pace of model releases is. And, I mean, the trend is for it to get faster and faster, where they improve post training and improve the data mixes. [00:47:14] S1: RSI, bro. It's RSI.
00:47:14 · Max Kan, Jordan Nanos
[00:47:14] S1: RSI, bro. It's RSI. [00:47:17] S0: Yeah. Can you talk about next steps? Like, where do you go from here? You think it's possible... Actually, one thing we didn't talk about at all is the concept of releasing one task and one rollout for the category, for each of the categories or, well, for each of the data vendors, so multiple from some of the categories. You talk about that approach that you guys took there of of the mix of kind of partially public, partially private, and how other benchmarks approach it comparison?
00:47:49 · Max Kan
[00:47:49] S1: So I would say the trade off that you have to make for any benchmark is the more public you make it, you know, obviously, the more experts are able to QA everything themselves and kinda verify that your stuff is actually good. At the same time, if you make it public, it just becomes, like, way too easy to hoe climb on a benchmark. And oftentimes, like, your benchmark no longer becomes a measure of, like, model capability and then more becomes a measure of, like, which lab bought more data that looks just like the task and my benchmark. And, again, I'm not gonna name any particular names, but I think there are a lot of public benchmarks that have already sort of, like, fit into that category. It's really just a measure of, like, okay. You know, which which lab spent the most effort to buy the most data of this shape? And priority number one for us is that we don't want people to yield the hill climber benchmark. We sort of want our benchmark to be an accurate measure of sort of real model capabilities. By the same time, we we obviously know that establishing credibility is important. And so releasing, like, the rollouts for one task per partner, we thought was a good trade off there where kind of people can get some visibility into each partner and have confidence of tasks are good. At the same time, sort of us going pretty in-depth in the actual methodology for, like, each partner and explaining how, like, you know, our finance guys actually use this, like, real deal room. Our our cyber guys, like, found their own zero days and further into the task. I think sort of buying all those details there is the other way that we can sort of instill confidence in the quality of the benchmark. Also, just like giving details in our QA process where we say, like, yo. You know, we ran our own agent to QA. And if you take Terminal-Bench four, literally flags, like, 50% of the task is broken. But then we ran out of every partner and we made sure that, like, if you couldn't pass this bar, you just weren't included in the benchmark. I think stuff like that is what allows us to sort of give people confidence in the quality of the bench without making all the tasks public. As for what's next, we...
00:49:57 · Max Kan, Jordan Nanos, Andrew Megalaa
Well, we really hope we can get sort of version one on some major new model release cards when they come out because we do think it's sort of, like, the best benchmark for measuring printer capabilities in the world today. At the same time, we're definitely, like, just getting started, and we plan to do, like, future versions as models become increasingly capable. They saturate the task in g p x v one. There are sort of more domains that get unlocked that can absorb lots of, like, frontier tokens and, like, be super valuable or in higher ROI. And so we definitely wanna release, like, future versions with more, you know, data partners, always using sort of the highest quality data partners at any given point in time, always picking the most important domains at any given point in time. Hopefully, we can do things like have more partners and have more tasks per domain and also run all the models at every reasoning level and stuff like that now that we have, like, you know, more clout and more traction with the benchmark. [00:50:59] S0: Awesome, man. Okay. I think you answered the final question about where we go from here. Andrew, any, like, closing thoughts from you about the future next steps, initial reaction from this stuff? Pretty fun week. [00:51:09] S2: Yeah. I mean, definitely agree with Max. Like, I do want this to be like the benchmark everyone thinks about one day. Think about like what model has which capabilities and where they stand. And this is only the first iteration. Like, we're gonna keep working on it. We're gonna get