SemiAnalysis Weekly
00:13:04 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:13:04] S2: Eventually, people actually started hearing about GDPx on their own and started reaching out to us to be included in [00:13:11] S1: the eval, was kinda interesting. Kinda just spread around the data space. Yeah. Yeah. Some of them were more opportunistic. Like, alignment is a good example where it's not something you'd probably immediately reach for if you're trying to make a benchmark that measures agentic capabilities. But I really do think it is, like, one of the most important things that you need to evaluate future models on. And so it sort of deserves its place in any comprehensive benchmark. At the same time, Fleet is sort of one of the biggest, most well respected data companies today. I have a few friends who work there. And so we're... When we were sort of brainstorming domains, like, that we could potentially include Fleet four, alignment ended up just being one, like, made a ton of sense. Yeah. [00:13:57] S0: Yeah. It makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like, first of all, these are clearly domains where people using Frontier AI today are burning a ton of tokens.
00:13:57 · Jordan Nanos, Andrew Megalaa, Max Kan
[00:13:57] S0: Yeah. It makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like, first of all, these are clearly domains where people using Frontier AI today are burning a ton of tokens. [00:14:11] S2: Absolutely. The domains are really interesting though because like it varies so much. Like some of these are like very established data companies who've been selling data for like a few years now. But then some of them like material science or just like one PhD who's just like really really good at material science. And they just like delivered like these insane tasks that are difficult enough for like even Astra. Like I found that like super cool. They just use, like, papers that, like, they wrote or, like, papers that, like, people that they know, like, wrote. So shout out for those. Like, those those are pretty cool emails. [00:14:45] S0: Yeah. My... Literally, my favorite part of the article is just going through and reading about these pretty esoteric domains that just had so much. It was just such a, it was so clear by reading through the details of the task that the person who had written it felt to me like a domain expert. They certainly knew more than me. And maybe you could talk a little bit about the reaction from the guys that you worked with since the release of the article and how it seems like a lot of people were surprised by the results. Like, I know Brian was specifically saying like, he's shocked that some of the tasks were solved [00:15:17] S1: by the future.
00:15:17 · Max Kan, Andrew Megalaa
[00:15:17] S1: by the future. [00:15:19] S2: Yeah. I think Brian Brian had like a brief moment where like, you know, initially, like, didn't actually expect, like, Astra to be so good at his tasks when we first, like, [00:15:28] S1: asked him for stuff. He's like, [00:15:28] S2: actually, yeah. Some of these workflows that I initially thought Astra was not good on, like, Once he sat down and made tasks, he realized that actually, no, I was just prompting it wrong or something, or I didn't provide enough context and it just ends up solving. So he had to make another set that was actually difficult and frontier level because I think a lot of people don't realize how good the models are. And he even said this, like, he said, like, this material science... Don't know how good the models are. And they think that the models can do certain things, but really they can't. [00:15:58] S1: Well, you should talk about the experience you had with software engineering at the start of the year and then how that relates to Brian.
00:15:58 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:15:58] S1: Well, you should talk about the experience you had with software engineering at the start of the year and then how that relates to Brian. [00:16:08] S2: Yeah. So Yeah. I think I went through a similar thing, like, essentially a year ago at this point. Like, you know, after I left my job to start, like, an RL company with Max here, like, at first, like, you know, I I was still of the opinion that, like, humans plus AI is just, like, a lot better. And, you know, I I can't really take software engineering job. Like, we'll still be coding by hand. Like, but like, you know, AI will automate the easy things. But eventually it just became like, know, Sonnet 3.7 was really good, Sonnet four point came out, Sonnet 4.5, and then Opus 4.6, and now Opus 4.6 was like a moment. So that was like weird. [00:16:42] S1: Opus 4.5, first one. [00:16:43] S2: Opus four five was really [00:16:44] S0: good, but Opus 4.6
00:16:44 · Jordan Nanos, Andrew Megalaa
[00:16:44] S0: good, but Opus 4.6 [00:16:45] S2: was also really good. And like those two models were just like... Yeah, like I think, you know, give it a few more iterations. Like there's... There'll be no need for me to look at a lot of this code. And it was like a very clear trend. And I had to like reconcile that myself because I really like coding at the time. And that was like my thing. Like, was just like good at coding and systems. But yeah, I just had to like figure out like, oh, actually, you know, I have to take the higher level things now. Like, this is no longer a job that like I'll always be doing, like writing code by hand. Though I still think, like, you know, human plus AI is still better than AI alone just because, like, there's, like, a little... This piece of entropy that you can provide that the models don't have themselves. And so, you know, still valuable.
00:17:28 · Max Kan
[00:17:28] S1: Yeah. But funny enough, the the reason why I wanted Andrew to talk about that is I think Brian had a very similar moment for material science when he made the task for a benchmark because he was like, holy shit. You know? If if any human material scientist could do what, like, Astra just did, they're probably, you know, a top... Like, they're definitely better than the media material scientist already. Like, I'm like, I just watched Astra do this. And so I think we're probably going to sort of have these similar moments in lots of other... In domains like chip design and material science and sort of all the other GDPX domains in the imminent future. And that's something that you also hope you'll be able to see sort of in the actual benchmark results as we're able to evaluate, you know, Opus 5.5, Astra 6.1, 6.2, or whatever.