SemiAnalysis Weekly
00:13:57 · Jordan Nanos, Andrew Megalaa, Max Kan
[00:13:57] S0: Yeah. It makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like, first of all, these are clearly domains where people using Frontier AI today are burning a ton of tokens. [00:14:11] S2: Absolutely. The domains are really interesting though because like it varies so much. Like some of these are like very established data companies who've been selling data for like a few years now. But then some of them like material science or just like one PhD who's just like really really good at material science. And they just like delivered like these insane tasks that are difficult enough for like even Astra. Like I found that like super cool. They just use, like, papers that, like, they wrote or, like, papers that, like, people that they know, like, wrote. So shout out for those. Like, those those are pretty cool emails. [00:14:45] S0: Yeah. My... Literally, my favorite part of the article is just going through and reading about these pretty esoteric domains that just had so much. It was just such a, it was so clear by reading through the details of the task that the person who had written it felt to me like a domain expert. They certainly knew more than me. And maybe you could talk a little bit about the reaction from the guys that you worked with since the release of the article and how it seems like a lot of people were surprised by the results. Like, I know Brian was specifically saying like, he's shocked that some of the tasks were solved [00:15:17] S1: by the future.
00:15:17 · Max Kan, Andrew Megalaa
[00:15:17] S1: by the future. [00:15:19] S2: Yeah. I think Brian Brian had like a brief moment where like, you know, initially, like, didn't actually expect, like, Astra to be so good at his tasks when we first, like, [00:15:28] S1: asked him for stuff. He's like, [00:15:28] S2: actually, yeah. Some of these workflows that I initially thought Astra was not good on, like, Once he sat down and made tasks, he realized that actually, no, I was just prompting it wrong or something, or I didn't provide enough context and it just ends up solving. So he had to make another set that was actually difficult and frontier level because I think a lot of people don't realize how good the models are. And he even said this, like, he said, like, this material science... Don't know how good the models are. And they think that the models can do certain things, but really they can't. [00:15:58] S1: Well, you should talk about the experience you had with software engineering at the start of the year and then how that relates to Brian.
00:15:58 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:15:58] S1: Well, you should talk about the experience you had with software engineering at the start of the year and then how that relates to Brian. [00:16:08] S2: Yeah. So Yeah. I think I went through a similar thing, like, essentially a year ago at this point. Like, you know, after I left my job to start, like, an RL company with Max here, like, at first, like, you know, I I was still of the opinion that, like, humans plus AI is just, like, a lot better. And, you know, I I can't really take software engineering job. Like, we'll still be coding by hand. Like, but like, you know, AI will automate the easy things. But eventually it just became like, know, Sonnet 3.7 was really good, Sonnet four point came out, Sonnet 4.5, and then Opus 4.6, and now Opus 4.6 was like a moment. So that was like weird. [00:16:42] S1: Opus 4.5, first one. [00:16:43] S2: Opus four five was really [00:16:44] S0: good, but Opus 4.6
00:16:44 · Jordan Nanos, Andrew Megalaa
[00:16:44] S0: good, but Opus 4.6 [00:16:45] S2: was also really good. And like those two models were just like... Yeah, like I think, you know, give it a few more iterations. Like there's... There'll be no need for me to look at a lot of this code. And it was like a very clear trend. And I had to like reconcile that myself because I really like coding at the time. And that was like my thing. Like, was just like good at coding and systems. But yeah, I just had to like figure out like, oh, actually, you know, I have to take the higher level things now. Like, this is no longer a job that like I'll always be doing, like writing code by hand. Though I still think, like, you know, human plus AI is still better than AI alone just because, like, there's, like, a little... This piece of entropy that you can provide that the models don't have themselves. And so, you know, still valuable.
00:17:28 · Max Kan
[00:17:28] S1: Yeah. But funny enough, the the reason why I wanted Andrew to talk about that is I think Brian had a very similar moment for material science when he made the task for a benchmark because he was like, holy shit. You know? If if any human material scientist could do what, like, Astra just did, they're probably, you know, a top... Like, they're definitely better than the media material scientist already. Like, I'm like, I just watched Astra do this. And so I think we're probably going to sort of have these similar moments in lots of other... In domains like chip design and material science and sort of all the other GDPX domains in the imminent future. And that's something that you also hope you'll be able to see sort of in the actual benchmark results as we're able to evaluate, you know, Opus 5.5, Astra 6.1, 6.2, or whatever.
00:18:22 · Andrew Megalaa, Jordan Nanos
[00:18:22] S2: I think we got, like, a really good tweet. Someone asked about whether, like, you know, GDPval had human baselines for how long tasks take, you know, to like put things in And this was really funny because like I responded to this and I was like, actually, like it doesn't even make sense to have human baselines anymore. Like in the material science tasks, like Astra solved a subtask of a task better than like a published state of the art reference. Like when the models are like essentially past expert level, how do you even put a human baseline to that? And And some of these chip design tasks are just so long horizon and would take teams at companies to even do... And those, there's no human baseline there. How do you even put a number on it? [00:19:07] S0: It's a lot of money. Whatever the hourly equivalent is in human work hours is a lot of money. And I just threw up on screen the the cost per task for the material science works where the biggest ones, like was at 40 or $50 per task, and many of the models are solving these tasks, some of them in $10, which is I think, a pretty unbelievable statement. Maybe you can talk about reading some of the rollouts in more detail. What were some of the pieces of model behavior that surprised you the most?