SemiAnalysis Weekly
00:10:55 · Jordan Nanos
[00:10:55] S0: Sweet. Okay. Maybe we can talk through the specific domains in a little bit more detail. For those who don't know, there's a website up, gpx.seminalysis.com. I'll share it on screen here for a sec. And as we scroll through the domains, self improvement or RSI, chip design, cybersecurity, biology, finance, legal, quant trading, hardware design, material science, alignment, and software engineering. I think this is not really a breakdown that a lot of people have on their tip of their tongue when they're talking about like the domains AI is coming. Even just introducing each of these one by one, think would be interesting. But maybe you can start by just saying, how did you arrive at these categories broadly? And then how did you find the specific data vendors that you were going with in the given categories that you're focusing on?
00:11:45 · Max Kan, Andrew Megalaa
[00:11:45] S1: Yeah. Yeah. So arriving at the domains was a lot of just sort of brainstorming ourselves, trying to pick the domains that are verifiable and also the ones that seem like they would have of the capacity to absorb a lot of intelligence. And after we had our initial list, we talked to a bunch of people at the labs, people working on similar problems, and then we did this process for maybe a few weeks until we finally arrived at our final set of domains. In terms of how we pick the partners, we're sort of fortunately decently well connected in the data space, and we actually just happen to know sort of a lot of RL environment companies and what each of their specialties are. So we reached out to sort of a bunch of people, who work in the various domains. We got sample tasks from them. We QA ed them. We sort of dropped the ones that are bad. In the cases where we had, like, more than one good person who wanted to be exclusive with us in a particular domain, we would tie break based on things like tastes and how interesting we thought the problems were and representative of their their domain. And that was... Yeah. That was basically the whole process there. [00:13:04] S2: Eventually, people actually started hearing about GDPx on their own and started reaching out to us to be included in
00:13:04 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:13:04] S2: Eventually, people actually started hearing about GDPx on their own and started reaching out to us to be included in [00:13:11] S1: the eval, was kinda interesting. Kinda just spread around the data space. Yeah. Yeah. Some of them were more opportunistic. Like, alignment is a good example where it's not something you'd probably immediately reach for if you're trying to make a benchmark that measures agentic capabilities. But I really do think it is, like, one of the most important things that you need to evaluate future models on. And so it sort of deserves its place in any comprehensive benchmark. At the same time, Fleet is sort of one of the biggest, most well respected data companies today. I have a few friends who work there. And so we're... When we were sort of brainstorming domains, like, that we could potentially include Fleet four, alignment ended up just being one, like, made a ton of sense. Yeah. [00:13:57] S0: Yeah. It makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like, first of all, these are clearly domains where people using Frontier AI today are burning a ton of tokens.
00:13:57 · Jordan Nanos, Andrew Megalaa, Max Kan
[00:13:57] S0: Yeah. It makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like, first of all, these are clearly domains where people using Frontier AI today are burning a ton of tokens. [00:14:11] S2: Absolutely. The domains are really interesting though because like it varies so much. Like some of these are like very established data companies who've been selling data for like a few years now. But then some of them like material science or just like one PhD who's just like really really good at material science. And they just like delivered like these insane tasks that are difficult enough for like even Astra. Like I found that like super cool. They just use, like, papers that, like, they wrote or, like, papers that, like, people that they know, like, wrote. So shout out for those. Like, those those are pretty cool emails. [00:14:45] S0: Yeah. My... Literally, my favorite part of the article is just going through and reading about these pretty esoteric domains that just had so much. It was just such a, it was so clear by reading through the details of the task that the person who had written it felt to me like a domain expert. They certainly knew more than me. And maybe you could talk a little bit about the reaction from the guys that you worked with since the release of the article and how it seems like a lot of people were surprised by the results. Like, I know Brian was specifically saying like, he's shocked that some of the tasks were solved [00:15:17] S1: by the future.
00:15:17 · Max Kan, Andrew Megalaa
[00:15:17] S1: by the future. [00:15:19] S2: Yeah. I think Brian Brian had like a brief moment where like, you know, initially, like, didn't actually expect, like, Astra to be so good at his tasks when we first, like, [00:15:28] S1: asked him for stuff. He's like, [00:15:28] S2: actually, yeah. Some of these workflows that I initially thought Astra was not good on, like, Once he sat down and made tasks, he realized that actually, no, I was just prompting it wrong or something, or I didn't provide enough context and it just ends up solving. So he had to make another set that was actually difficult and frontier level because I think a lot of people don't realize how good the models are. And he even said this, like, he said, like, this material science... Don't know how good the models are. And they think that the models can do certain things, but really they can't. [00:15:58] S1: Well, you should talk about the experience you had with software engineering at the start of the year and then how that relates to Brian.
00:15:58 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:15:58] S1: Well, you should talk about the experience you had with software engineering at the start of the year and then how that relates to Brian. [00:16:08] S2: Yeah. So Yeah. I think I went through a similar thing, like, essentially a year ago at this point. Like, you know, after I left my job to start, like, an RL company with Max here, like, at first, like, you know, I I was still of the opinion that, like, humans plus AI is just, like, a lot better. And, you know, I I can't really take software engineering job. Like, we'll still be coding by hand. Like, but like, you know, AI will automate the easy things. But eventually it just became like, know, Sonnet 3.7 was really good, Sonnet four point came out, Sonnet 4.5, and then Opus 4.6, and now Opus 4.6 was like a moment. So that was like weird. [00:16:42] S1: Opus 4.5, first one. [00:16:43] S2: Opus four five was really [00:16:44] S0: good, but Opus 4.6