SemiAnalysis Weekly
00:07:29 · Max Kan
[00:07:29] S1: Yeah. So both benchmarks are bad. That's sort of one thing to know. And then we had SemiAnalysis. We knew we wanted to make a benchmark ourselves because it's kind of the only way you can consistently say interesting things about new model releases because the team is not gonna sort of spend meaningful time and effort trying out, I don't know, the new minimax model when it comes out to see if it's actually good or not. No no disrespect to minimax, but, like, you know, we're just not gonna do that. And so and so you really want an eval. And then we're not gonna do, like... Like, no one fucking tested, you know, GT 6.1 Terra either. Right? But if you have sort of a really high quality eval that you normally do models on, it can go a long way in sort of tracking the performance there. But at the same time, like, we also recognized that a lot of public benchmarks are bad sort of not because of lack of effort on people's parts. It is actually also just really hard to make, like, a task that is fair, but also sufficiently difficult and also interesting or reflective of, like, real work. And it would have been sort of very naive of us to think that, like, oh, SemiAnalysis is gonna be able to come up with, like, 200 really high quality hacks ourselves. And so the strategy we took is we partnered with 14 different sort of specialized data companies who use, like, a lot of their expertise to actually supply the task for each domain. But we were responsible for sort of actually QA ing all of them. We actually... We, like, dropped more data providers than we actually included in the final benchmark because a lot of them just failed QA. And, also, we are sort of very heavily involved in... With, like, task iteration for the ones who passed QA as well, as we kinda mentioned at the the start of the episode. But that was sort of, like, I guess, the philosophy, methodology for how we actually made the benchmark. In terms of what the benchmark covers, this is obviously sort of another kind of big question we need to figure out because you can... I mean, you can you can kinda pick any domain. Right? AI is affecting everything these days.
00:09:44 · Max Kan
But one of the things that we sort of pretty strongly believe on the tokenomics team is that the first trillion dollars of AI revenue are going to be highly concentrated within a few domains that can actually absorb sort of increasingly smarter frontier intelligence and earn high ROI. I think a lot of people recently have seen sort of these token budgeting trends or people talking about how, like, oh, you actually don't need frontier intelligence for majority of tasks, and they view that as bearish for the AI trade. We sort of totally agree with the fact that, you know, most tasks, a lot of tasks will not need frontier intelligence. But we think there are gonna be a lot of domains like chip design or biology or cybersecurity where we actually are just gonna need sort of the increasingly smarter frontier model, and you will continue to get high ROI from table six, table seven, table eight, whatever. And so we decided to pick domains for our benchmark, GDPx, which sort of match these domains that can absorb frontier intelligence. And that was kind of our guiding philosophy there.
00:10:55 · Jordan Nanos
[00:10:55] S0: Sweet. Okay. Maybe we can talk through the specific domains in a little bit more detail. For those who don't know, there's a website up, gpx.seminalysis.com. I'll share it on screen here for a sec. And as we scroll through the domains, self improvement or RSI, chip design, cybersecurity, biology, finance, legal, quant trading, hardware design, material science, alignment, and software engineering. I think this is not really a breakdown that a lot of people have on their tip of their tongue when they're talking about like the domains AI is coming. Even just introducing each of these one by one, think would be interesting. But maybe you can start by just saying, how did you arrive at these categories broadly? And then how did you find the specific data vendors that you were going with in the given categories that you're focusing on?
00:11:45 · Max Kan, Andrew Megalaa
[00:11:45] S1: Yeah. Yeah. So arriving at the domains was a lot of just sort of brainstorming ourselves, trying to pick the domains that are verifiable and also the ones that seem like they would have of the capacity to absorb a lot of intelligence. And after we had our initial list, we talked to a bunch of people at the labs, people working on similar problems, and then we did this process for maybe a few weeks until we finally arrived at our final set of domains. In terms of how we pick the partners, we're sort of fortunately decently well connected in the data space, and we actually just happen to know sort of a lot of RL environment companies and what each of their specialties are. So we reached out to sort of a bunch of people, who work in the various domains. We got sample tasks from them. We QA ed them. We sort of dropped the ones that are bad. In the cases where we had, like, more than one good person who wanted to be exclusive with us in a particular domain, we would tie break based on things like tastes and how interesting we thought the problems were and representative of their their domain. And that was... Yeah. That was basically the whole process there. [00:13:04] S2: Eventually, people actually started hearing about GDPx on their own and started reaching out to us to be included in
00:13:04 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:13:04] S2: Eventually, people actually started hearing about GDPx on their own and started reaching out to us to be included in [00:13:11] S1: the eval, was kinda interesting. Kinda just spread around the data space. Yeah. Yeah. Some of them were more opportunistic. Like, alignment is a good example where it's not something you'd probably immediately reach for if you're trying to make a benchmark that measures agentic capabilities. But I really do think it is, like, one of the most important things that you need to evaluate future models on. And so it sort of deserves its place in any comprehensive benchmark. At the same time, Fleet is sort of one of the biggest, most well respected data companies today. I have a few friends who work there. And so we're... When we were sort of brainstorming domains, like, that we could potentially include Fleet four, alignment ended up just being one, like, made a ton of sense. Yeah. [00:13:57] S0: Yeah. It makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like, first of all, these are clearly domains where people using Frontier AI today are burning a ton of tokens.
00:13:57 · Jordan Nanos, Andrew Megalaa, Max Kan
[00:13:57] S0: Yeah. It makes sense. Maybe another way that I was thinking about it when I was reading through some of these descriptions in here was like, first of all, these are clearly domains where people using Frontier AI today are burning a ton of tokens. [00:14:11] S2: Absolutely. The domains are really interesting though because like it varies so much. Like some of these are like very established data companies who've been selling data for like a few years now. But then some of them like material science or just like one PhD who's just like really really good at material science. And they just like delivered like these insane tasks that are difficult enough for like even Astra. Like I found that like super cool. They just use, like, papers that, like, they wrote or, like, papers that, like, people that they know, like, wrote. So shout out for those. Like, those those are pretty cool emails. [00:14:45] S0: Yeah. My... Literally, my favorite part of the article is just going through and reading about these pretty esoteric domains that just had so much. It was just such a, it was so clear by reading through the details of the task that the person who had written it felt to me like a domain expert. They certainly knew more than me. And maybe you could talk a little bit about the reaction from the guys that you worked with since the release of the article and how it seems like a lot of people were surprised by the results. Like, I know Brian was specifically saying like, he's shocked that some of the tasks were solved [00:15:17] S1: by the future.