SemiAnalysis Weekly
00:31:08 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:31:08] S1: the same. That one email, [00:31:10] S2: you if were not on top of it, would have gotten yelled at for sure. You would have said, yeah, this is unacceptable behavior in a real banking environment. [00:31:17] S0: I know some people on the tokenomics team who I think are possibly gonna be listening to this podcast having some flashbacks, [00:31:23] S2: PTS out of the [00:31:25] S0: yard stares. You talk about the grading in this case? Because I think a lot of people would be confused about the partial credit versus binary grading approach you have to apply to some of these things that don't necessarily have a binary outcome.
00:31:40 · Max Kan
[00:31:40] S1: Yeah. Yeah. So in the case... So I'll say our finance tasks use binary grading. So the model either gets a zero or a one. But there's sort of a continuous sort of scoring component there because might have a task where the final deliverable is, like, some Excel workbook. Maybe it's like a quality of earnings book that, like, the accountant had to make for this private equity deal. And, obviously, like, that workbook can kinda be decomposed into a bunch of different parts, you know, different cells, and the model has to find the right number for. And you kind of assign partial grading for each of these atomic criteria. But then we still use kind of binary grading at the end because if you were a real accounting associate and then you gave your boss a workbook where, like, half the numbers are right and half of them are wrong, they're not gonna be like, oh, man. Like, great job, little Jimmy. Like, you got half of it right. Like, I'll take them here. They're gonna be like, dude, what the fuck, man? Like, I'm firing your ass right now. So I think in those cases, like, it makes sense to do binary grading even though there is, like, a partial score component. Yeah, maybe, Jordan, throwing a show here that, like, we have 0% when you use binary in the finance, basically, except for Carbon who kinda got, like, one task right. But when you switch to partial, and then you sort of get much more words to go on how the models actually did. That being said... Okay. Actually, I'll say, one, a lot of people on Twitter were kind of shitting on us using binary grading because there are other benchmarks that kind of use that to hide the fact that their tasks are actually quite easy. I won't name any particular names here, but there are some popular benchmarks where you look at binary grading and it's like, oh, 25% on Opus. Like, that is probably pretty hard. But then you switch to partial grading, it's like 97% Opus. And it's like, okay. Like that that last 33%
00:33:36 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:33:36] S2: is probably just, like, some pathetic requirement that there's no reason you shouldn't have expected them. [00:33:40] S1: Yeah. Or your task is just, like, fucking unfair and broken in some way. And, yeah, so, like, I agree. If you look at our finance task and it was, like, 95% on Opus or partial grading, they're probably a bad task. But you saw it was, like, 50 something percent. Same with our legal. The legal was all zeros, but then the best model, even when you search for partial grading, I think is, like, it's in the thirties or forties. At the same time, most of our domains actually use continuous grading because it makes sense for those. You know? You have a chip design task where maybe the model is sort of asked to do some, like, PPA optimization, And then you just have, like, zero as some naive solution, and then you have one as sort of some, like, max theoretical achievable solution. And you kinda just give the model partial credit depending on how much it can optimize, like, your chip design. We sort of picked the grading strategy that, like, makes the most sense given the particular domain. We did not do, you know, binary grading for everything so we could kind of artificially lower the score. [00:34:40] S0: Yeah. And I think this is... Maybe goes back to the comment about frontier intelligence versus day to day tasks. I mean, I don't know what frontier in finance necessarily represents the same way in science where it's pretty clear, like you're pursuing a new discovery or something like that. Finance, like the frontier is being accurate. And
00:34:40 · Jordan Nanos, Max Kan
[00:34:40] S0: Yeah. And I think this is... Maybe goes back to the comment about frontier intelligence versus day to day tasks. I mean, I don't know what frontier in finance necessarily represents the same way in science where it's pretty clear, like you're pursuing a new discovery or something like that. Finance, like the frontier is being accurate. And [00:35:02] S1: Yeah. I think I think one of the things to really note for finance is that maybe sort of at first glance, you think it's like different from, you know, chip design or material science. But we actually think it is going to be one of the domains that sort of spends very heavily on tokens in 2027. It is actually already one of the main domains that people spend tokens on in 2026, largely because, like, the margin structures for the businesses sort of supports it. Like, they can actually afford to spend a ton of money on tokens. But then also, there are... And this is actually... This is even more true for legal, but there are some sort of, like, PVP components there where if we if we take legal, for example, if the AI sort of could theoretically do a better job, like, winning this lawsuit or, like, redlining some contract, like, actually does have implications, especially if, like, maybe your your, like, opponent in the scenario is not using the superintelligent AI to do these things. So I actually think finance and legal, like, are domains that will be able to absorb a lot of frontier intelligence, whereas maybe things like real estate or, like, government or something might not be.
00:36:21 · Jordan Nanos, Andrew Megalaa
[00:36:21] S0: Makes sense. Just to... If you could expand on that a little bit more, because I think I know the backdrop. You're specifically talking about the reason for focusing on these domains specifically, rather than... Like, the choice to do that, rather than just taking a cross section of all knowledge work in the economy and saying we're gonna have tasks for every single thing that people do across the economy. Right? Yeah, I think the thing that the listeners are probably thinking about right now is just the concept of AI driven job loss. If people are using Frontier Intelligence and they're gaining these productivity improvements, or they're making more money, or they're winning more things, maybe you can talk about how does running through a benchmark like this inform your point of view on whether AI is going to yeah, cause job loss in the next year or two years, even five year time horizon? [00:37:22] S2: I actually think
00:37:25 · Max Kan
[00:37:25] S1: they're like orthogonal in some sense because we don't... We especially choose not to cover a lot of sort of the white collar knowledge work domains that people are probably most afraid of drop loss occurring, like, I don't know, customer service or something like that. In terms of whether or not we will have, like, massive job loss from AI, it's really hard to say. I think a lot of the doomers have probably been very pleasantly surprised by sort of how AI plus human, like, coworking has evolved over the past year where it's like every... Any any software engineer that sort of, like, has their eyes on the ball will probably tell you that Opus, when given sort of like a well specified problem, is just like a better software engineer than me, period. But at the same time, like, good software engineers are more valuable today than they've ever been. Now is it possible we're sort of in this, like, Goldilocks zone where the humans can... Like, a lot of humans can still provide kind of that, like, taste and supervision and judgment that makes it so that, like, them plus the model is, like, way, way more effective than just the model. But then, like, maybe two years from now, you actually only need, like, one dude at each company who's, like, supervising all the software engineering and, like, basically, everyone else got fired. Maybe. I don't know. I think it's possible that's the world we end up in. But it's also possible it's not. I think we just have to... You really gotta pay close attention to the model capabilities to see. I would