SemiAnalysis Weekly
00:33:36 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:33:36] S2: is probably just, like, some pathetic requirement that there's no reason you shouldn't have expected them. [00:33:40] S1: Yeah. Or your task is just, like, fucking unfair and broken in some way. And, yeah, so, like, I agree. If you look at our finance task and it was, like, 95% on Opus or partial grading, they're probably a bad task. But you saw it was, like, 50 something percent. Same with our legal. The legal was all zeros, but then the best model, even when you search for partial grading, I think is, like, it's in the thirties or forties. At the same time, most of our domains actually use continuous grading because it makes sense for those. You know? You have a chip design task where maybe the model is sort of asked to do some, like, PPA optimization, And then you just have, like, zero as some naive solution, and then you have one as sort of some, like, max theoretical achievable solution. And you kinda just give the model partial credit depending on how much it can optimize, like, your chip design. We sort of picked the grading strategy that, like, makes the most sense given the particular domain. We did not do, you know, binary grading for everything so we could kind of artificially lower the score. [00:34:40] S0: Yeah. And I think this is... Maybe goes back to the comment about frontier intelligence versus day to day tasks. I mean, I don't know what frontier in finance necessarily represents the same way in science where it's pretty clear, like you're pursuing a new discovery or something like that. Finance, like the frontier is being accurate. And
00:34:40 · Jordan Nanos, Max Kan
[00:34:40] S0: Yeah. And I think this is... Maybe goes back to the comment about frontier intelligence versus day to day tasks. I mean, I don't know what frontier in finance necessarily represents the same way in science where it's pretty clear, like you're pursuing a new discovery or something like that. Finance, like the frontier is being accurate. And [00:35:02] S1: Yeah. I think I think one of the things to really note for finance is that maybe sort of at first glance, you think it's like different from, you know, chip design or material science. But we actually think it is going to be one of the domains that sort of spends very heavily on tokens in 2027. It is actually already one of the main domains that people spend tokens on in 2026, largely because, like, the margin structures for the businesses sort of supports it. Like, they can actually afford to spend a ton of money on tokens. But then also, there are... And this is actually... This is even more true for legal, but there are some sort of, like, PVP components there where if we if we take legal, for example, if the AI sort of could theoretically do a better job, like, winning this lawsuit or, like, redlining some contract, like, actually does have implications, especially if, like, maybe your your, like, opponent in the scenario is not using the superintelligent AI to do these things. So I actually think finance and legal, like, are domains that will be able to absorb a lot of frontier intelligence, whereas maybe things like real estate or, like, government or something might not be.
00:36:21 · Jordan Nanos, Andrew Megalaa
[00:36:21] S0: Makes sense. Just to... If you could expand on that a little bit more, because I think I know the backdrop. You're specifically talking about the reason for focusing on these domains specifically, rather than... Like, the choice to do that, rather than just taking a cross section of all knowledge work in the economy and saying we're gonna have tasks for every single thing that people do across the economy. Right? Yeah, I think the thing that the listeners are probably thinking about right now is just the concept of AI driven job loss. If people are using Frontier Intelligence and they're gaining these productivity improvements, or they're making more money, or they're winning more things, maybe you can talk about how does running through a benchmark like this inform your point of view on whether AI is going to yeah, cause job loss in the next year or two years, even five year time horizon? [00:37:22] S2: I actually think
00:37:25 · Max Kan
[00:37:25] S1: they're like orthogonal in some sense because we don't... We especially choose not to cover a lot of sort of the white collar knowledge work domains that people are probably most afraid of drop loss occurring, like, I don't know, customer service or something like that. In terms of whether or not we will have, like, massive job loss from AI, it's really hard to say. I think a lot of the doomers have probably been very pleasantly surprised by sort of how AI plus human, like, coworking has evolved over the past year where it's like every... Any any software engineer that sort of, like, has their eyes on the ball will probably tell you that Opus, when given sort of like a well specified problem, is just like a better software engineer than me, period. But at the same time, like, good software engineers are more valuable today than they've ever been. Now is it possible we're sort of in this, like, Goldilocks zone where the humans can... Like, a lot of humans can still provide kind of that, like, taste and supervision and judgment that makes it so that, like, them plus the model is, like, way, way more effective than just the model. But then, like, maybe two years from now, you actually only need, like, one dude at each company who's, like, supervising all the software engineering and, like, basically, everyone else got fired. Maybe. I don't know. I think it's possible that's the world we end up in. But it's also possible it's not. I think we just have to... You really gotta pay close attention to the model capabilities to see. I would
00:39:05 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:39:05] S2: say it's definitely raised the bar for the kind of person you're looking for to like work like on stock. Like when you're when you're looking at like software engineering, for example, [00:39:17] S1: like I feel bad for the interns. [00:39:19] S2: Yes. Essentially. Like, you have to be really good to like provide value higher than just pure like Opus at this point as a software engineer. [00:39:26] S1: Yeah. Because I think a lot of people viewed interns as like, oh, I'm gonna... These are the people who I give my well specified, well defined problems I don't wanna do and they're gonna go do them for me. But obviously, now you just tell Claude to do that. And then sort of all the additional overhead of managing an intern oftentimes becomes sort of just like unbearable in those cases. [00:39:47] S2: I know one of the big quant trading firms had a post around this. It was like... Yeah. Really? Yeah. They they said their interns, like, finish their projects in the first few weeks this year compared to previous years. [00:39:59] S1: Hey, least interns, how do you say Ireland? That's good. Yeah. And [00:40:03] S0: I think the typical response from people is that you need to get more ambitious. It makes sense. I mean, going back to cybersecurity, where I've been doing some work recently. We saw this recently just with the increase in CVEs. I posted this on October 1. It's been a month. I don't have an update. There's a lot more than just this at this point.
00:40:03 · Jordan Nanos, Max Kan, Andrew Megalaa
[00:40:03] S0: I think the typical response from people is that you need to get more ambitious. It makes sense. I mean, going back to cybersecurity, where I've been doing some work recently. We saw this recently just with the increase in CVEs. I posted this on October 1. It's been a month. I don't have an update. There's a lot more than just this at this point. [00:40:26] S1: Yeah. Dude, is is the whole Internet just like broken now? Dude, what's what's gonna happen? What are you gonna do, Jordan? [00:40:33] S0: Well, I think it's a... The... Yeah. I can [00:40:36] S2: I mean? [00:40:36] S0: Do that in a lot more detail. Let's save that for another time, but let's use it as a way to make the point that there's, I think there's a lot more demand for cyber security experience right now, even when using the models. [00:40:51] S2: Yeah, for sure. [00:40:52] S0: And I think that's one of the first domains as a product of software engineering that's getting knocked down. But if you're pushing the frontiers of material science, for example, you're probably not automating away the jobs of the material sciences. You're probably starting to discover a whole bunch of new materials that need to go through the whole discovery phase and a [00:41:12] S1: lot Well, no, think it is very possible that you have automated the job of your PhD candidate assistant if you're a cracked material scientist. [00:41:20] S2: I think this is actually exactly what we've heard from [00:41:23] S1: our partners.