SemiAnalysis Weekly
00:28:52 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:28:52] S2: And it was organized or disorganized exactly how they would Yeah. [00:28:55] S1: It was like it was like the real the real, like, files for this real I [00:29:00] S0: was looking at this. And anybody who has worked in a VDR, like a a data room, for any m and a transaction has seen exactly this. And [00:29:11] S1: So they had they had the real files. And then, obviously, they sort of, like, deanonymized it. But along with the files, it actually has sort of all of the real tasks that, you know, analysts and associates and other people, like, working on this deal had to do. And they essentially converted this, like, you know, actual deal room into an environment they can evaluate the models in. And then they, like, found the emails that, you know, some accounting partner told their associate to do as part of this task, and then they had the model do it. I think this is this is obviously sort of just like a categorically different in terms of realism and interestingness and quality versus like just taking some random, you know, banker off the street and being like, come up with a task
00:29:11 · Max Kan, Jordan Nanos
[00:29:11] S1: So they had they had the real files. And then, obviously, they sort of, like, deanonymized it. But along with the files, it actually has sort of all of the real tasks that, you know, analysts and associates and other people, like, working on this deal had to do. And they essentially converted this, like, you know, actual deal room into an environment they can evaluate the models in. And then they, like, found the emails that, you know, some accounting partner told their associate to do as part of this task, and then they had the model do it. I think this is this is obviously sort of just like a categorically different in terms of realism and interestingness and quality versus like just taking some random, you know, banker off the street and being like, come up with a task [00:29:59] S0: for Okay. One more thing about this, Max, is the life cycle. I thought this was fascinating, which is they could effectively replay what the data room looked like when that task was signed. Right? So, you're gonna add in these thousand files over time, but not every task is assigned at that same time in the data room. I thought that was like, yeah, obviously super fascinating that this is the depth at which people can can go to to try to make this task realistic. And some of
00:29:59 · Jordan Nanos, Andrew Megalaa, Max Kan
[00:29:59] S0: for Okay. One more thing about this, Max, is the life cycle. I thought this was fascinating, which is they could effectively replay what the data room looked like when that task was signed. Right? So, you're gonna add in these thousand files over time, but not every task is assigned at that same time in the data room. I thought that was like, yeah, obviously super fascinating that this is the depth at which people can can go to to try to make this task realistic. And some of [00:30:28] S2: the failure modes are so esoteric when I looked at them. But then I had to ask my finance friends and they told me, yeah, that you would totally be expected to just have done this. It'd be like some email from somebody who was like, no. Actually, adjust these numbers to, like, whatever q four or something. It's it's just one email where it mentioned this. And if the model didn't find that email, it would fail because, you know, it had it had to find that email. Yeah. To, like, actually evaluate all the context relevant to the task. And the models didn't look hard enough sometimes. And so they just miss it and then have the wrong numbers go out. [00:31:00] S1: And if the human associate failed, would have gotten yelled at. So it's [00:31:02] S2: the Yeah. Apparently, that's actually That's exactly how it would have worked. Like, if you were an analyst and you did not read it all [00:31:08] S1: the same. That one email,
00:31:08 · Max Kan, Andrew Megalaa, Jordan Nanos
[00:31:08] S1: the same. That one email, [00:31:10] S2: you if were not on top of it, would have gotten yelled at for sure. You would have said, yeah, this is unacceptable behavior in a real banking environment. [00:31:17] S0: I know some people on the tokenomics team who I think are possibly gonna be listening to this podcast having some flashbacks, [00:31:23] S2: PTS out of the [00:31:25] S0: yard stares. You talk about the grading in this case? Because I think a lot of people would be confused about the partial credit versus binary grading approach you have to apply to some of these things that don't necessarily have a binary outcome.
00:31:40 · Max Kan
[00:31:40] S1: Yeah. Yeah. So in the case... So I'll say our finance tasks use binary grading. So the model either gets a zero or a one. But there's sort of a continuous sort of scoring component there because might have a task where the final deliverable is, like, some Excel workbook. Maybe it's like a quality of earnings book that, like, the accountant had to make for this private equity deal. And, obviously, like, that workbook can kinda be decomposed into a bunch of different parts, you know, different cells, and the model has to find the right number for. And you kind of assign partial grading for each of these atomic criteria. But then we still use kind of binary grading at the end because if you were a real accounting associate and then you gave your boss a workbook where, like, half the numbers are right and half of them are wrong, they're not gonna be like, oh, man. Like, great job, little Jimmy. Like, you got half of it right. Like, I'll take them here. They're gonna be like, dude, what the fuck, man? Like, I'm firing your ass right now. So I think in those cases, like, it makes sense to do binary grading even though there is, like, a partial score component. Yeah, maybe, Jordan, throwing a show here that, like, we have 0% when you use binary in the finance, basically, except for Carbon who kinda got, like, one task right. But when you switch to partial, and then you sort of get much more words to go on how the models actually did. That being said... Okay. Actually, I'll say, one, a lot of people on Twitter were kind of shitting on us using binary grading because there are other benchmarks that kind of use that to hide the fact that their tasks are actually quite easy. I won't name any particular names here, but there are some popular benchmarks where you look at binary grading and it's like, oh, 25% on Opus. Like, that is probably pretty hard. But then you switch to partial grading, it's like 97% Opus. And it's like, okay. Like that that last 33%
00:33:36 · Andrew Megalaa, Max Kan, Jordan Nanos
[00:33:36] S2: is probably just, like, some pathetic requirement that there's no reason you shouldn't have expected them. [00:33:40] S1: Yeah. Or your task is just, like, fucking unfair and broken in some way. And, yeah, so, like, I agree. If you look at our finance task and it was, like, 95% on Opus or partial grading, they're probably a bad task. But you saw it was, like, 50 something percent. Same with our legal. The legal was all zeros, but then the best model, even when you search for partial grading, I think is, like, it's in the thirties or forties. At the same time, most of our domains actually use continuous grading because it makes sense for those. You know? You have a chip design task where maybe the model is sort of asked to do some, like, PPA optimization, And then you just have, like, zero as some naive solution, and then you have one as sort of some, like, max theoretical achievable solution. And you kinda just give the model partial credit depending on how much it can optimize, like, your chip design. We sort of picked the grading strategy that, like, makes the most sense given the particular domain. We did not do, you know, binary grading for everything so we could kind of artificially lower the score. [00:34:40] S0: Yeah. And I think this is... Maybe goes back to the comment about frontier intelligence versus day to day tasks. I mean, I don't know what frontier in finance necessarily represents the same way in science where it's pretty clear, like you're pursuing a new discovery or something like that. Finance, like the frontier is being accurate. And