Latent Space: The AI Engineer Podcast
00:08:32 · Jacob Lauritzen, Alessio Fanelli, Shawn Wang
[00:08:32] S0: That would have been... Yeah. Sonnet 4.6, Opus 4.6 is probably right. [00:08:37] S2: What were the agents, like, not good at at that point? That they're much better at now, even though it's only a little, nine months later? [00:08:44] S0: Sub agent delegation was one of them that we've gotten working really well now for fan out type retrieval queries. Works extremely well for us now. That was difficult before. We basically track... So we have a team that's just doing evals, and so we track all of the model's performance in different practice areas, which is subgroups of legal and different tasks across all of those. We basically have like a big heat map, and we're like, and then we can see them move, and it's like, usually they improve most stuff, but there's actually a lot of model releases where they improve some and regress in others. [00:09:15] S1: Okay. [00:09:16] S0: And that actually happens a lot more than you'd think. [00:09:18] S2: Like, what are some of those tasks? Like, are you familiar with how you guys are running a lot of the evals? Like, what are those different sub segments, I guess, that you look at?
00:09:18 · Alessio Fanelli, Jacob Lauritzen
[00:09:18] S2: Like, what are some of those tasks? Like, are you familiar with how you guys are running a lot of the evals? Like, what are those different sub segments, I guess, that you look at? [00:09:25] S0: Yeah. So, like, there's there's different just, like, framing stuff. So, we have different, basically, like, worlds, like, RL environments, essentially, which is, like, here's a... Because because lawyers work in projects always, more or less, so it's, here's 10,000 file data room, you're doing your due diligence, in the end, a GRIT report looks like this. Like, it contains all these red flags, and, like, it doesn't contain whatever weird stuff, and, like, we want it to be concise, etcetera, etcetera. That's kind of how we've built it. So there's different tasks, which is, part of it is drafting a report. Like, does the draft look good? Does it contain all the right stuff? Is the wording sound the analysis sound? There's also the part of finding the red flags, like the analysis before that, so we can kinda, like, subdivide it a little bit. And that... And that's how we run the evals. [00:10:07] S2: Interesting. Wait. What does your evals team look like? Like, do you hire lawyers to, like, do that?
00:10:07 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:10:07] S2: Interesting. Wait. What does your evals team look like? Like, do you hire lawyers to, like, do that? [00:10:12] S0: So, yes, it's a mix. We have engineers, and then we have lawyers, and there's one team. So there's, like, three engineers and, like, three lawyers right now, and then we have a pool of what we call legal engineers, which is people that have worked in big law for, like, three or four years. Typically maybe they have a CS degree, and like they're very tech forward, and then they come to us because they wanna like help change the industry. [00:10:34] S1: Are they IT in big law? No. They're lawyers in big law. [00:10:38] S0: They're lawyers in big law. [00:10:39] S1: With CS degrees. Yes. My god. Must be [00:10:41] S0: really hard It's a Venn diagram that, yeah, exactly. That's like our hottest person to hire for, but they're also such a force multiplier, because we can pull on them for the eval stuff and for the product development, which is like you have a pool of a 100 experts that that use the product every day and go and talk to clients every day, and also, like, they live this and have lived this, and we just pull on them to do EVAS, and just, like, help us shape the product, and it's such a such an advantage. [00:11:03] S2: And curious, like, when did you guys start that team?
00:11:03 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:11:03] S2: And curious, like, when did you guys start that team? [00:11:05] S0: EVAS grew out of the assistant agent team at some point. I think there was a point up until recently where it was like, honestly, it's just not really worth it, because, like, all models just got better. So, it was like, do we wanna spend a lot of our time doing really complex EVALs, or do we just, like, wanna build... Because there was so much to be gained from building a better harness and just sort of leaning into everything's gonna improve, and everything kinda just improved. [00:11:30] S1: Yeah, like every two, three months you [00:11:32] S0: could Yeah, just like, a [00:11:33] S1: exactly. Just pass it through. [00:11:34] S0: Exactly. I mean, we had one problem where it wouldn't cite properly, and so it would be difficult to follow, and it was like, next model release solved, it went from 70% to 99. It's just like, we did nothing, and great. [00:11:44] S1: Yeah. Maybe the real work is keeping all the data private, making it taggable, sortable, add... Adding new features like the the tabular view. Is that the... You know, may may You [00:11:55] S0: mean over and above ChatGPT? [00:11:57] S1: Yes. But, obviously, now you're also doing evals. I assume your your use of open models is also increasing. We can talk about that. So, yeah, I don't know what I'm saying here apart from, like, you know, what is the job? The job is Yep. Maybe not... In the early days, it was not so much to do do evals because it was mostly exposed model capability to lawyers.
00:12:18 · Jacob Lauritzen, Alessio Fanelli
[00:12:18] S0: Yep. The huge part of it is just figuring out what does the legal profession look like in five years, ten years, with all of this stuff. Like, coding is maybe a year or two ahead, so we can kinda look to that a little but, like, what are the primitives? How do people work? How do you collaborate if it's not a 200 thread email with, like, v five word files back and forth? Like, what does that look like? And that's, like, not even necessarily purely AI. That's just, like, build a much better product experience Yeah. And make sure you do it with, like, audit logs and data retention and all of those sort of really important things in regulated industry. And then there's the the AI part, which is, like, make the work faster and better and proactive agents that react to stuff when it comes in so you don't have to, you know, kick off everything. So I think a lot of the work is just figuring out what do we think it's gonna look like in five years. If, you know, model capabilities keep increasing, if price keeps coming down, what could we do? Can you log in and it's like command center where it's just like, you need to react to these five things that the agents have been doing lately and, like, review and check and improve the system? Maybe that's where we're going, or it is where we're going. [00:13:21] S2: Yeah. Yeah. Yeah. I was gonna ask, like, could you go into that in a little more depth? Like, yeah, what what is the, like, law profession gonna look like in a few years?
00:13:21 · Alessio Fanelli, Jacob Lauritzen
[00:13:21] S2: Yeah. Yeah. Yeah. I was gonna ask, like, could you go into that in a little more depth? Like, yeah, what what is the, like, law profession gonna look like in a few years? [00:13:30] S0: I should preface this by saying I don't know. I mean, who knows, really? But I think we can look to coding a lot. I think all lawyers are gonna be way more productive than they were before. They're gonna be able to do a lot more stuff at a higher quality for lower cost. I think the profession itself, the work itself, is gonna improve a lot. It's gonna be a lot more... I think lawyers are gonna spend more of their time doing the stuff that they became lawyers to do, you know, figuring out what's the right risk profile, what's manageable risk, what's like all of their judgment, strategic judgment, and, like, working with clients, they're gonna do that a lot more and much less of the, like, trolling through big documents to, like, find the smoking gun or, like, do all this menial, super annoying work that a lot of lawyers have been doing. [00:14:18] S2: What do you think is gonna happen in the partnership model? Like, do you think that stays or like what happens?