Latent Space: The AI Engineer Podcast
00:06:33 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:06:33] S2: Yes. [00:06:34] S0: And make sure that it's it's great all the time, and you don't have to dive into all the details. Because, like, one of the things that I've... Exactly to this point, if you go to a lawyer and you ask them, oh, you're gonna draft this contract based on this president, do you think that's a sonnet high or an opus mid task? [00:06:48] S1: Actually, they won't know. Just give me this [00:06:50] S0: can part of this model. Just make sure it works. [00:06:52] S1: Our hourly rate is thousand dollars an hour. Exactly. Just make it work. So, yes, I originally came in with that. I was like, oh, yep. This is this is what Devin does differently. And now we have five things in the model picker, including Fable and Opus and and Astra. Yes. And people wanna experiment with the fancy new model. Do you... Is that the same? [00:07:11] S0: I I think so. I think it's prob... It's it's definitely way less in legal than it is in Coding. Engineering. Yeah. But there are definitely people that would just want table one right away, and, you know, people were asking us about mythos and everything [00:07:23] S1: Yeah. Yeah. [00:07:24] S0: Even though I think, realistically, a lot of legal work does not require mythos level intelligence. Yeah. There is [00:07:30] S1: So so exactly. Right? Like, I think basically you're like, okay. We'll give you the option. You you ask for this Exactly. But you're gonna run down your credits, like, 10 x faster than them. Yeah.
00:07:30 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:07:30] S1: So so exactly. Right? Like, I think basically you're like, okay. We'll give you the option. You you ask for this Exactly. But you're gonna run down your credits, like, 10 x faster than them. Yeah. [00:07:40] S0: I would have loved... We we fought introducing it for a very long time, and then in the end, we we just kinda caved in and we're like, okay. If you want this level of control Yes. You know, then we'll give it to you. [00:07:48] S2: Yeah. So you think the models are already, like, as good as lawyers need? Like, there's no need for them to get better? [00:07:53] S0: It's a... I think it's a power distribution. So, like, there's probably, 60% of the work Yeah. We can do really well right now, and the... No. No. From now on, it's like a cost and latency thing. But then there's the rest of the the thing where, especially for, like, very complex analysis, we we still need more intelligence. And then I think there's also just the work expanse kind of. It's like that's the work that we have right now, but also the work with Fort Lauderdale, thus, probably expands with intelligence [00:08:20] S1: Yeah. [00:08:20] S0: I imagine. [00:08:21] S2: Early on, like, when you guys first started, I guess, transitioning to the agents Mhmm. Or building the agents, like, what were... Like, what models were you guys using? I guess it would have been like Opus four six or something? [00:08:32] S0: That would have been... Yeah. Sonnet 4.6, Opus 4.6 is probably right.
00:08:32 · Jacob Lauritzen, Alessio Fanelli, Shawn Wang
[00:08:32] S0: That would have been... Yeah. Sonnet 4.6, Opus 4.6 is probably right. [00:08:37] S2: What were the agents, like, not good at at that point? That they're much better at now, even though it's only a little, nine months later? [00:08:44] S0: Sub agent delegation was one of them that we've gotten working really well now for fan out type retrieval queries. Works extremely well for us now. That was difficult before. We basically track... So we have a team that's just doing evals, and so we track all of the model's performance in different practice areas, which is subgroups of legal and different tasks across all of those. We basically have like a big heat map, and we're like, and then we can see them move, and it's like, usually they improve most stuff, but there's actually a lot of model releases where they improve some and regress in others. [00:09:15] S1: Okay. [00:09:16] S0: And that actually happens a lot more than you'd think. [00:09:18] S2: Like, what are some of those tasks? Like, are you familiar with how you guys are running a lot of the evals? Like, what are those different sub segments, I guess, that you look at?
00:09:18 · Alessio Fanelli, Jacob Lauritzen
[00:09:18] S2: Like, what are some of those tasks? Like, are you familiar with how you guys are running a lot of the evals? Like, what are those different sub segments, I guess, that you look at? [00:09:25] S0: Yeah. So, like, there's there's different just, like, framing stuff. So, we have different, basically, like, worlds, like, RL environments, essentially, which is, like, here's a... Because because lawyers work in projects always, more or less, so it's, here's 10,000 file data room, you're doing your due diligence, in the end, a GRIT report looks like this. Like, it contains all these red flags, and, like, it doesn't contain whatever weird stuff, and, like, we want it to be concise, etcetera, etcetera. That's kind of how we've built it. So there's different tasks, which is, part of it is drafting a report. Like, does the draft look good? Does it contain all the right stuff? Is the wording sound the analysis sound? There's also the part of finding the red flags, like the analysis before that, so we can kinda, like, subdivide it a little bit. And that... And that's how we run the evals. [00:10:07] S2: Interesting. Wait. What does your evals team look like? Like, do you hire lawyers to, like, do that?
00:10:07 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:10:07] S2: Interesting. Wait. What does your evals team look like? Like, do you hire lawyers to, like, do that? [00:10:12] S0: So, yes, it's a mix. We have engineers, and then we have lawyers, and there's one team. So there's, like, three engineers and, like, three lawyers right now, and then we have a pool of what we call legal engineers, which is people that have worked in big law for, like, three or four years. Typically maybe they have a CS degree, and like they're very tech forward, and then they come to us because they wanna like help change the industry. [00:10:34] S1: Are they IT in big law? No. They're lawyers in big law. [00:10:38] S0: They're lawyers in big law. [00:10:39] S1: With CS degrees. Yes. My god. Must be [00:10:41] S0: really hard It's a Venn diagram that, yeah, exactly. That's like our hottest person to hire for, but they're also such a force multiplier, because we can pull on them for the eval stuff and for the product development, which is like you have a pool of a 100 experts that that use the product every day and go and talk to clients every day, and also, like, they live this and have lived this, and we just pull on them to do EVAS, and just, like, help us shape the product, and it's such a such an advantage. [00:11:03] S2: And curious, like, when did you guys start that team?
00:11:03 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:11:03] S2: And curious, like, when did you guys start that team? [00:11:05] S0: EVAS grew out of the assistant agent team at some point. I think there was a point up until recently where it was like, honestly, it's just not really worth it, because, like, all models just got better. So, it was like, do we wanna spend a lot of our time doing really complex EVALs, or do we just, like, wanna build... Because there was so much to be gained from building a better harness and just sort of leaning into everything's gonna improve, and everything kinda just improved. [00:11:30] S1: Yeah, like every two, three months you [00:11:32] S0: could Yeah, just like, a [00:11:33] S1: exactly. Just pass it through. [00:11:34] S0: Exactly. I mean, we had one problem where it wouldn't cite properly, and so it would be difficult to follow, and it was like, next model release solved, it went from 70% to 99. It's just like, we did nothing, and great. [00:11:44] S1: Yeah. Maybe the real work is keeping all the data private, making it taggable, sortable, add... Adding new features like the the tabular view. Is that the... You know, may may You [00:11:55] S0: mean over and above ChatGPT? [00:11:57] S1: Yes. But, obviously, now you're also doing evals. I assume your your use of open models is also increasing. We can talk about that. So, yeah, I don't know what I'm saying here apart from, like, you know, what is the job? The job is Yep. Maybe not... In the early days, it was not so much to do do evals because it was mostly exposed model capability to lawyers.