Latent Space: The AI Engineer Podcast
00:04:15 · Jacob Lauritzen, Shawn Wang, Alessio Fanelli
[00:04:15] S0: So, I mean, we've all... We basically had this switch from the assistant to the agent Yeah. We call it. Right? I guess that the agent... I don't know. I wanna say a year ago, probably. [00:04:25] S1: Yeah. So this is a rough timeline for people who are understanding YC twenty twenty three. Yes. I know you are you are YC founder as four. YC twenty '4. [00:04:34] S0: Yeah. I was in YC '23. Yeah. Legora was '24. Okay. [00:04:37] S1: And then... So so there is... This is like a little bit pre a... Pre, like, agents really taking up. [00:04:41] S0: Exactly. [00:04:42] S1: And then, like, using models obviously changed everything. Yes. I mean, you know, in one in one sense today, you know, we had a preview of the products. It does look... Well, preview. We had a private thing. Well, we will put up It's a video on mostly like ChatGPT for lawyers. Right? Like, like, very fair to say that. But then, like, the sidebar has a lot of nice custom tools like the tabular view, like the intake, which I really liked, and a bunch of other things. Exactly. [00:05:08] S0: That's exactly what it is. It's coworker or chat GPT, just like really, really good for legal work. [00:05:13] S1: And the for either in house corporate lawyers, people looking for accounting regulations, all these things. And we can talk about more of the the products there. [00:05:23] S2: Yeah. What models are you guys using underneath? Like, can you talk about that? [00:05:27] S0: I mean, we're using bunch of different ones depending on tasks. Right? That's Okay.
00:05:27 · Jacob Lauritzen, Alessio Fanelli, Shawn Wang
[00:05:27] S0: I mean, we're using bunch of different ones depending on tasks. Right? That's Okay. [00:05:31] S2: And then that's not available to the lawyer to see? You guys can't the agent? [00:05:36] S0: Yeah. Well, we've been back and forth on this. Yeah. Just like everyone else, I think. In the agent, there's a model selector for the primary model, but there are certain tools that we use different models in, and there's, like, tabular view, and, like, we we try... I think we found it most effective to subdivide the problem and and and route on, like, as we subdivide the task, but we do right now have a a picker in the agent. Mhmm. I'm not sure if that's gonna stay there always, to be honest. So... Yeah. [00:06:03] S1: Yeah. I mean, so I bring a little bit of context to this because I'm an adviser for Cognition. Yep. And when I joined... You know, part of my my thesis for what an agent lab is... An agent lab is, you know, ideally a domain specific GPT wrapper. Yep. Right? Like, we we we run the... We are the domain experts. We will pay attention to you. We'll we'll serve your needs way, way more than the Frontier Labs, and we'll give you the best of the Frontier [00:06:26] S0: Labs We'll have to track your development. [00:06:28] S1: Yeah. So our job is to decide for you because you're not... Your job is not to Exactly. [00:06:32] S0: We'll always keep you on the Pareto frontier [00:06:33] S2: Yes.
00:06:33 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:06:33] S2: Yes. [00:06:34] S0: And make sure that it's it's great all the time, and you don't have to dive into all the details. Because, like, one of the things that I've... Exactly to this point, if you go to a lawyer and you ask them, oh, you're gonna draft this contract based on this president, do you think that's a sonnet high or an opus mid task? [00:06:48] S1: Actually, they won't know. Just give me this [00:06:50] S0: can part of this model. Just make sure it works. [00:06:52] S1: Our hourly rate is thousand dollars an hour. Exactly. Just make it work. So, yes, I originally came in with that. I was like, oh, yep. This is this is what Devin does differently. And now we have five things in the model picker, including Fable and Opus and and Astra. Yes. And people wanna experiment with the fancy new model. Do you... Is that the same? [00:07:11] S0: I I think so. I think it's prob... It's it's definitely way less in legal than it is in Coding. Engineering. Yeah. But there are definitely people that would just want table one right away, and, you know, people were asking us about mythos and everything [00:07:23] S1: Yeah. Yeah. [00:07:24] S0: Even though I think, realistically, a lot of legal work does not require mythos level intelligence. Yeah. There is [00:07:30] S1: So so exactly. Right? Like, I think basically you're like, okay. We'll give you the option. You you ask for this Exactly. But you're gonna run down your credits, like, 10 x faster than them. Yeah.
00:07:30 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:07:30] S1: So so exactly. Right? Like, I think basically you're like, okay. We'll give you the option. You you ask for this Exactly. But you're gonna run down your credits, like, 10 x faster than them. Yeah. [00:07:40] S0: I would have loved... We we fought introducing it for a very long time, and then in the end, we we just kinda caved in and we're like, okay. If you want this level of control Yes. You know, then we'll give it to you. [00:07:48] S2: Yeah. So you think the models are already, like, as good as lawyers need? Like, there's no need for them to get better? [00:07:53] S0: It's a... I think it's a power distribution. So, like, there's probably, 60% of the work Yeah. We can do really well right now, and the... No. No. From now on, it's like a cost and latency thing. But then there's the rest of the the thing where, especially for, like, very complex analysis, we we still need more intelligence. And then I think there's also just the work expanse kind of. It's like that's the work that we have right now, but also the work with Fort Lauderdale, thus, probably expands with intelligence [00:08:20] S1: Yeah. [00:08:20] S0: I imagine. [00:08:21] S2: Early on, like, when you guys first started, I guess, transitioning to the agents Mhmm. Or building the agents, like, what were... Like, what models were you guys using? I guess it would have been like Opus four six or something? [00:08:32] S0: That would have been... Yeah. Sonnet 4.6, Opus 4.6 is probably right.
00:08:32 · Jacob Lauritzen, Alessio Fanelli, Shawn Wang
[00:08:32] S0: That would have been... Yeah. Sonnet 4.6, Opus 4.6 is probably right. [00:08:37] S2: What were the agents, like, not good at at that point? That they're much better at now, even though it's only a little, nine months later? [00:08:44] S0: Sub agent delegation was one of them that we've gotten working really well now for fan out type retrieval queries. Works extremely well for us now. That was difficult before. We basically track... So we have a team that's just doing evals, and so we track all of the model's performance in different practice areas, which is subgroups of legal and different tasks across all of those. We basically have like a big heat map, and we're like, and then we can see them move, and it's like, usually they improve most stuff, but there's actually a lot of model releases where they improve some and regress in others. [00:09:15] S1: Okay. [00:09:16] S0: And that actually happens a lot more than you'd think. [00:09:18] S2: Like, what are some of those tasks? Like, are you familiar with how you guys are running a lot of the evals? Like, what are those different sub segments, I guess, that you look at?
00:09:18 · Alessio Fanelli, Jacob Lauritzen
[00:09:18] S2: Like, what are some of those tasks? Like, are you familiar with how you guys are running a lot of the evals? Like, what are those different sub segments, I guess, that you look at? [00:09:25] S0: Yeah. So, like, there's there's different just, like, framing stuff. So, we have different, basically, like, worlds, like, RL environments, essentially, which is, like, here's a... Because because lawyers work in projects always, more or less, so it's, here's 10,000 file data room, you're doing your due diligence, in the end, a GRIT report looks like this. Like, it contains all these red flags, and, like, it doesn't contain whatever weird stuff, and, like, we want it to be concise, etcetera, etcetera. That's kind of how we've built it. So there's different tasks, which is, part of it is drafting a report. Like, does the draft look good? Does it contain all the right stuff? Is the wording sound the analysis sound? There's also the part of finding the red flags, like the analysis before that, so we can kinda, like, subdivide it a little bit. And that... And that's how we run the evals. [00:10:07] S2: Interesting. Wait. What does your evals team look like? Like, do you hire lawyers to, like, do that?