Latent Space: The AI Engineer Podcast
00:55:50 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:55:50] S1: How did they do it Yeah. [00:55:51] S0: So I am the law firm, and I'm about to draft a huge... Whatever. NDA is very simple, but, like, there are contracts that are way more complex, and then you need to find the right precedent first, because lawyers just don't don't sit and start typing stuff. They figure out what clauses have we used in similar deals, and similar can mean a lot of different things, as I'm sure you know in search, it could be like same industry, same type of companies, same whatever. And so you need to identify the right person, and you might have like multiple documents that you then try to figure out which part should I take from what document, and that's actually nontrivial. So it's essentially also a [00:56:24] S1: search problem. Which model's best at drafting? Yeah. So [00:56:28] S0: I'll say... I was told that was farfetched this yesterday. Definitely Fable and Ophas are extremely chatty. I'm sure you guys have known this. They're very verbose. [00:56:36] S1: So that's not good. [00:56:37] S0: That's generally not good. No. [00:56:38] S2: I noticed Astra's been pretty good at writing. [00:56:40] S0: Astra's, yeah, pretty good at writing. It problem? More concise. Are you running like [00:56:45] S1: a 100 different prompts to [00:56:46] S0: Yeah. Be sure. We can, yeah, can do certain things, but I think the chattiness verbosity is actually a little bit difficult for us to do. Wow. So this is, I think, you know, let's see, but probably the next model from Anthropic is gonna be much better at this, Astra is actually very concise.
00:56:46 · Jacob Lauritzen, Shawn Wang, Alessio Fanelli
[00:56:46] S0: Yeah. Be sure. We can, yeah, can do certain things, but I think the chattiness verbosity is actually a little bit difficult for us to do. Wow. So this is, I think, you know, let's see, but probably the next model from Anthropic is gonna be much better at this, Astra is actually very concise. [00:57:00] S1: Yeah, and I assume they all have a Slack channel with you and you're telling them this, right? Is not news to them. [00:57:06] S0: No. No. They know this. [00:57:07] S1: Then the other thing that I really care about is hallucination. Do they make stuff up? [00:57:11] S0: So, we have a system of... A citation system where we basically check what they're writing has a link, so the user has a link, they can click, and it highlights with a bounding box, like in the PDF or whatever, where where this information was taken from, and we can sort of make sure that what it says always is citable somewhere, even multiple layers down. So, like, if the agent writes something based on a tabular review, you know, it can cite the cell in the tabular review or the row in the tabular review, so it can actually, like, flake multiple layers down and find, you know, sort of validate it all the way up. [00:57:41] S2: Yeah. Are you finding that lawyers are verifying that these citations are [00:57:45] S0: correct Initially, or kinda like of them work, but now... Yeah. Exactly. [00:57:49] S1: It's the [00:57:49] S2: same thing. I guess [00:57:50] S1: the same People shit like hallucinated links to Supreme Court. They don't care.
00:57:50 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:57:50] S1: the same People shit like hallucinated links to Supreme Court. They don't care. [00:57:54] S0: That happens. Yeah. Exactly. That happens. Yeah. I think it's like coding, you know, you have engineers that just don't give a shit and just like send agents, and you have people that like sit and read every comma, same description. [00:58:03] S2: There's just like so much that like, how can one human do that? Right? [00:58:06] S0: Yeah. But I think they probably, it's just like we know in code review what to look for, like what is the system or whatever, I think they also know, okay, this is the thing, these are the red, like, this is the thing that's important, let me just, like, double check these things, and, like, these sound reasonable, so it's probably fine. [00:58:19] S1: Okay. Interesting. [00:58:19] S0: Just like if you receive something from a junior lawyer, it's the same thing you do, right? Just like, I should double check this, and, like, this probably is fine. [00:58:25] S1: Any other elements of, let's call it, legal reasoning, or what's the the the legal frontier that models are not good enough yet that you want from the next generation of models?
00:58:25 · Shawn Wang, Jacob Lauritzen
[00:58:25] S1: Any other elements of, let's call it, legal reasoning, or what's the the the legal frontier that models are not good enough yet that you want from the next generation of models? [00:58:35] S0: Nothing I can think of specifically right now. I think long document reasoning is still difficult. Long, like, million token token contracts is what Exactly. You have a huge contract. Yeah. And, like, you're making edits across it. It's a difficult thing to do for a model today even. And, like, remember that you changed all these things up here, and, like, the definitions changed, and that you... The ordering changed, whatever else. [00:58:55] S1: So the simplest would be one shot, you put... Feed everything in and then one shot output. Yes. You don't do that, you break it up into chunks and then you orchestrate edits. Yeah. Exactly. Exactly. [00:59:05] S0: And this is like one of those things that are funny, like, that are fun to be part of in this industry, right? So, word ad, and it was like, we've done, you know, we've done MapReduce, and we've changed, and there was one big one, and there was like one big one in multiple times, it's like, we keep changing the scaffolding that we build around as models change. [00:59:20] S1: Yes. I I would say I I've recently... I do a lot of news reporting, obviously, because of what I do, and I've recently shifted from MapReduce to throwing the time into context. Because it's Yeah. Exactly.
00:59:20 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:59:20] S1: Yes. I I would say I I've recently... I do a lot of news reporting, obviously, because of what I do, and I've recently shifted from MapReduce to throwing the time into context. Because it's Yeah. Exactly. [00:59:33] S0: But it's also... It's different. I mean, we had this problem with search where it's like, you're searching for certain things in a huge contract, and you can map reduce it, but then it might, if you're summarizing any sort of thing, then it might lose some important nuance in between the chunks, essentially, if you just summarize the individual parts up. So, this has been a problem where it's like, we can do two types of search, we can also do the other one where it's like, you need to have reasoning in the different steps there in between. So, I'm trying to think of a really good example here. Yeah, there's some documents where they talk about something at length over, like, let's say two pages or whatever, and then each individual chunk is not necessarily relevant to the search term or what they're trying to figure out, but together it's extremely relevant. And so, yeah, these are some of the, you know, problems where it's [01:00:15] S2: And it must be hard, like you said earlier, to actually make the models better at that, because you can't even look at the contracts, right? Exactly. Do you guys ever ask them for like, hey, can you make example contracts that are similar to Yeah. The real
01:00:15 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[01:00:15] S2: And it must be hard, like you said earlier, to actually make the models better at that, because you can't even look at the contracts, right? Exactly. Do you guys ever ask them for like, hey, can you make example contracts that are similar to Yeah. The real [01:00:30] S0: Yeah. So we feedback, and can receive feedback from them, and if they're willing engage, yeah, exactly. They're just like, hey, this is like super important to us and it's not working well, and then we're like, okay, great. Can you like anonymize the documents and like the problem, whatever, and we'll... If we can recreate it, then we can fix it. [01:00:44] S1: Yeah. I mean, so one version of this is you buy a law firm Yep. Which some people are considering. [01:00:50] S0: Yes. But there's some stuff there on whether their data is actually their data or if it's the client's data. So, if their clients are all gone, then maybe it's fine. I actually don't know. Interesting. Yeah. But it's a... Yes. It's it's a way to get a lot of data. [01:01:05] S1: Yeah. People... Like, Google's buying Spiritualized. Exactly. Right? [01:01:09] S0: Yeah. They bought Yeah. They bought the... Yeah. Yeah. Yeah. [01:01:13] S1: Like, you know, if you have a failed company, you still have your Slack. You can sell the Slack. [01:01:17] S0: Think there's like I [01:01:18] S1: a YC company that buys your failed company. [01:01:20] S0: Yes, there is. [01:01:21] S1: Yeah. Because it's just like, oh, I just want real Exactly. People doing And then, yeah, just was Exactly. Pull stuff out.