Latent Space: The AI Engineer Podcast
00:54:00 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:54:00] S1: There's still a lot of, let's say, search and retrieval research or rag research to do that is maybe less glamorous. It's not actually even model training sometimes, it's just like architecture and pipelines. You you you did a case study with Turbopuffer, but, like, you know, there's also, like, do you do late interaction of [00:54:19] S0: Yes. Absolutely. You you do that. Yeah. We actually don't do that, but it's... You're completely right that this is... So there's one guy on the team, son, who's... He just really cares about search, and he basically comes to me all the time, and he's like, oh, we should do these things, like, there's these new embedding models, and like, we can do this late interaction. He actually talked to Turbopuffer teams to get them to build multivector so that we could do these types of things. But no, there's loads of low hanging fruit generally, even before models. And even in the harness, honestly, like, you just wanna create a better output for our clients, there's like so much stuff that we can do that has immediate impact before training models. [00:54:51] S2: Like, what makes a good harness for, like, the law industry, I guess? Like, you guys are making, I'm assuming, like, your own custom harness. [00:54:57] S0: Yep. Yeah. [00:54:57] S2: Of course.
00:54:57 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:54:57] S2: Of course. [00:54:58] S0: Yep. Way that we edit files is important. The way that we do an analysis is important. The way you do legal research is all baked into it. So if, like, you might have a problem that comes up or sees something, then it knows how to go and do the legal research to figure out what the answer is, what the precedent is. That's all very important. And then, of course, there's all the governance stuff around it as well. [00:55:17] S2: Yeah. It seems like the search is probably, like, would you say the most important tool that you guys are, like, customizing? Because if you can't pull in the right information Yeah. And then then you're [00:55:26] S0: screwed. [00:55:26] S2: You're screwed. [00:55:27] S0: It's... Yeah. I think it's searching like a lot of documents. And analysis. Yeah. Yeah. It's search, it's analysis, and it's drafting, probably. Okay. Because like drafting word, drafting huge contracts is actually also not super trivial. [00:55:39] S2: Like, what about it is not trivial? [00:55:40] S0: Because you have to... Let's say you're drafting a huge contract from scratch, then you have to find precedents of like, how did we do this previously? Do we need this type of clause [00:55:49] S2: A or specific law firm? [00:55:50] S1: How did they do it Yeah.
00:55:50 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:55:50] S1: How did they do it Yeah. [00:55:51] S0: So I am the law firm, and I'm about to draft a huge... Whatever. NDA is very simple, but, like, there are contracts that are way more complex, and then you need to find the right precedent first, because lawyers just don't don't sit and start typing stuff. They figure out what clauses have we used in similar deals, and similar can mean a lot of different things, as I'm sure you know in search, it could be like same industry, same type of companies, same whatever. And so you need to identify the right person, and you might have like multiple documents that you then try to figure out which part should I take from what document, and that's actually nontrivial. So it's essentially also a [00:56:24] S1: search problem. Which model's best at drafting? Yeah. So [00:56:28] S0: I'll say... I was told that was farfetched this yesterday. Definitely Fable and Ophas are extremely chatty. I'm sure you guys have known this. They're very verbose. [00:56:36] S1: So that's not good. [00:56:37] S0: That's generally not good. No. [00:56:38] S2: I noticed Astra's been pretty good at writing. [00:56:40] S0: Astra's, yeah, pretty good at writing. It problem? More concise. Are you running like [00:56:45] S1: a 100 different prompts to [00:56:46] S0: Yeah. Be sure. We can, yeah, can do certain things, but I think the chattiness verbosity is actually a little bit difficult for us to do. Wow. So this is, I think, you know, let's see, but probably the next model from Anthropic is gonna be much better at this, Astra is actually very concise.
00:56:46 · Jacob Lauritzen, Shawn Wang, Alessio Fanelli
[00:56:46] S0: Yeah. Be sure. We can, yeah, can do certain things, but I think the chattiness verbosity is actually a little bit difficult for us to do. Wow. So this is, I think, you know, let's see, but probably the next model from Anthropic is gonna be much better at this, Astra is actually very concise. [00:57:00] S1: Yeah, and I assume they all have a Slack channel with you and you're telling them this, right? Is not news to them. [00:57:06] S0: No. No. They know this. [00:57:07] S1: Then the other thing that I really care about is hallucination. Do they make stuff up? [00:57:11] S0: So, we have a system of... A citation system where we basically check what they're writing has a link, so the user has a link, they can click, and it highlights with a bounding box, like in the PDF or whatever, where where this information was taken from, and we can sort of make sure that what it says always is citable somewhere, even multiple layers down. So, like, if the agent writes something based on a tabular review, you know, it can cite the cell in the tabular review or the row in the tabular review, so it can actually, like, flake multiple layers down and find, you know, sort of validate it all the way up. [00:57:41] S2: Yeah. Are you finding that lawyers are verifying that these citations are [00:57:45] S0: correct Initially, or kinda like of them work, but now... Yeah. Exactly. [00:57:49] S1: It's the [00:57:49] S2: same thing. I guess [00:57:50] S1: the same People shit like hallucinated links to Supreme Court. They don't care.
00:57:50 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:57:50] S1: the same People shit like hallucinated links to Supreme Court. They don't care. [00:57:54] S0: That happens. Yeah. Exactly. That happens. Yeah. I think it's like coding, you know, you have engineers that just don't give a shit and just like send agents, and you have people that like sit and read every comma, same description. [00:58:03] S2: There's just like so much that like, how can one human do that? Right? [00:58:06] S0: Yeah. But I think they probably, it's just like we know in code review what to look for, like what is the system or whatever, I think they also know, okay, this is the thing, these are the red, like, this is the thing that's important, let me just, like, double check these things, and, like, these sound reasonable, so it's probably fine. [00:58:19] S1: Okay. Interesting. [00:58:19] S0: Just like if you receive something from a junior lawyer, it's the same thing you do, right? Just like, I should double check this, and, like, this probably is fine. [00:58:25] S1: Any other elements of, let's call it, legal reasoning, or what's the the the legal frontier that models are not good enough yet that you want from the next generation of models?
00:58:25 · Shawn Wang, Jacob Lauritzen
[00:58:25] S1: Any other elements of, let's call it, legal reasoning, or what's the the the legal frontier that models are not good enough yet that you want from the next generation of models? [00:58:35] S0: Nothing I can think of specifically right now. I think long document reasoning is still difficult. Long, like, million token token contracts is what Exactly. You have a huge contract. Yeah. And, like, you're making edits across it. It's a difficult thing to do for a model today even. And, like, remember that you changed all these things up here, and, like, the definitions changed, and that you... The ordering changed, whatever else. [00:58:55] S1: So the simplest would be one shot, you put... Feed everything in and then one shot output. Yes. You don't do that, you break it up into chunks and then you orchestrate edits. Yeah. Exactly. Exactly. [00:59:05] S0: And this is like one of those things that are funny, like, that are fun to be part of in this industry, right? So, word ad, and it was like, we've done, you know, we've done MapReduce, and we've changed, and there was one big one, and there was like one big one in multiple times, it's like, we keep changing the scaffolding that we build around as models change. [00:59:20] S1: Yes. I I would say I I've recently... I do a lot of news reporting, obviously, because of what I do, and I've recently shifted from MapReduce to throwing the time into context. Because it's Yeah. Exactly.