Latent Space: The AI Engineer Podcast
00:56:46 · Jacob Lauritzen, Shawn Wang, Alessio Fanelli
[00:56:46] S0: Yeah. Be sure. We can, yeah, can do certain things, but I think the chattiness verbosity is actually a little bit difficult for us to do. Wow. So this is, I think, you know, let's see, but probably the next model from Anthropic is gonna be much better at this, Astra is actually very concise. [00:57:00] S1: Yeah, and I assume they all have a Slack channel with you and you're telling them this, right? Is not news to them. [00:57:06] S0: No. No. They know this. [00:57:07] S1: Then the other thing that I really care about is hallucination. Do they make stuff up? [00:57:11] S0: So, we have a system of... A citation system where we basically check what they're writing has a link, so the user has a link, they can click, and it highlights with a bounding box, like in the PDF or whatever, where where this information was taken from, and we can sort of make sure that what it says always is citable somewhere, even multiple layers down. So, like, if the agent writes something based on a tabular review, you know, it can cite the cell in the tabular review or the row in the tabular review, so it can actually, like, flake multiple layers down and find, you know, sort of validate it all the way up. [00:57:41] S2: Yeah. Are you finding that lawyers are verifying that these citations are [00:57:45] S0: correct Initially, or kinda like of them work, but now... Yeah. Exactly. [00:57:49] S1: It's the [00:57:49] S2: same thing. I guess [00:57:50] S1: the same People shit like hallucinated links to Supreme Court. They don't care.
00:57:50 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:57:50] S1: the same People shit like hallucinated links to Supreme Court. They don't care. [00:57:54] S0: That happens. Yeah. Exactly. That happens. Yeah. I think it's like coding, you know, you have engineers that just don't give a shit and just like send agents, and you have people that like sit and read every comma, same description. [00:58:03] S2: There's just like so much that like, how can one human do that? Right? [00:58:06] S0: Yeah. But I think they probably, it's just like we know in code review what to look for, like what is the system or whatever, I think they also know, okay, this is the thing, these are the red, like, this is the thing that's important, let me just, like, double check these things, and, like, these sound reasonable, so it's probably fine. [00:58:19] S1: Okay. Interesting. [00:58:19] S0: Just like if you receive something from a junior lawyer, it's the same thing you do, right? Just like, I should double check this, and, like, this probably is fine. [00:58:25] S1: Any other elements of, let's call it, legal reasoning, or what's the the the legal frontier that models are not good enough yet that you want from the next generation of models?
00:58:25 · Shawn Wang, Jacob Lauritzen
[00:58:25] S1: Any other elements of, let's call it, legal reasoning, or what's the the the legal frontier that models are not good enough yet that you want from the next generation of models? [00:58:35] S0: Nothing I can think of specifically right now. I think long document reasoning is still difficult. Long, like, million token token contracts is what Exactly. You have a huge contract. Yeah. And, like, you're making edits across it. It's a difficult thing to do for a model today even. And, like, remember that you changed all these things up here, and, like, the definitions changed, and that you... The ordering changed, whatever else. [00:58:55] S1: So the simplest would be one shot, you put... Feed everything in and then one shot output. Yes. You don't do that, you break it up into chunks and then you orchestrate edits. Yeah. Exactly. Exactly. [00:59:05] S0: And this is like one of those things that are funny, like, that are fun to be part of in this industry, right? So, word ad, and it was like, we've done, you know, we've done MapReduce, and we've changed, and there was one big one, and there was like one big one in multiple times, it's like, we keep changing the scaffolding that we build around as models change. [00:59:20] S1: Yes. I I would say I I've recently... I do a lot of news reporting, obviously, because of what I do, and I've recently shifted from MapReduce to throwing the time into context. Because it's Yeah. Exactly.
00:59:20 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:59:20] S1: Yes. I I would say I I've recently... I do a lot of news reporting, obviously, because of what I do, and I've recently shifted from MapReduce to throwing the time into context. Because it's Yeah. Exactly. [00:59:33] S0: But it's also... It's different. I mean, we had this problem with search where it's like, you're searching for certain things in a huge contract, and you can map reduce it, but then it might, if you're summarizing any sort of thing, then it might lose some important nuance in between the chunks, essentially, if you just summarize the individual parts up. So, this has been a problem where it's like, we can do two types of search, we can also do the other one where it's like, you need to have reasoning in the different steps there in between. So, I'm trying to think of a really good example here. Yeah, there's some documents where they talk about something at length over, like, let's say two pages or whatever, and then each individual chunk is not necessarily relevant to the search term or what they're trying to figure out, but together it's extremely relevant. And so, yeah, these are some of the, you know, problems where it's [01:00:15] S2: And it must be hard, like you said earlier, to actually make the models better at that, because you can't even look at the contracts, right? Exactly. Do you guys ever ask them for like, hey, can you make example contracts that are similar to Yeah. The real
01:00:15 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[01:00:15] S2: And it must be hard, like you said earlier, to actually make the models better at that, because you can't even look at the contracts, right? Exactly. Do you guys ever ask them for like, hey, can you make example contracts that are similar to Yeah. The real [01:00:30] S0: Yeah. So we feedback, and can receive feedback from them, and if they're willing engage, yeah, exactly. They're just like, hey, this is like super important to us and it's not working well, and then we're like, okay, great. Can you like anonymize the documents and like the problem, whatever, and we'll... If we can recreate it, then we can fix it. [01:00:44] S1: Yeah. I mean, so one version of this is you buy a law firm Yep. Which some people are considering. [01:00:50] S0: Yes. But there's some stuff there on whether their data is actually their data or if it's the client's data. So, if their clients are all gone, then maybe it's fine. I actually don't know. Interesting. Yeah. But it's a... Yes. It's it's a way to get a lot of data. [01:01:05] S1: Yeah. People... Like, Google's buying Spiritualized. Exactly. Right? [01:01:09] S0: Yeah. They bought Yeah. They bought the... Yeah. Yeah. Yeah. [01:01:13] S1: Like, you know, if you have a failed company, you still have your Slack. You can sell the Slack. [01:01:17] S0: Think there's like I [01:01:18] S1: a YC company that buys your failed company. [01:01:20] S0: Yes, there is. [01:01:21] S1: Yeah. Because it's just like, oh, I just want real Exactly. People doing And then, yeah, just was Exactly. Pull stuff out.
01:01:21 · Shawn Wang, Jacob Lauritzen
[01:01:21] S1: Yeah. Because it's just like, oh, I just want real Exactly. People doing And then, yeah, just was Exactly. Pull stuff out. [01:01:29] S0: Exactly. The problem with buying law firm data is, well, you get all the inputs, the real inputs, which is great. The outputs are a little bit more difficult, because you just get what their end result was, and it's difficult to figure out if, like, the way... You have to basically, like, reverse engineer how they got there. So in our product, when we get real traces from really users, it's extremely high value, but if we just got this output, and we don't know if it's good or bad, or how it was created, it's a little bit difficult. [01:01:53] S1: We're gonna wrap up soon. The last thing I'll point people to is also, there's a lot of alpha left on embeddings. You're a rag heavy workload, you're just looking at MTEB and you're taking, like, top model, you're leaving a lot on the table, you should look at Voyage or some of the custom embeddings. You've gone into this. Yeah, yeah, So basically, don't take default embeddings, even OpenAI embeddings, Gemini embeddings, doesn't matter, customize your embeddings. Yeah. Right? Yes. You will you will support that. [01:02:18] S0: Yes. I think that's right. I think that's right. The only thing is you need to make sure that you're multi language. So, us, we we serve so many languages, so that's [01:02:25] S1: I gonna be mean, you're one of of this. [01:02:27] S0: Yeah. Yeah. Exactly. Mr. Au, helpful there? No. Until...