Latent Space: The AI Engineer Podcast
00:28:49 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:28:49] S2: But, like Yep. How do they think about their data being sent over to Anthropic or Yeah. Yeah. It must be, like, a huge thing. [00:28:56] S0: All of our infrastructure is on Azure Yeah. Primarily for this reason, because, like, when we started, law firms were like, AWS? No. Like, [00:29:04] S1: no way. Law firms don't trust AWS? [00:29:07] S0: Yeah. Yeah. It's different now. No. But there was like Amazon, Google, we can't trust them within our data, so basically, we had to host everything on Microsoft because they already have three sixty five. [00:29:16] S2: Oh, so they're okay with Microsoft? [00:29:17] S0: Exactly. Because they've already done, like, whatever security review, etcetera, for all of their documents and all of their Word and whatever, so Microsoft is an approved vendor in their mind, and so we had to host everything on Azure, and that's changed over the past few years. So, now people understand, okay, we need to get on cloud, we need to do all this kind of stuff. There was also a lot more people talking about on prem back then. That's also disappeared completely. People understand that, like, you should just do it in the cloud, it's and it's safe, and it's probably safer than what you're doing. Yeah. So that's changed [00:29:43] S1: a lot. [00:29:43] S2: But I
00:29:43 · Alessio Fanelli, Shawn Wang, Jacob Lauritzen
[00:29:43] S2: But I [00:29:44] S1: was And by the way, it's only been three years, so, like, it's actually, like... Yeah. It's a... Yeah. Yeah. It's a shoot shift. It's it's it's great. I mean, good for you guys, but yeah. I I don't expect the, like, you know, the legal firm, legal industry has somehow ignored this for the last, like, ten years. Yeah. Yeah. I think [00:29:58] S0: we needed the AI kicking the butt for everyone to understand that we need to change Okay. Our ways a little Okay. [00:30:04] S2: So also then on, like, open source models now are getting really good. So, how are you thinking about that with the... [00:30:10] S0: Yeah. We're just thinking about, to to your point, just bring our clients along on, like, the Pareto frontier. Yeah. And, like, the the map of all the models and what they're good at, and their latency, and their cost, and their governance, and and and their regional availability, how do we just optimize across all of that? So, some open weights models are great for certain tasks, and then we'll use them there. Some will post train, and and some will just use the frontier models. [00:30:36] S1: Yeah. You call... You call it model arbitrage. Yeah. You have different favors. You you did some research on this where you were firmly anthropic at the start of the year. Yeah. Grok is improving a lot. Yeah. You wanna give an example of a model that wasn't ranked highly overall, but, like, you wanna shout it out as surprisingly good?
00:30:36 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:30:36] S1: Yeah. You call... You call it model arbitrage. Yeah. You have different favors. You you did some research on this where you were firmly anthropic at the start of the year. Yeah. Grok is improving a lot. Yeah. You wanna give an example of a model that wasn't ranked highly overall, but, like, you wanna shout it out as surprisingly good? [00:30:55] S0: Yeah. I think not overall good. Gemini, one of the early Geminis is, yeah, extremely good for certain types of transcription and speaker attribution. So we use that for trans... Transcription actually. [00:31:07] S1: Oh, so you have a [00:31:09] S2: Yeah. What you gonna transcribe? [00:31:09] S0: Like a granola? We don't have... We actually probably will build, like, a meeting thing Yeah. But this is just, like, talk to speech, speech to text, text to speech, just like in in any other app that has this. And we do actually also transcribe files. If you upload hearings or whatever, we'll then transcribe them, put them into text, and so we can do all of our analysis on it. ElevenLabs. ElevenLabs, for DPA reasons. Same thing as like the data processing agreement, I should say. So like, our data processing agreement has very few vendors on it. [00:31:37] S1: Yeah. You just need to keep [00:31:38] S0: it Exactly. Need to keep it super tight on where any data goes. And so
00:31:38 · Jacob Lauritzen, Shawn Wang, Alessio Fanelli
[00:31:38] S0: it Exactly. Need to keep it super tight on where any data goes. And so [00:31:43] S1: I think, no, that that is one of the benefits of, like, a Gemini and OpenAI where they're like, you you come to us, we have most of the things that you need, and that is actually very valuable. Even if it's like... It doesn't have to be, like, state of the art. It's always just, like, mostly good enough for all tasks Yep. Then who cares? [00:31:59] S0: Yep. Yeah. No. Absolutely. I should actually shout out also Microsoft's transcription model. They made one Transcribe? Yeah. Okay. It's great. [00:32:08] S1: Okay. [00:32:08] S0: It's better than the old Gemini. [00:32:10] S1: Yeah. Yeah. Yeah. They they they did some good results. I don't know... Don't even know. Like, I think, like, usually, all of them are, like, top two, top three, top four in terms of, like, the the arena scores, and that's good enough. [00:32:20] S0: Yeah. Exactly. And it's like, you have all the governance and it's super scalable, and it's like, then that's probably good [00:32:24] S2: enough. Yeah. I was gonna say, like, at this point, are pretty much the speech to text models, like, good enough to where you're like, oh, one word here or there, if it's off, [00:32:32] S1: that's No? [00:32:33] S0: No? No. [00:32:33] S1: For sure. Like, I've done this a lot.
00:32:33 · Shawn Wang, Jacob Lauritzen
[00:32:33] S1: For sure. Like, I've done this a lot. [00:32:35] S0: And particularly in legal, there's like some... We We had this thing where I think it's... I forget if it's like in Australia where there's like a certain name for like a person in court, and it would consistently prescribe it as condom. Oh. So there, yeah, there's certain things that you don't want there. [00:32:52] S1: Yeah. But... So usually you run an LLM layer on top of it, and then you you extract some keywords and then you run the transcription model again with the keywords. Like, this is kind of my my workflow so far. But, yeah, if you... If if you've just done this on any large scale, you will pick up tons and tons of transcription errors on any domain that is not, like, average day to day conversation. [00:33:12] S0: Yeah. Yeah. Absolutely. If there's any sort of jargon in there Yeah. It's gonna be tough. [00:33:18] S1: Yeah. Like, I think Whisper and Eleven Labs still transcribe cloud code as cloud code. Yeah. And, like, it's Yeah. I've I've actually had that already. Very common. Yeah. So so, like, that's that's how you know there's, like, a lot more to improve. Yeah. Absolutely. I do want to cover a little bit on the the agent product side. So we've covered a few of, like, the the top level stuff. Right? We've already named job tabular review. We named jobs intake. Yep. And you've you've got some other stuff that we haven't even touched on yet, but I just wanna cover the the big two. Let's call it like Yep. It... That is is that fair to say that they are the big two?
00:33:18 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:33:18] S1: Yeah. Like, I think Whisper and Eleven Labs still transcribe cloud code as cloud code. Yeah. And, like, it's Yeah. I've I've actually had that already. Very common. Yeah. So so, like, that's that's how you know there's, like, a lot more to improve. Yeah. Absolutely. I do want to cover a little bit on the the agent product side. So we've covered a few of, like, the the top level stuff. Right? We've already named job tabular review. We named jobs intake. Yep. And you've you've got some other stuff that we haven't even touched on yet, but I just wanna cover the the big two. Let's call it like Yep. It... That is is that fair to say that they are the big two? [00:33:52] S0: Agent and Tabular Review? And our projects. Probably. Yeah. And our Word add in Okay. Is also something... Because that's... A lot of people work work inside of Word, so our Word add in is is a huge... And and Outlook add in also. [00:34:03] S1: Yeah. So I just wanna cover, like, the the big ones that everyone must know. Yep. So Tabular Review is basically, like, fan out of, like, a lot across each document. You can ask the same question across all all documents, select it on the table Yeah. And then work back and forth. And then Yep. A million is impressive. What's... Any other, like, engineering considerations or performance things? Or, like, what... What's hard about this? [00:34:24] S0: I feel [00:34:24] S1: like before we do [00:34:25] S2: that, can you just, like, walk through, like, what is a user experience for a lawyer? And then, like, actually, you wanna you use a tabular view? Yeah.