Latent Space: The AI Engineer Podcast
00:25:03 · Alessio Fanelli, Jacob Lauritzen
[00:25:03] S2: and What's an example of something they would wanna build themselves? [00:25:06] S0: Integrations into their own... Like, some of these customers build their own CRM and their own document management system and their own all sorts of stuff, and they want to build that, and then they want to build triggers that, like, you do a review in in Salesforce and it goes to this one person for whatever reason, and they do something, and then it goes into Legora, and then it goes back in, And there was... So that's, like, the simple end, and there are some that are extremely ambitious, whereas, like, a whole new product that they wanna build with us, and I... You know, we can't really do that. [00:25:31] S2: Yeah. Yeah. [00:25:32] S0: But but that's definitely a thing where, like, American companies really want to build their own more than European ones do. [00:25:39] S2: And the European ones are like, hey, like, you know, they give you product suggestions or... [00:25:43] S0: Yeah. For sure. I mean, we have great collaboration with all of them. Yeah. So they'll they'll say, hey, this sucks. This is great. We want more of this, and we'll just build a a product that does that. [00:25:52] S2: Okay. So they don't see, like, an existential threat here in their business? I mean, they Whereas the American ones do, or
00:25:52 · Alessio Fanelli, Jacob Lauritzen
[00:25:52] S2: Okay. So they don't see, like, an existential threat here in their business? I mean, they Whereas the American ones do, or [00:25:58] S0: I don't know if that's the reason that they want a custom build versus not I custom think they both know that this is, like, extremely important and key to their firm's success and their corporate success. So they... It's both extremely important. I think it's just a different way to achieve it. [00:26:12] S2: Okay. Another question I think maybe you you wanna bring this up is, like, product metrics. Like, how do you know that, like, we're actually doing something good here? [00:26:21] S0: Yeah. Yeah. [00:26:22] S2: How do you think about that? I'm assuming that's also related to evals.
00:26:22 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:26:22] S2: How do you think about that? I'm assuming that's also related to evals. [00:26:25] S0: So we have the regular product metrics that we care a lot about. Are people spending a lot of time in our product? Do they seem to get a lot of value out of it? Do they renew it? Do they come back a lot? We have really good numbers for an enterprise software of, like, our DAO over MAU and our, like, you know, hours spent per month in the product are extremely high. And then we do some sort of... We do online emails also where it's, like, try to figure out generally, like, did we solve the problem here? Just with, like, super cheap LMS address, try to just, have an online metric, and then we can use that also when we do rollouts of new things. We can just, like, compare feature flag wise, okay, does it did it improve or, like, did it catastrophically decline? Because one thing actually that might be interesting, that's difficult for our business is we can't look at any traces, basically. We can't look at any data because it's all super confidential, and that's, like, extremely frustrating to work with because you you basically don't know what people are doing. So it's like, here's an agent, and it can do so many amazing things, and I have no idea what people are actually [00:27:19] S1: doing. [00:27:20] S2: I would have assumed you guys signed some sort of agreement with the law firm, and they're like, oh yeah,
00:27:20 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:27:20] S2: I would have assumed you guys signed some sort of agreement with the law firm, and they're like, oh yeah, [00:27:24] S0: I wish, yeah, I wish, no, but the thing is, like, a lot of the data that they have is not even theirs, right, it's their clients, and it's super confidential, so we can't look at it. And so, have an evalist team that's, like, trying to approximate what we think everyone [00:27:36] S1: is doing [00:27:36] S0: and, like, build that into eval sets. Wow. So that's just, like, a super annoying kink of the business. [00:27:41] S1: Can you do paraphrase? For example, ChatGPT has like a slash feedback thing that directly fully sets its full trajectory. But obviously, that would... That is not possible for you guys. But you can use an element to paraphrase, like, from this real thing Yep. Make a synthetic thing that is, like, close enough, but change all the names, change all the details, but, like, still...
00:27:41 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:27:41] S1: Can you do paraphrase? For example, ChatGPT has like a slash feedback thing that directly fully sets its full trajectory. But obviously, that would... That is not possible for you guys. But you can use an element to paraphrase, like, from this real thing Yep. Make a synthetic thing that is, like, close enough, but change all the names, change all the details, but, like, still... [00:28:03] S0: Yeah. Yeah. We can't... There's... I mean, there's always a range, right, of, like, if you if you paraphrase all the way down to, three values and you pick one of them, then obviously that's fine. If you if you just remove the names, then that's maybe maybe not fine. Yeah. It's like, we are generally pick the the most safe way. Yeah. Because, like, we we restrict ourselves as most possible because we, you know, there should be no risk at at all. Yeah. But there are things where it's like, okay, we know that the failure modes are probably these 10 or 20, so we can classify it into one of those, and that's a helpful signal. We can classify it into the types of tasks and the practice areas also, and then, like, we have an idea. Something that we're always trying to to figure out how to do in a smart way. [00:28:42] S2: Also, like, on the privacy front, I guess, I'm assuming even two years ago, it must have been, like, insanely difficult to get the law firms to [00:28:49] S0: be okay with this. Yes. [00:28:49] S2: But, like Yep. How do they think about their data being sent over to Anthropic or Yeah. Yeah. It must be, like, a huge thing.
00:28:49 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:28:49] S2: But, like Yep. How do they think about their data being sent over to Anthropic or Yeah. Yeah. It must be, like, a huge thing. [00:28:56] S0: All of our infrastructure is on Azure Yeah. Primarily for this reason, because, like, when we started, law firms were like, AWS? No. Like, [00:29:04] S1: no way. Law firms don't trust AWS? [00:29:07] S0: Yeah. Yeah. It's different now. No. But there was like Amazon, Google, we can't trust them within our data, so basically, we had to host everything on Microsoft because they already have three sixty five. [00:29:16] S2: Oh, so they're okay with Microsoft? [00:29:17] S0: Exactly. Because they've already done, like, whatever security review, etcetera, for all of their documents and all of their Word and whatever, so Microsoft is an approved vendor in their mind, and so we had to host everything on Azure, and that's changed over the past few years. So, now people understand, okay, we need to get on cloud, we need to do all this kind of stuff. There was also a lot more people talking about on prem back then. That's also disappeared completely. People understand that, like, you should just do it in the cloud, it's and it's safe, and it's probably safer than what you're doing. Yeah. So that's changed [00:29:43] S1: a lot. [00:29:43] S2: But I