Latent Space: The AI Engineer Podcast
00:48:35 · Shawn Wang, Jacob Lauritzen
[00:48:35] S1: I have toyed with this. It's just generally good for all companies to have a platform team and a product team, because the product team, you have one, but then you can easily spin up a second one as long as the platform is not super tied to product. Yeah. Yeah. Yeah. You know, like, a very simple one is web app mobile app. Right? Or, like, web app mobile app RTP and and then, like, you know, whatever, like, other other integrations. Then your FTEs just are one more client to the overall platform anyway. [00:49:01] S0: Yes. Yeah. And I think even the agent, you can think about it this way. There's like a harness team or that runs the agent, and then that you can even build a product around it with skills and automations and whatever else. [00:49:11] S1: Yeah. This is very famously the Amazon web services mandate, like Bezos was like, no more backdoor access. You only offer other teams your APIs, the same APIs you offer customers. If you don't have it, you build it. Yep. And then the customers also get it.
00:49:11 · Shawn Wang, Jacob Lauritzen
[00:49:11] S1: Yeah. This is very famously the Amazon web services mandate, like Bezos was like, no more backdoor access. You only offer other teams your APIs, the same APIs you offer customers. If you don't have it, you build it. Yep. And then the customers also get it. [00:49:26] S0: Yeah. I think that's also changing a little bit. A lot of APIs now, like, one thing that we're toying with is, can the agent be the API? I know it sounds weird, but, like, for some of this stuff, what they actually want to do is they want to trigger certain things to happen. For some part of this, you could just be, I'm not saying that we're doing this, but it's like a thing that I'm toying with in my head is, could that just be like, okay, you trigger the agent and ask it to do this thing. Like, obviously it's way less efficient than if you just had like an API that you always hit and it's like, you know, but it's something that I'm toying with is like how much of our product can just be the agent versus all the other stuff? [00:49:58] S1: Well, you know, unbounded execution and costs and latency, [00:50:01] S0: that's the... That's... [00:50:02] S1: Is that the only whole batch? Yeah. [00:50:05] S0: And then I think deterministic, maybe in, you know, a 100 out of... Or in two out of a 100 times, it's gonna do something whack that we didn't want it [00:50:12] S1: to do. We're gonna go into, like, harness and and model fine tuning in a bit. But I think there's a there's a product question about restraint about what you don't do [00:50:20] S0: Yep.
00:50:20 · Jacob Lauritzen, Shawn Wang
[00:50:20] S0: Yep. [00:50:20] S1: Because you could do all that. And obviously, there's a prioritization issue. Like, you at more at a point of let's say no to most things and then only do a short number of stuff? Or, no, actually, we wanna do more things and we just need to hire like 100 people? [00:50:35] S0: I was very strongly the first thing for a very long time, let's just say no to everything, like, keep the AI team. I I think I've said this before, but, like, one of our presentations today, like, after I've been at Legora for half a year, I I pulled up this slide of, you know, the 300 Spartans and and and Yeah. For the movie, and I was like, we're just gonna be 20 engineers for the rest of the company's history, and, like, we're gonna beat everyone who's much bigger, and we're a 100 engineers now, and it feels like we're way too small. I wanted to do that, but I think in the real world, we're in, like, fierce competition, people want so much stuff. It's like, I could be a puritan and just be like, no, we have to stay 20 people, and we can only build, you know, whatever, but then we just lose, and, like, people would go elsewhere. So, we have to build more, and so I think my job is to figure out how do I build a system where we can build a lot, but actually only... Like, where we can... The customer feels like we've built so much stuff, but we've actually only built a few things. That's that's how I think about my job right now. [00:51:27] S1: Yeah. I mean, primitives is right, you know? Yeah. Exactly.
00:51:27 · Shawn Wang, Jacob Lauritzen, Alessio Fanelli
[00:51:27] S1: Yeah. I mean, primitives is right, you know? Yeah. Exactly. [00:51:30] S0: I think that's gonna be... That's really powerful. [00:51:32] S2: Do testing new products, just seeing if like, hey, there's like any pull here, and then you just kinda like... You're like, no, let's just focus on [00:51:37] S0: our main thing or... We haven't made... I mean, we've made certain strategic bets [00:51:43] S1: Yeah. [00:51:43] S0: But we've not tried anything that we don't have a lot of confidence in already, mostly because we just have, like, amount... Enough demand. Yeah. It's like, we want all of these things, and so it doesn't... It hasn't really made sense for us to try to be like, hey, what... Maybe this thing could work, because, there's so many people screaming for the other things, and so we mostly build those, and then sometimes we'll disagree with clients, we'll be like, actually, I think this thing over here will make a lot of sense strategically, and then we'll try to build that anyway. [00:52:07] S2: Like, strategically for you? [00:52:08] S0: Yeah. Exactly. [00:52:08] S1: What are [00:52:09] S2: the ones that
00:52:09 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:52:09] S2: the ones that [00:52:09] S0: you see? One of those things was the agent, actually. So, like, no one was actually asking for, like, I want it to be... I want the assistant that I'm using to have more agency and do things without me asking it. Like, no one ever... But we were just like, this is definitely where the industry is going. This is gonna empower you to do so much cool stuff. Collaboration was another one where we're like, we think the industry is going in a place where you can collaborate inside of where you're working, which is Legora and not in, like, emails and text messages. So, yeah, there there are there are a few of those. [00:52:39] S2: And then how... Like, once you say, okay, yeah, we want you guys to use agents, I'm assuming, like, internally at the law firm, you know, it's gonna take a lot for them to actually learn how to do that. Yep. And so that's where you kinda, like, say, hey, FTEs, go in there and teach them what [00:52:52] S1: they need. [00:52:52] S0: Yeah. Legal engineers, mostly Oh, legal. Go be... Yeah. That makes sense. But they're like, people call them forward deployed legal engineers [00:52:57] S2: if you [00:52:57] S0: want, but that's essentially what they are. Then they go in, yeah, and help and help them structure their workflows around the capabilities of the agent. [00:53:05] S2: You guys said previously, I think Max said this, but that early on specifically, you guys weren't fine tuning your own models. Is that still the case?
00:53:05 · Alessio Fanelli, Jacob Lauritzen, Shawn Wang
[00:53:05] S2: You guys said previously, I think Max said this, but that early on specifically, you guys weren't fine tuning your own models. Is that still the case? [00:53:14] S0: Yeah. So we haven't fine tuned any models up until I think now in the past few months, open weights models have become extremely capable, and so we will do it for certain tasks where we think there's a huge delta between what a frontier model can do at the cost and latency versus what we think a post trained model can do. Legal research is a great example. It's like a... You have to make... It's a it's a fan out retrieval problem, and you can make a really effective model to those queries super quickly and super cheaply at low latency. [00:53:43] S2: So, specifically, you're post training them to be better Very specific [00:53:48] S0: Yeah, bounded exactly. Okay. Yeah. We're not trying to, like, train our own fable to be the main agent. I don't think that's like a... We won't win that, you know, versus Anthropic, most likely. [00:54:00] S1: There's still a lot of, let's say, search and retrieval research or rag research to do that is maybe less glamorous. It's not actually even model training sometimes, it's just like architecture and pipelines. You you you did a case study with Turbopuffer, but, like, you know, there's also, like, do you do late interaction of