Latent Space: The AI Engineer Podcast
00:02:41 · Richard MacManus, Aman Singh
[00:02:41] S1: Okay. So let's get into the... Some of the technical aspects of it. Which models are you using to power your agents and how do you decide which model is best for for which action? [00:02:52] S0: Yeah. So today we're using the Frontier models from from the Frontier labs like Anthropic and and OpenAI. Claude is kind of our primary model has been, but Mhmm. We've gone through several iterations of models over the years. We used to be in Gemini way back when and then OpenAI, and then and then Anthropic. And what's guiding all of our decisions are our benchmarks that we have internally for all of the various capabilities of our general harness that it has to express in in running our business. And so just recently, for example, despite the external benchmarks claiming great things for Llama, it didn't really quite work out for us as as... Well, as we had expected, it was way too slow or... And it was costing a lot. And so our internal benchmark is gonna guide our model decisions. And as a principle, we we want to be model agnostic and and sort of not, you know, take a dependency on on a a certain provider or a certain harness from one of these labs and kind of keep that that optionality open for us. [00:04:01] S1: Okay. So you just mentioned harnesses. So you're using different harnesses depending on which model you're using? [00:04:08] S0: No. We have we have our our own Okay. Kind of built internally that is more model agnostic. Okay.
00:04:08 · Aman Singh, Richard MacManus
[00:04:08] S0: No. We have we have our our own Okay. Kind of built internally that is more model agnostic. Okay. [00:04:14] S1: So, you know, when when you're... When a financial adviser is doing something like a taxation report or something like that, how does the harness decide which model to use? Is that automated? [00:04:25] S0: Well, yes. So we have certain tiers of models based on different complexities of of [00:04:31] S1: tasks. Mhmm. [00:04:32] S0: So a task like assisting with a tax plan or doing a tax analysis would get our, like, heaviest tier of models. Inside of that tier, it is a little bit flexible, so you could use... So, like, an opus tier model or a sonnet tier model or a a a heavy GPT model. Mhmm. But based on... There's a lot of, like, background activity that our agents are participating in. Right? That is using the... A lower tier of model based on the complexity of the task. So that's kind of how we how we make decisions Yeah. On on which model serves which task. [00:05:04] S1: Okay. So is it the the financial adviser that's actually making that decision while they're using the product? [00:05:11] S0: Not really. They're they're they're not aware of Okay. Of that happening. We've abstracted this away Okay. From them entirely. They kind of just interact with our product as they would, and those decisions are made in... Internally by us. Okay.
00:05:11 · Aman Singh, Richard MacManus
[00:05:11] S0: Not really. They're they're they're not aware of Okay. Of that happening. We've abstracted this away Okay. From them entirely. They kind of just interact with our product as they would, and those decisions are made in... Internally by us. Okay. [00:05:25] S1: So in your presentation at at the conference earlier today, talked about a tax bill example. I just wondered if you could maybe walk us through an example of adjusting a client's portfolio, investment portfolio, you know, from identifying a need to maybe rebalance the portfolio to executing trades. Can you talk us through, you know, what parts of that process the agent does and and what parts the human advisor does? [00:05:52] S0: Right. So just like in the talk, this is a thing that will span ten days. And so agents are ultimately responsible for delivering that outcome, but there are certain parts that they do and certain parts that they do not. So in this example, when when the client emails us with this whole tax bill, there's a there's a piece of the analysis of, like, where is this client's money allocated today, and what is the most optimal way to liquidate some of that to to produce a $100,000 that is best suited for agents Right. To take on as an analysis task. [00:06:27] S1: Yeah. So that they do exactly that. Do they have certain limits though that they can, you know, they can execute a trade at [00:06:32] S0: Correct. [00:06:32] S1: X amount.
00:06:32 · Richard MacManus, Aman Singh
[00:06:32] S1: X amount. [00:06:33] S0: Yeah. So the analysis part is fully automated, but they're unable to actually execute any sort of money movement or liquidation automatically. Mhmm. We have a very deep integration with our custodian that actually gives us APIs to be able to do that, but we have strict guardrails around human in the loop for things that would either move money or liquidate assets or even directly communicate with our with our end clients because there's a level of service that that... Service service and quality that we expect. So all of the, like, internal analysis will be done by agents, but the the... They will also pull in the right person to be able to execute the trade when needed. Mhmm. And they'll give it... Give the person a plan that they... They're able to review and kind of, like, adjust. And, yeah, go from there. Yeah. [00:07:20] S1: Yeah. So you mentioned that these processes tend to be, you know, ten days or at least a longer sort of running time than most sort of agent stuff these days. So when an agent kind of resumes a task maybe days later than they, you know, than they started it, how do you restore... How do you check that the plan still fits the client's, you know, needs?
00:07:42 · Aman Singh
[00:07:42] S0: Yeah. So this is an interesting interesting one to think about is we have tons of these signals like I described in the talk. Some are, for example, you can imagine there's an email thread that is receiving responses and there's chatter happening. Right? You can imagine that that any additional response to that goes to the same agent that's kind of like reading that that email thread and reacting to it. Right? But a a fresh signal of... It's like a like a phone call that was made in the middle, That is a brand new signal that isn't attached with any other ongoing process. And so there's like a fresh agent that wakes up to figure out what this means. Right? And the way they way they understand what is happening right now is our internal memory systems. Mhmm. So we have we have this this persistence of... For every client, what is happening right now? What are we trying to service? Who owns what? What has happened so far? What are we tracking for next? Right, what what needs to happen immediately in the next couple of days, what are we tracking for the long term, right, which individual is responsible for which action. All of this is being tracked by our memory systems Mhmm. That our agents contribute to, But, also, there is a a kind of really sort of dreaming process that reflects on the state of the world as it relates to Richard, the client Mhmm. And is able to kind of, like, update these memory systems to to persist long horizon kind of tasks. So that agent that received a new signal quickly ramps up on what the situation is with the client, and he'll know, okay. We're we're actually executing a big tax bill, and we'll be able to see, okay. My signal relates to that tax bill, so let me kind of contribute to the to to the state of that world.
00:09:22 · Richard MacManus
[00:09:22] S1: Can you talk a little bit about the the safety sort of aspects that you've maybe built into your product or harness to to to make sure, obviously, that no money is lost along the way?