Latent Space: The AI Engineer Podcast
00:13:45 · Richard MacManus, Aman Singh
[00:13:45] S1: So earlier in the conference, Anthropic did a workshop about their managed agents infrastructure and a lot of it, even the orchestration is becoming more automated now. So do you foresee kind of using that type of functionality? Because it obviously presents a danger and that the human is not necessarily always in the loop anymore. So have you had any thoughts about using that technology to automate even more of the process? [00:14:11] S0: I think we are very bullish on the models from from Anthropic and and labs like OpenAI, but less so on on taking a dependency on a product like managed agents. Because managed agents, one of their their key things is how the models kind of persist and and contribute memory as to what is happening in the world and all of the operational knowledge that they need to be... Need to use to to be successful. Yes. And so letting them own that part of the business is kind of like, a a risk because, owning that is... Owning your intelligence is kind of, gives you the ability to impact the business. If you if you hand off the intelligence to to another provider, you lose some of that control that you need to kind of, like, build something that is unique and tailored to your business to such a great degree that it can actually impact your business. [00:15:07] S1: And just on Anthropic, they've they've also recently released financial services, so they're partnering with a whole bunch of different firms. Mhmm. Do you think what they're doing is kind of competitive to what you're trying to do, or do you see them different things?
00:15:07 · Richard MacManus, Aman Singh
[00:15:07] S1: And just on Anthropic, they've they've also recently released financial services, so they're partnering with a whole bunch of different firms. Mhmm. Do you think what they're doing is kind of competitive to what you're trying to do, or do you see them different things? [00:15:20] S0: Yeah. I think, I think Anthropic getting into our kind of industry is is very validating for for our industry. But the way I think about it is that it is very different to be an, a a services firm that has access to AI tools versus a firm that was built to be AI native. Mhmm. The firm with AI tools requires their users to prompt the use... The agents kind of to to produce work or to do work. But here, like I said, in... At Arca, agents prompt people more than people prompt agents, and it is a proactive kind of, service that is running for them twenty four seven, and it's kind of like running the business in the most standard way that we that we expect at the level of service that we that we that we aspire to, which looks very different if you're if you're a... An AI AI... Sorry, a company with AI tools. [00:16:09] S1: Yes. Yes. Okay. So a little bit more about on the security side. So can you explain, like, what in your processes runs inside a sandbox and how do you verify like a change to a financial plan, you know, before anyone acts on it? Mhmm.
00:16:27 · Aman Singh
[00:16:27] S0: Yeah. So the the the way we use sandboxes is that the the sandbox is one component of our harness. Our agent isn't actually inside of the sandbox. It uses the sandbox to offload computation, and we've we've built systems to make it safe for code inside of that sandbox to access our data. So we kind of, like, have this thing which translates the coded road into a strict format that we expect, throw away the code, and you, kinda instill guardrails on that format before it reaches your back end. Right. So we allow the agents kind of to use code to query our data, which means that they can, like, aggregate it, they can munch it, they can kind of, like, work with it as they... As to see fit Mhmm. To produce such a thing like a like a financial plan. Inside of the same sandbox, they produce this, you know, beautiful artifacts that we we can share with clients that has, like, it's a PDF of, like, here's your financial plan and here's how we're planning to kind of, like, approach your life financially. And once that is produced is where we run our verifiers. So it's a certain type of artifact that is produced from the sandbox. Yep. The moment that is produced, we run bunch of different different verifications on it. Some deterministic that check for presence of, like, hey, did you just put, like, a raw account number in there? You should not do that. So deterministically, we can check that. But also fast models like like the the smaller models or JEV to analyze for presence of things that might make this artifact noncompliant. Like, for example, you really should not be, like, promising returns in in that artifact. So that's another thing that we will check before it is it is kind of publishable. And the last thing I'll say on that is that doing this is actually... Makes our kind of compliance better than if humans were doing it because AI is tireless at at at finding even the smallest mention of a non compliant piece of writing in there. Yeah. So, yeah, we'll continue to invest in in in more of that and kind of like constrain at the capability edges.
00:18:39 · Richard MacManus, Aman Singh
[00:18:39] S1: Yes. Yeah, you mentioned JEV there, so you've started to use decision models as part of your process. What other products other than JEV are you sort of looking at? [00:18:51] S0: Yeah, so what we've noticed is when models are working with financial data, right, when they're answering questions that require financial knowledge, they are... They're very confident in in in the knowledge that they have. So a small example is that models will very happily use the tax brackets from 2025 to to come up with an analysis and be like, alright. That is it. But that is that is obviously incorrect. And so that is one of the ways in which we're using decision models in in our harness is to to check when the the model is about to make a financial recommendation or or anything... Any action that requires financial rigor will detect, and will nudge the model to kind of use our knowledge base of verified facts about the world that is true today and or use their kind of web search. So it kinda makes it... This is kind of a a a point about the the harness and and owning the harness that lets you do things like this that that help your your models be be accurate where it matters. [00:19:51] S1: And do you use decision models and for example, recovery if like a trade fails to execute, is there a decision model involved there or is there another path to to get the agent back [00:20:01] S0: on track? No, not today. That is not a use case that we're exploring today. Okay.
00:20:01 · Aman Singh, Richard MacManus
[00:20:01] S0: on track? No, not today. That is not a use case that we're exploring today. Okay. [00:20:06] S1: So, you know, if if a step... A trade does fail, for example, how does the agent recover at the moment [00:20:12] S0: in your process? So at the moment, we are not doing autonomous trading with with agents. It is very much a... The agents will propose that we we make trades. I see. Yep. And it is fully up to kinda our our experts to figure out if that is the right trade to be making and to execute the trade. Okay. [00:20:33] S1: So on the on the same sort of line of that, your slides say that every staff edit or cancellation comes with a y and come... It becomes an eval. So how does that feedback get turned back into the system as a measurable improvement?
00:20:50 · Aman Singh
[00:20:50] S0: Yeah. That is another one of the advantages of us being the business and not a SaaS company is that we have we have the ability to have to require them to give us feedback. And, yeah, the way that we loop the feedback back into improvements is something that you would you would expect is all of that feedback gets piped into a data store that that couples that feedback with the trace of the agent that produced it. Yeah. And so we're able to kind of, like, take that take take that trace, put it put it next to the the feedback. And every night, we run this process where we take... Collect traces from the real world, from our agents in the wild, and run run a meta agent on top that suggests changes to our harness. Mhmm. So maybe tools that were misleading or things that were were errored or there's missing knowledge in our knowledge base. Mhmm. And they'll surface this information to us, then we'll we'll be able to kind of improve it. Today, we still have a a human in in... Like, a human engineer in the loop before we actually ship a prompt change. Yes. I've noticed that that agents will will make really nasty prompt changes if you leave them leave them do it. Every little fix becomes the whole prompt. Mhmm. And they're they're unable to stay meta and kind of generalize the fix into into a prompt that generalizes. Mhmm. So... But, yeah, we have we have feedback loops from our product that we collect directly as first class product features to collect evals into into improvements. Mhmm.