00:00:00 · Kris Bennatti, Brett Caughran
[00:00:00] S0: For the foreseeable future, if you want to make sure that your numbers are definitely correct, you do need some pre processing and some built in checks. [00:00:12] S1: How have you thought about that transition of the instructions from the investor to the portal? [00:00:17] S0: Prompting is becoming becoming a lot less important. And I believe we're the only people on the market that that are willing to do a no hallucination guarantee, which does give you a bit of a sense of how much risk there is in the average financial AI software. [00:00:40] S1: Alright. Welcome to the next episode of Invest with AI, where we explore the intersection of investing and fundamental research. I'm really excited to have Chris Bonatti here with us today. Chris was the co instructor for the first iteration of the Fundamental Edge AI Accelerator, has been someone who's really taught me a lot about deploying AI in investment research, and is a founder and CEO of Hudson Labs. So thanks so much for being here, Chris. And maybe start just telling us a little bit about your background, how you got into AI and finance.
00:01:22 · Kris Bennatti
[00:01:22] S0: Awesome. Thanks so much for having me. As always, a pleasure. Yeah, I'm Chris Bonatti. I've been around for a minute now. I got my start way back when as a data scientist at a corporate governance advisory firm where I focused on sort of pre LLM securities filing processing and was building my career as AI was making really big moves, in LLM specifically in the research community and this would be, would have been back in say 2017, 2018, we started seeing auto suggest on our phones. The research community was getting super excited about LLMs, and yet at my workplace, I was being hailed as, you know, like the ultimate innovator running essentially a word based counts on Canadian corporate disclosure. Saw that and recognized the fact that oftentimes, at least historically in finance, in corporate governance and the corporate fields, we're many years behind consumer technology and we were not adopting LLMs nearly as quickly as, you know, iPhone, Gmail, all of these cool applications. Quit my job. Co founded Hudson Labs with the LLM guy here in Canada who is Suhas Pai. If you haven't purchased his book Designing LLM Applications, I highly recommend doing it. It's published with O'Reilly. It's now been translated into over 10 languages, including Polish and Suhas does a lot of really cool training and onboarding around AI agentic modeling right now, so do give him a follow and attend some of his seminars. Yeah. And then we launched what was then Bedrock AI, now Hudson Labs, and we focus on AI for institutional finance. Our big contributions to AI research have to do with materiality ranking and precision. So, being able to maintain accuracy over very large contexts, be able to pull numbers, pull guidance, pull sentiment in a way that's consistent, etcetera. We also started our business with a forensic risk score and for many years now have been popular among the short community, among plaintiff class auction lawyers and D and O insurers, as well as buy side firms.
00:04:16 · Brett Caughran
[00:04:16] S1: Great, great. Maybe I've always found I always get super excited on things, and then I call you and you're like, well, let's talk about the precision of these elements. Precision is so critical for finance use cases. You can sort of win so many AI things, create these impressive looking demos. But if the numbers aren't at a level of precision, they're just not institutionally useful. How do you think about just the broad challenge of precision with large language models, which are nondeterministic by nature? What are a few of the key vectors that you've thought about for years about making these tools more precise?
00:05:02 · Kris Bennatti
[00:05:02] S0: Yeah, it's a really tricky problem and it's been a persistent problem. It's definitely getting better from a generalist perspective, but for the foreseeable future, if you want to make sure that your numbers are definitely correct, you do need some pre processing and some built in checks. It's pretty much impossible to just take a generalist model, apply it to finance, and be like, okay, great, this is all going to be right. For reference, we did a test on all of the state of the art models a few months ago now. So, it's been before, before 5.5, whatever the Claude and OpenAI models were a few months ago, and whenever you asked for a metric for more than four to eight periods, you'd get an incorrect number 30% of the time. So, when you think about hallucination right now, there's mostly a problem or the biggest problem when you start getting beyond comfortable context limits, And that's one of the big problems we focus on. It will continue to get better, but as you've seen so far, it's getting better much, much, much more slowly than things we've seen like reasoning, mathematical elements, all of those things. We did, as of yesterday, formally launch our no hallucination guarantee. So we're pretty feeling incredibly confident about our ability to not pull incorrect numbers, restatement adjusted error free numbers across a five year period. That's all done using AI architecture rather than just thinking about the last generative step. And, you know, it's taken us a while to be able to do that five year period, restatement adjusted perfectly, but we're feeling really excited about that. And I believe we're the only people on the market are willing to do a no hallucination guarantee, which does give you a bit of a sense of how much risk there is in the average financial AI software.
00:07:25 · Brett Caughran
[00:07:25] S1: What does that entail to gain the conviction to offer a guarantee like that? Like you said, pre processing and built in checks. Like what is that? How hard is that problem? What are the big work streams to sort of take the general models and convert them to that hallucination free layer?
00:07:48 · Kris Bennatti
[00:07:48] S0: Yeah. If you want to think about hallucination, you, I think it's chapter 11 of Suhas's textbook, he talks about the place where you end up with hallucination is when the model doesn't know the answer and it doesn't know it doesn't know the answer. That is the highest risk. Time when you're not feeding the model the information it needs and it doesn't know that. So, as you can imagine, when you start moving beyond these comfortable context limits, that's when you start seeing hallucination in a generalist cell. How do we overcome that? We talked about pre processing. So once you get towards more than, say, four, eight quarters, you need to start using search and retrieval to get the right information. And one of the reasons so many tools, I'm not going to name names, but have issues in that long time horizon is because search and retrieval is really hard. If you're just looking for revenue and you're searching for it, again, across 20 press releases, you're going to be pulling in a lot of noise. And if you think about how complicated getting revenue right, depending on what you're asking for, can be, you know, maybe you're looking for a subsegment nine month and instead you're going to be getting numbers in your retrieved element that are consolidated and three month. So those sort of long horizon complex queries, in order to fix that problem, you do need some pre processing where your embeddings need to have information about what is this, what time period is it related to, and that allows the model to be able to select the information it needs and actually use it. And when we talk about guardrails or checks, it's about making sure you have specific pathways, that exist so that the model knows that you're asking for a KPI poll, and it should be able to inspect this information. And if it's not getting it, it just gives you an error.