00:05:02 · Kris Bennatti
[00:05:02] S0: Yeah, it's a really tricky problem and it's been a persistent problem. It's definitely getting better from a generalist perspective, but for the foreseeable future, if you want to make sure that your numbers are definitely correct, you do need some pre processing and some built in checks. It's pretty much impossible to just take a generalist model, apply it to finance, and be like, okay, great, this is all going to be right. For reference, we did a test on all of the state of the art models a few months ago now. So, it's been before, before 5.5, whatever the Claude and OpenAI models were a few months ago, and whenever you asked for a metric for more than four to eight periods, you'd get an incorrect number 30% of the time. So, when you think about hallucination right now, there's mostly a problem or the biggest problem when you start getting beyond comfortable context limits, And that's one of the big problems we focus on. It will continue to get better, but as you've seen so far, it's getting better much, much, much more slowly than things we've seen like reasoning, mathematical elements, all of those things. We did, as of yesterday, formally launch our no hallucination guarantee. So we're pretty feeling incredibly confident about our ability to not pull incorrect numbers, restatement adjusted error free numbers across a five year period. That's all done using AI architecture rather than just thinking about the last generative step. And, you know, it's taken us a while to be able to do that five year period, restatement adjusted perfectly, but we're feeling really excited about that. And I believe we're the only people on the market are willing to do a no hallucination guarantee, which does give you a bit of a sense of how much risk there is in the average financial AI software.
00:07:25 · Brett Caughran
[00:07:25] S1: What does that entail to gain the conviction to offer a guarantee like that? Like you said, pre processing and built in checks. Like what is that? How hard is that problem? What are the big work streams to sort of take the general models and convert them to that hallucination free layer?
00:07:48 · Kris Bennatti
[00:07:48] S0: Yeah. If you want to think about hallucination, you, I think it's chapter 11 of Suhas's textbook, he talks about the place where you end up with hallucination is when the model doesn't know the answer and it doesn't know it doesn't know the answer. That is the highest risk. Time when you're not feeding the model the information it needs and it doesn't know that. So, as you can imagine, when you start moving beyond these comfortable context limits, that's when you start seeing hallucination in a generalist cell. How do we overcome that? We talked about pre processing. So once you get towards more than, say, four, eight quarters, you need to start using search and retrieval to get the right information. And one of the reasons so many tools, I'm not going to name names, but have issues in that long time horizon is because search and retrieval is really hard. If you're just looking for revenue and you're searching for it, again, across 20 press releases, you're going to be pulling in a lot of noise. And if you think about how complicated getting revenue right, depending on what you're asking for, can be, you know, maybe you're looking for a subsegment nine month and instead you're going to be getting numbers in your retrieved element that are consolidated and three month. So those sort of long horizon complex queries, in order to fix that problem, you do need some pre processing where your embeddings need to have information about what is this, what time period is it related to, and that allows the model to be able to select the information it needs and actually use it. And when we talk about guardrails or checks, it's about making sure you have specific pathways, that exist so that the model knows that you're asking for a KPI poll, and it should be able to inspect this information. And if it's not getting it, it just gives you an error.
00:10:20 · Khe Hy, Kris Bennatti
[00:10:20] S2: Can I ask a lazy layperson question? If you think of the... I definitely appreciate the context window challenge. Does Like, how does something like NotebookLM solve this where presumably, as a user that doesn't understand the technical in and outs of it, it feels like the context window is much larger. Just like, can give it 500 files. It seems to be more accurate in the citation nature of it. It's not as intelligent. It's like pure retrieval. But can you how does that compare? Like, what's happening under the hood in a Notebook. Lm versus a generalist model versus a Hudson's Labs approach? [00:11:07] S0: So, one of the reasons that Notebook LLM feels better is because it has a more intelligent search. Yeah. So, it's going to be, it's not finance specific, obviously. It's not numeric. It's not built for pulling KPIs. You are going to still, even with Notebook. Llm, are going to run into that sort of eight quarter plus failure mode. I don't know if you've tried it. So for the same reasons around search, but it is using a more advanced search mechanism. So it's not about... It's not... It doesn't have a better context window. It just does better search. [00:11:48] S2: Got it. Yeah. And so, Hudson, being finance experts, you can basically customize that search process way better than any, than a generalist approach.
00:11:48 · Khe Hy, Kris Bennatti, Brett Caughran
[00:11:48] S2: Got it. Yeah. And so, Hudson, being finance experts, you can basically customize that search process way better than any, than a generalist approach. [00:12:01] S0: So, if you want to think about Hudson Labs versus Notebook LM versus just having a Claude instance, what's happening, the back end is a big part of the difference. So whenever we take in an SEC filing, a transcript, etc, it's, we use vector databases. So we're converting a sentence, a word, a paragraph to an embedding. Are you familiar with? [00:12:31] S2: Yeah. [00:12:31] S0: Or a vector. Yeah. So there's lots of different ways that you can do that and you should be doing that, but if you're using Claude, your data isn't in a vector database, which is one of the reasons that it can be so expensive. And if you're using NotebookLM, it's essentially search rather than a vector database. [00:13:00] S1: Got it. [00:13:01] S0: But if you have a vector database or you have a more sophisticated backend, you can do a lot of really cool things in pre processing that facilitate both accuracy, relevance, and completeness on the front end. Got it. In addition to, to be able to, if you have these things as, as embeddings, you can give metadata to the model in addition to containing better information about what that sentence is, And you can do cool things with your embeddings. For instance, the way that we store embeddings, we have meta declarations not just about is this a KPI or not, is it forward looking historical, but also information about sentiment. [00:13:54] S1: Got it.
00:13:54 · Brett Caughran, Kris Bennatti
[00:13:54] S1: Got it. [00:13:54] S0: Which is why I believe we're the only platform where you can do sentiment based screening. So ask like who are the most stressed out CEOs? Those types of queries. Because if you're using like LM and it's just using search, how does it search for stress? There's no keyword that you can use there. But if you have embeddings that have information about sentiment, you can do really cool searches. [00:14:17] S1: This was a huge problem for the institutional finance use cases of chatbots in default 25. And in all of my experimentation going into ChatGPT or Claude, almost anything quantitative was not helpful due to this sort of accuracy, like 60, 70% benchmark. Part of the way the ecosystem is aimed to solve that is through MCP and connectors into an agentic workspace. And so now I can connect into, rather than web searching for a number, I can connect into the MCP of a Dilupa or a FactSet, etcetera, through my agentic workspace. Is that, how do you think about sort of what you're doing in sort of you've aimed to solve this problem in the way you just explained? How would you evaluate the way that the industry has tried to solve this problem with that MCP and connector movement into agents?