00:04:16 · Brett Caughran
[00:04:16] S1: Great, great. Maybe I've always found I always get super excited on things, and then I call you and you're like, well, let's talk about the precision of these elements. Precision is so critical for finance use cases. You can sort of win so many AI things, create these impressive looking demos. But if the numbers aren't at a level of precision, they're just not institutionally useful. How do you think about just the broad challenge of precision with large language models, which are nondeterministic by nature? What are a few of the key vectors that you've thought about for years about making these tools more precise?
00:05:02 · Kris Bennatti
[00:05:02] S0: Yeah, it's a really tricky problem and it's been a persistent problem. It's definitely getting better from a generalist perspective, but for the foreseeable future, if you want to make sure that your numbers are definitely correct, you do need some pre processing and some built in checks. It's pretty much impossible to just take a generalist model, apply it to finance, and be like, okay, great, this is all going to be right. For reference, we did a test on all of the state of the art models a few months ago now. So, it's been before, before 5.5, whatever the Claude and OpenAI models were a few months ago, and whenever you asked for a metric for more than four to eight periods, you'd get an incorrect number 30% of the time. So, when you think about hallucination right now, there's mostly a problem or the biggest problem when you start getting beyond comfortable context limits, And that's one of the big problems we focus on. It will continue to get better, but as you've seen so far, it's getting better much, much, much more slowly than things we've seen like reasoning, mathematical elements, all of those things. We did, as of yesterday, formally launch our no hallucination guarantee. So we're pretty feeling incredibly confident about our ability to not pull incorrect numbers, restatement adjusted error free numbers across a five year period. That's all done using AI architecture rather than just thinking about the last generative step. And, you know, it's taken us a while to be able to do that five year period, restatement adjusted perfectly, but we're feeling really excited about that. And I believe we're the only people on the market are willing to do a no hallucination guarantee, which does give you a bit of a sense of how much risk there is in the average financial AI software.
00:07:25 · Brett Caughran
[00:07:25] S1: What does that entail to gain the conviction to offer a guarantee like that? Like you said, pre processing and built in checks. Like what is that? How hard is that problem? What are the big work streams to sort of take the general models and convert them to that hallucination free layer?
00:07:48 · Kris Bennatti
[00:07:48] S0: Yeah. If you want to think about hallucination, you, I think it's chapter 11 of Suhas's textbook, he talks about the place where you end up with hallucination is when the model doesn't know the answer and it doesn't know it doesn't know the answer. That is the highest risk. Time when you're not feeding the model the information it needs and it doesn't know that. So, as you can imagine, when you start moving beyond these comfortable context limits, that's when you start seeing hallucination in a generalist cell. How do we overcome that? We talked about pre processing. So once you get towards more than, say, four, eight quarters, you need to start using search and retrieval to get the right information. And one of the reasons so many tools, I'm not going to name names, but have issues in that long time horizon is because search and retrieval is really hard. If you're just looking for revenue and you're searching for it, again, across 20 press releases, you're going to be pulling in a lot of noise. And if you think about how complicated getting revenue right, depending on what you're asking for, can be, you know, maybe you're looking for a subsegment nine month and instead you're going to be getting numbers in your retrieved element that are consolidated and three month. So those sort of long horizon complex queries, in order to fix that problem, you do need some pre processing where your embeddings need to have information about what is this, what time period is it related to, and that allows the model to be able to select the information it needs and actually use it. And when we talk about guardrails or checks, it's about making sure you have specific pathways, that exist so that the model knows that you're asking for a KPI poll, and it should be able to inspect this information. And if it's not getting it, it just gives you an error.
00:10:20 · Khe Hy, Kris Bennatti
[00:10:20] S2: Can I ask a lazy layperson question? If you think of the... I definitely appreciate the context window challenge. Does Like, how does something like NotebookLM solve this where presumably, as a user that doesn't understand the technical in and outs of it, it feels like the context window is much larger. Just like, can give it 500 files. It seems to be more accurate in the citation nature of it. It's not as intelligent. It's like pure retrieval. But can you how does that compare? Like, what's happening under the hood in a Notebook. Lm versus a generalist model versus a Hudson's Labs approach? [00:11:07] S0: So, one of the reasons that Notebook LLM feels better is because it has a more intelligent search. Yeah. So, it's going to be, it's not finance specific, obviously. It's not numeric. It's not built for pulling KPIs. You are going to still, even with Notebook. Llm, are going to run into that sort of eight quarter plus failure mode. I don't know if you've tried it. So for the same reasons around search, but it is using a more advanced search mechanism. So it's not about... It's not... It doesn't have a better context window. It just does better search. [00:11:48] S2: Got it. Yeah. And so, Hudson, being finance experts, you can basically customize that search process way better than any, than a generalist approach.
00:11:48 · Khe Hy, Kris Bennatti, Brett Caughran
[00:11:48] S2: Got it. Yeah. And so, Hudson, being finance experts, you can basically customize that search process way better than any, than a generalist approach. [00:12:01] S0: So, if you want to think about Hudson Labs versus Notebook LM versus just having a Claude instance, what's happening, the back end is a big part of the difference. So whenever we take in an SEC filing, a transcript, etc, it's, we use vector databases. So we're converting a sentence, a word, a paragraph to an embedding. Are you familiar with? [00:12:31] S2: Yeah. [00:12:31] S0: Or a vector. Yeah. So there's lots of different ways that you can do that and you should be doing that, but if you're using Claude, your data isn't in a vector database, which is one of the reasons that it can be so expensive. And if you're using NotebookLM, it's essentially search rather than a vector database. [00:13:00] S1: Got it. [00:13:01] S0: But if you have a vector database or you have a more sophisticated backend, you can do a lot of really cool things in pre processing that facilitate both accuracy, relevance, and completeness on the front end. Got it. In addition to, to be able to, if you have these things as, as embeddings, you can give metadata to the model in addition to containing better information about what that sentence is, And you can do cool things with your embeddings. For instance, the way that we store embeddings, we have meta declarations not just about is this a KPI or not, is it forward looking historical, but also information about sentiment. [00:13:54] S1: Got it.