00:07:48 · Kris Bennatti
[00:07:48] S0: Yeah. If you want to think about hallucination, you, I think it's chapter 11 of Suhas's textbook, he talks about the place where you end up with hallucination is when the model doesn't know the answer and it doesn't know it doesn't know the answer. That is the highest risk. Time when you're not feeding the model the information it needs and it doesn't know that. So, as you can imagine, when you start moving beyond these comfortable context limits, that's when you start seeing hallucination in a generalist cell. How do we overcome that? We talked about pre processing. So once you get towards more than, say, four, eight quarters, you need to start using search and retrieval to get the right information. And one of the reasons so many tools, I'm not going to name names, but have issues in that long time horizon is because search and retrieval is really hard. If you're just looking for revenue and you're searching for it, again, across 20 press releases, you're going to be pulling in a lot of noise. And if you think about how complicated getting revenue right, depending on what you're asking for, can be, you know, maybe you're looking for a subsegment nine month and instead you're going to be getting numbers in your retrieved element that are consolidated and three month. So those sort of long horizon complex queries, in order to fix that problem, you do need some pre processing where your embeddings need to have information about what is this, what time period is it related to, and that allows the model to be able to select the information it needs and actually use it. And when we talk about guardrails or checks, it's about making sure you have specific pathways, that exist so that the model knows that you're asking for a KPI poll, and it should be able to inspect this information. And if it's not getting it, it just gives you an error.
00:10:20 · Khe Hy, Kris Bennatti
[00:10:20] S2: Can I ask a lazy layperson question? If you think of the... I definitely appreciate the context window challenge. Does Like, how does something like NotebookLM solve this where presumably, as a user that doesn't understand the technical in and outs of it, it feels like the context window is much larger. Just like, can give it 500 files. It seems to be more accurate in the citation nature of it. It's not as intelligent. It's like pure retrieval. But can you how does that compare? Like, what's happening under the hood in a Notebook. Lm versus a generalist model versus a Hudson's Labs approach? [00:11:07] S0: So, one of the reasons that Notebook LLM feels better is because it has a more intelligent search. Yeah. So, it's going to be, it's not finance specific, obviously. It's not numeric. It's not built for pulling KPIs. You are going to still, even with Notebook. Llm, are going to run into that sort of eight quarter plus failure mode. I don't know if you've tried it. So for the same reasons around search, but it is using a more advanced search mechanism. So it's not about... It's not... It doesn't have a better context window. It just does better search. [00:11:48] S2: Got it. Yeah. And so, Hudson, being finance experts, you can basically customize that search process way better than any, than a generalist approach.
00:11:48 · Khe Hy, Kris Bennatti, Brett Caughran
[00:11:48] S2: Got it. Yeah. And so, Hudson, being finance experts, you can basically customize that search process way better than any, than a generalist approach. [00:12:01] S0: So, if you want to think about Hudson Labs versus Notebook LM versus just having a Claude instance, what's happening, the back end is a big part of the difference. So whenever we take in an SEC filing, a transcript, etc, it's, we use vector databases. So we're converting a sentence, a word, a paragraph to an embedding. Are you familiar with? [00:12:31] S2: Yeah. [00:12:31] S0: Or a vector. Yeah. So there's lots of different ways that you can do that and you should be doing that, but if you're using Claude, your data isn't in a vector database, which is one of the reasons that it can be so expensive. And if you're using NotebookLM, it's essentially search rather than a vector database. [00:13:00] S1: Got it. [00:13:01] S0: But if you have a vector database or you have a more sophisticated backend, you can do a lot of really cool things in pre processing that facilitate both accuracy, relevance, and completeness on the front end. Got it. In addition to, to be able to, if you have these things as, as embeddings, you can give metadata to the model in addition to containing better information about what that sentence is, And you can do cool things with your embeddings. For instance, the way that we store embeddings, we have meta declarations not just about is this a KPI or not, is it forward looking historical, but also information about sentiment. [00:13:54] S1: Got it.
00:13:54 · Brett Caughran, Kris Bennatti
[00:13:54] S1: Got it. [00:13:54] S0: Which is why I believe we're the only platform where you can do sentiment based screening. So ask like who are the most stressed out CEOs? Those types of queries. Because if you're using like LM and it's just using search, how does it search for stress? There's no keyword that you can use there. But if you have embeddings that have information about sentiment, you can do really cool searches. [00:14:17] S1: This was a huge problem for the institutional finance use cases of chatbots in default 25. And in all of my experimentation going into ChatGPT or Claude, almost anything quantitative was not helpful due to this sort of accuracy, like 60, 70% benchmark. Part of the way the ecosystem is aimed to solve that is through MCP and connectors into an agentic workspace. And so now I can connect into, rather than web searching for a number, I can connect into the MCP of a Dilupa or a FactSet, etcetera, through my agentic workspace. Is that, how do you think about sort of what you're doing in sort of you've aimed to solve this problem in the way you just explained? How would you evaluate the way that the industry has tried to solve this problem with that MCP and connector movement into agents?
00:15:13 · Kris Bennatti
[00:15:13] S0: So I know the last time we talked about this, I had complained about the Claude MCP introducing hallucinations into Hudson Labs results, which had been very frustrating. And I would say I've been very, very impressed with how MCPs have evolved since. They're a lot less brittle than they used to be. I, you know, I think I think it's a good approach. Think having connectors that deal with the vectorization, the preprocessing, doing all of this back end work for you and pulling that into your own universe is a good idea. There's definitely some trade offs if you're using MCP. If you're using Hudson Labs via MCP, it's gonna be a lot less efficient. There's sort of a... One of the ways that Claude has become less brittle is by using more calls. So it so it is a much less efficient way of using Hudson Labs, but it is much better than it... I don't know the last time we talked about this, Brett, but it's it's a lot better than it was before. I mean, there's still some situations where it becomes harder to control. We don't offer a no hallucination guarantee if you are using Hudson Labs through MCP because it's not deterministic, right? It's using another model on top of ours, and sometimes it can introduce things that we didn't expect or you didn't expect. But at this point, we feel that the benefits outweigh the risks, which wasn't true a year ago.
00:17:07 · Brett Caughran, Kris Bennatti
[00:17:07] S1: Interesting. Interesting. Yeah. And that's principally like the evolution has principally been what? Just the model's getting better, the engineering around it? Like how has that brittleness problem been mitigated? [00:17:21] S0: Model is getting better, more guidance around how to set up an MCP. I would say a lot of issues right now with usage of, say, any connector, they're not... They're mostly solvable through better API documentation, better MCP documentation, facilitating more endpoints. So, it's because there's a little bit more documentation and more flexibility, it becomes easier to guide these servers to make better use of your own data. [00:18:01] S1: And I hear, and I don't fully understand this, I hear some people say like, Oh, XYZ vendor has a, their API is great, or their MCP is great, their MCP is not so good. Like what is the skill in sort of like for a neophyte, what's the process of taking your data into an MCP and where is the skill in that? How do some vendors sort of do that well? How do some vendors fail in your assessment?