00:32:36 · Brett Caughran, Kris Bennatti
[00:32:36] S1: Yeah. [00:32:36] S0: So if you're thinking about getting a complete lay of the land using AI search, you need a way to discover information that is related but not easily searchable. And that's true for a lot of AI style search. One reason I often point to sentiment is because it becomes really clear in that example. If you go, okay, find frustrated managers or find managers that are, show evidence of deflection. That's not something that you can type in and search for. And if you think of the AI model, how is it going to find it if you can't search for it either? And you need some way of making that information findable so that when you pull that into the context window, the model has something to work with. [00:33:35] S1: Got it. Yeah. Talk to us a little bit about, you're an accountant, the forensic accounting score, which is part of the one of the delightful capabilities of AI is to take a previously highly cumbersome process, like doing a deep forensic accounting analysis and sort of distill that into a score, which has the quantitative elements, but also the unstructured data elements. So this was sort of something that caught my attention very early on, and I've been a big fan of the Hudson risk score approach. Can you sort of just crack that open for us and walk through what you've been doing there?
00:33:35 · Brett Caughran, Kris Bennatti
[00:33:35] S1: Got it. Yeah. Talk to us a little bit about, you're an accountant, the forensic accounting score, which is part of the one of the delightful capabilities of AI is to take a previously highly cumbersome process, like doing a deep forensic accounting analysis and sort of distill that into a score, which has the quantitative elements, but also the unstructured data elements. So this was sort of something that caught my attention very early on, and I've been a big fan of the Hudson risk score approach. Can you sort of just crack that open for us and walk through what you've been doing there? [00:34:18] S0: Yeah. So, to take a quick step back, people have been trying to predict securities fraud since the beginning of time. And historically, when we used to try to do fraud projection, we would use generally things like days sales outstanding, how quickly is revenue growing, accruals, these financial metrics. Have you ever tried doing that, Brett? [00:34:51] S1: Oh, yeah.
00:34:52 · Kris Bennatti
[00:34:52] S0: Yeah. Yeah. And the problem is that it does find fraudulent companies, but it also finds a ton of just normal companies. It's a very noisy way of predicting fraud. It's the reason why when you read, you know, a Hiddenberg report or when you're pitching a short to your boss, you're not going, these sales outstanding increased by x percentage. You're saying, hey, look, the CFO quit. They're selling the product to their mom. There's millions of dollars of off balance sheet debt. The market thinks is really excited about this contract, but the contract's actually with a related party and the market doesn't know that. So those are the things that create a good short seat thesis. So we take that information, integrity of the management team, turnover at the top, you know, the number of times they're changing their segment disclosure, aggressive accounting policies, off balance sheet risk, related party transactions, dependence, governance. Fun fact, dual class governance is highly predictive of both growth and fraud. All of these elements and using LLMs, we're able to take all of these previously very qualitative elements of risk that, you know, maybe you'd be able to look at and go, Hey, that's sketchy. But we can convert that into a mathematical representation of risk. How important is this weird family related party relationship? And then use those quantifications to predict fraud. So, our model looks at all of these different types of qualitative risks and says given that they've restated once in the last three years and the CFO quits and they changed their useful life estimate, what's the likelihood they'll be subject to an SEC enforcement action or litigation related to securities fraud?
00:37:11 · Brett Caughran, Kris Bennatti
[00:37:11] S1: How have you... And part the reason I love that is sort of like heretofore, like tracking segment, you know, disclosure changes over time at scale. There was just no way to do that. All these little signals into a collective mosaic, like, yes, you get this sort of footprint of this feels like there's some obfuscation here, right? And there was no way to scale before. How do you think about like the actual algorithm of like putting that into a score? Like how do you, how have you thought about weighting those various components? [00:37:57] S0: We let machine learning do the weighting. So we have a forensic model that classifies or represents the underlying risk. And then we take those and create variables, quantitative variables, and then we run a standard ML process where we have a big database of companies that were demonstrably fraudulent, either because they had an SEC investigation or a settled class action lawsuit related to fraud, and we let the machine learning model figure out the weights, which are often not the weights that a human short seller would apply, which is somewhat interesting. [00:38:47] S1: Interesting. Long has that model been live for, and how do you track, like, efficacy? Is there any way to sort of distill that down into a batting average or hit rate or alpha, etcetera?
00:38:47 · Brett Caughran, Kris Bennatti
[00:38:47] S1: Interesting. Long has that model been live for, and how do you track, like, efficacy? Is there any way to sort of distill that down into a batting average or hit rate or alpha, etcetera? [00:38:58] S0: Yeah. So it's been a while since we've done a backtest versus share price. But I believe there's about like a 14 differential in performance. So, underperforming by about 15% for high risk companies back But when we did we've been doing this since 2019. Very popular among actuarial teams at D and O insurers. So, we do a lot of tracking based on SCA securities class actions, which is the most important fraud indicator for them. So, high risk companies are three times more likely to be subject to an SCA of any kind and every company that has a score of 70 or higher has about a one in three chance of SEC enforcement. So, it's very high precision. Yeah. And, of course, the other two out of three also likely frauds just subject to SEC enforcement. [00:40:04] S1: Yeah. Yeah. Yeah. Conversation we had one time is just how sort of natively good LLMs have been at identifying risk. And I think the sort of conclusion was there's a lot of evidence in the training corpus to sort of pattern recognize. Am remembering that conversation correctly? Like, why why why have LLMs been sort of... Why why has this been sort of a a strong native skill of of AI, the sort of risk dimension dimension? [00:40:34] S0: We've had mixed results actually with out of the box LLMs. [00:40:41] S1: Okay.
00:40:41 · Brett Caughran, Kris Bennatti
[00:40:41] S1: Okay. [00:40:41] S0: So some things that have worked well are find headwinds, those types of requests. Where we've seen issues is with more complex elements. So I've actually been somewhat disappointed with out of the box LLMs and being able to understand, say, bank's risk or off balance sheet risk in a way that's a bit more nuanced. So, yeah, I think there's still a couple gaps in understanding that personally I would have expected to be fixed by now. Yeah. I think the reason for that, though, is that there is just not a lot of good forensic information on the web to learn from. [00:41:44] S1: Okay. [00:41:44] S0: Where there is a lot of good information about general risks, like operational risks, etcetera, but there's not a ton of good forensic accounting stuff out there. Up until two years ago, until we wrote a blog post on it, if you Googled what is a critical audit matter, it would tell you the wrong thing. [00:42:03] S1: Yeah. There's sort of a weird circ too. Like when I'll search for things about buy side and AI, it'll bring my own blog post back to me. You probably... Yeah. [00:42:13] S0: Yeah. No. [00:42:14] S1: Wait, no, I want someone smarter than me telling me about it. I don't want to read my own. Yeah. Read my own blogs, but a lot of Hudson Labs stuff shows up there too. It's funny sort of getting your own documents fed back to you.