00:34:52 · Kris Bennatti
[00:34:52] S0: Yeah. Yeah. And the problem is that it does find fraudulent companies, but it also finds a ton of just normal companies. It's a very noisy way of predicting fraud. It's the reason why when you read, you know, a Hiddenberg report or when you're pitching a short to your boss, you're not going, these sales outstanding increased by x percentage. You're saying, hey, look, the CFO quit. They're selling the product to their mom. There's millions of dollars of off balance sheet debt. The market thinks is really excited about this contract, but the contract's actually with a related party and the market doesn't know that. So those are the things that create a good short seat thesis. So we take that information, integrity of the management team, turnover at the top, you know, the number of times they're changing their segment disclosure, aggressive accounting policies, off balance sheet risk, related party transactions, dependence, governance. Fun fact, dual class governance is highly predictive of both growth and fraud. All of these elements and using LLMs, we're able to take all of these previously very qualitative elements of risk that, you know, maybe you'd be able to look at and go, Hey, that's sketchy. But we can convert that into a mathematical representation of risk. How important is this weird family related party relationship? And then use those quantifications to predict fraud. So, our model looks at all of these different types of qualitative risks and says given that they've restated once in the last three years and the CFO quits and they changed their useful life estimate, what's the likelihood they'll be subject to an SEC enforcement action or litigation related to securities fraud?
00:37:11 · Brett Caughran, Kris Bennatti
[00:37:11] S1: How have you... And part the reason I love that is sort of like heretofore, like tracking segment, you know, disclosure changes over time at scale. There was just no way to do that. All these little signals into a collective mosaic, like, yes, you get this sort of footprint of this feels like there's some obfuscation here, right? And there was no way to scale before. How do you think about like the actual algorithm of like putting that into a score? Like how do you, how have you thought about weighting those various components? [00:37:57] S0: We let machine learning do the weighting. So we have a forensic model that classifies or represents the underlying risk. And then we take those and create variables, quantitative variables, and then we run a standard ML process where we have a big database of companies that were demonstrably fraudulent, either because they had an SEC investigation or a settled class action lawsuit related to fraud, and we let the machine learning model figure out the weights, which are often not the weights that a human short seller would apply, which is somewhat interesting. [00:38:47] S1: Interesting. Long has that model been live for, and how do you track, like, efficacy? Is there any way to sort of distill that down into a batting average or hit rate or alpha, etcetera?
00:38:47 · Brett Caughran, Kris Bennatti
[00:38:47] S1: Interesting. Long has that model been live for, and how do you track, like, efficacy? Is there any way to sort of distill that down into a batting average or hit rate or alpha, etcetera? [00:38:58] S0: Yeah. So it's been a while since we've done a backtest versus share price. But I believe there's about like a 14 differential in performance. So, underperforming by about 15% for high risk companies back But when we did we've been doing this since 2019. Very popular among actuarial teams at D and O insurers. So, we do a lot of tracking based on SCA securities class actions, which is the most important fraud indicator for them. So, high risk companies are three times more likely to be subject to an SCA of any kind and every company that has a score of 70 or higher has about a one in three chance of SEC enforcement. So, it's very high precision. Yeah. And, of course, the other two out of three also likely frauds just subject to SEC enforcement. [00:40:04] S1: Yeah. Yeah. Yeah. Conversation we had one time is just how sort of natively good LLMs have been at identifying risk. And I think the sort of conclusion was there's a lot of evidence in the training corpus to sort of pattern recognize. Am remembering that conversation correctly? Like, why why why have LLMs been sort of... Why why has this been sort of a a strong native skill of of AI, the sort of risk dimension dimension? [00:40:34] S0: We've had mixed results actually with out of the box LLMs. [00:40:41] S1: Okay.
00:40:41 · Brett Caughran, Kris Bennatti
[00:40:41] S1: Okay. [00:40:41] S0: So some things that have worked well are find headwinds, those types of requests. Where we've seen issues is with more complex elements. So I've actually been somewhat disappointed with out of the box LLMs and being able to understand, say, bank's risk or off balance sheet risk in a way that's a bit more nuanced. So, yeah, I think there's still a couple gaps in understanding that personally I would have expected to be fixed by now. Yeah. I think the reason for that, though, is that there is just not a lot of good forensic information on the web to learn from. [00:41:44] S1: Okay. [00:41:44] S0: Where there is a lot of good information about general risks, like operational risks, etcetera, but there's not a ton of good forensic accounting stuff out there. Up until two years ago, until we wrote a blog post on it, if you Googled what is a critical audit matter, it would tell you the wrong thing. [00:42:03] S1: Yeah. There's sort of a weird circ too. Like when I'll search for things about buy side and AI, it'll bring my own blog post back to me. You probably... Yeah. [00:42:13] S0: Yeah. No. [00:42:14] S1: Wait, no, I want someone smarter than me telling me about it. I don't want to read my own. Yeah. Read my own blogs, but a lot of Hudson Labs stuff shows up there too. It's funny sort of getting your own documents fed back to you.
00:42:14 · Brett Caughran, Kris Bennatti
[00:42:14] S1: Wait, no, I want someone smarter than me telling me about it. I don't want to read my own. Yeah. Read my own blogs, but a lot of Hudson Labs stuff shows up there too. It's funny sort of getting your own documents fed back to you. [00:42:27] S0: Honestly, the reason we created a blog post about critical audit matters is because it was bothering me because I kept reading short reports that would reference the Google result for critical audit matter, which had clearly been written by AI. There's all of these short reports going out being like, oh, you know, this company has a critical audit matter related to revenue, ergo fraud. And I'm going, you know, single software company has a critical audit matter related to revenue. This is... We need to we need to clear this up. So Oh, okay. I I I do think there is still still a few gaps in understanding much, much better than it was a year ago. But in some of the more nuanced disclosure, there's... I'm still finding some reasoning error.
00:43:23 · Brett Caughran
[00:43:23] S1: Okay. Makes sense. Yeah. What... Super, super, super interesting, I think sort of like a very low calorie way for investors to just make better decisions, like having some sort of like systematic... If I sort of think about like the agent path and the agent stack, some sort of like forensic accounting risk element in that agent stack is critical. Like if I can do this in a seamless way before I make a trading decision, having sort of a risk checklist, forensic risk score, like, there's a one in three chance of this thing being a fraud. Like, I'd like to know that before I make the trading decision instead of after I make the trading decision. Odds of restatement risk, etcetera, which those are sort of very painful eight Ks to sort of wake up to. Having the systematizing your decision dashboard just seems like an absolute no brainer to me.