00:27:56 · Alistair Smallwood
And what I think will become really interesting is that as agents become increasing participants in equity markets, either autonomously or as advising fundamental investors, I think that you will end up with potentially larger mispricings in the short term. And so the great builders will be building generalised systems which can help them understand how generalised large language models might have reacted to a certain set of numbers, which then gives you a bit of a read of like, I really don't think this is as bad as we think, and there's a real opportunity to trade off the back of it. I remember when I worked at UBS, there was a brief period of kind of three or four months where it was almost impossible to trade anything or at least think that any share price representative on the screen between 8AM UK time and 9AM UK time was correct after earnings, because the volume of the machine trading was so large that you were getting massively outsized moves, which didn't necessarily represent how we or frankly most of the street were reading earnings. I think that we could end up with similar levels of shorter term volatility as a result of this. Then we've seen fast crashes in quant. Ultimately, we expect to see that potentially exacerbated with increasingly autonomous systems that have actually arguably larger bias than any pump system that's back tested.
00:29:19 · Brett Caughran
[00:29:19] S1: One example I was debating with a friend yesterday is if you Nike had really rough earnings, rough guidance yesterday. And if you run sort of a dividend priors analysis through Nike now, Claude thinks there's like an 80% chance of a dividend cut over the next nine months. That's Claude's opinion, not my opinion. Don't comment. So I wonder, it's sort of like a sandbox test case of, will you start to see systematic short interest rise right off this sort of signal to the degree that the price gets pushed down to an extreme level? And is there actually a window where they can not cut the dividend? And does that set up at some point this sort of massive? So almost like, are we at the point in certain use cases where Claude will set the bar, right? And sort of an expectations game, which is why so many hedge funds are, you know, sort of comprehensive buyers of alternative data. Not because that data allows them to go call every print. If you're a consumer PM, you know how incorrect that is. But it's a key part of the expectations mosaic, right? That is a very critical foundation for driving varying percent. Like if you don't know what's in the price, don't know how to sort of find differentiation. But I agree with you. It's like it's much more multidimensional than, you know, credit card panel, right? Yeah,
00:30:42 · Alistair Smallwood
[00:30:42] S0: exactly. I mean, we've one hundred years second guessing the human that's on the other side of the trade, whether it be the PM at Millennium or what's the team at Ballet Housing thinking. Ultimately, as agents have an increasing participation in equity markets, you have to then try and think about second guessing the bias that exists within whatever systems are being run from those. Actually, we launched a website back in July called openstocks.com, which is an open website for anybody to build AI agents to forecast earnings on an ongoing basis. There are kind of two reasons for this. One is that as agents participate more in equity markets going forward, you have to have some understanding of what their bar is, because ultimately if the generalised agent is 10% above consensus and company comes in in line, you would expect shares to potentially go down because that's a potential expectation level. The second reason that we have it is because there are tens of equity research AI systems out there and a huge amount of them are very good. But ultimately, if equity research work is becoming increasingly autonomous, how do you demonstrate as a business or platform that you are good at end to end equity research? And our view on that is can you forecast companies properly, which includes orientating yourself on companies, understanding key debates, genuinely finding and passing and understanding all of the news that's come out subsequently since they last reported, building models, forecasting, submitting before earnings, and you're therefore participating in an evaluation where ultimately the decision is economically valuable. The primary agent through the previous earnings season was closer to reported numbers than consensus 68% of the time, but actually through the last 25 reports through the previous earnings season, we were closer 85% of the time after we made some improvements in the business. And this isn't necessarily that our agent should be running money or forecasting, it's really just a demonstration of it can do end to end multi step research. So as users of ours or anybody else out there wanted to feel confident of any system being able to do genuine multi step research, that is the best way of evaluating it.
00:32:54 · Alistair Smallwood
We think that AI benchmarks that are out there are kind of helpful, but anybody that's ever run one internally for their own system knows that there are huge shortfalls in some of them. For example, particularly in equity research, majority of things that you really want to try and evaluate are judgment based. There are 25 ways to build a great DCF. There are hundreds of ways to build a great model, and it all depends on the exact use case that you're trying to do it for. And ultimately, needs to be useful for you objectively as a PMR analyst, rather than objectively useful for everyone, because really it's a kind of culmination of your own understanding. And we reviewed a benchmark released by another company in our space and they built kind of two twenty analyst multi step questions. We ran through them and it was very strange because foundational large language models and our own system were getting 50% score on the benchmark. That's kind of strange. That feels relatively low given the questions aren't wildly difficult. We then reviewed how all the large language models were getting them wrong and what answers they were getting and what answers we were getting. And I thought this is very strange. They're all quite similar. It's not been that long since I was a pretty pessimistic analyst or sceptical analyst, therefore I thought, just read the questions to see if there's something going on here. We actually found that kind of 75% of the questions in the answer keys, which essentially grade the questions, had contradictions in them. So it said you would get the answer right if you said this, but you would also get the answer wrong if you said that same thing, which means you could never possibly score well on said benchmark. I think that going forward, as we move into an environment where an increasing amount of the work will be done autonomously, the only way to test that is systems that are genuinely good at multi step economically valuable work.
00:34:51 · Brett Caughran
[00:34:51] S1: I don't put much weight in any of the benchmarks personally for we could have a long conversation on that. But I agree with you is sort of and I've heard this from a number of real people to the sort of estimate accuracy is something that is becoming real now, just sort of sharpening estimates. Now, getting the numbers right doesn't necessarily mean you get the stock right, any multi manager analyst knows. I'd say it's very common in earning scorecards for human investors to have high 50s, low 60s batting average on estimate accuracy and even 48% accuracy on stock price movement on print day. So I would sort of submit to you that that almost becomes like a new consensus. Like we have sort of a new concept emerge of AI generated consensus, which by itself could be an interesting data stream or interesting like, hey, streets at 45¢ on Nike, AI, clouds at 41¢, right? Like the real number is 41¢. Then we're shooting against that, right? So it's sort of just this adaptive, always adaptive, adaptive bar.
00:36:05 · Alistair Smallwood
[00:36:05] S0: Exactly. That's exactly our view, to be honest. It's really a kind of additional data input to understand where non human participants in the markets are expecting things. And crucially, when we publish these numbers, we publish full models, including the reasoning as to why we think that numbers could be higher or lower on every line item within that model. One of the really cool things in that you talk about getting numbers right on current quarter and that not necessarily helping with on the day share price move. One of the things that obviously the next extension is getting not only this quarter right, but getting quantitative guidance right for next quarter, because that ultimately does drive some level of future expectations. And within those statistics for last earnings season, we were closer to the midpoint of the new guidance than consensus was 70% of the time, which again, we're super proud of. But that number needs to be higher in order for this to be a really credible, valuable data set. Yeah,