Invest with AI
00:30:52 · Doug Clinton
[00:30:52] S0: Yeah. We do a ton of benchmarking internally. We've actually started to publish some of this work as well externally literally today. So it'll be a few days, I guess, maybe prior to when this gets launched. But we launched something called the IA 500, which is a nod to the S and P 500, obviously. But it's a it's a stock picking benchmark where we have a quarterly rebalance structure just like most indexes, and we test 10 of the top models. Top meaning, you know, most influential. It would be all the names that you would expect to kinda be in that group. And we look at their ability to pick a portfolio of about 200 stocks, And then we actually ensemble all of the picks from the 10 models into a single portfolio of about 500 stocks. And it's performed quite well versus the S and P 500 over the past year since we've been kind of putting this data together. So you can check it out at ia500.com. But the the kind of answer to your question, though, Brett, is we're always doing these tests on an ongoing basis to see what models sort of stand out where. And what I would tell you is this. There, in our view, there is a separation that is notable between the closed source models and the open weight models that we see on most of our benchmarks. In general, I would say when we're talking about investment tasks, not just stock picking, the the closed source models and GPT has consistently been the best one that we test in this section. They always have a little bit of outperformance as a group relative to the open weight models. So we do see some superiority there. And I know, like you said, Claude has been, probably the hottest one this year. And, this is a contrarian take, but but I do think that GPT is probably a better model overall for the stock related work that we do. It generally is our top performing model in most of the things we look at. So that's piece number one. On the open source stuff, I actually think that, to your point, is is really interesting because Kimi's had a moment. Nematron is is actually, I think, quite good as well on the American side. GLM, I know, was was hot before Kimi came out with k three. I think we're gonna see kind of the same thing there in in this open source space, which is just like the closer...
00:33:17 · Doug Clinton, Brett Caughran
Or the the closed source models, very rapidly sort of pushing the boundary up but maybe not quite hitting the closed source. That's sort of my bet is that we'll see a persistent gap between closed and open, mainly because I think a lot of it is, you know, based on distillation, how fast some of the the the open source stuff is moving, which is totally fine. But if you really wanna be on the frontier, and I think you do want that in the investment space where 1% edge is important, I think you really do have to spend more time looking at the closed source models. [00:33:53] S2: Yeah. Okay. How have you thought about this? I mean, you've spent a lot of time with Claude specific training, Claude coworking in particular, which has been sort of the default UI for, you know, mid to small size funds. Super powerful, particularly when you sort of set it up with the right connectors and the right skills. If funds have adopted Cloud Desktop, Cloud Cowork Code, etcetera, and you see this model evolution, what's next? What's the counter punch that evolution?
00:33:53 · Brett Caughran, Khe Hy, Doug Clinton
[00:33:53] S2: Yeah. Okay. How have you thought about this? I mean, you've spent a lot of time with Claude specific training, Claude coworking in particular, which has been sort of the default UI for, you know, mid to small size funds. Super powerful, particularly when you sort of set it up with the right connectors and the right skills. If funds have adopted Cloud Desktop, Cloud Cowork Code, etcetera, and you see this model evolution, what's next? What's the counter punch that evolution? [00:34:27] S1: Folks are scratching their heads right now. I think that the smaller funds that I work with are they're not ready to go to the open source. I mean, a lot of these funds use managed, you know, managed IT services. Like they don't even have a dedicated IT person in house, let alone a software engineer in house. So I think the open source question is not for the kind of the long tail of smaller funds without the technical expertise. I think that there is a huge pull into Claude. I think some folks are still using ChatGPT web because they haven't like Codex was very scary, you know, just the name intimidated people. But Codex is having its co work moment. I mean, it had it and it's very, very good. The new Codex or ChatGPT work, whatever. I don't know. I don't even know what to call it anymore. Yeah. They need to [00:35:26] S0: fix that. Got a branding problem. You're right?
00:35:28 · Khe Hy
[00:35:28] S1: They've got big branding problem. I still call it Codex just so people know what I'm talking about. And I think some folks are starting, the folks that kept their ChatGPT subscription open are starting to dabble. I've found that in the folks I talk to, intelligence, so much of the way LMs are being used is like the end to end workflow or as much of the workflow that you can capture. So yes, obviously you want the model that is 2%, 3% better. But if the harness isn't there, then that raw intelligence doesn't really matter to my clients. And so they are kind of poking around on Codex. I think that until they fix the branding problem, though, no one's really pulled the trigger because no one understands what Codex is, but it's really co work. And I think that's the problem that people haven't had. And I think that just the issue of cost, I think one of the things that's really interesting about the chat GB, the codex versus co work conversation is that, as Doug said, the 5.6 model series is very good. It's very good at agentic work. I can't speak at the nuances of financial analysis, the edges of financial analysis, but it's also significantly cheaper. And I just as a tangent, I've got a $200 Claude Max Plan that I personally that I get close to using up regularly. And I've got a $20 GPT plan. And they're generous on their resets, but it takes a long time for me to hit the $20 limit on the GPT plan. So I think that folks are trying to figure it out. I think eventually they'll get to a dual place, but that's really like an IT and a mind share question until there's like a 20% delta in performance that's not there at the moment.
00:37:32 · Brett Caughran
[00:37:32] S2: Yeah. Question for you, Doug, perspective on this two ks. Like in other areas, we've seen the sort of wrapper businesses become ascending again, like Harvey and Ligora. It sort of solves a lot of these, like, you know, model router and data wholesale aggregation, you know, commercial questions. You know, we've seen that in certain areas of finance, investment banking with Rogo really sort of ascendant. We haven't seen that yet in investing yet. Is that due to just the heterogeneity of our craft? Or do you think that's just a timing issue that we'll start to see that moment where you see some leaders, know, AlphaSense is sort of most adjacent to that, FactSet's building out an AI workspace, etcetera. Bloomberg's sort of perpetually eighteen months behind the frontier. But I'm curious if you've heard any interesting vendor stories you think we'll start to see know, sort of the right you... Your workspace being this finance specific workspace. I
00:38:36 · Doug Clinton
[00:38:36] S0: haven't heard any sort of under the radar companies or workspaces yet, but I wouldn't be surprised if somebody does try something here, and maybe it is an AlphaSense or or or somebody like that. It would sort of make sense they might try. We've talked to... And and I know, Kaye, you've talked to a ton of people in this space too. We've talked to a handful of pretty big asset managers over the past year just about, you know, other ways we could help them solve problems. And and one of the things that always is an issue or a sticking point with us working with them in any capacity is how do you get through compliance related issues? How do you get through sort of data sovereignty related issues and making sure that, you know, if they're sharing data with us in some way, our systems aren't using that data. That data's not available to any of their competitors maybe that are using the same system. And so these are all, I think, solvable problems, but I think they're they're easier to solve technically than they are to to solve organizationally, if that makes sense. And I wonder if that level of headache is the thing that's maybe holding us back more than, you know, just the reality that the technology is probably ready. It's, you know, are the institutions ready and and are very large financial institutions, which which I would argue and I hope you guys would agree are are maybe not the most aggressive as it pertains to, being innovative are just really slow to try to figure it out too.