00:33:46 · Ying Hua
[00:33:46] S0: I think it's a great question. So I think in a dashboard example, I actually, in most of the cases that we build, I don't think we're actually relying on LLM to make judgments. We basically will give a rubric flag me when these criteria hits. So it's still a human filter on what matters and what not. Obviously they're disadvantaged of that. It could be a new environment I haven't thought of. I didn't really flag that because of that. So that's I think but most of that is going to be a human, rubric that LLM flags. Judgment is interesting. I would say at the current state, given the fact that a few things, number one is I don't believe models are truly reasoning or truly understanding. There is no true sense of understanding. It's like a mimic sense of understanding or mimic sense of reasoning. Given that there is no continuous learning, meaning, anything changes doesn't get included, and given that I don't think models are trained in the right way to solve financial puzzle. And by the way, think financial puzzle is also one of the hardest ones, right? I think like compared to AlphaGo, compared to poker, like this is much much harder. Given all those limitations, I actually agree with the fact that judgment should rely on with humans. So let me give you an example how we think about it. Number one is very early on we have instructed our platform to not give subjective things unless they're asked to do so. So everything should be relied on checkable verifiable objective things. That's number one. Number two is if you think about like the knowledge layer we built. The knowledge bay we built is purely facts, things that can be verified with facts. It can be synthesis to your point, right? It can be like how credible is management? Can give you a score, but the score is backable by two stats. Like every time, you know, 30% chance they actually missed the guidance they gave. That's not a very good track record. Even though it's giving opinion but it's backable by track. So all of the things we focus on are these trackable factual things. Subjective stuff we try to stay away with. I don't personally think LLM is at a point where it can do that yet. We have some ideas on how it can get better. We're playing with some stuff. It's kind of like how human analysts start from scratch, build intuitions, right? It's a very much I form a hypothesis.
00:36:16 · Ying Hua, Brett Caughran
I test it out. I look at the stock reaction. Did I get it right? Did I get it wrong? Then learn something new. Okay, I'm try it again. I think, we can definitely build that but I don't think the model is there yet. So I would generally agree to leave the judgment to human. And the other thing is I think human is just naturally better at forming patterns, especially new patterns. I have a three year old comparing AI with a three year old. You'll see like how much better a human is better at like just connecting the dots and actually like spit out a hypothesis or like oh actually that's pretty reasonable versus AI is very much like I think too much backward looking, too much like pattern finding. So I generally agree with that sentiment. [00:37:03] S1: Yeah. Yeah. One of the conversations we've had in the past has always been sort of thorny issue for quants and sort of persists in AI is sort of this issue of materiality. Even I was sort of going through SpaceX today and my AI agent was really convinced that Starlink ARPU was the core thing that's going to matter. I'm like, well, what about the perceived terminal value of space travel and AI? Why do these systems still have native materiality limitations in your mind?
00:37:40 · Ying Hua
[00:37:40] S0: That's the domain knowledge. Like to give you an example, if you ask that same question what is most important for SPAY X? But precede it by providing a lot of context. This is what's going on in the market right now. These are the valuation of AI stocks. These are all twists that Elon said. These are all the things. If you give it enough context, will know starting is probably not the most important thing. But that context rely on domain knowledge and that's the same reason why quants, historically struggle with that too. Once have talked to quants and they can mimic, they can make a automatic do the model line by line, But they still don't know which line item matters for a given quarter. So they cannot bet every single line. They have to pick three but they don't know which one. I think again this goes back to domain knowledge. And how do you form that domain knowledge? I think that's the hardest part. Think this is the part that human analyst kind of the advantage of human analysis based on all the domain that they based on all the history they had all the domain knowledge they have a mental map of what's important what's not important. The other thing is I think to be fair, like I think it's a hard thing. I remember towards the end of my time at BAM, sometimes I would be like, hey if you give me, the press release, earnings release the day before, even if you were to do that, I don't think my hit rate on how the stock is going react is going to be 100%. Even humans have a hard time doing that. What matters? What's factored in? So I think it's a hard part. So I think that's a domain knowledge issue And it really depends on when you ask that question, what other relevant contexts that LLMN need to have a sense of what's material, what's not.
00:39:27 · Brett Caughran
[00:39:27] S1: Yeah. One other thorny issue that I see... I I sort of run into in my build out is sort of the current primacy of Excel as a sort of the investors' workbench. And I'm sort of sort of learning that Excel is not very token efficient, and I sort of would historically push back on the AI people saying Excel isn't the great, isn't sort of the best quantitative dashboard, needs to move into Python, etcetera. How do we solve Excel sort of ingress, egress, like the sort of the brain of the investment process today? How do we how do we solve that? And what's the future look like from a spreadsheet perspective?
00:40:12 · Ying Hua
[00:40:12] S0: Yeah. That's a great question. I'll give you a few kind of bulleted thoughts on some of the things you mentioned. Number one, I don't think the functionality or the advantage of Excel can be replicated by dashboards or Python completely. I think a lot of it can, like if you're doing time series you should do that in python and dashboard, should not do in excel. But if you're doing a model where it's like interactive, you can change your one assumption, you want to see how everything else flows, that should stay in excel. Like there's no better platform to do it it in Excel. So that's number one. Do you think Excel is necessary? Number two, is, I think the current methodology that most vendor out there is doing, which is let me treat Excel as a coding tool and let me just write code to write excel has natural limitations. And there are few limitations some come from the LLM side, some come from Excel side. LLM side is very easy. So let's say I want a model to remember, to update a model, right, well, Excel to... Agent to update a model. Then what agent has to do is go read a document, it has to remember all of the numbers, and then it has to remember where to put that number. Right, so there is like a remembering this, putting over here. Here becomes tricky. Language model next token slash next war prediction. When you have a stream of random numbers together, very likely it's going to miss something because it's no there's no pattern. How do you how do you do next number? I don't know if you ever run to it. So when we build, so sometimes when we do a even like a simple or like a temporary file that have a long string of random numbers words and sometimes like LLM would just miss that or mess up a letter a number and also nothing works. But that's the same issue. You're doing numbers to remember numbers to put into Excel. I think that's an issue. Then there are issues coming from limitation of Excel. So in my dream world, I think the process should be when earnings release hit someone should update all of my models. I should be focusing on reading, doing other things, whatever. But I often have companies 50 like five to 10 companies reporting at the same time. What when I have an Excel add in, what am I supposed to do? I'm supposed to like let AI take control of my computer and open 10 Excels and try to update it?
00:42:39 · Ying Hua, Brett Caughran
Like I don't think that's very practical, right? So our vision is there should be an Excel exist. One, it should be a better Excel, an AI native Excel that exists, that solves the first problem. Second thing is there should be an AI native Excel that exists that exists in the cloud, so they can metal update stuff behind, like your move away from a computer can auto update. So think about when company reports earnings, it can auto update. When company is doing call, it should be able to update your forward number and tell you, hey, at the end of call, Brett, your SM needs to move up 5% based on the consensus estimate. So it should be doing all of that in the cloud. Now you should be able to download it in Excel to play around with things. But I think that is the ultimate form. [00:43:28] S1: The last that gave me sort of the idea of, like, almost a Google Sheets. Right? Where it's like it could be it could be updated in the cloud. [00:43:36] S0: Yeah. Shared. [00:43:38] S1: Yeah. Shared. How does that architecture emerge in the Microsoft Excel ecosystem?