Invest with AI
00:57:17 · Brett Caughran, Thomas Li
[00:57:17] S1: And the labs, you know, have really built their modeling corpus toward like tuned to investment banking, right, which is a very different which a very different modeling than build a hedge fund model to forecast revenues, EBITDA, EPS. [00:57:32] S0: Yeah, and the other thing is like, if you think about it, this might be something that you and I take for granted. Getting access to hedge fund grade models, even knowing what they look like is not obvious, right? Because you can't just go onto the internet and download a model that is hedge fund grade. You can download a model that looks like a hedge fund grade model, but it's very different because what makes it a high quality model is not what it looks like. I mean, how it looks like is important, but it's how it's wired, right? It's the thought that went through the whole thing is why did he forecast seasonality versus year on year growth? Why use incremental costs? Why say the company is increasing SG and A by $10,000,000, as opposed to percentage of revenue? Like, why are each of these micro decisions being made? That's what makes a good buy side model a buy side model. Right, but very, it's not readily available on the internet. And I think most product managers don't even know, like how, what makes like, if I showed a really good product manager two models, one is a really good one, and one is an okay one, and they look identical on the formatting side, it's not clear to me that your average product person will be able to discern it without having done the job before.
00:58:47 · Brett Caughran
[00:58:47] S1: One other question I'm wrangling with is sort of you we've been on this pathway of abstraction with the models like Claude is a three from Anthropoc has talked about the system prompt going down 80%. And I'm sort of in the process of sort of again narrowing down my skills architecture. With that, I'm sort of going from like a chunking, like visibility, like many, many skills where I can see the outputs of each skill, the more verifiable to the model can now take sort of one shot. With that, it sort of feels like evals become much more critical because you have less visibility. How have you thought about building and designing evals and managing sort of skills creation process through this trend of abstraction and better model capability?
00:59:35 · Thomas Li
[00:59:35] S0: So we've about a year ago, maybe a little more than a year ago, we changed the way we think about how to run product organizations. So historically, you run product organizations by saying, Hey, here's my PRD, here's what I'm going do, here's why, here's how I'm going to build it, and then you go build it. About a little over a year ago, we changed it. We basically say, you start every PRD with a set of evals. So you ask questions. I care a lot about the orthogonality of the questions. So you don't want to ask the same questions five different times. You want to make sure every question is designed to go reflect a particular thing a customer will do. And a lot of times we actually ask our customers for these questions. So we get evals from our customers. And then everything we then do is like, as we build a product, we run through all the evals. So I actually have a whole team in my business, a very large team, their job is to do evals, right? So you're constantly running evals, you're constantly testing. Sometimes an eval takes a week and we don't have time. So we just run a small version of it. Sometimes we have to run a very, very broad scope one, we run it across all the different models, across orchestrations, across MCPs, across token costs. So I think it's a cultural thing. When we first made the shift, it was a little bit awkward. But now we're like very used to it. And I think that's just how product is going to have to be built in the future. This is like the new paradigm of thinking, just like how Amazon created a sort of like the old paradigm, like start with the output in mind, the press release, and then work backwards. Today, instead of starting with the press release, you're starting with the end eval, and you're working backwards towards that eventual, like let's say you need 100% score on all the evals, you're working towards that.
01:01:18 · Brett Caughran
[01:01:18] S1: Thomas, what's on your bingo card for the next three to nine months of AI and investing? Like where do you think we'll be, you know, by spring of twenty seven? What will have changed?
01:01:32 · Thomas Li
[01:01:32] S0: I think the biggest change is going to come in how people think about team structure. So I think we all have a good sense of, you know, the Excel problem. Hopefully, the loop is the one that gets the enterprise grade Excel solution out there. But, you know, there will be an enterprise grade Excel solution. There will be more data vendors into the space. The adoption rates will be higher. I think all of that are predictable. I think the part that people aren't spending enough time thinking about is how you hire, how you train, and how you do career laddering in finance, especially on the buy side. Because when I think about the job of an analyst, right, and this loosely tracks with like the career, you start with fact gathering, get your numbers right, get your facts right, know where the documents lie. If Amazon's reporting at 4PM, know that, you know, facts. And then it goes from facts to analysis. Right now you're a senior analyst, now you're trusted with building models that you own the model, you own the analysis, you own the pitch, to ultimately judgment, right? What factor risk am I taking? What stocks am I taking going long? What am I going short for? How long? How much leverage do I take? Right, there's a taste. I mean, obviously you get tools, but there's a huge taste element to that. We're starting to see the middle layer collapse, right, where a) well set up agents can run analysis basically at the same quality humans can. The difference is that it can run 100 times the amount of analysis a human can because it doesn't need to sleep, and I can open up a bunch of different agents to go do it. So then what that changes is the humans gets, I wouldn't say forced into, but the human now needs to spend time on how do I set up the infrastructure for the content side, because an agent is not just going to magically go out there and sign a contract for you to, you know, purchase fundamental data from Dilupa, credit card data from Consumer Edge or whatever, you still need to go do that. How do you hold your vendors accountable? How do you hold your internal people accountable to creating a high quality infrastructure? And ultimately, how do you spend more and more of your time on collecting content that cannot be easily collected? So let's say calls with IR teams, right?
01:03:44 · Thomas Li, Brett Caughran
A good agent isn't going to help you win more calls with IR teams, or talk to the right experts, like you still have to go do that work on your own. And then like, how do you process judgment? Can you create frameworks for like creating more judgments for yourself? Like, how do you increase the ability for yourself to pass judgment? How do you even know you've gone better? Right? How do you meditate on how much of a decision is emotional versus logical? Right? Has your thesis slipped? Do you think your thesis has slipped because you let it? Or because didn't know? So like, I think there is a lot more of the hiring training side that's going to go into the judgment piece, but a lot more time is going to be spent on the infrastructure piece. Whereas historically, the bulk of the time is actually spent on the analysis piece. [01:04:33] S1: Yeah. Yeah. Great. Well, thank you so much for being with us, Thomas. Any last thoughts or anything you're wrangling with right now?