Invest with AI
00:58:47 · Brett Caughran
[00:58:47] S1: One other question I'm wrangling with is sort of you we've been on this pathway of abstraction with the models like Claude is a three from Anthropoc has talked about the system prompt going down 80%. And I'm sort of in the process of sort of again narrowing down my skills architecture. With that, I'm sort of going from like a chunking, like visibility, like many, many skills where I can see the outputs of each skill, the more verifiable to the model can now take sort of one shot. With that, it sort of feels like evals become much more critical because you have less visibility. How have you thought about building and designing evals and managing sort of skills creation process through this trend of abstraction and better model capability?
00:59:35 · Thomas Li
[00:59:35] S0: So we've about a year ago, maybe a little more than a year ago, we changed the way we think about how to run product organizations. So historically, you run product organizations by saying, Hey, here's my PRD, here's what I'm going do, here's why, here's how I'm going to build it, and then you go build it. About a little over a year ago, we changed it. We basically say, you start every PRD with a set of evals. So you ask questions. I care a lot about the orthogonality of the questions. So you don't want to ask the same questions five different times. You want to make sure every question is designed to go reflect a particular thing a customer will do. And a lot of times we actually ask our customers for these questions. So we get evals from our customers. And then everything we then do is like, as we build a product, we run through all the evals. So I actually have a whole team in my business, a very large team, their job is to do evals, right? So you're constantly running evals, you're constantly testing. Sometimes an eval takes a week and we don't have time. So we just run a small version of it. Sometimes we have to run a very, very broad scope one, we run it across all the different models, across orchestrations, across MCPs, across token costs. So I think it's a cultural thing. When we first made the shift, it was a little bit awkward. But now we're like very used to it. And I think that's just how product is going to have to be built in the future. This is like the new paradigm of thinking, just like how Amazon created a sort of like the old paradigm, like start with the output in mind, the press release, and then work backwards. Today, instead of starting with the press release, you're starting with the end eval, and you're working backwards towards that eventual, like let's say you need 100% score on all the evals, you're working towards that.
01:01:18 · Brett Caughran
[01:01:18] S1: Thomas, what's on your bingo card for the next three to nine months of AI and investing? Like where do you think we'll be, you know, by spring of twenty seven? What will have changed?
01:01:32 · Thomas Li
[01:01:32] S0: I think the biggest change is going to come in how people think about team structure. So I think we all have a good sense of, you know, the Excel problem. Hopefully, the loop is the one that gets the enterprise grade Excel solution out there. But, you know, there will be an enterprise grade Excel solution. There will be more data vendors into the space. The adoption rates will be higher. I think all of that are predictable. I think the part that people aren't spending enough time thinking about is how you hire, how you train, and how you do career laddering in finance, especially on the buy side. Because when I think about the job of an analyst, right, and this loosely tracks with like the career, you start with fact gathering, get your numbers right, get your facts right, know where the documents lie. If Amazon's reporting at 4PM, know that, you know, facts. And then it goes from facts to analysis. Right now you're a senior analyst, now you're trusted with building models that you own the model, you own the analysis, you own the pitch, to ultimately judgment, right? What factor risk am I taking? What stocks am I taking going long? What am I going short for? How long? How much leverage do I take? Right, there's a taste. I mean, obviously you get tools, but there's a huge taste element to that. We're starting to see the middle layer collapse, right, where a) well set up agents can run analysis basically at the same quality humans can. The difference is that it can run 100 times the amount of analysis a human can because it doesn't need to sleep, and I can open up a bunch of different agents to go do it. So then what that changes is the humans gets, I wouldn't say forced into, but the human now needs to spend time on how do I set up the infrastructure for the content side, because an agent is not just going to magically go out there and sign a contract for you to, you know, purchase fundamental data from Dilupa, credit card data from Consumer Edge or whatever, you still need to go do that. How do you hold your vendors accountable? How do you hold your internal people accountable to creating a high quality infrastructure? And ultimately, how do you spend more and more of your time on collecting content that cannot be easily collected? So let's say calls with IR teams, right?
01:03:44 · Thomas Li, Brett Caughran
A good agent isn't going to help you win more calls with IR teams, or talk to the right experts, like you still have to go do that work on your own. And then like, how do you process judgment? Can you create frameworks for like creating more judgments for yourself? Like, how do you increase the ability for yourself to pass judgment? How do you even know you've gone better? Right? How do you meditate on how much of a decision is emotional versus logical? Right? Has your thesis slipped? Do you think your thesis has slipped because you let it? Or because didn't know? So like, I think there is a lot more of the hiring training side that's going to go into the judgment piece, but a lot more time is going to be spent on the infrastructure piece. Whereas historically, the bulk of the time is actually spent on the analysis piece. [01:04:33] S1: Yeah. Yeah. Great. Well, thank you so much for being with us, Thomas. Any last thoughts or anything you're wrangling with right now?
01:04:33 · Brett Caughran, Thomas Li
[01:04:33] S1: Yeah. Yeah. Great. Well, thank you so much for being with us, Thomas. Any last thoughts or anything you're wrangling with right now? [01:04:45] S0: I mean, funny thing is, this is a conversation we've had recently, the single biggest challenge for us and a lot of peers is hiring, and specifically is in hiring engineers. And if you think about it, it's ironic because a year ago, there was SaaSpocalypse, and everyone was like engineers are going be out of a job like ClockCode is able to code now. And now we're sitting here saying, well, yes, coding is a solved problem. But aligning what you're coding with the company's direction is not, right? Communicating to make sure that your evals represent your customers' problems, like that's not solved agentically. So while we agentically solve the coding problem, we did not solve the engineering problem. Right? So the engineering problem now became bigger because we can write more code, and every single tech company is desperately trying to hire more engineers, which I think is very ironic, but totally logical of like how we ended And up I wonder if that's going to be the same thing in finance.