00:06:26 · Khe Hy
[00:06:26] S2: Do you think that, you know, there's a big push, especially in like the power user community, AI community of, you know, really good folder structures with markdown files, with metadata, with indices, kind of bypassing bypassing the very kind of heavily structured, you know, relational databases. Do you think that that works you know, that works well for an individual contributor, but does the folder structure with, you know, good markdown files and tagging, does that, can that expand beyond, you know, a small group of individuals?
00:07:04 · Matt Stockton
[00:07:04] S1: Yeah, yeah. It's an interesting topic. And basically, like, just kind of drilling out a level, right, when we talk about these models, know, they obviously have to have access to data. And the models, it turns out they're really good at exploring folder structures, like using the tools that are kind of baked into them to open folders and look through things and search files. And that's actually really it's really useful for kind of unstructured data. You know, you think about creating some taxonomy where you're organizing your data by company or by theme or something like that, and have these tools be able to search that. That's super useful. I think there's definitely still a spot for structured relational data. Like you still need databases and data warehouses, especially for, you know, the numbers. So it's kind of a mix, right? And basically, that's when you're building these tools or designing these systems, it's kind of figuring out the right mix of how structure your data and how you kind of tie these tools or these agents into that data so that they can kind of find what they need.
00:08:17 · Brett Caughran
[00:08:17] S0: One of the things I've been wondering is, if I have PDFs, PowerPoint slides, etcetera, in a folder structure, understanding how that can connect, most investors live their lives with their nose in Microsoft Excel. And I did an assessment of one of my models and it was 1,200,000 tokens. And so when I initially connected SharePoint to ChatGPT, I'm like, Oh, great. ChatGPT is going to be able to talk to all my financial models. This is amazing. And it was nowhere close, right? Is that a token issue? And how you see firms handling that Microsoft Excel issue, like extracting and speaking to you know, 17 tab, 1,200,000 token financial models?
00:09:08 · Matt Stockton
[00:09:08] S1: Yeah, yeah, know it's interesting. Admittedly, I haven't spent a ton of time connecting Excel workbooks, right? A lot of the work that I've done is kind of taking more unstructured information, like transcripts and filings and things like that. Yeah, I mean, there's definitely, you know, with these models, right, like, you have to figure out how to both preprocess the information you're giving to them and also kind of, like, segment your tasks that you're giving to them. There's this concept of a context window with these models, right, which is how many tokens the model can pay attention to. And that is clearly growing, right? There's models out there with million token context windows. But the fact is, it's like the quality degrades, the more things these models have to look at. So when I ultimately, like, if you kind of step back a level and say, Hey, I have this data that's gigantic, right? My mental model is how do I split apart the task so that I can kind of segment the task to make it a smaller, a smaller set of tasks that feed into the end result? Just kind of breaking apart the task to make it more easier for the model to accomplish each task. Right? So like, for example, one project that I I've worked on is taking these thousand page PDFs of unstructured information and having to build a process that goes into there and actually, like, distills the information, pulls out very specific aspects of that PDF, you definitely don't wanna just point Claude code at the PDF, right? Like, you want to preprocess it. You want to pull out the text. You potentially want to put that text in a vector database and and be... Have the model be able to search it. I don't think there's, like, a silver bullet, but I think my mental model is if it's not working, if it if it seems like it's not working when you use the model, how can you break apart that workflow, that thing that you're trying to accomplish into smaller pieces?
00:11:15 · Brett Caughran, Matt Stockton
[00:11:15] S0: That makes sense. And you've talked a lot about the bitter lesson of AI engineering on Twitter. Just walk through sort of your experience with the bitter lesson. Like to what degree have many of these fixes been obsoleted by the next iteration of the model? And where do you think we are on that obsolescence curve? [00:11:36] S1: Yeah, yeah, it's interesting. So it's kind of like, you know, the speed of iteration in this space is so fast that basically you kind of have to be positioned to kind of like throw away a lot of your work or repurpose it or redo it. And I guess like a couple of good examples of this in my experience is, you know, way back when we're getting these early models like GPT-four-zero, you'd have to do a lot of things to instruct the model on how to get work done. You'd have to be very specific. You'd have to overly prescribe what the model model does, right? And now the newer models, GPT-five 0.5, you don't actually want to constrain them as much, right? You actually want to let them breathe a little bit when you provide instructions. And, you know, ultimately, you keep having to reinvent things when you're building with these tools, which is hard because it's the kind of the life cycle of building is shrinking in that you kind of have to reevaluate how you're using these tools. And I think that's just going to continue to happen. You kind of have to have your, you have to be aware of the techniques that are useful for the time period in which you're building, right? And those change relatively quickly.
00:13:05 · Khe Hy
[00:13:05] S2: How do you advise folks, especially ones that don't have the in house expertise, you're mentioning things like Rag and vector databases. I know enough to be dangerous to ask one or two follow-up questions on that topic. But a full time CIO, you know, small team of PMs, they can't invest that time. And so how do you, like, how does someone from the outside know kind of the unknown unknowns to be dangerous from an infrastructure, AI infrastructure perspective, while not getting lost in the weeds, you know, besides hiring three of us?