Business Breakdowns
· David Plon
And then in terms of transferring domain knowledge, I have like a set of bullet points that I try to encapsulate – hre's how I think about being an analyst and some of the things you might learn through your experience. One of those is I always remind the model, like, "Look, you're going to largely be reading commentary from management teams. They are always biased positively. It's important you take a skeptical eye to anything they're saying." And even something that simple really helps because the models have been trained to want to be helpful, and that usually influences them to be more positive than they otherwise should be. So just little things like that, where you could imagine taking an eager college student who's excited and believes in the good, and trying to impart a little wisdom – "Hey, actually management teams tend to inflate things a bit." Just to wrap that up, defining the task, the context behind the task, some specific outputs if I want it, some guidelines, and some kind of domain context, I think is a really good starting point. And if you can send that to a human and they read it and understand the task, you're probably going to do pretty well with the model as well.
· Matt Reustle
[00:00:00] S0: Yeah, the level of instruction, if you almost humanize it, that was one of the initial things that I think you mentioned months and months ago that made a material change for me. And then on the idea of context and where --- and we'll use an LLM, assume off-the-shelf LLM for this question – but when you're thinking about when to just give them massive freedom versus when to load in documents. How does that change the scale and the output quality in your experience?
· David Plon, Matt Reustle
[00:00:00] S1: So I think it's very task-specific. There's a few different dimensions which it depends upon. One is, is the nature of the task something defined and structured and maybe more quantitative? In which case yeah, it can be a pain if you're using an off-the-shelf solution, but upload the documents. Because otherwise, generally speaking, an off-the-shelf model is going to have to go crawl the web, and that's an imperfect, inefficient way to gather a bunch of structured information, and you'll most likely get hallucinations. Which is why, of course, companies like Portrait exist because all that stuff's preloaded. But to the extent that the goal is something more exploratory or creative, it can still be really useful to upload the relevant documents, but I would explicitly instruct the model, "Hey, I don't really know which way this answer is going to go, so pull on threads, follow it, chase down leads." And in that case, you can steer the model – especially modern ones that are quite responsive – to do a lot of web searching. Where this matters as well is how important the task is with respect to accuracy to usefulness. There are certain tasks where if you're 99% accurate, you're 0% useful, and that's like self-driving cars. Where the cost of an error is exceptionally low. So I think of model building like, if I presented a model that had an obvious flaw in it to my boss in any of my past roles, that would be pretty bad. I certainly did that on occasion. [00:00:00] S0: That makes two of us.
· Matt Reustle, David Plon
[00:00:00] S0: That makes two of us. [00:00:00] S1: Yeah. Maybe you're just doing an industry survey – "What's the history of this industry and how has it evolved to these three players?" If there's a factual error here or there and it's earlier in the research process and it's more of a triage exercise, that's okay, I'm fine with that. Having some calibration depending on where the task is in the process, the importance of the accuracy, and how structured you want the output to be, I would vary and experiment. [00:00:00] S0: Yeah, it's quite interesting to see where things matter based on precision versus being directionally right is where all the value comes from. On your point on experimentation, I'm curious how much this matters. And this is something that I mentally struggle with sometimes as well. You mentioned the technology is constantly changing, so by experimenting or getting better with certain skill sets that eventually the models are going to be able to do on their own – how much difference do you see in investors or yourself when you experiment with different things in creative ways, when you know that there's a standard, decent-quality way that you can solve it already? I know it's a broad question just about experimentation as a skill set, but I am curious if you've noticed any patterns with the clients that you work with, or you yourself and some of your teammates.
· David Plon
[00:00:00] S1: In an evolving technology like this, I think spending some 15% of your time on experimentation is really important, because unlike past technologies – one, the underlying models are changing, but two, the capability frontier is evolving as well and is quite jagged, and there are a lot of undiscovered, for lack of a better term, capabilities that you can pick up on well before others do. And so having a process by which folks experiment and really spending some time pushing the models can be one, helpful in its own right, but two, gives you a better intuition of where the models are today, and maybe where they're going. So, to add a little specificity to that, I and we at Portrait, have plenty of tasks like this, and even me personally, I have some tasks where I know the models can't do this today. Or maybe they can do them a small percentage of the time. But anytime there's a new model that comes out, I run my suite of 10 different exercises across it. And one, I can get a feel for how much of a leap this model is, but two, I might now need to update where I think the edge of that capability frontier is. And a lot of times once you start doing that process, it kind of builds on itself, where you might have an experiment – maybe it is building a cost curve and how that's shifting. That's a type of task that can take a long time and be challenging, and now the model can start doing it. Well, great. Now you start having ideas of, "Well, I wonder if I layer in the long-term cost curve versus a marginal cost curve," and there are different things you can start coming up with for ways you could actually apply it within the research process today. So I would say for anybody – certainly companies do this, but individual investors – have a suite of 10 different things where if the models could do it, it would be great, and just constantly run these against the model. And once they are able to do it, save that template and reuse it as part of your research process when working on a name. So I think those are some of the things that where experimentation can pay a lot of dividends.
· Matt Reustle
[00:00:00] S0: I would bucket this in experimentation, but one of the things that you laid out when you talked to Brett from Fundamental Edge, was just writing prompts in different forms. So start out with the most simple prompt that you know is probably going to give you a very broad, mediocre answer. Then take it a layer down in more detail and specify more detail, and then see how the responses change based on how you're instructing it via prompts. I did it a bit, and it made me appreciate how it's sensitive to maybe the ordering of questions or if I specify the ordering of questions and what I'm loading into the context window. And it doesn't just impact that single exercise; it makes you appreciate that this is a technology that really doesn't have many buttons that tell you, "Oh, this is four-wheel drive, that I use in this condition." It is just a chat, and you have to play around with it yourself.