Latent Space: The AI Engineer Podcast
00:11:16 · Joseph Kim
And therefore, we need to shape the security profile of that to look like a person. And so there's a few things I'll call out there. One is there's this notion of read, write, you know, even for people. Like, you're a new software engineer at a company. You oftentimes... You know, when I was at Google, you need you need something called readability. And so there's these kind of, like, checks and balances and systems in place before you can start mucking around and touching the system. And so there's already systems in place to monitor the work of people. So if mod... If models behave more like people, then you can leverage those systems, And that makes it very easy to kind of talk about and work through this with these financial institutions. So that's something we're leading into pretty heavily coming up. And the very last thing I would say is, at least in the context of banking, is where we kind of started, there's already this system in place for junior analysts doing a lot of the groundwork. Their work gets passed off to associates. Associates are reviewing the work. Then the associates are kind of adding their secret sauce, their magic, their reasoning. That gets passed off to the VP. VP goes up to the director, managing director. So there's this kind of like a hierarchical system where there's this implicit work checking that's happening that's already been built out to stop... To try to prevent against mistakes that humans make. Right? Like, humans make just as many mistakes as models, and so those systems are built there. And so we're also leaning heavily in terms of those systems as it as it pertains to building financial products.
00:12:36 · S1, Joseph Kim, S2
[00:12:36] S1: Okay. I'd say there's this, like, sort of misconception where in the finance world, one mistake really compounds like your your whole career is at risk if you submit something wrong. How should people think about getting started and using it? So if you're an analyst, how should you use these tools? And do have like any analogies towards where we're at right now? So there's like levels of autonomy and self driving. One of your slides, think said like 2027, you see no more PowerPoint. A lot of people will be using AI tools in the loop. So how should people think about where we are now and where things are going and how should they adopt these tools if they have that misconception of like one mistake my career is over. Obviously there's human in the loop, but how should they also think about it going forward as they need to offload more and more? [00:13:18] S0: Or Yeah. No. That's that's wonderful. I I have two perspectives on it, and they come... They start with different assumptions. I think I I think the notion that, like, my career is over if you make one mistake is an incorrect notion, And I have a line of thought that, like, follows that. But let me... Let's first assume that it's true. Right? And and in the case that it's true, what I would say is, like, not all mistakes are created equally [00:13:38] S2: Yeah.
00:13:38 · Joseph Kim
[00:13:38] S0: And not all model capabilities are created equally. And so when you think about the mistake that a model or a human can make that could be career ending, you know, it's it's usually once you get up to a certain level. Like, if you're a, you know, VP, the stakes are higher than when you're a junior analyst because there's a presumption of work having been checked up to that point. So if an m t... MD sees a mistake, that's a huge problem. A lot of the safety nets have been, you know, have failed for that mistake to make it up to an MD. Right? But what I would say is like, you know, the career ending mistake is something where you get a piece of information factually incorrect. And that piece of factual information changes the conclusion. Like that is obviously like the kind of the biggest type of mistake that you can make. And what I would say is we're seeing a lot of progress in in attention to detail and you know, the hallucination used to be big term that used to come up in 2022, 2023 that doesn't come up nearly as often nowadays. I would say that, like, we're starting to see huge improvements in the reliability of models as it pertains to factual mistakes. So the the kind of egregious mistake that would get you fired, models are getting much better at and continue to get much better. And so I feel actually quite confident that a combination of that continual improvements plus the existing system checks in place are... Will make it such that, you know, that type of mistake, the chance of that propagating up to an ND is just very low probability. Right? What what I would say the models are lacking today and they still need to improve is actually not the career ending mistake, but it's the it's the bit that involves human judgment and taste. So it's the... This slide looks ugly. Right? And it's it's not wrong, but it's formatted. Like, the colors look a little bit off. Something about it looks sloppy, and therefore, you know, this is not a good slide. And and that's that's not a true ending mistake in the way that in in a... Like a factual mistake is, but it still matters. It still matters in terms of how useful that work is. And so as we think about the progression in going into 2027, what I would say is I expect the model to get much better at the human taste component.
00:15:29 · Joseph Kim, S1
And that's what I was kind of alluding to in my keynote is I think the model will get much better human taste on things that are softer objective functions. It's not is it right or wrong? It's comparative. Is there something that's better than this? And I think models are gonna get much better along that dimension moving forward. How [00:15:45] S1: so? How do we solve taste? How do we train taste into the models?
00:15:48 · Joseph Kim
[00:15:48] S0: Yeah. So it's funny. Like, very early on in the, like, early RHF days, the only thing we trained models on were essentially taste. Right? Like, there there was reinforcement learning through human feedback, and the human feedback was always a pairwise signal. It was like given two options, which one's better? And so the early days of reinforcement learning were built on the notion of training on human taste. And then obviously things like verifiable rewards and environments and, you know, we built a lot of machinery to do other types of rewards and other types of RL. And so people kind of forgot that a lot... The early days of of reinforcement learning were all around human taste. And so how do you train on human taste? It's... I think the technical answer is quite simple. I think the, like, practical answer obviously is more complicated. The practical answer is you just train a reward model. Right? You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works. So, like, fun fact, like, we did this for slides. Right? Like, I mean, how do you how do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better. And humans are actually really, really good at aesthetic judgment. Right? Actually much better than models is something that we discovered. It's like a human can look at two slides and within seconds say, this one looks better. And if you have a model like a grading grading model, try to go about doing that. It's gonna spend a whole bunch of reasoning to get into this pixels this way, this thing's that, and then maybe come to the wrong conclusion. And so humans are still really valuable in making these models better. And so we rely... We will rely a lot on human preference to imbue things like text. So that's one. The last thing I'll kind of add is there's precedent for this, which makes us confident that this strategy will work. And that precedent, I would say, is coding. The models have gotten... I would say coding is not fully solved, but, like, let's say we're 80% of the way there and we're making progress on the on the last 20%. Coding is a combination of verifiable rewards and human taste. Right?
00:17:33 · Joseph Kim
And so the verifiable rewards are, you know, does a does a code compile? Are there syntax errors? Is the output of the function expected? And so there's these verifiable rewards that are in place. But a lot of good software engineering is, is this clean system design? Did you build a system that's maintainable over time? Is it easy to unit test? Is it following paradigms that the vast majority of the community or the industry thinks is the right paradigms? And if you look at it from that lens, financial... A lot of financial work looks very similar. It's the... You know, I have an Excel model. Does it sum up? Are there reference errors? There's a whole bunch of verifiable things that we can check to make sure that the work is passing this, you know, first pass sanity check. But there's also a lot of, is this the way I would have built this model? Something I learned about as I started working in finance is there's this notion of like the feel of the model, which is, you know, you you kind of perturb the inputs and you look at the outputs. And an experienced financial professional can kind of perturb the inputs, look at the outputs, and kind of understand does the data flow through this model in a way that feels good. Right? And that is a human preference that looks a lot like does this code look clean? Is it well written? And I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well.