Latent Space: The AI Engineer Podcast
00:15:29 · Joseph Kim, S1
And that's what I was kind of alluding to in my keynote is I think the model will get much better human taste on things that are softer objective functions. It's not is it right or wrong? It's comparative. Is there something that's better than this? And I think models are gonna get much better along that dimension moving forward. How [00:15:45] S1: so? How do we solve taste? How do we train taste into the models?
00:15:48 · Joseph Kim
[00:15:48] S0: Yeah. So it's funny. Like, very early on in the, like, early RHF days, the only thing we trained models on were essentially taste. Right? Like, there there was reinforcement learning through human feedback, and the human feedback was always a pairwise signal. It was like given two options, which one's better? And so the early days of reinforcement learning were built on the notion of training on human taste. And then obviously things like verifiable rewards and environments and, you know, we built a lot of machinery to do other types of rewards and other types of RL. And so people kind of forgot that a lot... The early days of of reinforcement learning were all around human taste. And so how do you train on human taste? It's... I think the technical answer is quite simple. I think the, like, practical answer obviously is more complicated. The practical answer is you just train a reward model. Right? You just train a reward model. You get a bunch of humans to look at pairs of things and say which one is better. And this still works. So, like, fun fact, like, we did this for slides. Right? Like, I mean, how do you how do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better. And humans are actually really, really good at aesthetic judgment. Right? Actually much better than models is something that we discovered. It's like a human can look at two slides and within seconds say, this one looks better. And if you have a model like a grading grading model, try to go about doing that. It's gonna spend a whole bunch of reasoning to get into this pixels this way, this thing's that, and then maybe come to the wrong conclusion. And so humans are still really valuable in making these models better. And so we rely... We will rely a lot on human preference to imbue things like text. So that's one. The last thing I'll kind of add is there's precedent for this, which makes us confident that this strategy will work. And that precedent, I would say, is coding. The models have gotten... I would say coding is not fully solved, but, like, let's say we're 80% of the way there and we're making progress on the on the last 20%. Coding is a combination of verifiable rewards and human taste. Right?
00:17:33 · Joseph Kim
And so the verifiable rewards are, you know, does a does a code compile? Are there syntax errors? Is the output of the function expected? And so there's these verifiable rewards that are in place. But a lot of good software engineering is, is this clean system design? Did you build a system that's maintainable over time? Is it easy to unit test? Is it following paradigms that the vast majority of the community or the industry thinks is the right paradigms? And if you look at it from that lens, financial... A lot of financial work looks very similar. It's the... You know, I have an Excel model. Does it sum up? Are there reference errors? There's a whole bunch of verifiable things that we can check to make sure that the work is passing this, you know, first pass sanity check. But there's also a lot of, is this the way I would have built this model? Something I learned about as I started working in finance is there's this notion of like the feel of the model, which is, you know, you you kind of perturb the inputs and you look at the outputs. And an experienced financial professional can kind of perturb the inputs, look at the outputs, and kind of understand does the data flow through this model in a way that feels good. Right? And that is a human preference that looks a lot like does this code look clean? Is it well written? And I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well.
00:18:40 · S1, Joseph Kim, S2
[00:18:40] S1: Yeah. We can shout out Frontier Code from Cognition. So it's not only like, SWE-bench is very, you know, is the answer correct? Right. This is also would you merge the code? So you can have multiple answers that are proper PRs that pass. Right. [00:18:53] S0: But which [00:18:53] S1: one would you merge? And that's why you sometimes see the charts of lower reasoning is better, but it's a taste benchmark as well, right? [00:18:59] S2: Yeah. And one of your other 2027 predictions I thought was interesting was that, you know, the models that you're training will become much better at, you know, investment thesis and sort of generating ideas for investment and trading. But obviously, you know, that's something that humans are are experienced kind of financial professionals are very good at, and they kinda have an intuition for that kind of thing, but it's often counter to the prevailing wisdom out there. Otherwise, know, there's less of a chance of it making a great return. So how are you tackling that problem in terms of training the agents to do that kind of work?
00:19:33 · Joseph Kim
[00:19:33] S0: Yeah. I would say that like up until date, vast the majority of applications of AI and finance have been really focused on, I would say like the junior analysts work. And I think that's a... And I think that's for two reasons. One is you have to build the product based on the like capabilities of the model of today. And so if you look at the capabilities of the model today, they're really quite good at taking pretty concrete instructions of what to do and kind of executing. Right? So build me a slide, I want it to look this way, you know, make it look good, use our template, these types of things. Model's The quite capable of doing those things and so therefore we've built product experiences around that. And the other thing I would call out is there's a strong preference to make the model good at things that humans are not good at or don't like to do. Right? And so human... You know, most humans I talk to, there's obviously exceptions, but a lot of the humans I talk to like humans I talk... People I talk to, you know, they really don't like building Excel models. It's it's a tedious kind of work and it's very detail oriented. It's easy to get wrong. And similarly with slides, you know, you're you're copying logos, you're placing them very precisely on a specific part of the page. That's very tedious work that lot of people don't like to do. And so that's been part of the impetus for us to go and focus on some of these tasks. Right? It's a combination of two of those things. There's a... Like the existing molecule is pretty good at certain types of tasks and also human preferences of what they like to do and not like to do helps drive the direction of the product as well. Right? It's a combination of those two things. I think that like we've solved a lot of the low hanging from those two things. And so going into 2027, we're gonna start to move up the stack. Right? We're gonna go from the task that a GG analyst can do to some of the tasks that an associate can do to some of the tasks that an MD would do and move up that chain. And as you move up the chain, what I would say is the objectives become much fuzzier. It goes from this is clearly a good or bad slide or something that would put in front of a client or not to, well, I don't know if this investment idea is good. And it's...
00:21:19 · Joseph Kim
And it would take me, you know, three years of waiting to really be able to give you a clear concrete answer of like what the market has done. And so the objectives start to become fuzzier. And so the the way you go about training the model and the way you think about how to make the model better at those things becomes more difficult. It becomes more difficult from a research perspective. And so... But it's not impossible is what I would say. And so what it takes is a lot of research focus. It takes a lot of wonderful data. It takes a lot of feedback from experts in the field. We really wanna get the the, you know, the p 99.9 type of feedback to make the model, you know, as good as humans. But I don't think it's an insurmountable problem. And I'd... And what it takes is continued scaling the base model and focus. Focus on the research labs to think that this is an important problem to go solve. And one of the things that I'm viewing on my team is making this an area of focus and priority for OpenAI. And and I think because of the combination of those two things, I feel confident we'll make a lot of strides in 2027.