Latent Space: The AI Engineer Podcast
00:07:07 · Joseph Kim
[00:07:07] S0: Yeah. It's a... Yeah. I like the question a lot. It it... You know, I try to allude to this in my keynote as well, but the, like, interesting thing is, you know, it's fun to talk about the post training stuff because it's unique to what we do and gets people excited. But 95% of what we built was something that absolutely a startup could do, another company could do. And we didn't really have to go leverage our ability to go and chain... Make changes in the model to go about building fantastic product. Right? And so I think it's a great call to kind of mention that What I would say is like few fold. One is most of the product that we built was built on top of the codex or I should say ChatGPT now plug in architecture and plug in primitives. And that is something that we've been investing a lot of time and energy in. You know, OpenAI is like an interesting company where, you know, we have... We kind of face the same problems internally that a startup would face if they were to build on top of OpenAI. So in very... In a lot of ways, like, team is kind of like a startup that's building on top of OpenAI technology. Right? And so even internally, we need to find scalable ways for teams like myself to stand on the shoulders of giants and build on top of Codex. Right? Codex is fantastic. It's got great memory. It's got good skills. Like, there's a... You know, it's persistent. There's a lot of things that are wonderful about Codex. But the thesis of my team was Codex was... Codex will get you about 80% of the way to a product that would be successful in the market. But, you know, the 20% is missing. And so we need to find some way for my team to go build that 20% to actually go because the other half of the thesis was if you just took Codex and launched it as a finance product, it would fail. And so that 20% is actually pivotal. So what I would say is we we wouldn't be built on top of the plug in architecture. It's using the same primitives that are available to any of the builders that wanna go build on top of ChatGPT. And, you know, I had a line in my presentation, but I I really do think that there's gonna be a billion dollar business built on top of a a ChatGPT plugin. Right? Like, it's... Like, you're gonna be able to leverage all of that horizontal functionality and just add your secret sauce on top.
00:08:57 · Joseph Kim, S1, S2
And that secret sauce might be data, it might be UI changes, it could be an integration, but, you know, that secret sauce is really valuable. And I think there's there's a billion dollars worth of value to be built [00:09:07] S1: there. So, [00:09:09] S2: yeah, it's an interesting time for the industry because we're at a a time now where orchestration is becoming a lot more automated, you know, teams of agents going out and doing things. Yep. But for finance, obviously, there are a lot of risks involved in that. So can you talk a bit about about the safety sort of that you've baked into the product?
00:09:26 · Joseph Kim
[00:09:26] S0: Yeah. Safety is super, super important. I would say that like... And it's only going to get more important because the the, you know, the agents are already getting more autonomous and more independent. And I think that trend we expect it to continue into '27 into '28. I would say for the for the product that we built and launched last month, we... There was a bunch of paradigms that already existed as it pertains to safety and, you know, being thoughtful around auditing the work, making sure that the model is not making a mistake that's gonna impact business outcomes. And we're able to lean heavily into those. And and what I would say is the existing product that we launched is a productivity tool for knowledge workers working in the space. And so the idea behind it is there's a deep assumption that there's always a human in the loop. Right? Like, so the idea there is, you know, you're a financial analyst and you have a a laundry list of things you wanna get done that day. And and just like how a software engineers do with Codex, for every one of those items, the first thing you do is try to build it with ChatGPT. Right? And it might get you 30% of the way there. It might get you 80% of way there depending on the but there's always this presumption of humans in the loop that are presiding over that work. And that made it very easy for us to to build a product where we could kind of pass off the responsibility of auditing review into people's hands and into people's responsibility. And that worked for us well and it was easy to communicate. So that was wonderful. I will say we're thinking very deeply about what comes next and and what comes next looks different than than a productivity tool that sits and and is always being monitored by a human. And I think as we think about what comes next, there's kind of two things we're thinking about. One is how do we kind of shape the security safety profile of the product into something that people understand intuitively? And some... I would say something that people understand very intuitively is how to work with other people. And so our thesis is if we make the product very similar to how you work with your your coworkers, your interns, people around you, that is something that people can pick up and learn very quickly, very easily. And we...
00:11:16 · Joseph Kim
And therefore, we need to shape the security profile of that to look like a person. And so there's a few things I'll call out there. One is there's this notion of read, write, you know, even for people. Like, you're a new software engineer at a company. You oftentimes... You know, when I was at Google, you need you need something called readability. And so there's these kind of, like, checks and balances and systems in place before you can start mucking around and touching the system. And so there's already systems in place to monitor the work of people. So if mod... If models behave more like people, then you can leverage those systems, And that makes it very easy to kind of talk about and work through this with these financial institutions. So that's something we're leading into pretty heavily coming up. And the very last thing I would say is, at least in the context of banking, is where we kind of started, there's already this system in place for junior analysts doing a lot of the groundwork. Their work gets passed off to associates. Associates are reviewing the work. Then the associates are kind of adding their secret sauce, their magic, their reasoning. That gets passed off to the VP. VP goes up to the director, managing director. So there's this kind of like a hierarchical system where there's this implicit work checking that's happening that's already been built out to stop... To try to prevent against mistakes that humans make. Right? Like, humans make just as many mistakes as models, and so those systems are built there. And so we're also leaning heavily in terms of those systems as it as it pertains to building financial products.
00:12:36 · S1, Joseph Kim, S2
[00:12:36] S1: Okay. I'd say there's this, like, sort of misconception where in the finance world, one mistake really compounds like your your whole career is at risk if you submit something wrong. How should people think about getting started and using it? So if you're an analyst, how should you use these tools? And do have like any analogies towards where we're at right now? So there's like levels of autonomy and self driving. One of your slides, think said like 2027, you see no more PowerPoint. A lot of people will be using AI tools in the loop. So how should people think about where we are now and where things are going and how should they adopt these tools if they have that misconception of like one mistake my career is over. Obviously there's human in the loop, but how should they also think about it going forward as they need to offload more and more? [00:13:18] S0: Or Yeah. No. That's that's wonderful. I I have two perspectives on it, and they come... They start with different assumptions. I think I I think the notion that, like, my career is over if you make one mistake is an incorrect notion, And I have a line of thought that, like, follows that. But let me... Let's first assume that it's true. Right? And and in the case that it's true, what I would say is, like, not all mistakes are created equally [00:13:38] S2: Yeah.
00:13:38 · Joseph Kim
[00:13:38] S0: And not all model capabilities are created equally. And so when you think about the mistake that a model or a human can make that could be career ending, you know, it's it's usually once you get up to a certain level. Like, if you're a, you know, VP, the stakes are higher than when you're a junior analyst because there's a presumption of work having been checked up to that point. So if an m t... MD sees a mistake, that's a huge problem. A lot of the safety nets have been, you know, have failed for that mistake to make it up to an MD. Right? But what I would say is like, you know, the career ending mistake is something where you get a piece of information factually incorrect. And that piece of factual information changes the conclusion. Like that is obviously like the kind of the biggest type of mistake that you can make. And what I would say is we're seeing a lot of progress in in attention to detail and you know, the hallucination used to be big term that used to come up in 2022, 2023 that doesn't come up nearly as often nowadays. I would say that, like, we're starting to see huge improvements in the reliability of models as it pertains to factual mistakes. So the the kind of egregious mistake that would get you fired, models are getting much better at and continue to get much better. And so I feel actually quite confident that a combination of that continual improvements plus the existing system checks in place are... Will make it such that, you know, that type of mistake, the chance of that propagating up to an ND is just very low probability. Right? What what I would say the models are lacking today and they still need to improve is actually not the career ending mistake, but it's the it's the bit that involves human judgment and taste. So it's the... This slide looks ugly. Right? And it's it's not wrong, but it's formatted. Like, the colors look a little bit off. Something about it looks sloppy, and therefore, you know, this is not a good slide. And and that's that's not a true ending mistake in the way that in in a... Like a factual mistake is, but it still matters. It still matters in terms of how useful that work is. And so as we think about the progression in going into 2027, what I would say is I expect the model to get much better at the human taste component.