Latent Space: The AI Engineer Podcast
00:00:52 · Joseph Kim
[00:00:52] S0: Sure. Yeah. I can give a quick thirty second intro to kind of what we just chatted about. One, I would say, like, we just released a new product ChatGPT for financial services. This is an enterprise focused product that's really trying to bring the... Mix the power of ChatGPT, kind of the tools and things that you already, you know, like to use in the consumer product, but also add in the enterprise layer. So that's firm specific templates, you know, sources like the loop, S&P, etcetera. And so really exciting. We've been working with our design partners Morgan Stanley and Evercore on that and reception to that's been incredible. So that was kind of the first half of the talk. And the second half of the talk really much more forward looking. So thinking deeply about some of the trend lines that we're seeing in terms of model improvements and drawing those trends line out to the future and trying to think a little bit about, okay, like what are these models gonna be able to do at the end of, you know, mid twenty seven, end of twenty seven? And what does that mean for folks that are building products on top of that model? And how can you kind of start to plan ahead to the intelligence that's coming in, you know, just a matter of months?
00:01:47 · S1
[00:01:47] S1: I think it's a fun time because you actually come from the research side of things. So you now lead a lot of product and engineering, but tell us what it takes to make a financial specific product. You mentioned in the keynote that there was sort of an uptick in users when you launched ChatGPT work. So Codex started to become available to the masses. People started using it for finance and now you're shipping your first finance product. Yeah. What is like financial reasoning data like? What goes into finance specific models?
00:02:14 · Joseph Kim
[00:02:14] S0: Yeah. And it's a good question. I mean, one thing I will say is that like, you know, are all of the training data that we do for financial, you know, reasoning analysis, we do upstream into what we call the big run, which is the the main kind of run that becomes, you know, something like six ASHA or six one ASHA, etcetera. And so even though we're adding a bunch of financial data to the model, it's going into the main main model. And so therefore, it's available to folks that wanna build on top of it. And that is something that's very important to us. You know, at OpenAI, we really do want that intelligence to be accessible not only to our products that we're building in house and first party ways, but also for folks that want to build startups, build businesses on top of that, as well as, you know, financial firms that wanna integrate that directly into their companies. And so we will make all that intelligence accessible there. I think you touched a little bit on product and research and I think that's like really core to what we do at OpenAI. Internally, call ourselves a research and deployment company. And so what that... And we kind of structure that in order on purpose like research comes first and then deployment comes second. And so as we think about, you know, at the end of the day trying to change the way that people do knowledge work, we kind of work in that order. Right? We think about the model capabilities, where they're headed, what kind of new things can they enable, so that's one. And then two, we think very deeply about the product features and surface areas that we should build to make that model intelligence accessible to more people. And I think you alluded to to a slide in my my keynote where, you know, we have this chart of codex adoption within OpenAI. And it's funny because you can you can see the the software engineering adoption curve and it's just like up into the right, pretty steady. People are picking it up and and incorporating sort of daily routines. But if you look at finance, legal, and sales, you know, it's pretty much at zero and suddenly it kind of jumps up to 90%. And And if you and if you look at that line and try to see what happened around the time where it jumped up, we we had released Codex desktop app, you know, in February and that was when that adoption jumped up.
00:04:04 · Joseph Kim, S2
And so I do think it kind of shows the importance of co developing those things and the the purpose of the product is to make that model intelligence accessible to as many users as possible. [00:04:13] S2: So in the, you know, that uptick of data for using codex for financial, you know, applications, what what sort of things did you learn that you then applied into the post training of financial data?
00:04:26 · Joseph Kim
[00:04:26] S0: Yeah. Mean, it's it's a good question. I would say like... So, yeah, could talk a little bit about how we went about building ChatGPT for financial services. It's it's very unique. We we haven't built a whole lot of first party products. I would say like Codex is really like our biggest and main one. And it was the first time we developed a product where the product, the model harness, and the the research was all kind of under one house and on one roof and really gives a lot of flexibility to go about building things. I think the biggest takeaway that I have from having worked on this for the last six months or so is in enterprise, it's, you know, you have to deal with the reality of the industry that you're in and go build around that. And I think that piece is... It's easy to ignore. Like if you just think about it from first base principles, you can come to a conclusion that makes sense and you talk about it and you can defend it with, you know, with logic and reason. But if you ignore the state of the world that you're in when you start building the forces that are at play within the industry to try, you know, it's either competitive forces, people have their own objectives. And if you just don't think about that, you're not gonna build the right product. Right? And so we saw that with ChatGPT for financial service and we thought about that a lot. I'll give you one example. Data. Data is like a huge part of how we thought about building the product. And, you know, and and data was one of those things where on first principles, it was a little bit tough when I was talking about both working on this product to defend because, you know, you say, hey, like I wanna go and bring quarter earnings transcripts into the product and bundle it. And the first question that... The first thing people are gonna say are is, look, there's an MCP connection in the store. Like, what is what is the incremental value that you're providing by bundling that in? On paper, it looks bad. It's like you can... They might already have a license for it, so they might feel like they're paying twice for it and they might have an MCP. And so like why are you building those? And the reality is we went and we tested all these MCPs and what we found was the error rates were were really really high.
00:06:07 · Joseph Kim, S1
Some some of the MCPs that we connected, some big names, 40% plus error rate on their MCPs. Right? And so MCPs is one of those things where it it looks great on paper, but you have to go see... Look at the data and see what's actually going on. And so for data, we're like, hey. Look. This is really bad. The only way we can go about fixing this at scale is to bring the data in house, like just control that experience end to end. And the extra money that you pay on top to bundle that data is worth it. Right? So that was a product decision that we had to make that was very industry specific, and it's been resonating with customers. Customers love it. You know, the number one thing they call out is the data, and so it worked. [00:06:40] S1: So you had good success in this. Tell us about the technical side of things for people building a harness adaptation for a specific service. Right? So you guys went out, you changed codex, you did post training, you did whatever you did to make ChatGPT for financial services. If someone was to the... To do the same for another domain, say legal sales, whatever, what are the best practices there? What did you learn? What goes into just taking off the shelf and actually, you know, making a certain product around it?