The Frontier with Brendan Foody
00:35:20 · Jonathan Frankle, Brendan Foody
So the all this is to say that like it's very interesting that we've gone from this very optimistic posture, kind of as a field, as an industry, and as business in general. I feel like everybody's kind of like battening down the hatches and getting very afraid. And I hate that in some sense. Like I feel like there's an in-between we can live in where it's been an interesting spring. Like some stuff has happened. It's been hard and complicated and we've all made some mistakes. [00:35:42] S1: Yeah, I definitely feel like in the first four or five months of this year, it was all about token maxing. It was this first phase of deployment of how do we just plug the models into the context and the tools and see what they can do. And then as Athropic and OpenAI revenue skyrocketed, naturally the question became how do we optimize this? How do we find the proto-frontier of cost and performance? And how do we understand whether we actually need to be using the largest model to accomplish this task? And obviously, the foundation for all of that optimization is the Eval, which a lot of it is flowing back to. So that makes a lot of sense. But I think as models improve so much in the economy, it's also begging this important question of what role will humans play. I'm curious how you think about that and the role that humans will play in the enterprise uh over the next five to 10 years.
00:36:34 · Jonathan Frankle
[00:36:34] S0: Yeah, I think this is one of those things where people have these really oversimplified views of like all the humans will be replaced, or none of the humans will be replaced, or that, you know, entire job functions will go away. We won't have any more, I don't know, I'm picking on lawyers. I can say this because my parents are both lawyers. We'll get rid of all the lawyers or something. Not only is that like an oversimplified view that's wrong, but it's also not helpful to tell that story. Like the idea that we're going around telling that story is wrong. And like I can understand if someone says, yes, we're going to replace all the lawyers, I can understand why a lawyer might say, well, let's not have any more data centers. I think this is, you know, if you take a very extreme oversimplified position, I can understand why someone would oppose that position. I think we're actually headed for a nice Goldilocks scenario where work is complicated. The more time I spend with my customers, the more I learn that like there's no such thing as AI for finance. Finance is thousands of different individual little tasks, each of which is both complicated and like nuanced and also different at every single organization because you have all the legacy of how your business operates and how it's structured. And there may be three humans in the world, if that, who understand the details of any one task. The phrase I like to use is like the world is fractal in complexity. You look at any problem and you see the same level of complexity as you look more closely and more closely and more closely. So the idea of a finance benchmark or a biology benchmark or a medical benchmark is just to me so absurd and honestly disrespectful to the folks who are actually doing that work every day. And I think what's likely to happen is as with digital computers, as with calculators, as with the internet, work is going to change because there's a lot of stuff we don't need to do anymore. You no longer need to pull out your typewriter and typeset something, copy, edit it, and then retype it. That's gone. We no longer need to have someone who listens to audio tapes and transcribes them. That's gone. That's really boring work that nobody wants to do.
00:38:28 · Jonathan Frankle, Brendan Foody
And instead, the task of doing law no longer requires a typewriter and document preparation in the like word processing sense is no longer an issue. Instead, people are doing more law. So I think it's going to push humans both up a level, but it's also going to push humans to the boundary of tasks. If the really simple, obvious stuff in any domain is relatively automatable, that doesn't mean we no longer need you any more than I think you were just mentioning to me, like, cool, we've like agriculture has become efficient. So now 95% of humans are out of work because they used to all work on agriculture. No, we've pushed to the boundaries of more interesting, more sophisticated things. A lawyer who can spend their time on the most nuanced, challenging issues, or who can help a lot more people. We're not in a world where we have an abundance of legal counsel. How many people in the world, do you have an estate plan right now? Not yet. And you're the kind of person who should definitely have an estate plan, but it takes a while. [00:39:20] S1: Yeah. [00:39:21] S0: Like what percent of people have an estate plan, and yet we're all going to need an estate plan someday. [00:39:27] S1: Yeah. [00:39:27] S0: That is one universal human thing. Yeah. We're all going to need an estate plan someday. And I say that my dad's an estate lawyer, so this is you know near and dear to my heart. But every single person is going to need one and it's expensive and complicated. Wouldn't it be great if everyone could have one?
00:39:27 · Jonathan Frankle, Brendan Foody
[00:39:27] S0: That is one universal human thing. Yeah. We're all going to need an estate plan someday. And I say that my dad's an estate lawyer, so this is you know near and dear to my heart. But every single person is going to need one and it's expensive and complicated. Wouldn't it be great if everyone could have one? [00:39:40] S1: If we were to itemize all of your customers' problems and even the broader economy into tasks, those tasks obviously require reasoning, but more importantly, they also require context. Contextual to that business, contextual to that problem, et cetera. What percent of those tasks do you think that the models have actually automated so far?
00:39:40 · Brendan Foody, Jonathan Frankle
[00:39:40] S1: If we were to itemize all of your customers' problems and even the broader economy into tasks, those tasks obviously require reasoning, but more importantly, they also require context. Contextual to that business, contextual to that problem, et cetera. What percent of those tasks do you think that the models have actually automated so far? [00:40:00] S0: Very, very, very little. I would say we're in the single-digit percents and probably the very low single-digit percents. I think some of it may be very difficult to automate for a really long time. The fundamental breakthrough we need, in my view, is not more intelligent models, but more cost-effective e-file development. And cost effective, not in the sense of like you spend less money, but that it is for the three humans who know how to do this one tiny enterprise task, it's not miserable for them to do this. And they feel like maybe it's even delightful. And maybe it's not just an extractive task where they feel like they are getting the knowledge pulled out of them into an agent, but they are growing and learning and they become more sophisticated, what they want to do. This is kind of, you know, in some sense, the EM algorithm in AI, you know, you, you know, expectation maximization. It's a back and forth process where things get better and where, you know, the idea of building an eval is also a learning process if you structure it that way, if you can make it delightful. And to me, that is the ultimate place where we need a breakthrough right now. If we can do that, we can unleash AI and unleash humans.
00:40:00 · Jonathan Frankle, Brendan Foody
[00:40:00] S0: Very, very, very little. I would say we're in the single-digit percents and probably the very low single-digit percents. I think some of it may be very difficult to automate for a really long time. The fundamental breakthrough we need, in my view, is not more intelligent models, but more cost-effective e-file development. And cost effective, not in the sense of like you spend less money, but that it is for the three humans who know how to do this one tiny enterprise task, it's not miserable for them to do this. And they feel like maybe it's even delightful. And maybe it's not just an extractive task where they feel like they are getting the knowledge pulled out of them into an agent, but they are growing and learning and they become more sophisticated, what they want to do. This is kind of, you know, in some sense, the EM algorithm in AI, you know, you, you know, expectation maximization. It's a back and forth process where things get better and where, you know, the idea of building an eval is also a learning process if you structure it that way, if you can make it delightful. And to me, that is the ultimate place where we need a breakthrough right now. If we can do that, we can unleash AI and unleash humans. [00:40:57] S1: Yeah. [00:40:58] S0: I don't think this is mutually exclusive any more than, you know, to what you told me earlier, like making agriculture efficient did unleash humans. We had more food, we didn't need as many people to do manual labor. And instead, we built our modern world.