The Frontier with Brendan Foody
00:40:58 · Jonathan Frankle, Brendan Foody
[00:40:58] S0: I don't think this is mutually exclusive any more than, you know, to what you told me earlier, like making agriculture efficient did unleash humans. We had more food, we didn't need as many people to do manual labor. And instead, we built our modern world. [00:41:12] S1: What do you see that work looking like in five years? Like, do you think that people spend a lot of time building evals, updating the context of agents, uh just operating at higher levels of abstraction, managing those agents? What will that look like as we think about this category for training agents and how it develops?
00:41:29 · Jonathan Frankle
[00:41:29] S0: God, I wish I knew. Um, and I can't wait to find out. I can tell you my best guess right now, knowing that we'll laugh at this in three years, just as much as we can laugh that I said you'd never need more than eight GPUs back in 2022. But I think, you know, there will be a little bit of like all humans will become more like managers a little bit. We're all gonna have to learn the skills of like how to declaratively describe a task and then check whether the task was done well selectively, know which questions to ask, kind of almost context-free to figure out whether something was done well. We're all gonna be writing a lot of process. We're all gonna become kind of program managers in a sense. We're gonna write process and figure out how to build pipelines of agents. So we're all gonna kind of like become our own architects of teams. But I think we're all really gonna have to become critical thinkers and get pushed to the boundary of what we do. Nothing is gonna be wrote anymore because if it's wrote, we can have an agent do it, which on the one hand is a little bit scary because some of us are very comfortable with our rote task. I know how to do performance reviews really well at this point. I know how to be a good research manager. Um, and that's scary for me because I'm gonna have to be more creative every day. But it's also gonna force each of us to like really get to the hard and interesting stuff that makes our work worthwhile. In the same way that, like, if you don't have to sit down with a typewriter and typeset, or you don't have to figure out the punch card and walk to the mainframe and plug it in. You can think about building software that's really creative. I was meeting with, you know, someone who used to be high up at a bank who used to work on mergers in the 80s, and he was telling me what they were like, because I'd mentioned the punch card example. He was like, Yeah, a merger used to be a stack of punch cards. And I asked him, so how many mergers did you do a year? Three. And I asked him how sophisticated were the mergers, and he was like, they were all the same structure. What about today? And he was like, Yeah, an associate can do three mergers, and every merger is creative and interesting.
00:43:12 · Jonathan Frankle
And it's led to more dynamism in the economy because we can now conceive of transactions that wouldn't have made sense because we can handle the scale and complexity of it because we have computers. And I think that's what we're going to see is just the sophistication of the tasks. An example I love is like in the insurance world, the idea that, like, okay, in California, there are wildfires, so we will cancel all insurance policies in California for homeowners is such a 2026 thing to do. In 2030, I would hope that with AI, every single home is being like evaluated by a drone that's taking footage and looking at the environment and analyzing it multiple times a year. And every policy is custom written. And we would never cancel all the policies in California because there was a wildfire in LA. We would look at the risk specific to each home. An insurance adjuster is going to be thinking about not like rubber stamp this policy. Oh, this policy doesn't meet this one criteria. Let me follow the process. Cool, AI can do that. They'll be thinking about how do I ensure. Person's home that I could never have ensured before. Yeah. We'll push to the boundaries of work. And I don't know, to me, that's I hate using the word abundance in like I don't mean it in the political sense, and I don't mean it in the um, you know, AGI sense. I mean it in the sense of like a lot of things will be easier to create and demand will go up for them. We'll need more people to work on them. The work will be more interesting, and more people will have access to things like an estate plan at a really low cost. We'll all have access to better tax preparation, like these very boring things that we actually spend a lot of time on, but the life of an accountant will also get a lot more interesting. And I'm really excited about that world.
00:44:36 · Brendan Foody
[00:44:36] S1: Right now, creating Evals or training agents is a small fraction of a percent of the economy. How many multiples larger do you expect that to be in five years?
00:44:45 · Jonathan Frankle
[00:44:45] S0: I mean, you know, not to flatter you, but I think it's gonna get pretty big. Um, it's hard for me to say off the bat. I think these things can grow beyond, way beyond our imagination. As with everything, the way it's going to grow is that we have to reduce the cost. We have to make it easier, more accessible for everyone. The market size will grow. The cost per eval, I hope, will go down by several orders of magnitude. And the market size will go up by several more orders of magnitude, just as with any other good. The fact that digital computers have come down so much in cost has meant we have them in our pockets and our watches and now our smoke detectors. I hope that evals feel like that. Where in five years, you don't need to go to Mercore and say, here's tens of millions of dollars, please give me a really good eval for this task. You go to Mercore and you say, here's $1,000 for a license to some amazing product that I'm giving to this one person, but I've actually given it to every single employee at my company. And now I'm spending hundreds of millions or billions of dollars on Mercore and the market has grown, the technology has changed and become democratized, just as any digital technology does. I hope we get there the way that software has. It used to be that you needed to be at a university or a lab and know the right person so you could get access to the mainframe, learn how to use punch cards, and get your time on the computer, potentially in the middle of the night, because these things ran 24-7, so that you could get your program to run, find out what happened, and come back the next day to try again. That's what evals feel like today. I hope we get to the point where it feels like clawed code for building an eval. Yeah. The demand for software has gone way up because it's become really accessible. And I hope we get there with evals where the demand for evals looks like the demand for software, the market size gets big, and anyone can do this.
00:46:21 · Brendan Foody
[00:46:21] S1: If you had one final parting piece of advice for enterprises that are looking to adopt AI, what would it be?