Entrepreneurial Thought Leaders (ETL)
00:16:36 · Emily Ma, Brendan Foody
[00:16:36] S1: Okay. So let's talk about Apex. I think that was a huge win, and it's not just one Apex. I know you started with investment banking and management consulting and law and whatnot, but that's branched out to include AgenTic and include SWISE and whatnot. Why is that important? Where is this all going? [00:16:54] S2: Yeah. So it actually ties back to... I remember my first conversation with Sean, we talked about IMOmedalists. And in the early days of models, everyone was talking about very academic benchmarks to set the goals for models of how could we get IMO medalist to measure Olympiad math? How could we get PhDs in GPQA? All the way to how could we get, you know, IOI medalist for, various coding evals? And then there was this big transition in the market that we saw early on moving away from academic capabilities towards economically valuable capabilities of how do we measure the things that professionals do in their real jobs, How do we create the distribution of the prompts, the context, the rubrics or success criteria corresponding to that? And measure that distribution across investment banking, law, medicine, software engineering, and all sorts of the most economically valuable domains. And so we built Apex with that purpose, and it's quickly gotten adopted by a lot of the top labs as, becoming this industry standard for how labs determine whether their models are doing well at the capabilities that enterprises care about and how enterprises make buying decisions for what model they should be using. Yeah.
00:16:54 · Brendan Foody, Emily Ma
[00:16:54] S2: Yeah. So it actually ties back to... I remember my first conversation with Sean, we talked about IMOmedalists. And in the early days of models, everyone was talking about very academic benchmarks to set the goals for models of how could we get IMO medalist to measure Olympiad math? How could we get PhDs in GPQA? All the way to how could we get, you know, IOI medalist for, various coding evals? And then there was this big transition in the market that we saw early on moving away from academic capabilities towards economically valuable capabilities of how do we measure the things that professionals do in their real jobs, How do we create the distribution of the prompts, the context, the rubrics or success criteria corresponding to that? And measure that distribution across investment banking, law, medicine, software engineering, and all sorts of the most economically valuable domains. And so we built Apex with that purpose, and it's quickly gotten adopted by a lot of the top labs as, becoming this industry standard for how labs determine whether their models are doing well at the capabilities that enterprises care about and how enterprises make buying decisions for what model they should be using. Yeah. [00:18:11] S1: How do you find, like, how do you find the authors? How do you find the right people? [00:18:17] S2: The experts that help to create these datasets. [00:18:20] S1: Yep.
00:18:20 · Emily Ma, Brendan Foody, S3
[00:18:20] S1: Yep. [00:18:21] S2: The most important is making sure, a, that they're really high caliber. Right? Of we want to get the top people from any given program or organization, and b, that they cover a representative distribution of what our customers care about. [00:18:41] S3: Okay. [00:18:42] S2: So say our customers care about being better at consulting, We would break that down into what are all the different kinds of consulting, whether it's, management consulting or operational consulting, etcetera. And then what weighting do we wanna have in each of those categories, and how do we build out the distribution that corresponds to all of that? And so it's really just, like, the bottleneck to AI labs automating everything that happens in the economy [00:19:09] S1: Yeah. [00:19:10] S2: And we can do in all of our jobs is how do they build this network of human knowledge that covers the distribution of all of the context, all of the prompts, and all of the responses that correspond to that. And so that's what we focus on doing. [00:19:25] S1: Amazing. Okay. We're gonna ask the hard question. [00:19:28] S3: Mhmm. [00:19:28] S1: Right? Once all of this expertise is, in essence, built into the large language models, what comes next? Let me put it this way. We have a lot of students here who are wondering why should they study CS, and how are they gonna find their first job or their first adventure out of college. So what would you say?
00:19:28 · Emily Ma, Brendan Foody
[00:19:28] S1: Right? Once all of this expertise is, in essence, built into the large language models, what comes next? Let me put it this way. We have a lot of students here who are wondering why should they study CS, and how are they gonna find their first job or their first adventure out of college. So what would you say? [00:19:50] S2: Well, I think that in every revolution in productivity, people have underestimated how elastic productivity is. And what I mean by that is, I think, like, as an example, as a company, if we make all of our software engineers 10 times more productive, we'll hire more software engineers. Sure. People might not just be coding. They might be doing a lot of the things adjacent to software engineering of designing architectures, of having taste in the product that we wanna use, and setting the right goalposts for the agents. But but I think that the investment in human labor to do all the things that models can't do is gonna be profound. And while I would guess that probably within a few years, agents will be able to do a significant portion, maybe even a majority of what humans do in our jobs, I think it is a multi decade process to actually achieve super intelligence where we automate absolutely everything that people can do because models inevitably will just struggle with these super long horizon challenging to verify tasks. [00:20:55] S1: Yeah. Yeah. Okay. This is a fun one. We talk about artificial general intelligence. I know there's AISI, AGI. There's many ways of going about it. When that day comes, what will you be doing?
00:20:55 · Emily Ma, Brendan Foody, S3
[00:20:55] S1: Yeah. Yeah. Okay. This is a fun one. We talk about artificial general intelligence. I know there's AISI, AGI. There's many ways of going about it. When that day comes, what will you be doing? [00:21:06] S2: Well, I think I'll be doing the same thing. But but I do think there is such a large nuance between, I guess, AGI people define usually as models being able to do most economically valuable work versus a 100% of economically valuable work. And a great proxy for this is that the Bureau of Labor Statistics analyzed that over the last two hundred twenty five years, we've made the average, person 20 times more productive. So that's equivalent to automating 95% of what you do in your job. Yet, at the same time, we have more jobs in the economy than ever before. [00:21:45] S3: That's right. [00:21:46] S2: And I think that that will continue where so long as we don't automate a 100% and we just automate 95% of what everyone does, we're not gonna run out of things to do as a society. We're gonna have so much more work. We'll build a thousand times more software. We'll build rockets. We'll cure cancer, go to Mars, etcetera. And I think there's gonna be immense human involvement in all of those processes.
00:21:46 · Brendan Foody, Emily Ma
[00:21:46] S2: And I think that that will continue where so long as we don't automate a 100% and we just automate 95% of what everyone does, we're not gonna run out of things to do as a society. We're gonna have so much more work. We'll build a thousand times more software. We'll build rockets. We'll cure cancer, go to Mars, etcetera. And I think there's gonna be immense human involvement in all of those processes. [00:22:09] S1: You know, it's funny. We're experiencing these moments. The one that comes to mind for me is always the ATM. I think the banking industry was having a moment, thinking all these tellers would lose their jobs, but actually, financial industry evolved and became much broader and stronger as a result of... I mean, ATMs were just one thing in in innovation in that field. So we're seeing that widespread now, and it's helping people through this transition. [00:22:34] S2: I totally agree. Yeah. And I think there's also enormous job creation associated with people building data centers, people training agents for deployed engineers that need to go deep with every Fortune 500 to implement agents. And so with every revolution, there's been some degree of creative destruction that happens, and ultimately, humans always have and always will, play a meaningful part of the economy. [00:23:00] S1: Let me zoom out a little bit. What is the role of policy and government in a transition like this?