Lenny's Podcast: Product | Career | Growth
00:11:52 · Brendan Foody, Lenny Rachitsky
[00:11:52] S0: Yeah, precisely. And so it really ties to your first question around the era of evals that's framing all of this, which is that the lab's primary bottleneck to being able to improve models is how they can effectively have some way of measuring what success looks like for the model, both to use it as the eval for the tests that they're measuring their progress against, as well as the verifiers in an RL environment to then reward the model, improve capabilities, etcetera. And they need this across every domain for every capability that models don't know how to use. And the wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities where Merkur is sitting at the forefront, and sort of the primary bottleneck. [00:12:43] S1: Okay. What are these people actually doing? So what's an example of a kind of person that is sought after, and then what are they doing, like sitting there at the computer?
00:12:43 · Lenny Rachitsky, Brendan Foody
[00:12:43] S1: Okay. What are these people actually doing? So what's an example of a kind of person that is sought after, and then what are they doing, like sitting there at the computer? [00:12:51] S0: Effectively, the market is bound by the amount of things where humans can do something that models can't. So I'll make that very concrete. Say you have a model that you want to write, like, a red line for a contract in the way that a lawyer would, and it makes a handful of mistakes, misses a bunch of key points in doing so. What you could do is have a lawyer create a rubric, similar to how a professor might create a rubric to create a deliverable for what are the things we want the model to be able to do, so it can effectively score that, right? Like, you know, plus however much of it identifies this, or, you know, XYZ key point. And that's really the foundation to measuring what does progress look like for models? Is this model achieving the capabilities that these professionals want? As well as how do we use this as training data to reward and to reinforce a lot of the capabilities that people want models to achieve. [00:13:50] S1: Okay. So they're essentially writing evals just to connect it back to original conversation.
00:13:50 · Lenny Rachitsky, Brendan Foody
[00:13:50] S1: Okay. So they're essentially writing evals just to connect it back to original conversation. [00:13:55] S0: Exactly. Well, that's an interesting thing is everyone talks about RL environment. I feel like the two hot button things are RL environments and evals. But one thing Andre Carvathi has tweeted out about a bunch is there's not actually a nuance in the data type. It's more just a different semantic way of describing what it's being used for. But ultimately it's just some stasis point for how do you measure what good looks like? And you can use that either as the benchmark to the sales collateral, as Sarah was saying, to say, here is why our model's the best model in the world and here's the capabilities that we've been working towards. Or you can use it on the post training side to reward certain model trajectories and achieve those capabilities. [00:14:39] S1: Okay. So say this lawyer, so this person is writing, here's what a great red line contract looks like, and here's the rubric of what excellent is. And then are they also providing data, like actual examples of Redline documents as a part of that?
00:14:39 · Lenny Rachitsky, Brendan Foody
[00:14:39] S1: Okay. So say this lawyer, so this person is writing, here's what a great red line contract looks like, and here's the rubric of what excellent is. And then are they also providing data, like actual examples of Redline documents as a part of that? [00:14:54] S0: They may. So the data landscape historically has included two kinds of data. The first is supervised fine tuning data, which is input output. When people think about fine tuning in the historical sense, that's what it is. The second is RHF, where the model will generate a couple of examples, will choose which is the most popular example. What everyone is generally moving towards is reinforcement learning from AI feedback instead of human feedback, where you have instead the human define some sort of success criteria, some way to measure that. And examples in code, it could be a unit test, right? We can scalably measure success. In other domains, it could be a rubric. And then you use that to incentivize model capabilities. And it's far more scalable and data efficient, and so that's why a lot of the broader trend in the market across the board is moving towards RLAIF to both eval models as well as improved capabilities.
00:14:54 · Brendan Foody, Lenny Rachitsky
[00:14:54] S0: They may. So the data landscape historically has included two kinds of data. The first is supervised fine tuning data, which is input output. When people think about fine tuning in the historical sense, that's what it is. The second is RHF, where the model will generate a couple of examples, will choose which is the most popular example. What everyone is generally moving towards is reinforcement learning from AI feedback instead of human feedback, where you have instead the human define some sort of success criteria, some way to measure that. And examples in code, it could be a unit test, right? We can scalably measure success. In other domains, it could be a rubric. And then you use that to incentivize model capabilities. And it's far more scalable and data efficient, and so that's why a lot of the broader trend in the market across the board is moving towards RLAIF to both eval models as well as improved capabilities. [00:15:55] S1: I had the, one of the co founders of Anthropic on, he said exactly the same thing, that's what they've done at Anthropic, is move towards AI driven reinforcement learning. So essentially, if I can understand this correctly, I'm the lay person here trying to understand this on behalf of the audience. So essentially a lawyer is like, here's what correct looks like for redlining. And then AI is just on its own almost just like, here's all the... I'm gonna try to get this. I'm gonna try to improve on this. And I know if I'm heading the right direction based on this eval slash rubric I've been given.
00:15:55 · Lenny Rachitsky, Brendan Foody
[00:15:55] S1: I had the, one of the co founders of Anthropic on, he said exactly the same thing, that's what they've done at Anthropic, is move towards AI driven reinforcement learning. So essentially, if I can understand this correctly, I'm the lay person here trying to understand this on behalf of the audience. So essentially a lawyer is like, here's what correct looks like for redlining. And then AI is just on its own almost just like, here's all the... I'm gonna try to get this. I'm gonna try to improve on this. And I know if I'm heading the right direction based on this eval slash rubric I've been given. [00:16:27] S0: Exactly. Applying all of the criteria of what good looks like, similar to how, you know, the TA might apply the professor's criteria of does the, you know, student's response meet this criteria or this criteria, plus however many points, etcetera. [00:16:41] S1: Awesome. Okay. Let me shift to talking about the broader labor market here. So there's kind of two parts to this question as we talk about this. One is just how long will we need to do this? Is there a point where we don't need? Like, you guys grew so incredibly fast. Is there a point of like, okay, we don't need humans or we're capped out? So let's start there, and then I'll ask a broader question.