Lenny's Podcast: Product | Career | Growth
00:13:50 · Lenny Rachitsky, Brendan Foody
[00:13:50] S1: Okay. So they're essentially writing evals just to connect it back to original conversation. [00:13:55] S0: Exactly. Well, that's an interesting thing is everyone talks about RL environment. I feel like the two hot button things are RL environments and evals. But one thing Andre Carvathi has tweeted out about a bunch is there's not actually a nuance in the data type. It's more just a different semantic way of describing what it's being used for. But ultimately it's just some stasis point for how do you measure what good looks like? And you can use that either as the benchmark to the sales collateral, as Sarah was saying, to say, here is why our model's the best model in the world and here's the capabilities that we've been working towards. Or you can use it on the post training side to reward certain model trajectories and achieve those capabilities. [00:14:39] S1: Okay. So say this lawyer, so this person is writing, here's what a great red line contract looks like, and here's the rubric of what excellent is. And then are they also providing data, like actual examples of Redline documents as a part of that?
00:14:39 · Lenny Rachitsky, Brendan Foody
[00:14:39] S1: Okay. So say this lawyer, so this person is writing, here's what a great red line contract looks like, and here's the rubric of what excellent is. And then are they also providing data, like actual examples of Redline documents as a part of that? [00:14:54] S0: They may. So the data landscape historically has included two kinds of data. The first is supervised fine tuning data, which is input output. When people think about fine tuning in the historical sense, that's what it is. The second is RHF, where the model will generate a couple of examples, will choose which is the most popular example. What everyone is generally moving towards is reinforcement learning from AI feedback instead of human feedback, where you have instead the human define some sort of success criteria, some way to measure that. And examples in code, it could be a unit test, right? We can scalably measure success. In other domains, it could be a rubric. And then you use that to incentivize model capabilities. And it's far more scalable and data efficient, and so that's why a lot of the broader trend in the market across the board is moving towards RLAIF to both eval models as well as improved capabilities.
00:14:54 · Brendan Foody, Lenny Rachitsky
[00:14:54] S0: They may. So the data landscape historically has included two kinds of data. The first is supervised fine tuning data, which is input output. When people think about fine tuning in the historical sense, that's what it is. The second is RHF, where the model will generate a couple of examples, will choose which is the most popular example. What everyone is generally moving towards is reinforcement learning from AI feedback instead of human feedback, where you have instead the human define some sort of success criteria, some way to measure that. And examples in code, it could be a unit test, right? We can scalably measure success. In other domains, it could be a rubric. And then you use that to incentivize model capabilities. And it's far more scalable and data efficient, and so that's why a lot of the broader trend in the market across the board is moving towards RLAIF to both eval models as well as improved capabilities. [00:15:55] S1: I had the, one of the co founders of Anthropic on, he said exactly the same thing, that's what they've done at Anthropic, is move towards AI driven reinforcement learning. So essentially, if I can understand this correctly, I'm the lay person here trying to understand this on behalf of the audience. So essentially a lawyer is like, here's what correct looks like for redlining. And then AI is just on its own almost just like, here's all the... I'm gonna try to get this. I'm gonna try to improve on this. And I know if I'm heading the right direction based on this eval slash rubric I've been given.
00:15:55 · Lenny Rachitsky, Brendan Foody
[00:15:55] S1: I had the, one of the co founders of Anthropic on, he said exactly the same thing, that's what they've done at Anthropic, is move towards AI driven reinforcement learning. So essentially, if I can understand this correctly, I'm the lay person here trying to understand this on behalf of the audience. So essentially a lawyer is like, here's what correct looks like for redlining. And then AI is just on its own almost just like, here's all the... I'm gonna try to get this. I'm gonna try to improve on this. And I know if I'm heading the right direction based on this eval slash rubric I've been given. [00:16:27] S0: Exactly. Applying all of the criteria of what good looks like, similar to how, you know, the TA might apply the professor's criteria of does the, you know, student's response meet this criteria or this criteria, plus however many points, etcetera. [00:16:41] S1: Awesome. Okay. Let me shift to talking about the broader labor market here. So there's kind of two parts to this question as we talk about this. One is just how long will we need to do this? Is there a point where we don't need? Like, you guys grew so incredibly fast. Is there a point of like, okay, we don't need humans or we're capped out? So let's start there, and then I'll ask a broader question.
00:17:00 · Brendan Foody
[00:17:00] S0: So the key question is how long there's going to be things in the economy that humans can do that AI can't do. And I think there's certainly a bucket of people that say, we're going to have superintelligence within three years and humans won't play a role in the economy. And that's one school of thought. Our perspective is very different. Our perspective is that these models are extraordinary and automating a lot of things very quickly, but there's a lot of things that they're horrible at. Like even still, it can't schedule time on my calendar. It can't draft emails for me. It can't use basic tools. And we need evals for everything. For everything that the models can't do, we need evals for the tool use, evals for the long horizon reasoning. Like imagine in 10 when we want models to be able to go out and build a startup for thirty days. Like we need evals for that to effectively reward it. And I think that that road to improving models, will last for as long as there is anything in the economy that humans can do, which models can't, and be a huge portion of what the future of workflow looks like. And so our mission is creating the future of work. And I think that this is a really exciting industry in giving us a glimpse into the direction that everything is headed towards.
00:18:20 · Lenny Rachitsky, Brendan Foody
[00:18:20] S1: There's this tweet that you retweeted that I want to ask you about. If you really think about it, we were put on earth to create reinforcement learning training data for labs. Yeah. What does that mean to you? What is this person implying? And it's basically what you're saying is we're just helping train models. [00:18:37] S0: It speaks to conversations I've had with a lot of researchers and executives at TOP Labs, which is that it's highly likely that the entire economy will become an RL environment machine, building out all of these worlds and contexts for us to then have rubrics or other kinds of verifiers. And that is really exciting in so many ways. Because I think let's draw analog to other revolutions where when we had the industrial revolution, everyone was freaking out about losing their jobs. But there was this whole new class of jobs of how do we build the machines? How do we have knowledge work? How do we create everything new? And I think that the narrative in AI over the last three years has almost entirely been one of job displacement. ChatGPT is growing fast and it's very cool and everyone loves using it. But from an economic standpoint, people talking a lot about job displacement. But very few companies and people have talked about this new category of jobs that's being created, and what that's going to mean and how people can prepare and upscale for that. And I think that the most exciting thing possible is creating that future of how do humans fit into the economy and how will that evolve over time.