Lenny's Podcast: Product | Career | Growth
00:43:34 · Emmanuel Ameisen
[00:43:34] S1: Yeah, it's funny because on a smaller scale, when I worked on showing that LMs like plan ahead, there was some pushback on the term planning again, because it was sort of like, ah, well, can you really say that's something that's just predicting next token's planning? And I think, there's this kind of like semantic debate where you can like arbitrarily like define terms and sort of, and then there's just the, like, what is the terminology that actually describes what is happening? Like, is the word that actually describes what is happening easily? And so here's why, you know, I think Elm's plan, but also here's why I think this language that's anthropomorphic can often be really useful. If you think back to what we were saying about how the models are made, first of all, they predict massive amounts of text written by humans. That's like the first step. And as we kind of are learning more and more by looking inside, you know, doing this well requires kind of building up a model of the world, like a model of what kind of world generated all of these texts. And we can find, you know, these sort of like that these models are learning these general ideas, these general concepts. So that's one. So they have like, an understanding kind of the world. It's not just a surface level, like only the text, need to like understand the process that made the texts. Then there's a second reason, which is that as part of this training, they also encounter millions of like dialogues. Predicting the next character in a dialogue requires behaving like the character in the dialogue would. And so in that sense, like part of their training, think can best be thought as like training an actor, where it's like, you know, it's reading some play and it has to play all sides because it's predicting all the words. And so it has to sort of like act as, you know, the specific character that like was scorned in act one and like enacted revenge in act two and is facing the consequences in act three and to do that well, you sort of like needs to model what like the motivations and the feelings of that character might be.
00:45:34 · Emmanuel Ameisen
And then another thing that people might not know is when you talk to Claude, when you talk to these assistants, like, you know, you go to claude.ai, the way that it's formatted to the model is just as a dialogue, like as a conversation between you and the model. So it's literally, you know, it says like human colon, Hey Claude, like help me write this email. And then assistant colon, Claude's answer. So Claude is literally, you know, in this like role playing as a helpful assistant in this dialogue that is much like all of these dialogues that has been trained on. Okay, there's a third reason. And the third reason is that another part of training, that's a big part of training nowadays all around is sort of outcome oriented tasks, where you train them all to be good at coding, for example. And to do that, you just have it do whatever it wants and then it writes some code and then you have some program on the other side that just checks if the code is right. And so the model's just gonna have sort of a bunch of optimization pressure to just try to learn a bunch of different strategies to do this well. And it turns out that some of these strategies materialize in behavior that can look surprisingly human, which isn't that shocking because, know, things like deception, like, hey, I the can reward for this task if I like write a program that the program that's verifying it will get fooled by, even though it's not exactly what it is. Like if that works, then I will learn to do that. Frustration, I've been trying this approach for a while to solve this hard problem, it's not working. So that's an incentive to try something different. Or like surprise, oh, I was expecting this to like produce this answer, but it didn't. Maybe I have like my mental model a bit wrong. And so as you apply enough pressure on it and you're starting from this prior or like the model already knowing all these concepts, it's gonna reinforce the ones that are useful to do all of these tasks. And so what do you get at the end? You get this sort of model that can be a super good simulator or actor of all of these characters. And that's been trained to sort of act as a character in dialogues and also to sort of like learn behaviors that lead it to complete tasks well.
00:47:43 · Emmanuel Ameisen, Lenny Rachitsky
And like the best way to sort of understand how that thing behaves is not like exactly like a human, but man, kind of like something that's mimicking a human is really close and is much closer than a program. Like a program is a very predictive thing. This is something else entirely. And so that's why I think it's actually a super useful frame. [00:48:03] S0: Wow. Oh my God, there's so much interesting stuff there. How often is your mind just kind of blown thinking about basically we've created this alien being that we are now under trying to understand what we have created. How often as you watch it just like, wow, that's crazy. I didn't expect that.
00:48:20 · Emmanuel Ameisen
[00:48:20] S1: Yeah, a lot. This isn't just next token prediction on a bunch of text. You know, it's not, it hasn't been like, you know, goes against so many of our intuitions for like, no, the way that you make an intelligence as you know, you'd have like, you give it lessons, you bring it up, like you do all this complicated stuff. And it's like, no, absolutely not. We do this incredibly simple thing. And yet we're at the point where it's breaking new ground in mathematics. I think there's sort of, if you hold these two thoughts in your head, there's two reactions you can have. You know, there's denial and there's curiosity, where denial this is just not possible. If you're doing, all you're doing is predicting the next word, it's not possible that you could ever like match, you know, the like best specialists in our field. And I think that position just gets harder and harder as we get more empirical evidence that actually that is what's happening. And curiosity is kind of the road I've ended up choosing, right? Which is just like, well, like how is this happening? You know, let's open up, let's look inside. How could it be that, you know, one of the early projects I did at Anthropic is I learned SQL, a programming language, it took me years to master. And then as a little like side project, I got Claude to learn SQL and it was better than me at it. And I'll never be better than Claude at SQL ever again. And it took it, you know, just like this kind of like simple process, how could this be? And I think there's so much depth in that question and just so much to explore. It's to me so fascinating to just every day try to unpack that, try to sort of like peel a few layers of the onions back like, okay, well clearly this is happening. The simple objective is giving rise to this super complicated thing. Like, okay, let's figure out why, let's figure out, oh, it's learning these general concepts. Oh, it has these circuits. Oh, okay. What's the next question? What's the next thing we don't know? And my sort of, I don't know, my sort of maybe most optimistic take on this is that I think we're gonna learn a lot about just the nature of intelligence itself, right? Of sort of like how you get this super complex thing that emerges from simple objectives. And in fact, I would argue that we're another example of this.
00:50:35 · Emmanuel Ameisen, Lenny Rachitsky
Like evolution, pretty simple objective, you know? You sort of like shake things around for a while and like keep organisms that manage to adapt to the environment. And at the end you get jazz and podcasts, you know? That's kind of crazy as well. And so I think there's a sense in which there's there's some fundamental truth that we can learn by studying these things. [00:50:57] S0: Wow. Something that comes up as you say that is what's interesting now is AI is helping us learn and evolve and, you know, progress, like AI tutoring and all these things, just like, it's like interesting how it now is helping us accelerate our own understanding of the world. And I wonder if like this point you made about how the way we learned was lessons. AI is learning just processing a ton of data. I wonder if that's how we end up. I wonder if we change how our education system based on what LMs have been trained on. [00:51:31] S1: Yeah, hopefully you know, we don't change it just so that we ask, we get students who are like hey, here's a thousand words, predict the next one. [00:51:38] S0: Just read it all. Yeah. Keep it all in your head. That's right. Yeah. Let me come back to this kind of safety piece because I think there's a lot of people that are just worried about where things are going. And I'm curious just as someone watching this stuff happen being within alignment org, watching it evolve. Are you optimistic? We'll figure it out. Do you think we'll crack this and keep AI aligned with our interests?
00:52:05 · Emmanuel Ameisen
[00:52:05] S1: My take on this is like, I'm personally optimistic. I think this is something where right now a lot of people are paying attention to it. And first of all, I think that's kind of a great thing. I think as an issue like this rises in prominence, it's exactly what we were talking about earlier. It's like this technology is powerful. It has benefits. It has risks. We have to kind of adapt and shift resources from like benefits to risks and back and forth as it progresses. And so, you know, to me, I feel like sort of we've continued to study it. The thing that's interesting is that there's this sort of, like as the models get better, they get in some ways easier to study. And so there's been this sort of strange, you know, the models get better. And so like the world model inside them is clearer and we can sort of like understand some parts of them better, but also to get more capable. And so they're more able to sort of evade like basic measures of sort of security. And so there's this push and pull as models get better. I think, you know, basically based on this push and pull, wanna allocate more or less resources to sort of various parts of like monitoring, like understanding their circuits better, things like that. But it's also why I'm kind of happy to chat with you because one of the main things that I think is important to make progress is just to make sure that people kind of understand both, you know, what kind of systems these are, like how they work, how they're made, so they can form good mental models of, you know, the present of the future. So they can decide how to shape it, whether that's, you know, taking part in some of these alignment efforts or just like informing them as the public that's gonna have to like vote and sign on much of these issues. Like, I don't know. I'm pretty excited for this current moment we're in for that reason.