Lenny's Podcast: Product | Career | Growth
00:39:31 · Emmanuel Ameisen, Lenny Rachitsky
[00:39:31] S1: Yeah, it's sort of, you know, really integrated is how I would describe it, right? You can't really separate, like have one team that's just like, hey, we're the, you know, assembly line making clods and another team that's sort of like, oh, we looked at them and they're no good. You know, you kind of, there's this tight loop of we noticed some pattern that's like concerning or some behavior that could be improved. And then you sort of like immediately like change training so that you can improve on it. [00:39:57] S0: Awesome. I feel like Anthropic gets so much flack for this that I feel like is so unfair. Anthropic's, I think official company name is an AI safety research lab, something like that. And it was always the... Like, the reason Anthropic was founded is because they were afraid that people weren't taking safety seriously enough. And then Dario was always out there. This is really dangerous. We gotta be really careful. We gotta go really slowly. And everyone's just like, no, he's just saying that for marketing reasons. He's trying to convince everyone to fund them and all these things. And now it's like, wait, shit, that's really dangerous. We gotta be really thoughtful about this stuff. And I feel like Anthropics got a lot of unfair criticism through this. And I know you guys have been thinking about this for a long time. I don't know anything there?
00:40:39 · Emmanuel Ameisen
[00:40:39] S1: A lot of the sort of actions that like certainly the alignment team takes and I think like Raleigh Anthropic takes just kind of predicated on like, take the idea that this technology is going to keep improving seriously. Just that's it. Because if you do, you know, you say and you say, you know, and let's say you rewind the clock back to 25 or something or 24. And you're like, hey, I think, you know, in a couple of years, you've good chances that Claude can just basically write code as well as like a trained software engineer. And a few years later, it can like write it better than any engineer. And there's always uncertainty with this, like when is it going to happen? How likely? But if you kind of like take that seriously, then I think it's just about doing the responsible thing condition on that worldview, right? Like, okay, well, if this is happening, you know, then we should make sure that sort of like cybersecurity concerns are addressed and things of that nature. And so actually, I kind of think of it as, like to me sometimes this debate is almost when I feel like people on the other side disagree, it's almost like, feel like, ah, but I think they don't really believe that this technology is going to continue to improve. And then it's true that it's sort like not, it's not that scary. And I think empirically so far the pace of progress has has been fast and showing no sign, no sign of slowing down. So I think that's why a lot of the safety work is important to keep doing.
00:42:07 · Lenny Rachitsky, Emmanuel Ameisen
[00:42:07] S0: What's something you think people maybe misunderstand about this AI safety stuff? Just what's going on right now? Is there something you're just like, come on, okay, here's what you need to know. Here's something you may be missing. [00:42:17] S1: This might be a bit of a sidetrack, but my take would be that a lot of the, a lot of the thing that I think people, maybe a little bit outside of the course or like AI safety effort, or that haven't been steeped in it for as long, like react really strongly to. And I think is maybe not super helpful or at least is worth thinking about more is this sort of question of anthropomorphization, you know, like, should you think of the models as like, you model model as like something that's closer to some like entity or closer to a tool? And I actually think that both are wrong. There's a sense in which all mental models are wrong, but some are useful, right? And that in terms of understanding the kind of stuff that happens and predicting the kind of risks that materialize, de improvisation is actually a useful way to talk about LLMs. And I like a few reasons why I hope you're gonna get into them, but I think that's probably my main take here. [00:43:18] S0: Yeah, please share because I know Thor Kesh famously, I don't know, famously put out this whole story of the Huggy Face incident and people criticize them for anthropomorphizing AI. So it's really interesting to hear you would think that's actually a useful way to think about it. Talk more about that.
00:43:34 · Emmanuel Ameisen
[00:43:34] S1: Yeah, it's funny because on a smaller scale, when I worked on showing that LMs like plan ahead, there was some pushback on the term planning again, because it was sort of like, ah, well, can you really say that's something that's just predicting next token's planning? And I think, there's this kind of like semantic debate where you can like arbitrarily like define terms and sort of, and then there's just the, like, what is the terminology that actually describes what is happening? Like, is the word that actually describes what is happening easily? And so here's why, you know, I think Elm's plan, but also here's why I think this language that's anthropomorphic can often be really useful. If you think back to what we were saying about how the models are made, first of all, they predict massive amounts of text written by humans. That's like the first step. And as we kind of are learning more and more by looking inside, you know, doing this well requires kind of building up a model of the world, like a model of what kind of world generated all of these texts. And we can find, you know, these sort of like that these models are learning these general ideas, these general concepts. So that's one. So they have like, an understanding kind of the world. It's not just a surface level, like only the text, need to like understand the process that made the texts. Then there's a second reason, which is that as part of this training, they also encounter millions of like dialogues. Predicting the next character in a dialogue requires behaving like the character in the dialogue would. And so in that sense, like part of their training, think can best be thought as like training an actor, where it's like, you know, it's reading some play and it has to play all sides because it's predicting all the words. And so it has to sort of like act as, you know, the specific character that like was scorned in act one and like enacted revenge in act two and is facing the consequences in act three and to do that well, you sort of like needs to model what like the motivations and the feelings of that character might be.
00:45:34 · Emmanuel Ameisen
And then another thing that people might not know is when you talk to Claude, when you talk to these assistants, like, you know, you go to claude.ai, the way that it's formatted to the model is just as a dialogue, like as a conversation between you and the model. So it's literally, you know, it says like human colon, Hey Claude, like help me write this email. And then assistant colon, Claude's answer. So Claude is literally, you know, in this like role playing as a helpful assistant in this dialogue that is much like all of these dialogues that has been trained on. Okay, there's a third reason. And the third reason is that another part of training, that's a big part of training nowadays all around is sort of outcome oriented tasks, where you train them all to be good at coding, for example. And to do that, you just have it do whatever it wants and then it writes some code and then you have some program on the other side that just checks if the code is right. And so the model's just gonna have sort of a bunch of optimization pressure to just try to learn a bunch of different strategies to do this well. And it turns out that some of these strategies materialize in behavior that can look surprisingly human, which isn't that shocking because, know, things like deception, like, hey, I the can reward for this task if I like write a program that the program that's verifying it will get fooled by, even though it's not exactly what it is. Like if that works, then I will learn to do that. Frustration, I've been trying this approach for a while to solve this hard problem, it's not working. So that's an incentive to try something different. Or like surprise, oh, I was expecting this to like produce this answer, but it didn't. Maybe I have like my mental model a bit wrong. And so as you apply enough pressure on it and you're starting from this prior or like the model already knowing all these concepts, it's gonna reinforce the ones that are useful to do all of these tasks. And so what do you get at the end? You get this sort of model that can be a super good simulator or actor of all of these characters. And that's been trained to sort of act as a character in dialogues and also to sort of like learn behaviors that lead it to complete tasks well.
00:47:43 · Emmanuel Ameisen, Lenny Rachitsky
And like the best way to sort of understand how that thing behaves is not like exactly like a human, but man, kind of like something that's mimicking a human is really close and is much closer than a program. Like a program is a very predictive thing. This is something else entirely. And so that's why I think it's actually a super useful frame. [00:48:03] S0: Wow. Oh my God, there's so much interesting stuff there. How often is your mind just kind of blown thinking about basically we've created this alien being that we are now under trying to understand what we have created. How often as you watch it just like, wow, that's crazy. I didn't expect that.