Lenny's Podcast: Product | Career | Growth
00:35:36 · Emmanuel Ameisen
So there's this sort of, you know, trying to understand what kind of behavior we formed and trying to mold it and trying to check that we've molded the right way. And then the second part is this sort of, well, yeah, just observe the model and see if it's doing bad stuff. For the basics of that, you don't necessarily need interp. You can just like have another language model, you know, read transcripts and be like, hey, this model do anything bad here and stop it. But you can also use interpret to sort of try to get at parts that are, you know, these unverbalized parts like that, these parts that only exist in these intermediate results. Like models are able to represent stuff in their intermediate results that they don't say as we've established. And so one of the roles of Interp is to try to find, you know, try to identify cases when this happens and prevents them. And then one thing that my team works on that I think is, is along those lines is kind of a fun bit of alignment as I think of it as almost, know, being like a, maybe not a PI, but like they're like investigations or deep dives, you know. We'll see a model do something really weird and really perplexed by it. And we'll try to understand what happened. So a recent example of this is there was this model we were training and sometimes in math problems, it would go crazy. It would do some math, it would get to an answer. I think it was 12. And then it would it would immediately go back and say like, no, oh my God. The answer actually is for sure, 36. I should say 36. And they would do math again. It would get to 12. And then immediately it would say, actually the answer should be 36. And slowly it would like start descending into madness, writing in all caps, literally saying like, I'm possessed by a demon. And you're like, what is going on? Why is it losing its mind here? And if you look inside again, these intermediate results, what's going on in this example, you'll see that over the prompts that just described the math problem, like before the model even did anything, there was a huge like intermediate results activation for 36. Like the model's already obsessed with the number 36 before it's even had a chance to like understand what the problem is.
00:37:42 · Emmanuel Ameisen, Lenny Rachitsky
And what was happening is that the model was able to do math correctly, the answer correctly was 12, but it had been trained by accident on similar questions where the answer was incorrectly marked as 36. And so the model knew that the correct answer was 12, and it also had sort of been optimized to say 36. It kind of like had this pressure of saying 36 because that's what had been rewarded. And so it was sort of going haywire just because it had memorized this result. It's sort like, see this weird behavior and you look inside and you try to see if what you find inside can help you understand what's going on so that you can sort of improve the general alignment of this model and future models. [00:38:21] S0: Along those lines, there was famously the Golden Gate Clod, which was just obsessed with the Golden Gate and every answer kind of find some way to reference the Golden Gate Bridge. [00:38:30] S1: Yes, yes. Well, that one was an inter experiment, right? So that like the cool part of that one is we just found the intermediate results that maps the Golgi. And usually it's only on when the model like talks about the Golgi, but we just turned it on always. [00:38:44] S0: Always super intentional to see what would happen. [00:38:47] S1: Well, And sort of it's also one way that you convince yourself that you found the right thing, right? Like how do you know for sure that this thing we decoded that we think is the Golden Gate sign is actually the Golden Gate thing? It's like, turn it on all the ways and see if the model behaves like that. It did.
00:38:47 · Emmanuel Ameisen, Lenny Rachitsky
[00:38:47] S1: Well, And sort of it's also one way that you convince yourself that you found the right thing, right? Like how do you know for sure that this thing we decoded that we think is the Golden Gate sign is actually the Golden Gate thing? It's like, turn it on all the ways and see if the model behaves like that. It did. [00:39:00] S0: So funny. So coming back to this alignment stuff. So this is interesting. So your org is under the AI alignment org, and just thinking through how it takes to align AI to do things we want. What was the three components that you described? Honest, what are the kind of the? [00:39:17] S1: Oh, helpful, harmless, and honest. [00:39:20] S0: Helpful, harmful and harmless and honest. So there's like the train AI to be aligned. And then there's the, okay, let's watch it and see how it operates. Are those kind of the two sides of the coin of how we solve this problem? [00:39:31] S1: Yeah, it's sort of, you know, really integrated is how I would describe it, right? You can't really separate, like have one team that's just like, hey, we're the, you know, assembly line making clods and another team that's sort of like, oh, we looked at them and they're no good. You know, you kind of, there's this tight loop of we noticed some pattern that's like concerning or some behavior that could be improved. And then you sort of like immediately like change training so that you can improve on it.
00:39:31 · Emmanuel Ameisen, Lenny Rachitsky
[00:39:31] S1: Yeah, it's sort of, you know, really integrated is how I would describe it, right? You can't really separate, like have one team that's just like, hey, we're the, you know, assembly line making clods and another team that's sort of like, oh, we looked at them and they're no good. You know, you kind of, there's this tight loop of we noticed some pattern that's like concerning or some behavior that could be improved. And then you sort of like immediately like change training so that you can improve on it. [00:39:57] S0: Awesome. I feel like Anthropic gets so much flack for this that I feel like is so unfair. Anthropic's, I think official company name is an AI safety research lab, something like that. And it was always the... Like, the reason Anthropic was founded is because they were afraid that people weren't taking safety seriously enough. And then Dario was always out there. This is really dangerous. We gotta be really careful. We gotta go really slowly. And everyone's just like, no, he's just saying that for marketing reasons. He's trying to convince everyone to fund them and all these things. And now it's like, wait, shit, that's really dangerous. We gotta be really thoughtful about this stuff. And I feel like Anthropics got a lot of unfair criticism through this. And I know you guys have been thinking about this for a long time. I don't know anything there?
00:40:39 · Emmanuel Ameisen
[00:40:39] S1: A lot of the sort of actions that like certainly the alignment team takes and I think like Raleigh Anthropic takes just kind of predicated on like, take the idea that this technology is going to keep improving seriously. Just that's it. Because if you do, you know, you say and you say, you know, and let's say you rewind the clock back to 25 or something or 24. And you're like, hey, I think, you know, in a couple of years, you've good chances that Claude can just basically write code as well as like a trained software engineer. And a few years later, it can like write it better than any engineer. And there's always uncertainty with this, like when is it going to happen? How likely? But if you kind of like take that seriously, then I think it's just about doing the responsible thing condition on that worldview, right? Like, okay, well, if this is happening, you know, then we should make sure that sort of like cybersecurity concerns are addressed and things of that nature. And so actually, I kind of think of it as, like to me sometimes this debate is almost when I feel like people on the other side disagree, it's almost like, feel like, ah, but I think they don't really believe that this technology is going to continue to improve. And then it's true that it's sort like not, it's not that scary. And I think empirically so far the pace of progress has has been fast and showing no sign, no sign of slowing down. So I think that's why a lot of the safety work is important to keep doing.
00:42:07 · Lenny Rachitsky, Emmanuel Ameisen
[00:42:07] S0: What's something you think people maybe misunderstand about this AI safety stuff? Just what's going on right now? Is there something you're just like, come on, okay, here's what you need to know. Here's something you may be missing. [00:42:17] S1: This might be a bit of a sidetrack, but my take would be that a lot of the, a lot of the thing that I think people, maybe a little bit outside of the course or like AI safety effort, or that haven't been steeped in it for as long, like react really strongly to. And I think is maybe not super helpful or at least is worth thinking about more is this sort of question of anthropomorphization, you know, like, should you think of the models as like, you model model as like something that's closer to some like entity or closer to a tool? And I actually think that both are wrong. There's a sense in which all mental models are wrong, but some are useful, right? And that in terms of understanding the kind of stuff that happens and predicting the kind of risks that materialize, de improvisation is actually a useful way to talk about LLMs. And I like a few reasons why I hope you're gonna get into them, but I think that's probably my main take here. [00:43:18] S0: Yeah, please share because I know Thor Kesh famously, I don't know, famously put out this whole story of the Huggy Face incident and people criticize them for anthropomorphizing AI. So it's really interesting to hear you would think that's actually a useful way to think about it. Talk more about that.