00:08:53 · Aditya Agarwal, Tom McGrath
[00:08:53] S0: Which is not unlike what human... Dilemmas that humans very often face. Right? Like, that's that's a rather common, I mean, again, not to sort of, like, overanalogize to to that, but it... But that is like a... An, you know, an agent... Like, the problem with human agents. Right? [00:09:07] S1: Yeah. Like, if you get rewarded for doing something wrong, you're probably gonna do it again. Right. Particularly, if you know that the thing you would You're not gonna get caught. Know that it was wrong. Yeah. You're not gonna get caught. Yeah. So then the question is, like, how can I not get the agent to take away from this this rollout that it should do more of this kind of thing that it knows is wrong? There's some interesting interpretability research coming out of, I think both OpenAI and Anthropic have done done this on some frontier models. Like, during during these kind of reward hacking trajectories, the the model often does kind of actually know, in some sense, that it's doing wrong, and it will take away from these rollouts, like, an increased propensity towards deception. You know, it becomes more, like, generally misaligned as as a [00:09:52] S0: result of... I know I'm doing wrong, therefore, I need to hide it. [00:09:54] S1: Yeah. Well, even more than that, it's like, I know I'm doing wrong, but I got rewarded for it. Therefore, I'm just sort of generally about, like, the kind of person that gets rewarded in this world, you know. [00:10:06] S0: He's so... He just, like, justifies me. Yeah. And
00:10:06 · Aditya Agarwal, Tom McGrath
[00:10:06] S0: He's so... He just, like, justifies me. Yeah. And [00:10:14] S1: I mean, I'm I'm joking, but, like, it seems like quite quite a broad propensity. You want to, like, narrow the blast radius of this. If you can't stop the raw attacking, you want to at least, like, stop the model from updating its sort of personality towards being a generally evil guy. And this is almost inherently an act of intervening on the intervening on the, like, what gets learned. You know, it it is sort of in... Intervening on the training process itself. [00:10:42] S0: It's it's so funny that the way you're describing it, it just sounds so, human in so many way. Right? Like, it's it's very difficult not to anthropomorphize when, talking about, like, interpretability topics and... I mean, this is true of AI generally. Right? Like, surprise is trained on, you know, so much human data. But, like, I always find it actually most noticeable when talking about this sort of interpretability stuff because... I mean, it just brought to mind the the quote that, like, it's it's impossible to get someone to understand something when their salary depends on them not understanding it. Right? Like, you're kind of describing that. Right? This is a model being like, we... Well, yeah, I shouldn't do this, but my salary depends on it. You know? Like, my my reward depends on it. So, like, [00:11:26] S1: it's impossible to get a model [00:11:28] S0: Yeah. [00:11:28] S1: Not to to understand something if it's reward.
00:11:28 · Tom McGrath, Aditya Agarwal
[00:11:28] S1: Not to to understand something if it's reward. [00:11:31] S0: Yeah. So, I mean, is that sort of, like, the the dilemma that we find ourselves in? Because we haven't figured that out for people. [00:11:36] S1: Yeah. You know? But the advantage we have with models is, like, to the extent we have good interpretability, we that can actually kind of read their minds while they're doing this. [00:11:45] S0: Ah, we have... So you're saying lie detectors is the... [00:11:47] S1: That, but also, like, the ability to intervene. That, I think, is the thing that kind of separates the intentional design direction from more general, like, training run monitoring interpretability. Because you might imagine that you just sort of have the agent going around, then you put an fMRI on its head or something or some brain scanner, and you're like, oh, you know, it learned a bad thing there. Let's just kind of wind back. That seems like a pretty compelling move. But you also, like, if you have good interpretability tools that kind of respect the way that the model works, you also have the ability to kind of go and do a bit of rewiring or kind of say, oh, yes. Learn that bit. Like, you can sort of treat what is going to get taught as less like a package deal and more thing you can, like, choose from. I will learn these things, but not the deceptive part. Again, this depends on having very good interpretability. Like, probably, will depend on us having much better interpretability than we have as of mid twenty six.
00:11:47 · Tom McGrath, Aditya Agarwal
[00:11:47] S1: That, but also, like, the ability to intervene. That, I think, is the thing that kind of separates the intentional design direction from more general, like, training run monitoring interpretability. Because you might imagine that you just sort of have the agent going around, then you put an fMRI on its head or something or some brain scanner, and you're like, oh, you know, it learned a bad thing there. Let's just kind of wind back. That seems like a pretty compelling move. But you also, like, if you have good interpretability tools that kind of respect the way that the model works, you also have the ability to kind of go and do a bit of rewiring or kind of say, oh, yes. Learn that bit. Like, you can sort of treat what is going to get taught as less like a package deal and more thing you can, like, choose from. I will learn these things, but not the deceptive part. Again, this depends on having very good interpretability. Like, probably, will depend on us having much better interpretability than we have as of mid twenty six. [00:12:38] S0: Yeah. Alright. Let's talk a little bit more about that because I actually wanna keep kind of digging into this concept even more. One of the other bits of research that I found really interesting coming out of Goodfire recently is the sort of neural geometry stuff. And I I love this because when I first read some of the stuff you guys put out, it felt so intuitive to me. Again, I'm gonna... I'm at the risk of sort of, like, anticorporealizing here and [00:12:59] S1: Occupational hazard.
00:12:59 · Tom McGrath, Aditya Agarwal
[00:12:59] S1: Occupational hazard. [00:13:00] S0: Because to me, concepts seem like they have shapes. Right? And in fact, I I spent a lot of time working with founders at SPC, like, helping them with storytelling and and telling them, like, no. You know, the the the pitch is a logical argument. Right? There... It should... We... Even like the language we use, when you get something, it clicks. Yeah. Right? And so to me, there's always been, like, a shape to concepts that fit together neatly versus ones with... Which are discordant, or, there's a shape to, like, certain kind of recurring thoughts that happen. And so when I read the neural geometry stuff that actually the concepts within these language models have, like, a multidimensional manifold instead... They're not just, like, vectors. They're Yeah. They're this kind of... They're a shape. That felt kind of intuitively obvious. Like, is it obvious, or or am I just sort of, like, lucky that my my, like, intuition around similarities here with with, like, my own, you know, experience of cognition and qualia, sort of matches that?
00:13:57 · Tom McGrath, Aditya Agarwal
[00:13:57] S1: Or or were you at all surprised by the the results? I was surprised by the extent of them. I guess the question is less, are there any are there any shapes versus, like, how much seem to be shapes and, like, how intuitive and low dimensional are those shapes going to be? And I'm surprised by how how, like, they are and how much they seem to have, like, consistent patterns, like numbers love being a helix, say, or, like, things which have a sort of bound... Things like... Which have sort of a bounded range of them being reasonable, like temperature, just kind of a bit of a spirally arc or that kind of thing. Oh, even even one thing, this is crazy. [00:14:39] S0: Like, [00:14:40] S1: we we applied this to a robotics model, and one of the manifolds, there's the robe... The robot has its arm in the picture, right, sort of going around like this, and you can see that the... According to the the featurizer, like, this is... This... The hand corresponds to one sort of region in space, and the and the other bits of the arm correspond to other bits. And you can actually see, like, a little arm. Like, there's a little arm inside its representation space. And, you know, when the hand moves in the in the real world, like, essentially, the pixels corresponding to the hand move in three d space, which is just... I saw that, and I was like, this is crazy. [00:15:20] S0: Did you... Did at any point it cross your mind that, like, wow, Plato was right? Yeah. Like, man, platonic forms exist in the models. And, like Maybe I'm [00:15:36] S1: in there.