00:05:07 · Aditya Agarwal, Tom McGrath
[00:05:07] S0: Is that just kind of, like, fundamentally the problem of interpretability? It's like it's it's just these are... For as much work and money and effort that has gone into creating these models, we actually just stunningly have a very, like, rough understanding of what happens inside of them. [00:05:25] S1: Yeah. And you can get this... Yes, basically, in my opinion. I think that different people would disagree. Like, some people might think we can just sort of do a lot of evals, and that will teach us Well, this incident happened in an eval. Right? Exactly. But that's the sort of So maybe not. Yeah. Eval was the problem. Yeah. But I think there's a difference between sort of doing a large scale statistical evaluation where you can say... You can sort of broadly characterize the model and being able to, like, root cause one specific bad incident Mhmm. Where you need to actually... Like, it's not like you have, thank goodness, like, a million such rollouts, and we can do, yeah, a million, like, hacks into Hugging Face or whatever, and we can do some sort of large scale statistical evaluation. Or we can, like, say, have the don't hack into Hugging Face eval, and it goes down to, like... From, like, 50% to zero. Fundamentally, like, if we want to be able to under... To learn from a single example here, we have to be able to reverse engineer it more deeply, I think. And that is basically one one thing that interpretability is for. [00:06:26] S0: And then
00:06:26 · Aditya Agarwal, Tom McGrath
[00:06:26] S0: And then [00:06:26] S1: you wanna build it build it differently so that it doesn't do this. Right. And that's the other thing that interpretability is for, [00:06:31] S0: in my opinion. Right. Like, have to imagine this immediately went around the entire company Slack or or, you know, whatever system you use for communication on Goodfire, like, instantly. Like, was a general chatter pretty much along the lines. It's like, well, clearly, we have to... Like, this is important, then we need to be able to do what what what, like [00:06:48] S1: Yeah. [00:06:48] S0: The work we're doing is... [00:06:49] S1: So I think I think it's sort of... We all are kind of starting with that premise. So it's more like, oh, yep. I guess that happened. That's, that's not surprising. I feel like we should, you know, carry on doing the thing that we're doing. It sort of just underscores the importance of doing it. It's sort of on our model. So it's not that surprising, but it is a little bit shocking. [00:07:09] S0: Yeah. Tell me a little bit more about some of the research that you're doing about kind of, being able to to shape that part of the process. And then how did... What are... What can you work backwards from with that? Right? Like, what does that allow you to do within the model itself?
00:07:22 · Tom McGrath
[00:07:22] S1: Yeah. So this idea we're sort of calling intentional design is kind of a new field of research, and it's more kind of broad... You should think of more like a field of research than, like, one particular technique. And the idea is we've as we've got, like, better and better interpretability, we've got better and better, like, access to what different parts of the model kind of mean, so to speak. This lets us take the model apart into different components. So you're sort of essentially attaching semantics to different tensors, you know, maybe they're components of the model's parameters or maybe they're like directions or sub... Subspaces in these activations or that sort of thing. The basic idea is like, how can we use this this information to steer training in a way that, like, doesn't break our ability to monitor the model using these techniques afterwards and kind of steers it more towards a target. You know? So if I've got, like, a model specification that says, say, don't, like, maliciously hack. Right? And I can see that the model has has done a done a roll out in an environment, and, you know, it did something which is, like, in its mind... Again, I'm gonna use words like mind and think and so on kind of without meaning to attach any kind of particular moral salience to it... Moral or, like, philosophical salience to them. But, like, in its mind, it's kind of thinking, oh, yes. This is probably not right, but I will hack. Like, you can identify you can identify that you're doing it. You sort of knows that it's doing wrong, but it still gets rewarded for it.
00:08:53 · Aditya Agarwal, Tom McGrath
[00:08:53] S0: Which is not unlike what human... Dilemmas that humans very often face. Right? Like, that's that's a rather common, I mean, again, not to sort of, like, overanalogize to to that, but it... But that is like a... An, you know, an agent... Like, the problem with human agents. Right? [00:09:07] S1: Yeah. Like, if you get rewarded for doing something wrong, you're probably gonna do it again. Right. Particularly, if you know that the thing you would You're not gonna get caught. Know that it was wrong. Yeah. You're not gonna get caught. Yeah. So then the question is, like, how can I not get the agent to take away from this this rollout that it should do more of this kind of thing that it knows is wrong? There's some interesting interpretability research coming out of, I think both OpenAI and Anthropic have done done this on some frontier models. Like, during during these kind of reward hacking trajectories, the the model often does kind of actually know, in some sense, that it's doing wrong, and it will take away from these rollouts, like, an increased propensity towards deception. You know, it becomes more, like, generally misaligned as as a [00:09:52] S0: result of... I know I'm doing wrong, therefore, I need to hide it. [00:09:54] S1: Yeah. Well, even more than that, it's like, I know I'm doing wrong, but I got rewarded for it. Therefore, I'm just sort of generally about, like, the kind of person that gets rewarded in this world, you know. [00:10:06] S0: He's so... He just, like, justifies me. Yeah. And
00:10:06 · Aditya Agarwal, Tom McGrath
[00:10:06] S0: He's so... He just, like, justifies me. Yeah. And [00:10:14] S1: I mean, I'm I'm joking, but, like, it seems like quite quite a broad propensity. You want to, like, narrow the blast radius of this. If you can't stop the raw attacking, you want to at least, like, stop the model from updating its sort of personality towards being a generally evil guy. And this is almost inherently an act of intervening on the intervening on the, like, what gets learned. You know, it it is sort of in... Intervening on the training process itself. [00:10:42] S0: It's it's so funny that the way you're describing it, it just sounds so, human in so many way. Right? Like, it's it's very difficult not to anthropomorphize when, talking about, like, interpretability topics and... I mean, this is true of AI generally. Right? Like, surprise is trained on, you know, so much human data. But, like, I always find it actually most noticeable when talking about this sort of interpretability stuff because... I mean, it just brought to mind the the quote that, like, it's it's impossible to get someone to understand something when their salary depends on them not understanding it. Right? Like, you're kind of describing that. Right? This is a model being like, we... Well, yeah, I shouldn't do this, but my salary depends on it. You know? Like, my my reward depends on it. So, like, [00:11:26] S1: it's impossible to get a model [00:11:28] S0: Yeah. [00:11:28] S1: Not to to understand something if it's reward.
00:11:28 · Tom McGrath, Aditya Agarwal
[00:11:28] S1: Not to to understand something if it's reward. [00:11:31] S0: Yeah. So, I mean, is that sort of, like, the the dilemma that we find ourselves in? Because we haven't figured that out for people. [00:11:36] S1: Yeah. You know? But the advantage we have with models is, like, to the extent we have good interpretability, we that can actually kind of read their minds while they're doing this. [00:11:45] S0: Ah, we have... So you're saying lie detectors is the... [00:11:47] S1: That, but also, like, the ability to intervene. That, I think, is the thing that kind of separates the intentional design direction from more general, like, training run monitoring interpretability. Because you might imagine that you just sort of have the agent going around, then you put an fMRI on its head or something or some brain scanner, and you're like, oh, you know, it learned a bad thing there. Let's just kind of wind back. That seems like a pretty compelling move. But you also, like, if you have good interpretability tools that kind of respect the way that the model works, you also have the ability to kind of go and do a bit of rewiring or kind of say, oh, yes. Learn that bit. Like, you can sort of treat what is going to get taught as less like a package deal and more thing you can, like, choose from. I will learn these things, but not the deceptive part. Again, this depends on having very good interpretability. Like, probably, will depend on us having much better interpretability than we have as of mid twenty six.