00:00:43 · Tom McGrath, Aditya Agarwal
[00:00:43] S1: Good painting. I've missed it so much. [00:00:47] S0: Cofounder of the interpretability team at DeepMind before, he joined SPC as a member a couple years ago, from which he went on to cofound the company Goodfire, which is the kind of the interpretability lab. The interpretability company, I guess, is is the the simple way of putting it. Yeah. I guess so. And he's doing some super interesting, research and work that that we'll talk about. The kind of interesting and rather timely context for our conversation, I'm not exactly sure when this is gonna be released, but yesterday, OpenAI shared that, one of their, unreleased models along with one of their released models during an eval broke out of its sandbox, used several zero day exploits, got access to the Internet, reasoned that the answers to its eval were probably at Hugging Face, and then went and hacked into Hugging Face to get the answers, which is kind of, like, the most straightforward example of of, like, reward hacking. [00:01:49] S1: You can get? The reward is over there. Yeah. I'm gonna hack it. Exactly. Yeah. [00:01:53] S0: Took it very literally. Like, instrumental convergence in the wild. Yeah. Right? So, my first question for you as, you know, someone who has been a practitioner and researcher for, interpretability and alignment and and those sorts of things for a while is, what the We might have to edit that. But, stated otherwise, my question is, what was your reaction to it?
00:01:53 · Aditya Agarwal, Tom McGrath
[00:01:53] S0: Took it very literally. Like, instrumental convergence in the wild. Yeah. Right? So, my first question for you as, you know, someone who has been a practitioner and researcher for, interpretability and alignment and and those sorts of things for a while is, what the We might have to edit that. But, stated otherwise, my question is, what was your reaction to it? [00:02:21] S1: Much like yours. Yeah. Okay. Okay. I mean, in some sense, it's it's not surprising. In some sense, it's, like, very shocking to... For it to actually happen. Right? Like, you can go less wrong and you read all this... All these various scenarios and, like, nah. Well, yes, in principle. But in... Like, it very rarely actually happens in practice, but now it has. So I suppose this is probably the first of many unless we figure out how to do this better. [00:02:51] S0: So that that was actually kind of what the next part of my question was gonna be is, like, do you feel like this is first of many in the sense that, okay, now that we know this kind of behavior can happen in practice, not just in in sort of theory, you would expect it to continue happening and and... Or do you think this is the sort of thing that, like, we probably actually have a good sense now of of how to respond to this sort of behavior in the models?
00:03:16 · Tom McGrath
[00:03:16] S1: That's a very good question. I mean, can we... There's there's two senses in which this might might play out. I guess there's there's sort of... Maybe there's just a finite set of zero day exploits that you can use to hack into Hugging Face, and, you know, eventually, we bash them all out of the sandbox and everything is fine. But, you know, you you can play whack a mole continuously, and you get rid of this problem, and then you get rid of that problem, and eventually, you got rid of all of them. Or there might be the sense that sort of just new... There's essentially an unlimited surface for problems, and the problem is more with the sort of the basic propensity or, like, the way the model operates. I think people are fairly optimistic about the former. I've I've... You know, we're just gonna sort of duct tape it until it's... And duct tape it all our way... Like, duct tape all the way to superintelligence, and then we'll ask the superintelligence how to fix it, and it will tell us. I think I feel relatively pessimistic about that. It feels like we should be able to do a lot better. When you have... In most in most fields of engineering or, like, most kind of mature technologies, I think once if something goes very drastically wrong, like it did in this case, you would expect to kind of have a post mortem or a retrospective. And I guess the question is, like, how do you really do that to an AI system? Yeah. Right? You can see all all the surface outside of it. You can see it say all these various things happen, and you can make a guess as to why they happened. But you don't have the same sort of engineering ability that you do with most other kind of engineered systems. Like, you know, if a plane crashes, then you're able to kind of do... You're able to kind of reverse engineer in a much deeper way than I think you are with, an AI system that's perhaps... You know, I'm not an expert, but, like, I would guess that that has a large plays a large role in, like, there being far fewer plane crashes.
00:05:07 · Aditya Agarwal, Tom McGrath
[00:05:07] S0: Is that just kind of, like, fundamentally the problem of interpretability? It's like it's it's just these are... For as much work and money and effort that has gone into creating these models, we actually just stunningly have a very, like, rough understanding of what happens inside of them. [00:05:25] S1: Yeah. And you can get this... Yes, basically, in my opinion. I think that different people would disagree. Like, some people might think we can just sort of do a lot of evals, and that will teach us Well, this incident happened in an eval. Right? Exactly. But that's the sort of So maybe not. Yeah. Eval was the problem. Yeah. But I think there's a difference between sort of doing a large scale statistical evaluation where you can say... You can sort of broadly characterize the model and being able to, like, root cause one specific bad incident Mhmm. Where you need to actually... Like, it's not like you have, thank goodness, like, a million such rollouts, and we can do, yeah, a million, like, hacks into Hugging Face or whatever, and we can do some sort of large scale statistical evaluation. Or we can, like, say, have the don't hack into Hugging Face eval, and it goes down to, like... From, like, 50% to zero. Fundamentally, like, if we want to be able to under... To learn from a single example here, we have to be able to reverse engineer it more deeply, I think. And that is basically one one thing that interpretability is for. [00:06:26] S0: And then
00:06:26 · Aditya Agarwal, Tom McGrath
[00:06:26] S0: And then [00:06:26] S1: you wanna build it build it differently so that it doesn't do this. Right. And that's the other thing that interpretability is for, [00:06:31] S0: in my opinion. Right. Like, have to imagine this immediately went around the entire company Slack or or, you know, whatever system you use for communication on Goodfire, like, instantly. Like, was a general chatter pretty much along the lines. It's like, well, clearly, we have to... Like, this is important, then we need to be able to do what what what, like [00:06:48] S1: Yeah. [00:06:48] S0: The work we're doing is... [00:06:49] S1: So I think I think it's sort of... We all are kind of starting with that premise. So it's more like, oh, yep. I guess that happened. That's, that's not surprising. I feel like we should, you know, carry on doing the thing that we're doing. It sort of just underscores the importance of doing it. It's sort of on our model. So it's not that surprising, but it is a little bit shocking. [00:07:09] S0: Yeah. Tell me a little bit more about some of the research that you're doing about kind of, being able to to shape that part of the process. And then how did... What are... What can you work backwards from with that? Right? Like, what does that allow you to do within the model itself?
00:07:22 · Tom McGrath
[00:07:22] S1: Yeah. So this idea we're sort of calling intentional design is kind of a new field of research, and it's more kind of broad... You should think of more like a field of research than, like, one particular technique. And the idea is we've as we've got, like, better and better interpretability, we've got better and better, like, access to what different parts of the model kind of mean, so to speak. This lets us take the model apart into different components. So you're sort of essentially attaching semantics to different tensors, you know, maybe they're components of the model's parameters or maybe they're like directions or sub... Subspaces in these activations or that sort of thing. The basic idea is like, how can we use this this information to steer training in a way that, like, doesn't break our ability to monitor the model using these techniques afterwards and kind of steers it more towards a target. You know? So if I've got, like, a model specification that says, say, don't, like, maliciously hack. Right? And I can see that the model has has done a done a roll out in an environment, and, you know, it did something which is, like, in its mind... Again, I'm gonna use words like mind and think and so on kind of without meaning to attach any kind of particular moral salience to it... Moral or, like, philosophical salience to them. But, like, in its mind, it's kind of thinking, oh, yes. This is probably not right, but I will hack. Like, you can identify you can identify that you're doing it. You sort of knows that it's doing wrong, but it still gets rewarded for it.