00:00:00 · Aditya Agarwal, Tom McGrath
[00:00:00] S0: The way you're describing it, it just sounds so human. Is that sort of like the dilemma that we find ourselves in? Because we haven't figured that out for people. [00:00:08] S1: Yeah. You know? If you get rewarded for doing something wrong, you're probably gonna do it again. If you can't stop the reward hacking, you want to at least stop the model from updating its sort of personality towards being a generally evil guy. We're sort of speed running neuroscience here. The advantage we have with models is, like, to the extent we have good interpretability, we can actually kind of read their minds. [00:00:30] S0: Welcome to the Minus One podcast. This is where we talk to the most interesting people in the world about the winding journeys that led to what they're working on now. I'm excited to be joined by Tom McGrath, alum of SPC. [00:00:43] S1: Good painting. I've missed it so much.
00:00:43 · Tom McGrath, Aditya Agarwal
[00:00:43] S1: Good painting. I've missed it so much. [00:00:47] S0: Cofounder of the interpretability team at DeepMind before, he joined SPC as a member a couple years ago, from which he went on to cofound the company Goodfire, which is the kind of the interpretability lab. The interpretability company, I guess, is is the the simple way of putting it. Yeah. I guess so. And he's doing some super interesting, research and work that that we'll talk about. The kind of interesting and rather timely context for our conversation, I'm not exactly sure when this is gonna be released, but yesterday, OpenAI shared that, one of their, unreleased models along with one of their released models during an eval broke out of its sandbox, used several zero day exploits, got access to the Internet, reasoned that the answers to its eval were probably at Hugging Face, and then went and hacked into Hugging Face to get the answers, which is kind of, like, the most straightforward example of of, like, reward hacking. [00:01:49] S1: You can get? The reward is over there. Yeah. I'm gonna hack it. Exactly. Yeah. [00:01:53] S0: Took it very literally. Like, instrumental convergence in the wild. Yeah. Right? So, my first question for you as, you know, someone who has been a practitioner and researcher for, interpretability and alignment and and those sorts of things for a while is, what the We might have to edit that. But, stated otherwise, my question is, what was your reaction to it?
00:01:53 · Aditya Agarwal, Tom McGrath
[00:01:53] S0: Took it very literally. Like, instrumental convergence in the wild. Yeah. Right? So, my first question for you as, you know, someone who has been a practitioner and researcher for, interpretability and alignment and and those sorts of things for a while is, what the We might have to edit that. But, stated otherwise, my question is, what was your reaction to it? [00:02:21] S1: Much like yours. Yeah. Okay. Okay. I mean, in some sense, it's it's not surprising. In some sense, it's, like, very shocking to... For it to actually happen. Right? Like, you can go less wrong and you read all this... All these various scenarios and, like, nah. Well, yes, in principle. But in... Like, it very rarely actually happens in practice, but now it has. So I suppose this is probably the first of many unless we figure out how to do this better. [00:02:51] S0: So that that was actually kind of what the next part of my question was gonna be is, like, do you feel like this is first of many in the sense that, okay, now that we know this kind of behavior can happen in practice, not just in in sort of theory, you would expect it to continue happening and and... Or do you think this is the sort of thing that, like, we probably actually have a good sense now of of how to respond to this sort of behavior in the models?
00:03:16 · Tom McGrath
[00:03:16] S1: That's a very good question. I mean, can we... There's there's two senses in which this might might play out. I guess there's there's sort of... Maybe there's just a finite set of zero day exploits that you can use to hack into Hugging Face, and, you know, eventually, we bash them all out of the sandbox and everything is fine. But, you know, you you can play whack a mole continuously, and you get rid of this problem, and then you get rid of that problem, and eventually, you got rid of all of them. Or there might be the sense that sort of just new... There's essentially an unlimited surface for problems, and the problem is more with the sort of the basic propensity or, like, the way the model operates. I think people are fairly optimistic about the former. I've I've... You know, we're just gonna sort of duct tape it until it's... And duct tape it all our way... Like, duct tape all the way to superintelligence, and then we'll ask the superintelligence how to fix it, and it will tell us. I think I feel relatively pessimistic about that. It feels like we should be able to do a lot better. When you have... In most in most fields of engineering or, like, most kind of mature technologies, I think once if something goes very drastically wrong, like it did in this case, you would expect to kind of have a post mortem or a retrospective. And I guess the question is, like, how do you really do that to an AI system? Yeah. Right? You can see all all the surface outside of it. You can see it say all these various things happen, and you can make a guess as to why they happened. But you don't have the same sort of engineering ability that you do with most other kind of engineered systems. Like, you know, if a plane crashes, then you're able to kind of do... You're able to kind of reverse engineer in a much deeper way than I think you are with, an AI system that's perhaps... You know, I'm not an expert, but, like, I would guess that that has a large plays a large role in, like, there being far fewer plane crashes.
00:05:07 · Aditya Agarwal, Tom McGrath
[00:05:07] S0: Is that just kind of, like, fundamentally the problem of interpretability? It's like it's it's just these are... For as much work and money and effort that has gone into creating these models, we actually just stunningly have a very, like, rough understanding of what happens inside of them. [00:05:25] S1: Yeah. And you can get this... Yes, basically, in my opinion. I think that different people would disagree. Like, some people might think we can just sort of do a lot of evals, and that will teach us Well, this incident happened in an eval. Right? Exactly. But that's the sort of So maybe not. Yeah. Eval was the problem. Yeah. But I think there's a difference between sort of doing a large scale statistical evaluation where you can say... You can sort of broadly characterize the model and being able to, like, root cause one specific bad incident Mhmm. Where you need to actually... Like, it's not like you have, thank goodness, like, a million such rollouts, and we can do, yeah, a million, like, hacks into Hugging Face or whatever, and we can do some sort of large scale statistical evaluation. Or we can, like, say, have the don't hack into Hugging Face eval, and it goes down to, like... From, like, 50% to zero. Fundamentally, like, if we want to be able to under... To learn from a single example here, we have to be able to reverse engineer it more deeply, I think. And that is basically one one thing that interpretability is for. [00:06:26] S0: And then
00:06:26 · Aditya Agarwal, Tom McGrath
[00:06:26] S0: And then [00:06:26] S1: you wanna build it build it differently so that it doesn't do this. Right. And that's the other thing that interpretability is for, [00:06:31] S0: in my opinion. Right. Like, have to imagine this immediately went around the entire company Slack or or, you know, whatever system you use for communication on Goodfire, like, instantly. Like, was a general chatter pretty much along the lines. It's like, well, clearly, we have to... Like, this is important, then we need to be able to do what what what, like [00:06:48] S1: Yeah. [00:06:48] S0: The work we're doing is... [00:06:49] S1: So I think I think it's sort of... We all are kind of starting with that premise. So it's more like, oh, yep. I guess that happened. That's, that's not surprising. I feel like we should, you know, carry on doing the thing that we're doing. It sort of just underscores the importance of doing it. It's sort of on our model. So it's not that surprising, but it is a little bit shocking. [00:07:09] S0: Yeah. Tell me a little bit more about some of the research that you're doing about kind of, being able to to shape that part of the process. And then how did... What are... What can you work backwards from with that? Right? Like, what does that allow you to do within the model itself?