00:11:28 · Tom McGrath, Aditya Agarwal
[00:11:28] S1: Not to to understand something if it's reward. [00:11:31] S0: Yeah. So, I mean, is that sort of, like, the the dilemma that we find ourselves in? Because we haven't figured that out for people. [00:11:36] S1: Yeah. You know? But the advantage we have with models is, like, to the extent we have good interpretability, we that can actually kind of read their minds while they're doing this. [00:11:45] S0: Ah, we have... So you're saying lie detectors is the... [00:11:47] S1: That, but also, like, the ability to intervene. That, I think, is the thing that kind of separates the intentional design direction from more general, like, training run monitoring interpretability. Because you might imagine that you just sort of have the agent going around, then you put an fMRI on its head or something or some brain scanner, and you're like, oh, you know, it learned a bad thing there. Let's just kind of wind back. That seems like a pretty compelling move. But you also, like, if you have good interpretability tools that kind of respect the way that the model works, you also have the ability to kind of go and do a bit of rewiring or kind of say, oh, yes. Learn that bit. Like, you can sort of treat what is going to get taught as less like a package deal and more thing you can, like, choose from. I will learn these things, but not the deceptive part. Again, this depends on having very good interpretability. Like, probably, will depend on us having much better interpretability than we have as of mid twenty six.
00:11:47 · Tom McGrath, Aditya Agarwal
[00:11:47] S1: That, but also, like, the ability to intervene. That, I think, is the thing that kind of separates the intentional design direction from more general, like, training run monitoring interpretability. Because you might imagine that you just sort of have the agent going around, then you put an fMRI on its head or something or some brain scanner, and you're like, oh, you know, it learned a bad thing there. Let's just kind of wind back. That seems like a pretty compelling move. But you also, like, if you have good interpretability tools that kind of respect the way that the model works, you also have the ability to kind of go and do a bit of rewiring or kind of say, oh, yes. Learn that bit. Like, you can sort of treat what is going to get taught as less like a package deal and more thing you can, like, choose from. I will learn these things, but not the deceptive part. Again, this depends on having very good interpretability. Like, probably, will depend on us having much better interpretability than we have as of mid twenty six. [00:12:38] S0: Yeah. Alright. Let's talk a little bit more about that because I actually wanna keep kind of digging into this concept even more. One of the other bits of research that I found really interesting coming out of Goodfire recently is the sort of neural geometry stuff. And I I love this because when I first read some of the stuff you guys put out, it felt so intuitive to me. Again, I'm gonna... I'm at the risk of sort of, like, anticorporealizing here and [00:12:59] S1: Occupational hazard.
00:12:59 · Tom McGrath, Aditya Agarwal
[00:12:59] S1: Occupational hazard. [00:13:00] S0: Because to me, concepts seem like they have shapes. Right? And in fact, I I spent a lot of time working with founders at SPC, like, helping them with storytelling and and telling them, like, no. You know, the the the pitch is a logical argument. Right? There... It should... We... Even like the language we use, when you get something, it clicks. Yeah. Right? And so to me, there's always been, like, a shape to concepts that fit together neatly versus ones with... Which are discordant, or, there's a shape to, like, certain kind of recurring thoughts that happen. And so when I read the neural geometry stuff that actually the concepts within these language models have, like, a multidimensional manifold instead... They're not just, like, vectors. They're Yeah. They're this kind of... They're a shape. That felt kind of intuitively obvious. Like, is it obvious, or or am I just sort of, like, lucky that my my, like, intuition around similarities here with with, like, my own, you know, experience of cognition and qualia, sort of matches that?
00:13:57 · Tom McGrath, Aditya Agarwal
[00:13:57] S1: Or or were you at all surprised by the the results? I was surprised by the extent of them. I guess the question is less, are there any are there any shapes versus, like, how much seem to be shapes and, like, how intuitive and low dimensional are those shapes going to be? And I'm surprised by how how, like, they are and how much they seem to have, like, consistent patterns, like numbers love being a helix, say, or, like, things which have a sort of bound... Things like... Which have sort of a bounded range of them being reasonable, like temperature, just kind of a bit of a spirally arc or that kind of thing. Oh, even even one thing, this is crazy. [00:14:39] S0: Like, [00:14:40] S1: we we applied this to a robotics model, and one of the manifolds, there's the robe... The robot has its arm in the picture, right, sort of going around like this, and you can see that the... According to the the featurizer, like, this is... This... The hand corresponds to one sort of region in space, and the and the other bits of the arm correspond to other bits. And you can actually see, like, a little arm. Like, there's a little arm inside its representation space. And, you know, when the hand moves in the in the real world, like, essentially, the pixels corresponding to the hand move in three d space, which is just... I saw that, and I was like, this is crazy. [00:15:20] S0: Did you... Did at any point it cross your mind that, like, wow, Plato was right? Yeah. Like, man, platonic forms exist in the models. And, like Maybe I'm [00:15:36] S1: in there.
00:15:36 · Tom McGrath, Aditya Agarwal
[00:15:36] S1: in there. [00:15:36] S0: Well, yeah. Oh my goodness. Right. Exactly. There's [00:15:40] S1: Hopefully, need more than three dimensions. [00:15:42] S0: Yeah. You hope to be like a a... Yeah. We're like a truly multi... You you look in there and and you're like, wow. Look how complex my shape [00:15:48] S1: is. Yeah. I hope, like, seven. [00:15:50] S0: Yeah. It took seven dimensions to define me. It took Six for for that. You know? It's... We're not we're we're not comparing here. Right? Like, it's not a competition. I'm just more multidimensional. Right. Exactly. It's like that thing that went around on on, Twitter that was, like, your presence in the weights. Did you see that? Oh, yes. Yeah. Yeah. Right. Like, we're we're all starting to sort of measure the degree to which we will be, remembered in, by by our, artificial successors, so to speak. Okay. Why does this matter, though? Like like, what what is discovering and and sort of articulating the multidimensionality, the manifold nature of concepts within a model, what does that let you do?
00:16:35 · Tom McGrath
[00:16:35] S1: A few things. Like, one is just the basic science. Like, we're building these things. I feel like we should understand how they work. It feels kind of wrong, but we don't. That's just sort of, you know, as a scientist. But then it also lets you... And sort of going up in in, like, steadily increasing pragmatism, it actually gives you a much better ability to understand different components of the model because, like, sparse autoencoders, which are sort of method that we were kind of working with a lot beforehand, they tend to sort of fragment these these manifolds into a bit. So if I've got kind of an arc here, an SAE will, like, be like, oh, it's this bit of sky and that bit of sky and so on. And so they sort of look disconnected. But when you kind of put them together, they make a lot of sense. So, like, I mean, this this is sort of abstract. So the idea of temperature, like, there's this temperature curve. And if you look at a bit of it, it's like, oh, this this feature, like, this direction in space represents numbers between, like, seventy and seventy nine? Like, okay. That's a kind of weird concept to have. And the other one is like, oh, this represents numbers between, like, fifty and fifty four. These are like... You you... When you look at these, you're like, these don't really make very much sense. Mhmm. And then you're like, oh, actually, the whole arc... Oh, well, this is temperature. Right? Because you can see the whole thing. This is still not very practical. But the reason that I, like, care about it quite deeply from a practical sense is for this, like, purpose of intentional design. So when you are trying to steer steer the model, like, steer steer training, you generally want to be making, an intervention. You know, the model is... The model is sort of at this point in activation space, maybe it's like, I'm currently an honest person. And then I've got to stick with this, like, rolling example. Yeah. I've... I'm currently an honest person, and I've done this rollout. I was rewarded for being, like, dishonest. So I'm gonna move kind of a long list. Let's say there's, like, a personality manifold, and it's got one axis, and it's like... And one of the axis is, like, honesty.