00:13:57 · Tom McGrath, Aditya Agarwal
[00:13:57] S1: Or or were you at all surprised by the the results? I was surprised by the extent of them. I guess the question is less, are there any are there any shapes versus, like, how much seem to be shapes and, like, how intuitive and low dimensional are those shapes going to be? And I'm surprised by how how, like, they are and how much they seem to have, like, consistent patterns, like numbers love being a helix, say, or, like, things which have a sort of bound... Things like... Which have sort of a bounded range of them being reasonable, like temperature, just kind of a bit of a spirally arc or that kind of thing. Oh, even even one thing, this is crazy. [00:14:39] S0: Like, [00:14:40] S1: we we applied this to a robotics model, and one of the manifolds, there's the robe... The robot has its arm in the picture, right, sort of going around like this, and you can see that the... According to the the featurizer, like, this is... This... The hand corresponds to one sort of region in space, and the and the other bits of the arm correspond to other bits. And you can actually see, like, a little arm. Like, there's a little arm inside its representation space. And, you know, when the hand moves in the in the real world, like, essentially, the pixels corresponding to the hand move in three d space, which is just... I saw that, and I was like, this is crazy. [00:15:20] S0: Did you... Did at any point it cross your mind that, like, wow, Plato was right? Yeah. Like, man, platonic forms exist in the models. And, like Maybe I'm [00:15:36] S1: in there.
00:15:36 · Tom McGrath, Aditya Agarwal
[00:15:36] S1: in there. [00:15:36] S0: Well, yeah. Oh my goodness. Right. Exactly. There's [00:15:40] S1: Hopefully, need more than three dimensions. [00:15:42] S0: Yeah. You hope to be like a a... Yeah. We're like a truly multi... You you look in there and and you're like, wow. Look how complex my shape [00:15:48] S1: is. Yeah. I hope, like, seven. [00:15:50] S0: Yeah. It took seven dimensions to define me. It took Six for for that. You know? It's... We're not we're we're not comparing here. Right? Like, it's not a competition. I'm just more multidimensional. Right. Exactly. It's like that thing that went around on on, Twitter that was, like, your presence in the weights. Did you see that? Oh, yes. Yeah. Yeah. Right. Like, we're we're all starting to sort of measure the degree to which we will be, remembered in, by by our, artificial successors, so to speak. Okay. Why does this matter, though? Like like, what what is discovering and and sort of articulating the multidimensionality, the manifold nature of concepts within a model, what does that let you do?
00:16:35 · Tom McGrath
[00:16:35] S1: A few things. Like, one is just the basic science. Like, we're building these things. I feel like we should understand how they work. It feels kind of wrong, but we don't. That's just sort of, you know, as a scientist. But then it also lets you... And sort of going up in in, like, steadily increasing pragmatism, it actually gives you a much better ability to understand different components of the model because, like, sparse autoencoders, which are sort of method that we were kind of working with a lot beforehand, they tend to sort of fragment these these manifolds into a bit. So if I've got kind of an arc here, an SAE will, like, be like, oh, it's this bit of sky and that bit of sky and so on. And so they sort of look disconnected. But when you kind of put them together, they make a lot of sense. So, like, I mean, this this is sort of abstract. So the idea of temperature, like, there's this temperature curve. And if you look at a bit of it, it's like, oh, this this feature, like, this direction in space represents numbers between, like, seventy and seventy nine? Like, okay. That's a kind of weird concept to have. And the other one is like, oh, this represents numbers between, like, fifty and fifty four. These are like... You you... When you look at these, you're like, these don't really make very much sense. Mhmm. And then you're like, oh, actually, the whole arc... Oh, well, this is temperature. Right? Because you can see the whole thing. This is still not very practical. But the reason that I, like, care about it quite deeply from a practical sense is for this, like, purpose of intentional design. So when you are trying to steer steer the model, like, steer steer training, you generally want to be making, an intervention. You know, the model is... The model is sort of at this point in activation space, maybe it's like, I'm currently an honest person. And then I've got to stick with this, like, rolling example. Yeah. I've... I'm currently an honest person, and I've done this rollout. I was rewarded for being, like, dishonest. So I'm gonna move kind of a long list. Let's say there's, like, a personality manifold, and it's got one axis, and it's like... And one of the axis is, like, honesty.
00:18:37 · Aditya Agarwal, Tom McGrath
[00:18:37] S0: There's a literal character arc. [00:18:39] S1: Yeah. Exactly. Like, his character arc is like this. It's bending steadily towards evil. [00:18:45] S0: This this model's breaking bad. [00:18:47] S1: Yeah. And so, you know, you want to be able to intervene on this. You wanna be able to say, like, no, go that way Yeah. Or various other, like, more subtle techniques that... But they generally involve, like, interventions of one form or another. And so your ability to characterize, like, the shape of this determines your ability to intervene on it. That's kind of how we came to caring about this. To answer your earlier question about, like, is it surprising? Ish. So, like, one thing that's surprising about models is how often just straight lines seem to actually work quite nicely. Often, you do them, and they're kind of a bit janky, but they have roughly... They have kind of a desire... They're kind of a reasonable effect, and then you take them a bit further, and they go... Become incoherent. So it's quite surprising to see just how much... What specifically do you [00:19:32] S0: mean by straight lines in this case? [00:19:33] S1: Oh, sorry. Yes. So I've got... So if I'm a I'm a neural network, and I have some neurons, then you can sort of... This is for any machine that this will be, like, horribly simplified. [00:19:45] S0: Not sure.
00:19:45 · Aditya Agarwal, Tom McGrath
[00:19:45] S0: Not sure. [00:19:46] S1: Let's say that I have three neurons. Right? Neuron one can fire in some number. Neuron two can fire in some number. Neuron three can fire in some number. Now you can, like, represent this as a point in three d space. Right. So, you know, if my neuron one is firing a lot, I'm up here. [00:19:59] S0: Yeah. So this is this is a vector. This is [00:20:00] S1: a Sorry. I apologize. Yeah. Patronizing you. Yeah. You just got like... [00:20:04] S0: I'm never quite sure. I have enough to reveal how much I don't. I know enough to reveal how much I don't now. [00:20:13] S1: It is exactly like the vector of neuron activations or the vector of the residual stream in a transformer. [00:20:19] S0: Gotcha. Okay. And and so, like, the the sort of takeaway from sparse autoencoders is, like, that's... I mean, that's been a sort of direction of research for a while Yeah. In interpretability, in particular, is that, oh, hey. It seems like these... Just, like, actually kind of dropping down to the, like, most sparse kind of signal for a particular concept Yeah. Seems to work kind of surprisingly well. And you're saying it's like, yeah, but, like, not not well enough?
00:20:19 · Aditya Agarwal, Tom McGrath
[00:20:19] S0: Gotcha. Okay. And and so, like, the the sort of takeaway from sparse autoencoders is, like, that's... I mean, that's been a sort of direction of research for a while Yeah. In interpretability, in particular, is that, oh, hey. It seems like these... Just, like, actually kind of dropping down to the, like, most sparse kind of signal for a particular concept Yeah. Seems to work kind of surprisingly well. And you're saying it's like, yeah, but, like, not not well enough? [00:20:43] S1: Or Not well enough for the purposes of intervening. Gotcha. And the interpretation of, like, reliably intervening. Sometimes the interventions are cool. Like Golden Gate Claude, I think, is based on one of these linear interventions. But they do often make the model worse. I guess the thing that makes this, like, not surprising in some ways is, like, we're sort of speedrunning neuroscience here, where Yeah. Neuroscience, they had this funny funny thing where they're like, first of all, they'd be like, things are just horribly, impossibly distributedly represented, I think. And then they'd like, ah, neurons. Like, a neuron can... Maybe that part is wrong. So so we can just get rid of that if it's wrong. Then they had sort of the neuron doctrine, which was like, a neuron has some... Will often have a meaning, and can be understood if you study it. And then they move to, like, the population doctrine, and they... Which is, like, there are populations of neurons that collectively encode, like, higher dimensional or more complex phenomena. And we've basically just done the same thing.