00:18:37 · Aditya Agarwal, Tom McGrath
[00:18:37] S0: There's a literal character arc. [00:18:39] S1: Yeah. Exactly. Like, his character arc is like this. It's bending steadily towards evil. [00:18:45] S0: This this model's breaking bad. [00:18:47] S1: Yeah. And so, you know, you want to be able to intervene on this. You wanna be able to say, like, no, go that way Yeah. Or various other, like, more subtle techniques that... But they generally involve, like, interventions of one form or another. And so your ability to characterize, like, the shape of this determines your ability to intervene on it. That's kind of how we came to caring about this. To answer your earlier question about, like, is it surprising? Ish. So, like, one thing that's surprising about models is how often just straight lines seem to actually work quite nicely. Often, you do them, and they're kind of a bit janky, but they have roughly... They have kind of a desire... They're kind of a reasonable effect, and then you take them a bit further, and they go... Become incoherent. So it's quite surprising to see just how much... What specifically do you [00:19:32] S0: mean by straight lines in this case? [00:19:33] S1: Oh, sorry. Yes. So I've got... So if I'm a I'm a neural network, and I have some neurons, then you can sort of... This is for any machine that this will be, like, horribly simplified. [00:19:45] S0: Not sure.
00:19:45 · Aditya Agarwal, Tom McGrath
[00:19:45] S0: Not sure. [00:19:46] S1: Let's say that I have three neurons. Right? Neuron one can fire in some number. Neuron two can fire in some number. Neuron three can fire in some number. Now you can, like, represent this as a point in three d space. Right. So, you know, if my neuron one is firing a lot, I'm up here. [00:19:59] S0: Yeah. So this is this is a vector. This is [00:20:00] S1: a Sorry. I apologize. Yeah. Patronizing you. Yeah. You just got like... [00:20:04] S0: I'm never quite sure. I have enough to reveal how much I don't. I know enough to reveal how much I don't now. [00:20:13] S1: It is exactly like the vector of neuron activations or the vector of the residual stream in a transformer. [00:20:19] S0: Gotcha. Okay. And and so, like, the the sort of takeaway from sparse autoencoders is, like, that's... I mean, that's been a sort of direction of research for a while Yeah. In interpretability, in particular, is that, oh, hey. It seems like these... Just, like, actually kind of dropping down to the, like, most sparse kind of signal for a particular concept Yeah. Seems to work kind of surprisingly well. And you're saying it's like, yeah, but, like, not not well enough?
00:20:19 · Aditya Agarwal, Tom McGrath
[00:20:19] S0: Gotcha. Okay. And and so, like, the the sort of takeaway from sparse autoencoders is, like, that's... I mean, that's been a sort of direction of research for a while Yeah. In interpretability, in particular, is that, oh, hey. It seems like these... Just, like, actually kind of dropping down to the, like, most sparse kind of signal for a particular concept Yeah. Seems to work kind of surprisingly well. And you're saying it's like, yeah, but, like, not not well enough? [00:20:43] S1: Or Not well enough for the purposes of intervening. Gotcha. And the interpretation of, like, reliably intervening. Sometimes the interventions are cool. Like Golden Gate Claude, I think, is based on one of these linear interventions. But they do often make the model worse. I guess the thing that makes this, like, not surprising in some ways is, like, we're sort of speedrunning neuroscience here, where Yeah. Neuroscience, they had this funny funny thing where they're like, first of all, they'd be like, things are just horribly, impossibly distributedly represented, I think. And then they'd like, ah, neurons. Like, a neuron can... Maybe that part is wrong. So so we can just get rid of that if it's wrong. Then they had sort of the neuron doctrine, which was like, a neuron has some... Will often have a meaning, and can be understood if you study it. And then they move to, like, the population doctrine, and they... Which is, like, there are populations of neurons that collectively encode, like, higher dimensional or more complex phenomena. And we've basically just done the same thing.
00:20:43 · Tom McGrath, Aditya Agarwal
[00:20:43] S1: Or Not well enough for the purposes of intervening. Gotcha. And the interpretation of, like, reliably intervening. Sometimes the interventions are cool. Like Golden Gate Claude, I think, is based on one of these linear interventions. But they do often make the model worse. I guess the thing that makes this, like, not surprising in some ways is, like, we're sort of speedrunning neuroscience here, where Yeah. Neuroscience, they had this funny funny thing where they're like, first of all, they'd be like, things are just horribly, impossibly distributedly represented, I think. And then they'd like, ah, neurons. Like, a neuron can... Maybe that part is wrong. So so we can just get rid of that if it's wrong. Then they had sort of the neuron doctrine, which was like, a neuron has some... Will often have a meaning, and can be understood if you study it. And then they move to, like, the population doctrine, and they... Which is, like, there are populations of neurons that collectively encode, like, higher dimensional or more complex phenomena. And we've basically just done the same thing. [00:21:42] S0: Is it a mistake to analogize these networks to brains? Like like Yeah. Right? Like, I I always sort of, think that, oh, I'm probably... Right. Like, I shouldn't do that. Right? This is this is... No. These are alien intelligences. The architecture is not the same. And yet, it sure seems like it. Yeah. It keeps working. Right? Like... And you hear about how that... You know, one of these networks works. That's like, well, kinda feels like how my own cognition works in
00:21:42 · Aditya Agarwal, Tom McGrath
[00:21:42] S0: Is it a mistake to analogize these networks to brains? Like like Yeah. Right? Like, I I always sort of, think that, oh, I'm probably... Right. Like, I shouldn't do that. Right? This is this is... No. These are alien intelligences. The architecture is not the same. And yet, it sure seems like it. Yeah. It keeps working. Right? Like... And you hear about how that... You know, one of these networks works. That's like, well, kinda feels like how my own cognition works in [00:22:08] S1: many ways. Right? Like... So, you know, if if if analogizing networks to brains is, like, reliably helpful, whether it's true or not Yeah. You maybe you should keep doing it. [00:22:21] S0: Hey. You know what? If it works, it works. [00:22:23] S1: Yeah.
00:22:23 · Tom McGrath, Aditya Agarwal
[00:22:23] S1: Yeah. [00:22:24] S0: That's not really the approach that you're... You wanna take with the models. You're trying to understand why it works. Alright. So let's talk about how this all sort of comes together as, like, a company and a product. Yeah. Because Goodfire is not like a research organization in, in, like, a university. Right? It's a... It's a company. Yep. It was started as a company to go and and, like, make interpretability available as as a product. So how do you, like, kind of pull together these strands of research that you've been doing over the last few years into something that, you know, if we are right about, oh, suddenly this is going to be a must have, and more and more, companies are gonna want to post train their own models. We're gonna have, like, very capable open weight, open source models to post train on and and probably then more people might wanna pre train their own model. Right? Like, you would expect training to sort of, explode Yes. In in terms of demand. Well, combine that with more capable models, you wanna be able to control them as well. Right? So kind of like a rough outline of would interpretability and steerability, as a result be valuable, makes sense. How how do you actually pull that together as, like, a product? [00:23:31] S1: I mean, this was certainly a bit of a a squiggle. Yeah. But we we had a few occasions. [00:23:39] S0: Cut squiggle in [00:23:41] S1: the... No. The classic SPC squiggle. Cut to [00:23:44] S0: brand brand collateral there. Yeah. Yeah. You for