00:19:45 · Aditya Agarwal, Tom McGrath
[00:19:45] S0: Not sure. [00:19:46] S1: Let's say that I have three neurons. Right? Neuron one can fire in some number. Neuron two can fire in some number. Neuron three can fire in some number. Now you can, like, represent this as a point in three d space. Right. So, you know, if my neuron one is firing a lot, I'm up here. [00:19:59] S0: Yeah. So this is this is a vector. This is [00:20:00] S1: a Sorry. I apologize. Yeah. Patronizing you. Yeah. You just got like... [00:20:04] S0: I'm never quite sure. I have enough to reveal how much I don't. I know enough to reveal how much I don't now. [00:20:13] S1: It is exactly like the vector of neuron activations or the vector of the residual stream in a transformer. [00:20:19] S0: Gotcha. Okay. And and so, like, the the sort of takeaway from sparse autoencoders is, like, that's... I mean, that's been a sort of direction of research for a while Yeah. In interpretability, in particular, is that, oh, hey. It seems like these... Just, like, actually kind of dropping down to the, like, most sparse kind of signal for a particular concept Yeah. Seems to work kind of surprisingly well. And you're saying it's like, yeah, but, like, not not well enough?
00:20:19 · Aditya Agarwal, Tom McGrath
[00:20:19] S0: Gotcha. Okay. And and so, like, the the sort of takeaway from sparse autoencoders is, like, that's... I mean, that's been a sort of direction of research for a while Yeah. In interpretability, in particular, is that, oh, hey. It seems like these... Just, like, actually kind of dropping down to the, like, most sparse kind of signal for a particular concept Yeah. Seems to work kind of surprisingly well. And you're saying it's like, yeah, but, like, not not well enough? [00:20:43] S1: Or Not well enough for the purposes of intervening. Gotcha. And the interpretation of, like, reliably intervening. Sometimes the interventions are cool. Like Golden Gate Claude, I think, is based on one of these linear interventions. But they do often make the model worse. I guess the thing that makes this, like, not surprising in some ways is, like, we're sort of speedrunning neuroscience here, where Yeah. Neuroscience, they had this funny funny thing where they're like, first of all, they'd be like, things are just horribly, impossibly distributedly represented, I think. And then they'd like, ah, neurons. Like, a neuron can... Maybe that part is wrong. So so we can just get rid of that if it's wrong. Then they had sort of the neuron doctrine, which was like, a neuron has some... Will often have a meaning, and can be understood if you study it. And then they move to, like, the population doctrine, and they... Which is, like, there are populations of neurons that collectively encode, like, higher dimensional or more complex phenomena. And we've basically just done the same thing.
00:20:43 · Tom McGrath, Aditya Agarwal
[00:20:43] S1: Or Not well enough for the purposes of intervening. Gotcha. And the interpretation of, like, reliably intervening. Sometimes the interventions are cool. Like Golden Gate Claude, I think, is based on one of these linear interventions. But they do often make the model worse. I guess the thing that makes this, like, not surprising in some ways is, like, we're sort of speedrunning neuroscience here, where Yeah. Neuroscience, they had this funny funny thing where they're like, first of all, they'd be like, things are just horribly, impossibly distributedly represented, I think. And then they'd like, ah, neurons. Like, a neuron can... Maybe that part is wrong. So so we can just get rid of that if it's wrong. Then they had sort of the neuron doctrine, which was like, a neuron has some... Will often have a meaning, and can be understood if you study it. And then they move to, like, the population doctrine, and they... Which is, like, there are populations of neurons that collectively encode, like, higher dimensional or more complex phenomena. And we've basically just done the same thing. [00:21:42] S0: Is it a mistake to analogize these networks to brains? Like like Yeah. Right? Like, I I always sort of, think that, oh, I'm probably... Right. Like, I shouldn't do that. Right? This is this is... No. These are alien intelligences. The architecture is not the same. And yet, it sure seems like it. Yeah. It keeps working. Right? Like... And you hear about how that... You know, one of these networks works. That's like, well, kinda feels like how my own cognition works in
00:21:42 · Aditya Agarwal, Tom McGrath
[00:21:42] S0: Is it a mistake to analogize these networks to brains? Like like Yeah. Right? Like, I I always sort of, think that, oh, I'm probably... Right. Like, I shouldn't do that. Right? This is this is... No. These are alien intelligences. The architecture is not the same. And yet, it sure seems like it. Yeah. It keeps working. Right? Like... And you hear about how that... You know, one of these networks works. That's like, well, kinda feels like how my own cognition works in [00:22:08] S1: many ways. Right? Like... So, you know, if if if analogizing networks to brains is, like, reliably helpful, whether it's true or not Yeah. You maybe you should keep doing it. [00:22:21] S0: Hey. You know what? If it works, it works. [00:22:23] S1: Yeah.
00:22:23 · Tom McGrath, Aditya Agarwal
[00:22:23] S1: Yeah. [00:22:24] S0: That's not really the approach that you're... You wanna take with the models. You're trying to understand why it works. Alright. So let's talk about how this all sort of comes together as, like, a company and a product. Yeah. Because Goodfire is not like a research organization in, in, like, a university. Right? It's a... It's a company. Yep. It was started as a company to go and and, like, make interpretability available as as a product. So how do you, like, kind of pull together these strands of research that you've been doing over the last few years into something that, you know, if we are right about, oh, suddenly this is going to be a must have, and more and more, companies are gonna want to post train their own models. We're gonna have, like, very capable open weight, open source models to post train on and and probably then more people might wanna pre train their own model. Right? Like, you would expect training to sort of, explode Yes. In in terms of demand. Well, combine that with more capable models, you wanna be able to control them as well. Right? So kind of like a rough outline of would interpretability and steerability, as a result be valuable, makes sense. How how do you actually pull that together as, like, a product? [00:23:31] S1: I mean, this was certainly a bit of a a squiggle. Yeah. But we we had a few occasions. [00:23:39] S0: Cut squiggle in [00:23:41] S1: the... No. The classic SPC squiggle. Cut to [00:23:44] S0: brand brand collateral there. Yeah. Yeah. You for
00:23:48 · Tom McGrath
[00:23:48] S1: the plug. I try. So, yeah, it was definitely, like, a bit of a wiggly path. And what we settled on is this idea of Silico, and Silico is a a way of, like, empowering people to do interpretability and model AI research. You know, if you open up Silico, what you get is essentially, you can talk to an agent and it will kind of go off and use... Either use our libraries of interpretability and training, software that we spent a lot of time on and, you know, the sort of deep skills encoded in there from all the kind of collective wisdom of our researchers. And you can use that to do whatever AI or, you know, research. Like, we're specializing in interpretability, but you can also do training in there. You can also do, like, data research in there, that sort of thing. And the idea here is, like, if we went out and we individually taught everyone to do interpretability, we would just never, like, get enough of it done. Like, as you say, as as increasingly capable of models are coming out and people want to own their own own their own models, they are going to want more complicate... Like, more powerful training services. They won't gonna want want to be able to, like, understand the model they're training, control it during training, like, you know, for instance, like this predictive data debugging stuff, they... They've they've gonna have a load of their own data. Some of it is gonna be great. Some of it is gonna, by surprise, teach the model all sorts of things you didn't want it to teach. And so you're gonna want ways of, like, curating that data, debugging individual, like, rollouts from production back into the data or into the model and saying, like, how can we fix this? That sort of thing.