00:20:43 · Tom McGrath, Aditya Agarwal
[00:20:43] S1: Or Not well enough for the purposes of intervening. Gotcha. And the interpretation of, like, reliably intervening. Sometimes the interventions are cool. Like Golden Gate Claude, I think, is based on one of these linear interventions. But they do often make the model worse. I guess the thing that makes this, like, not surprising in some ways is, like, we're sort of speedrunning neuroscience here, where Yeah. Neuroscience, they had this funny funny thing where they're like, first of all, they'd be like, things are just horribly, impossibly distributedly represented, I think. And then they'd like, ah, neurons. Like, a neuron can... Maybe that part is wrong. So so we can just get rid of that if it's wrong. Then they had sort of the neuron doctrine, which was like, a neuron has some... Will often have a meaning, and can be understood if you study it. And then they move to, like, the population doctrine, and they... Which is, like, there are populations of neurons that collectively encode, like, higher dimensional or more complex phenomena. And we've basically just done the same thing. [00:21:42] S0: Is it a mistake to analogize these networks to brains? Like like Yeah. Right? Like, I I always sort of, think that, oh, I'm probably... Right. Like, I shouldn't do that. Right? This is this is... No. These are alien intelligences. The architecture is not the same. And yet, it sure seems like it. Yeah. It keeps working. Right? Like... And you hear about how that... You know, one of these networks works. That's like, well, kinda feels like how my own cognition works in
00:21:42 · Aditya Agarwal, Tom McGrath
[00:21:42] S0: Is it a mistake to analogize these networks to brains? Like like Yeah. Right? Like, I I always sort of, think that, oh, I'm probably... Right. Like, I shouldn't do that. Right? This is this is... No. These are alien intelligences. The architecture is not the same. And yet, it sure seems like it. Yeah. It keeps working. Right? Like... And you hear about how that... You know, one of these networks works. That's like, well, kinda feels like how my own cognition works in [00:22:08] S1: many ways. Right? Like... So, you know, if if if analogizing networks to brains is, like, reliably helpful, whether it's true or not Yeah. You maybe you should keep doing it. [00:22:21] S0: Hey. You know what? If it works, it works. [00:22:23] S1: Yeah.
00:22:23 · Tom McGrath, Aditya Agarwal
[00:22:23] S1: Yeah. [00:22:24] S0: That's not really the approach that you're... You wanna take with the models. You're trying to understand why it works. Alright. So let's talk about how this all sort of comes together as, like, a company and a product. Yeah. Because Goodfire is not like a research organization in, in, like, a university. Right? It's a... It's a company. Yep. It was started as a company to go and and, like, make interpretability available as as a product. So how do you, like, kind of pull together these strands of research that you've been doing over the last few years into something that, you know, if we are right about, oh, suddenly this is going to be a must have, and more and more, companies are gonna want to post train their own models. We're gonna have, like, very capable open weight, open source models to post train on and and probably then more people might wanna pre train their own model. Right? Like, you would expect training to sort of, explode Yes. In in terms of demand. Well, combine that with more capable models, you wanna be able to control them as well. Right? So kind of like a rough outline of would interpretability and steerability, as a result be valuable, makes sense. How how do you actually pull that together as, like, a product? [00:23:31] S1: I mean, this was certainly a bit of a a squiggle. Yeah. But we we had a few occasions. [00:23:39] S0: Cut squiggle in [00:23:41] S1: the... No. The classic SPC squiggle. Cut to [00:23:44] S0: brand brand collateral there. Yeah. Yeah. You for
00:23:48 · Tom McGrath
[00:23:48] S1: the plug. I try. So, yeah, it was definitely, like, a bit of a wiggly path. And what we settled on is this idea of Silico, and Silico is a a way of, like, empowering people to do interpretability and model AI research. You know, if you open up Silico, what you get is essentially, you can talk to an agent and it will kind of go off and use... Either use our libraries of interpretability and training, software that we spent a lot of time on and, you know, the sort of deep skills encoded in there from all the kind of collective wisdom of our researchers. And you can use that to do whatever AI or, you know, research. Like, we're specializing in interpretability, but you can also do training in there. You can also do, like, data research in there, that sort of thing. And the idea here is, like, if we went out and we individually taught everyone to do interpretability, we would just never, like, get enough of it done. Like, as you say, as as increasingly capable of models are coming out and people want to own their own own their own models, they are going to want more complicate... Like, more powerful training services. They won't gonna want want to be able to, like, understand the model they're training, control it during training, like, you know, for instance, like this predictive data debugging stuff, they... They've they've gonna have a load of their own data. Some of it is gonna be great. Some of it is gonna, by surprise, teach the model all sorts of things you didn't want it to teach. And so you're gonna want ways of, like, curating that data, debugging individual, like, rollouts from production back into the data or into the model and saying, like, how can we fix this? That sort of thing.
00:25:27 · Aditya Agarwal, Tom McGrath
[00:25:27] S0: Yeah. I mean, it kinda gets to... It's a useful segue to the the last thing I wanted to talk about, which is kind of research generally. Right? Like, you're almost talking about... Or the way you put it to me, I think, before we... When we were chatting earlier is, you know, it's it's not exactly correct to call it, like, a research platform. But also, you know, like research and product have become so kind of fuzzy, right, like with AI. And you even see this in the way that we sort of talk about certain types of companies. Right? Goodfire is a research lab. Right? The lab alone. Right? Like, the the, foundation model labs or that. Right? Everyone uses that term. But you... You've had kind of an interesting perspective as someone who has experienced research. First in academia, when you got your PhD, you were at ICL. Right? [00:26:14] S1: Yes.
00:26:14 · Tom McGrath, Aditya Agarwal
[00:26:14] S1: Yes. [00:26:15] S0: And then then you were at DeepMind. Mhmm. So you're, you know, kind of the the big tech version of, of an AI lab. And then you cofounded a company. Right? And and you're the chief scientist at a at a, small but growing start up. How has your relationship to the idea of doing AI research changed as you've moved through those? Because I I meet a lot of people who are coming from research backgrounds and are thinking about making similar sorts of transitions. Right? And in fact, very kind of weirdly, I've sort of ended up specializing in working with researcher founders because my background is in, like, storytelling. Mhmm. And it turns out that's, like, a very value... That's a particularly valuable skill when working with researcher founders. [00:26:58] S1: Yes. We we can be very, like, literal. [00:27:00] S0: Yes. Yeah. Right. Exactly. Kinda learns a little bit of the the, like, figurative magic. But also it's just, like, often kind of the most ambitious and sci fi stuff. Right? So, like, I I have an affinity to it. But I, you know, I often find myself talking to people about that that process of making the jump from kind of one version of those to the other and and and... Like, because of where I am at SPC to start ups. But, like, how... What has your experience been like making those transitions, and and how has your... How has research changed for you?