Lenny's Podcast: Product | Career | Growth
00:18:32 · Lenny Rachitsky
[00:18:32] S0: What's so interesting about this is how you have to kind of peek inside its little thought process. As a consumer of AI, kind of give it a prompt and then you wait for it to say something, and what you look at is something that happens between those two or kind of thinks. Talk a little bit about what that little space is, how you're able to observe these sorts of things.
00:18:53 · Emmanuel Ameisen
[00:18:53] S1: So the right mental model is you know again, between each word, there's like multiple steps sort of like computation. And in between each steps, there's like some intermediate result. And what is that result concretely? It's a list of numbers, right? Cause all these models do is just do, a lot of mathematical operations on many numbers. And so initially, if you look at it, it makes no sense. You just have a list of a 100 numbers, you know, the model is doing math. It's like, you look in the middle and there's like a 100 random numbers and then the model is writing poetry and there's a 100 numbers and maybe they kind of look different, but it's unclear, like if they mean anything. And so part of the job is we sort of decode this sort of list of numbers and we try to see if there's structure in it. And if you, you know, if you do this very quickly, you realize that there is structure in it. So one common technique that is used in interpretability is you pass a bunch of text, just all of the text that you can find, just random text through the model and you record these intermediate numbers for every example, you just record it. And then you just try to see if there's patterns, like are there numbers or, you know, like pairs of numbers or like specific series of numbers that like only occur when the model is like processing English text or French text or text about love or text about eyes or whatever. And I think of this as the equivalent of, you know, scan my brain and like put images in front of me and record everything that activates in my brain as I like watch a bunch of images and then be like, oh, you know, it seems like when you see images of a house, like this part of your brain lights up, but when you see images of like a dog, that part of your brain lights up. So that's basically like one of the ways that we do what we do. And if you do it, you do find basically that there's intermediate results that represent like all of these concepts that you'd expect. And so that's the first phase, you know, sort of like decode these intermediate results and now we have sort of like a guess as to what they mean. And then we can do a bunch of experiments. Like we can ask harder questions and then just like label all of the intermediate states and see like, oh, this is what it's doing. Or we can even go inside its activations and change it.
00:21:09 · Emmanuel Ameisen, Lenny Rachitsky
And then sort of see like, oh, what would happen, you know, if while the model is writing these intermediate results, I went and like secretly changed one of the results. And does that do what I would expect? And in fact, oftentimes it does. [00:21:22] S0: It's so interesting to hear you talk about this and how much of this is us just trying to figure out what the hell is going on versus we designed it this certain way and just how much of this has emerged. I don't know. Is that just like, I guess that's just what makes this such a fun job I imagine for you just like what is going on? How is this all working?
00:21:40 · Emmanuel Ameisen
[00:21:40] S1: Yeah, it's it's a fun and sci fi like job because I have been working in tech for a while. And before this, you know, I was working on more applied machine learning and there are a lot of what you do is about building systems and it's sort of like the practice of software engineering. And here a lot of what we do feels so much closer to biology than to engineering, right? Because it's just the sort of system arrives at you and you're being asked to say like, well, we've noticed that Claude can translate words or it can code in this way, or sometimes it behaves, sometimes there's like a weird behavior, like, and you're just asked to answer, well, like, why is it doing that? But you don't have access to the source code really like you do for a program. Like the source code is just this massive list of mathematical operations, which is just like incomprehensible. And so you end up, you know, in the position of someone kind of similar to someone being asked like, hey, you know, when like a mouse gets hungry, like it sort of like gets aggressive, what's going on with that, right? And you're just like, well, guess I'll like run some experiments to see what the factors are. And so it's a very different kind of study than anything that I think most of us on the team have done before.
00:23:02 · Lenny Rachitsky
[00:23:02] S0: Yeah. I imagine people are starting on this, they didn't expect to create this new kind of almost life life type of being. Do you think about consciousness as a part of this? Just like, do you think AI is conscious? Do you think it'll get to a place where we think it's conscious? Or is that just not even, not even worth our time to think about?
00:23:19 · Emmanuel Ameisen
[00:23:19] S1: I mean, I think it's sort of very hard to say one way or another, you know, some of the most interesting ways to get at this question is interpretability because you're sort of, you're not biased by sort of like what the model says. You can just look at the structure inside and try to determine whether it pushes one way or another on the question. I will say that the thing I believe pretty strongly is that most people underestimate the sort of complexity that emerges and that like, if you look inside, there's just the sort of, you know, the facts that the models, the out learns or these representations for all of the concepts that we have and that we know, that they manipulate them to solve problems, that they do so by like planning many tokens ahead, that they sort of have this mix between this higher order reasoning and lower reasoning. And all of that just kind of emerges something that I think is underappreciated by people. And to me, it gives me, let's say a deeper sense of like, oh, there's a lot to still understand about how they work. I think maybe that's the main thing is, is I find that the more you look inside, the less certain you are about how they work. And then oftentimes people that will confidently say like, ah, language models are just X or just Y are the people that actually haven't spent a while looking inside.