Lenny's Podcast: Product | Career | Growth
00:10:46 · Lenny Rachitsky, Emmanuel Ameisen
[00:10:46] S0: Yeah. The other the other... I forget who tweeted this idea that, like, humans, if you really boil it down, like we're basically next token predictors ourselves. We're just trying to figure out the next thing to say and if you meditate you start to kind of actually notice your brain wanting to do things and say things. I don't know how much meditation you've done, but it feels like you start to pay attention to your own next token predictor. [00:11:08] S1: Oh yeah. Yeah. And this is like, a thing I think about often is that, you know, like right now, as I'm answering you, there's sort of like two things going on in my mind. There's hopefully more than two, but at least two. One of them is, there's sort of like a point I'm leading you to. There's like a broad idea I'm trying to land the sentence on. And the other thing is, at the same time, I'm just predicting the next thing to say. I'm just saying one word at a time and I'm making sentences and I'm trying to make them make sense and I'm pausing when I think it makes sense. But at the end of the day, even though I'm just saying one word at a time, like all along, what I wanted to do is to compare this process to what language models do. [00:11:50] S0: I'm curious what else as you spend time paying attention to how LLMs work and operate. I feel like you've you're like in the top 0.1% of people that have studied and watched LMs operate and think. What else has surprised you about how they operate and think?
00:12:04 · Emmanuel Ameisen
[00:12:04] S1: There's so much. So maybe I'll go through like, rather than, you know, there's a bunch of findings, but then maybe there's sort of like what, yeah, how they've changed, how I think about models. So to go back to the, you know, capital example, this sort of like, you know, ask the model, you know, the capital of the state that contains Dallas is. So one question that people have had for a while is how do these models answer questions like this? Is it that they've seen just so much text that they've kind of memorized like lists of, you know, prompts and responses? Or is it that they're having some sort of model of like what you're actually asking about, that you're actually asking about like cities and cities are in states and states are in countries and things like that. And so one thing that we do on my team is we ask a question like this, and then we look inside the model at these intermediate results before it produces a given word, you know, as it's about to answer, we just look at the intermediate results and we basically have an immense amount of effort dedicated to trying to decode what these intermediate results mean. And if we decode them well, which we can do in some cases, we can see that models often have and represents these concepts. So when you ask that, you know, Dallas question, what you see is inside the model, you see the representation, you see an intermediate results that maps the representation of Texas that lights up and one to capitals. And then those two combine to activate an intermediate results like, the capital of Texas, that's Austin. They have representations like this for basically everything, like every concept that you can think of, or at least, you know, the vast majority of the ones we've tried, they sort of represent, you know, these concepts uniquely across context. So one thing I find really neat is there's this, there's this debate about, you know, our language models like thinking and then, you know, that's, that's a hard question. But one aspect of thought is like having unique representations that are shared when you think about different things. So, you know, you like know what the concept of an eye is and it could be like an eye in a photo or an eye in like a medical diagram or an eye in a little drawing. And we see that in language models as well.
00:14:18 · Emmanuel Ameisen
If you pass a prompt that has a photo of an eye, or you talk about eyes in any language, or you draw a little SVG with an eye in it, you look at the intermediate results inside the model, the same direction lights up and that direction is like, oh, this is an I. And so there's a sort of a sense in which they have these general concepts that they know about and that they use and reuse to solve problems. Another thing that sort of is surprising to people, it's kind of the mirror of what I was saying about, you know, I'm saying the sentence saying one word at a time, but I have some destination in mind. So one mystery of language models is, bam, if they're outputting only one word at a time, how is it even possible that they could, you know, write a coherent report or like write code well? It's sort of like no one on earth that is doing this as their job, you know, that's like writing reports or, you know, writing emails is literally just only thinking about the next word in the sentence with no idea, you know, like empty mind about anything that comes next. And so that was sort of a big question that we had for a while. And if you look inside these models, like one neat fact about these models is that all of the intermediate results that they've produced when they were predicting a word, you know, let's say like the fifth word that they predicted, they can be reused. It can be accessed and reused by all of the future words. So what does that mean? That means that in practice we find that models will like in their intermediate results, sometimes formulate a plan. The example we offer this is poetry. It's hard to rhyme without planning. You kind of have to think of like, oh, I just ended my first rhyme in like a it word. And so I should end my second line with another it word. You know, maybe the word I should end it with is rabbit. And we find that if you look at models as they're like writing poetry, you see exactly that. They write a line and then like in their intermediate results, they're like, oh, oh, oh, this is an it rhyme. Should I rhyme more with rabbit. And then every future word that they predict looks back to that plan they made. And so even though they're, you know, outputting one word at a time, just like I am right now, they can of course have a plan that they follow over many, many, many, many words.
00:16:34 · Emmanuel Ameisen, Lenny Rachitsky
And so all of a sudden, even though you have like a next word predictor, even though it's like technically right, you actually have something that's closer to being able to just do like, you know, higher order tasks where it's like, oh, well you just have a plan and you just kind of refer back to your plan and you're slowly executing on it. And you can do that without, without saying the plan. That's the cool part is like this plan that we find is in these intermediate results inside the model, but the model doesn't even have to sort of like say [00:17:06] S0: it out loud. Wow. I know that this exactly what you described is one of the biggest findings you guys have discovered about how AI thinks this this kind of assumption we all... Like it was originally built, the original idea of this... Of of how LMs operate was this next token predictor, and you're saying that that's not actually how it is anymore. They plan, they think ahead. It's not just, like you said, this idea of an empty mind picking the next word independent of everything else.
00:17:31 · Emmanuel Ameisen
[00:17:31] S1: That's right. And I think that, I think those two visions can like actually aren't, you know, as in conflict as you think and that you train the model by asking it to be like, hey, what happens next? What happens next? What happens next? What happens next? And then if you do that and the model gets good enough and you give it hard enough tasks, well then of course the best way to know what happens next is to think about like, you know, what happens way down the line. You know, you can think of even in this pre training phase when like the models are reading a story with characters, right? And it's like, oh, like what is this character gonna do next? Well, the best way to know that is to be like, okay, so this character, you know, is generally in love with that character. And they've been sort of like planning to like declare their love at this instant. And so, or in the future, and so to do that, they probably should do this thing. And so what they're going to say is that. And so even though you're only like predicting the like next instant, it's like, yeah, but to do that really well, you have to have like a really good understanding of what's going on, including what you think might happen in the future.
00:18:32 · Lenny Rachitsky
[00:18:32] S0: What's so interesting about this is how you have to kind of peek inside its little thought process. As a consumer of AI, kind of give it a prompt and then you wait for it to say something, and what you look at is something that happens between those two or kind of thinks. Talk a little bit about what that little space is, how you're able to observe these sorts of things.