Lenny's Podcast: Product | Career | Growth
00:06:30 · Emmanuel Ameisen, Lenny Rachitsky
[00:06:30] S1: I think it's really weird. I think that the best way to think about it is sort of like the reason we ended up here. And one of the big reasons is that we don't know how to do it the normal programming way, you know, so there's, there's like for you want to build an app or something you want to sort of do something pretty self contained. You can just have a programmer that's going to write a bunch of rules and say like, oh, well, if somebody clicks on this button, then this happens. You know, if this and that and so on. We don't know how to write that software for classifying, you know, images into categories or for writing, you know, for making a machine that can generally sort of take in arbitrary text and produce texts. It can take in like reports and summarize them, can write emails. And so, know, in a sense, it's weird that this very simple kind of training objective, like gets you all the way there. But in another sense, it's also, you know, I almost feel like there's a sense of like, it's impressive and means something. It means that what you think is a simple objective, this like, oh, we're just feeding data and predicting the next thing is probably deceptively complicated. Like we're probably doing more than what it sounds like. And I think a lot of the ways that maybe like I've updated on how these models work is by realizing how complicated it is to do what they do well. [00:07:56] S0: Say more about that. What do you think about when you say it's this kind of unexpected complication?
00:08:01 · Emmanuel Ameisen
[00:08:01] S1: Okay, so one way to think about next token prediction that I think a lot of people think about is your iPhone autocomplete, you know, where you write some text and then it'll like predict the next word and it's maybe like right, I don't know, 60% of the time and then kind of funny and weird the rest of the time. And I would think of that as sort of, you know, the very, the easiest version of predicting the next word. And unfortunately, I think that's the mental model that people have for these models. But if you get better at that and language models got better at predicting the next word than humans in around 2019, if you keep getting better at at that, you know, you start running out of easy words. So initially you just had to predict sort of like simple sentence structure, you know, later on the model has to predict like, oh, like simple facts about the world. Like if I complete a sentence like, oh, the capital of France is, and that's like, okay, that's Paris. And you know, once it's good at that and it keeps improving, the only thing that's left is sort of predicting, you know, the answer of some complex, physics equation or the sort of final paragraph or some PhD thesis. And so I think that you end up realizing if you think about this is that man, like predicting the hardest words is actually extremely hard to do. And it's a sort of mystery how you would be able to do that. And to me, this is a sort of, this has been a motivation of why you can start studying these models in the first place. So I was like, how can something so simple give rise to something so powerful? And it's like, well, if you think about the model that has to predict sort of, you know, the, one example I always like to use is like, ah, the capital of the states that contains Dallas is, you know, it'll say Austin. Well, to do that well, at least for you and me, you kind of have to have a mental model of like the geography at play, right? You have to be like, oh, where's Dallas? Dallas is in Texas, states have capitals, the capital of Texas is Austin and so on. And so as these models get better at the seemingly simple task, the sort of like best way to solve it, at least for us is to just kind of understand the underlying structure of the world.
00:10:18 · Emmanuel Ameisen, Lenny Rachitsky
And in fact, you know, some of the stuff we look at inside models shows that they do understand that and they do serve manipulate these concepts like we do. [00:10:27] S0: Oh man, there's so much there. One is just this kind of idea, I forget who said this, maybe Ilya, this idea that to predict the next token while you basically have to understand the entire universe. [00:10:36] S1: He has an example that I love to where he he he has the example, like, predict and the murderer was at the end of a mystery novel. Right? It's like, well, to do that, you kind of have to have understood the whole plot. [00:10:46] S0: Yeah. The other the other... I forget who tweeted this idea that, like, humans, if you really boil it down, like we're basically next token predictors ourselves. We're just trying to figure out the next thing to say and if you meditate you start to kind of actually notice your brain wanting to do things and say things. I don't know how much meditation you've done, but it feels like you start to pay attention to your own next token predictor.
00:10:46 · Lenny Rachitsky, Emmanuel Ameisen
[00:10:46] S0: Yeah. The other the other... I forget who tweeted this idea that, like, humans, if you really boil it down, like we're basically next token predictors ourselves. We're just trying to figure out the next thing to say and if you meditate you start to kind of actually notice your brain wanting to do things and say things. I don't know how much meditation you've done, but it feels like you start to pay attention to your own next token predictor. [00:11:08] S1: Oh yeah. Yeah. And this is like, a thing I think about often is that, you know, like right now, as I'm answering you, there's sort of like two things going on in my mind. There's hopefully more than two, but at least two. One of them is, there's sort of like a point I'm leading you to. There's like a broad idea I'm trying to land the sentence on. And the other thing is, at the same time, I'm just predicting the next thing to say. I'm just saying one word at a time and I'm making sentences and I'm trying to make them make sense and I'm pausing when I think it makes sense. But at the end of the day, even though I'm just saying one word at a time, like all along, what I wanted to do is to compare this process to what language models do. [00:11:50] S0: I'm curious what else as you spend time paying attention to how LLMs work and operate. I feel like you've you're like in the top 0.1% of people that have studied and watched LMs operate and think. What else has surprised you about how they operate and think?
00:12:04 · Emmanuel Ameisen
[00:12:04] S1: There's so much. So maybe I'll go through like, rather than, you know, there's a bunch of findings, but then maybe there's sort of like what, yeah, how they've changed, how I think about models. So to go back to the, you know, capital example, this sort of like, you know, ask the model, you know, the capital of the state that contains Dallas is. So one question that people have had for a while is how do these models answer questions like this? Is it that they've seen just so much text that they've kind of memorized like lists of, you know, prompts and responses? Or is it that they're having some sort of model of like what you're actually asking about, that you're actually asking about like cities and cities are in states and states are in countries and things like that. And so one thing that we do on my team is we ask a question like this, and then we look inside the model at these intermediate results before it produces a given word, you know, as it's about to answer, we just look at the intermediate results and we basically have an immense amount of effort dedicated to trying to decode what these intermediate results mean. And if we decode them well, which we can do in some cases, we can see that models often have and represents these concepts. So when you ask that, you know, Dallas question, what you see is inside the model, you see the representation, you see an intermediate results that maps the representation of Texas that lights up and one to capitals. And then those two combine to activate an intermediate results like, the capital of Texas, that's Austin. They have representations like this for basically everything, like every concept that you can think of, or at least, you know, the vast majority of the ones we've tried, they sort of represent, you know, these concepts uniquely across context. So one thing I find really neat is there's this, there's this debate about, you know, our language models like thinking and then, you know, that's, that's a hard question. But one aspect of thought is like having unique representations that are shared when you think about different things. So, you know, you like know what the concept of an eye is and it could be like an eye in a photo or an eye in like a medical diagram or an eye in a little drawing. And we see that in language models as well.
00:14:18 · Emmanuel Ameisen
If you pass a prompt that has a photo of an eye, or you talk about eyes in any language, or you draw a little SVG with an eye in it, you look at the intermediate results inside the model, the same direction lights up and that direction is like, oh, this is an I. And so there's a sort of a sense in which they have these general concepts that they know about and that they use and reuse to solve problems. Another thing that sort of is surprising to people, it's kind of the mirror of what I was saying about, you know, I'm saying the sentence saying one word at a time, but I have some destination in mind. So one mystery of language models is, bam, if they're outputting only one word at a time, how is it even possible that they could, you know, write a coherent report or like write code well? It's sort of like no one on earth that is doing this as their job, you know, that's like writing reports or, you know, writing emails is literally just only thinking about the next word in the sentence with no idea, you know, like empty mind about anything that comes next. And so that was sort of a big question that we had for a while. And if you look inside these models, like one neat fact about these models is that all of the intermediate results that they've produced when they were predicting a word, you know, let's say like the fifth word that they predicted, they can be reused. It can be accessed and reused by all of the future words. So what does that mean? That means that in practice we find that models will like in their intermediate results, sometimes formulate a plan. The example we offer this is poetry. It's hard to rhyme without planning. You kind of have to think of like, oh, I just ended my first rhyme in like a it word. And so I should end my second line with another it word. You know, maybe the word I should end it with is rabbit. And we find that if you look at models as they're like writing poetry, you see exactly that. They write a line and then like in their intermediate results, they're like, oh, oh, oh, this is an it rhyme. Should I rhyme more with rabbit. And then every future word that they predict looks back to that plan they made. And so even though they're, you know, outputting one word at a time, just like I am right now, they can of course have a plan that they follow over many, many, many, many words.