Lenny's Podcast: Product | Career | Growth
00:00:42 · Lenny Rachitsky, Emmanuel Ameisen
[00:00:42] S0: You are figuring it out the same way we figure out how the brain works, looking for patterns. You show this image, this part of the brain lights up, and you're doing the exact same thing with AI. Wow. This is so fascinating. [00:00:50] S1: There was this model we were training. It would go crazy. It would do some math. It would get to an answer. I think it was 12. It would immediately go back and say like, oh, Oh my god. The answer actually is for sure 36. It would do math again. It would get to 12. Slowly, would, like, start descending into madness, writing in all caps, literally saying, I am possessed by a demon. [00:01:07] S0: Humans, if you really boil it down, like, we're basically next token predictors ourselves. We're just trying to figure out the next thing to say. Dario is always out there. This is really dangerous. We gotta be really careful. And everyone's just like, no. He's just saying that for marketing reasons. And now it's like, wait. Shit. It's really dangerous. [00:01:21] S1: Take the idea that this technology is going to keep improving seriously. [00:01:25] S0: Do you think about consciousness as a part of this? Just like do you think AI is conscious? Oh, man. Emmanuel, thank you so much for being here, welcome to the podcast. [00:01:37] S1: Thanks for having me, Eleni. [00:01:38] S0: So you work on interpretability at Anthropic. Explain what that is. Explain what it is you do.
00:01:38 · Lenny Rachitsky, Emmanuel Ameisen
[00:01:38] S0: So you work on interpretability at Anthropic. Explain what that is. Explain what it is you do. [00:01:45] S1: Yeah. So I work in a team of interpretability researchers and our goal is to understand how LLMs like Claude work by looking inside them. One of the most important things to realize if you like use language models or think about them is just this like very bizarre fact that to make them is not to understand them. You make them by sort of like building this architecture, which is kind of like a scaffold, and then you pass a bunch of data through them and they sort of like grow in inscrutable ways. And so that's why it's even a thing that we could be wanting to do to study like how these models work. And so a lot of our research has been on, you know, when the models are about to produce one more word, there's actually a whole universe of stuff that happens and we try to kind of untangle as much of it as we can. [00:02:42] S0: There's gonna be a lot of mind expanding, I don't know, elements to this conversation, I expect. One is along these lines, this point that you make that models are not built, they're grown. Speak more to that.
00:02:55 · Emmanuel Ameisen
[00:02:55] S1: At a really high level, the way a model like Claude works, is you kind of train it in these successive phases. Like the first phase, which people refer to as pre training is you just take a large corpus of data. And over this corpus of data, you sort of feed the model as input, you know, the let's say first half of like a certain document and you ask it to predict the word that comes next. And then you do that again and again and again and again for sort of an ungodly amount of data. And after you do that, there's then this other phase where, and you sort of have this fine tuning phase where maybe instead of arbitrary data, you have a corpus of examples of like how the model should behave and that's what you train it on. So you have examples of dialogue of like helpful assistant telling you, you know, your favorite recipe or solving coding problems. And then there's sort of like further details here that I'm just gonna skip for a second, but then, okay. So what does that mean that you're doing this process? Like, is this thing that you're, that you're sort of training? All models are is, you know, they take this input, which is a bunch of words. They turn these words into numbers and then they do some math on these numbers and they get some intermediate results. Then they do more math and they get some intermediate results. And they do this again and again and again and again many times. And then at the end, they have a final result, which then convert back into a word. And so the math that they do, computations, that's the part that's grown. That's the part where it's just like through learning to do this task better, we sort of like update parameters in one direction or other. And then they learn some formulas. We have no idea what they are. And the goal of my team is to sort of like study, you know, both sort of like what these formulas mean in the end and also what these intermediate results are. Can we decode them and can we see like what the model is doing as it's doing all of this intermediate stuff?
00:04:49 · Lenny Rachitsky
[00:04:49] S0: This episode is brought to you by our season's presenting sponsor WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay, and hundreds of other winning companies all have in common? They are all powered by WorkOS. If you're building a product for the enterprise, you've felt the pain of integrating single sign on, SCIM, RBAC, audit logs, and other features required by large companies. WorkOS turns those deal blockers into drop in APIs with a modern developer platform built specifically for b to b SaaS. Literally every startup that I'm an investor in that starts to expand upmarket ends up working with Work OS. And that's because they are the best. Whether you are a seed stage startup trying to land your first enterprise customer or a unicorn expanding globally, Work OS is the fastest path to becoming enterprise ready and unblocking growth. It's essentially Stripe for enterprise features. Visit workos.com to get started or just hit up their Slack where they have actual engineers waiting to answer your questions. WorkOS allows you to build faster with delightful APIs, comprehensive docs, and a smooth developer experience. Go to workos.com to make your app enterprise ready today. Does it blow your mind as much as it blows mine that we have some code that we wrote that, first of all, we have to now kind of watch and study like like a neuroscientist studies a brain. Like, that's like a new thing that we need to be doing to understand what the hell is going on. And then this idea that as... Like this terminology using is how a lot of people think about this. It's like we've created this new life, artificial life, artificial intelligence. It just is... It's pretty mind boggling if you really think about it and kind of zoom out.
00:06:30 · Emmanuel Ameisen, Lenny Rachitsky
[00:06:30] S1: I think it's really weird. I think that the best way to think about it is sort of like the reason we ended up here. And one of the big reasons is that we don't know how to do it the normal programming way, you know, so there's, there's like for you want to build an app or something you want to sort of do something pretty self contained. You can just have a programmer that's going to write a bunch of rules and say like, oh, well, if somebody clicks on this button, then this happens. You know, if this and that and so on. We don't know how to write that software for classifying, you know, images into categories or for writing, you know, for making a machine that can generally sort of take in arbitrary text and produce texts. It can take in like reports and summarize them, can write emails. And so, know, in a sense, it's weird that this very simple kind of training objective, like gets you all the way there. But in another sense, it's also, you know, I almost feel like there's a sense of like, it's impressive and means something. It means that what you think is a simple objective, this like, oh, we're just feeding data and predicting the next thing is probably deceptively complicated. Like we're probably doing more than what it sounds like. And I think a lot of the ways that maybe like I've updated on how these models work is by realizing how complicated it is to do what they do well. [00:07:56] S0: Say more about that. What do you think about when you say it's this kind of unexpected complication?
00:08:01 · Emmanuel Ameisen
[00:08:01] S1: Okay, so one way to think about next token prediction that I think a lot of people think about is your iPhone autocomplete, you know, where you write some text and then it'll like predict the next word and it's maybe like right, I don't know, 60% of the time and then kind of funny and weird the rest of the time. And I would think of that as sort of, you know, the very, the easiest version of predicting the next word. And unfortunately, I think that's the mental model that people have for these models. But if you get better at that and language models got better at predicting the next word than humans in around 2019, if you keep getting better at at that, you know, you start running out of easy words. So initially you just had to predict sort of like simple sentence structure, you know, later on the model has to predict like, oh, like simple facts about the world. Like if I complete a sentence like, oh, the capital of France is, and that's like, okay, that's Paris. And you know, once it's good at that and it keeps improving, the only thing that's left is sort of predicting, you know, the answer of some complex, physics equation or the sort of final paragraph or some PhD thesis. And so I think that you end up realizing if you think about this is that man, like predicting the hardest words is actually extremely hard to do. And it's a sort of mystery how you would be able to do that. And to me, this is a sort of, this has been a motivation of why you can start studying these models in the first place. So I was like, how can something so simple give rise to something so powerful? And it's like, well, if you think about the model that has to predict sort of, you know, the, one example I always like to use is like, ah, the capital of the states that contains Dallas is, you know, it'll say Austin. Well, to do that well, at least for you and me, you kind of have to have a mental model of like the geography at play, right? You have to be like, oh, where's Dallas? Dallas is in Texas, states have capitals, the capital of Texas is Austin and so on. And so as these models get better at the seemingly simple task, the sort of like best way to solve it, at least for us is to just kind of understand the underlying structure of the world.