Lenny's Podcast: Product | Career | Growth
00:00:00 · Lenny Rachitsky, Emmanuel Ameisen
[00:00:00] S0: One of the big questions on people's minds these days is AI doing bad things. As someone that watches these models think and operate, why is it so hard to catch it thinking badly, trying to deceive us? [00:00:09] S1: We train Claude to be helpful, harmless, honest. Is it the case that it's behaving well, but if you look inside, actually, it's like, oh, I'm only behaving well because Lenny's watching right now. [00:00:18] S0: People's full time jobs is studying the model, how it thinks, how it operates. How long has this been a skill set and a job? [00:00:24] S1: One of the most important things to realize if you, like, use language models or think about them is just this, like, very bizarre fact that to make them is not to understand them. All these models do is just a lot of mathematical operations on many numbers. If you look at it, it makes no sense. You just have a list of a 100 numbers. You know, the model's doing math, and then you just try to see if there's patterns. [00:00:42] S0: You are figuring it out the same way we figure out how the brain works, looking for patterns. You show this image, this part of the brain lights up, and you're doing the exact same thing with AI. Wow. This is so fascinating.
00:00:42 · Lenny Rachitsky, Emmanuel Ameisen
[00:00:42] S0: You are figuring it out the same way we figure out how the brain works, looking for patterns. You show this image, this part of the brain lights up, and you're doing the exact same thing with AI. Wow. This is so fascinating. [00:00:50] S1: There was this model we were training. It would go crazy. It would do some math. It would get to an answer. I think it was 12. It would immediately go back and say like, oh, Oh my god. The answer actually is for sure 36. It would do math again. It would get to 12. Slowly, would, like, start descending into madness, writing in all caps, literally saying, I am possessed by a demon. [00:01:07] S0: Humans, if you really boil it down, like, we're basically next token predictors ourselves. We're just trying to figure out the next thing to say. Dario is always out there. This is really dangerous. We gotta be really careful. And everyone's just like, no. He's just saying that for marketing reasons. And now it's like, wait. Shit. It's really dangerous. [00:01:21] S1: Take the idea that this technology is going to keep improving seriously. [00:01:25] S0: Do you think about consciousness as a part of this? Just like do you think AI is conscious? Oh, man. Emmanuel, thank you so much for being here, welcome to the podcast. [00:01:37] S1: Thanks for having me, Eleni. [00:01:38] S0: So you work on interpretability at Anthropic. Explain what that is. Explain what it is you do.
00:01:38 · Lenny Rachitsky, Emmanuel Ameisen
[00:01:38] S0: So you work on interpretability at Anthropic. Explain what that is. Explain what it is you do. [00:01:45] S1: Yeah. So I work in a team of interpretability researchers and our goal is to understand how LLMs like Claude work by looking inside them. One of the most important things to realize if you like use language models or think about them is just this like very bizarre fact that to make them is not to understand them. You make them by sort of like building this architecture, which is kind of like a scaffold, and then you pass a bunch of data through them and they sort of like grow in inscrutable ways. And so that's why it's even a thing that we could be wanting to do to study like how these models work. And so a lot of our research has been on, you know, when the models are about to produce one more word, there's actually a whole universe of stuff that happens and we try to kind of untangle as much of it as we can. [00:02:42] S0: There's gonna be a lot of mind expanding, I don't know, elements to this conversation, I expect. One is along these lines, this point that you make that models are not built, they're grown. Speak more to that.
00:02:55 · Emmanuel Ameisen
[00:02:55] S1: At a really high level, the way a model like Claude works, is you kind of train it in these successive phases. Like the first phase, which people refer to as pre training is you just take a large corpus of data. And over this corpus of data, you sort of feed the model as input, you know, the let's say first half of like a certain document and you ask it to predict the word that comes next. And then you do that again and again and again and again for sort of an ungodly amount of data. And after you do that, there's then this other phase where, and you sort of have this fine tuning phase where maybe instead of arbitrary data, you have a corpus of examples of like how the model should behave and that's what you train it on. So you have examples of dialogue of like helpful assistant telling you, you know, your favorite recipe or solving coding problems. And then there's sort of like further details here that I'm just gonna skip for a second, but then, okay. So what does that mean that you're doing this process? Like, is this thing that you're, that you're sort of training? All models are is, you know, they take this input, which is a bunch of words. They turn these words into numbers and then they do some math on these numbers and they get some intermediate results. Then they do more math and they get some intermediate results. And they do this again and again and again and again many times. And then at the end, they have a final result, which then convert back into a word. And so the math that they do, computations, that's the part that's grown. That's the part where it's just like through learning to do this task better, we sort of like update parameters in one direction or other. And then they learn some formulas. We have no idea what they are. And the goal of my team is to sort of like study, you know, both sort of like what these formulas mean in the end and also what these intermediate results are. Can we decode them and can we see like what the model is doing as it's doing all of this intermediate stuff?
00:04:49 · Lenny Rachitsky
[00:04:49] S0: This episode is brought to you by our season's presenting sponsor WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay, and hundreds of other winning companies all have in common? They are all powered by WorkOS. If you're building a product for the enterprise, you've felt the pain of integrating single sign on, SCIM, RBAC, audit logs, and other features required by large companies. WorkOS turns those deal blockers into drop in APIs with a modern developer platform built specifically for b to b SaaS. Literally every startup that I'm an investor in that starts to expand upmarket ends up working with Work OS. And that's because they are the best. Whether you are a seed stage startup trying to land your first enterprise customer or a unicorn expanding globally, Work OS is the fastest path to becoming enterprise ready and unblocking growth. It's essentially Stripe for enterprise features. Visit workos.com to get started or just hit up their Slack where they have actual engineers waiting to answer your questions. WorkOS allows you to build faster with delightful APIs, comprehensive docs, and a smooth developer experience. Go to workos.com to make your app enterprise ready today. Does it blow your mind as much as it blows mine that we have some code that we wrote that, first of all, we have to now kind of watch and study like like a neuroscientist studies a brain. Like, that's like a new thing that we need to be doing to understand what the hell is going on. And then this idea that as... Like this terminology using is how a lot of people think about this. It's like we've created this new life, artificial life, artificial intelligence. It just is... It's pretty mind boggling if you really think about it and kind of zoom out.
00:06:30 · Emmanuel Ameisen, Lenny Rachitsky
[00:06:30] S1: I think it's really weird. I think that the best way to think about it is sort of like the reason we ended up here. And one of the big reasons is that we don't know how to do it the normal programming way, you know, so there's, there's like for you want to build an app or something you want to sort of do something pretty self contained. You can just have a programmer that's going to write a bunch of rules and say like, oh, well, if somebody clicks on this button, then this happens. You know, if this and that and so on. We don't know how to write that software for classifying, you know, images into categories or for writing, you know, for making a machine that can generally sort of take in arbitrary text and produce texts. It can take in like reports and summarize them, can write emails. And so, know, in a sense, it's weird that this very simple kind of training objective, like gets you all the way there. But in another sense, it's also, you know, I almost feel like there's a sense of like, it's impressive and means something. It means that what you think is a simple objective, this like, oh, we're just feeding data and predicting the next thing is probably deceptively complicated. Like we're probably doing more than what it sounds like. And I think a lot of the ways that maybe like I've updated on how these models work is by realizing how complicated it is to do what they do well. [00:07:56] S0: Say more about that. What do you think about when you say it's this kind of unexpected complication?