Lenny's Podcast: Product | Career | Growth
00:27:40 · Lenny Rachitsky
[00:27:40] S0: This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product life cycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity. Visit getdx.com/leni to get a demo of d x. That's getdx.com/leni. Externally, there's been these kind of big jumps seemingly in what AI can do for us. There's like I forget which version of Opus, where it just unlocked coding. Now as Boris Jurney famously said, coding is solved. And many people are just like holy shit, over I think winter break, we're just like wow, AI is really good at this now. So there's been this kind of these moments of big jumps in abilities. And you actually said internally, it's, as you just said, it's actually been this very smooth progression of AI power that happens to cross some threshold where now we can do something. Speak a little bit about that.
00:29:07 · Emmanuel Ameisen
[00:29:07] S1: You know, there's a term for this, which is sort of this idea of emergence, which is it seems like two things are true at once. Like there's this predictably smooth trend. And then, you know, before, like Opus 4.5, I really couldn't let Claude cook too much on my code without a short leash. And then after Opus 4.5, could. And so that's weird. Like, I feel like you would expect a more gradual transition. And the mental model I have for this is, there's kind of two ways of looking at it. So there's the way that I talked about briefly just earlier where it's like, if you're just doing this kind of next token prediction, you imagine that all the texts in the world has sort of like easy words and medium words and hard words in terms of how hard they are to predict. And so as you're going down this curve first, know, the model is bad at everything. And then at some point it gets all of the easy words, right. And that means, you know, maybe the easy words are like the structure of sentences. And so the model, like all of a sudden can write coherent sentences, you know, but just one sentence at a time. And so it's kind of cool and impressive, but like we can't really do anything with that. And then maybe the medium words are sort of, yeah, these sort of basic facts about the world. And as the model smoothly gets better at predicting words, at some points it predicts most of the medium words, right? And now all of a sudden it can write coherent sentences and sentences that are true. And so now maybe it's helpful to like write emails. Maybe it's helpful to sort of like help you write drafts of blog posts, right? Even though that was like a smooth improvement, it just takes it taking over to be helpful. And then I guess the hard words, which maybe are like, your select domain expertise like science or code. And another way to look at it is it's a question of kind of how many of these words do you get right and how many do you get wrong? You know, once you get to the point where you're getting like, I don't know, maybe 90% of the medium words, right? Then you're useful as like an email writer. But if you're at 50%, it's like, it's actually wasting my time to use you because I'm just gonna have to go in and fix it half the time. But if I have to go in and fix it, you know, 10% of the time it's better or 5% of time it's better.
00:31:16 · Emmanuel Ameisen
Different tasks have a different sort of like reliability thresholds, but you smoothly improve. And then at some point it's just reliable enough to be good. And again, for me, my lived experience is basically exactly that, right. Which is at some point, you know, I was just using the models to like help when I was stuck, I would just talk to them and then they would never, I would, they would never write my code. And then at some point they could write like small snippets of code basically all the time. And so now I don't write small snippets. And at some point they could write, you know, large snippets like this year, basically Claude has taken on most of my code writing. And that's because it went from like doing that correctly, you know, I don't know, 80% of the time, which was not useful to, you know, 99% of the time, which now is useful. And so, you know, combination of smooth improvement and then like each different task has a different threshold at which like now it becomes solved, I think gives you these big jumps, even though we're just smoothly going down that scale.
00:32:16 · Lenny Rachitsky
[00:32:16] S0: I think this is really interesting. This idea that the models are just smoothly getting smarter, but once they hit some threshold of they're this often good at this thing, it changes the world. So, I think internally, you would think it'd be like, oh wow, this next model is crazy, so much different, but it's like, no, okay, you can almost predict, give it these many GPUs, this much data, it's gonna get this smart. And it may not be obvious exactly what will change and what unlocks, I know another part of it is just like the data you feed it, there's all this RL work and all this specific data that you train it on, like become better writer, become better at math. I imagine it's part of it too. Okay, so one of the big questions I think on people's minds these days, a big debate happening is just like AI doing bad things, the potential damage it can do because as it gets smarter, been doing bad things. There's been bad things happening recently. So a big question for you is just as someone that watches these models think and operate, why is it so hard to catch it thinking badly, trying to deceive us and do something bad? Why is that not just a solution to the problem of AI alignment?
00:33:29 · Emmanuel Ameisen
[00:33:29] S1: In one sense, it's totally the solution. You know, you watch it and you catch if it does bad stuff. So I think, you know, there's maybe there's tiers to this, right? So I would think of like, maybe one part of it is pure pure engineering. You know, it's like, well, you're giving access to the model to certain things and certain things should be off limits. You know, in some context, it shouldn't have access to the internet. In some context, it should only have access to like a small narrow piece of your computer, stuff like that. So that's that's one part of the work and that's helpful. But then there's sort of there's two additional things that I think are squishier and harder and that we focus on more. So, you know, Interp is part of alignments and broadly I think of sort of two big goals. There's many more, but two that are important, I think, understand is you want to generally understand how the model behaves. What kind of thing is it? What are its propensities? You know, does it tend to do X when put in Y situation? And if so, why? You know, is it the case that the model will only behave well when it has some evidence that it's being observed and monitored, that it's in some evaluation. Then if it sort of like doesn't have that evidence, that it'll behave poorly. So there's this sort of like generally like characterizing what kind of thing you're interacting with, because then it's sort of, just like anything in security, you need to have like a mental model of sort of what's the risk? Like what can they do? What do they tend to do? And so, this is like broader character training. We've talked about the constitution and the docs where we sort of explicitly wants to shape the character of Claude, for example, of what kind of behavior should it be okay doing. And that's the goal. But then the role of interpreting other parts of alignment is then like trying to see like, well, did that work? You know, we train Claude to be helpful, harmless and honest. And we can sort of like look inside and see like, is it, you know, or is it the case that it's behaving well, but if you look inside actually it's like, oh, I'm only behaving well because Lenny's watching right now, you know, and if you weren't, it would do something else.
00:35:36 · Emmanuel Ameisen
So there's this sort of, you know, trying to understand what kind of behavior we formed and trying to mold it and trying to check that we've molded the right way. And then the second part is this sort of, well, yeah, just observe the model and see if it's doing bad stuff. For the basics of that, you don't necessarily need interp. You can just like have another language model, you know, read transcripts and be like, hey, this model do anything bad here and stop it. But you can also use interpret to sort of try to get at parts that are, you know, these unverbalized parts like that, these parts that only exist in these intermediate results. Like models are able to represent stuff in their intermediate results that they don't say as we've established. And so one of the roles of Interp is to try to find, you know, try to identify cases when this happens and prevents them. And then one thing that my team works on that I think is, is along those lines is kind of a fun bit of alignment as I think of it as almost, know, being like a, maybe not a PI, but like they're like investigations or deep dives, you know. We'll see a model do something really weird and really perplexed by it. And we'll try to understand what happened. So a recent example of this is there was this model we were training and sometimes in math problems, it would go crazy. It would do some math, it would get to an answer. I think it was 12. And then it would it would immediately go back and say like, no, oh my God. The answer actually is for sure, 36. I should say 36. And they would do math again. It would get to 12. And then immediately it would say, actually the answer should be 36. And slowly it would like start descending into madness, writing in all caps, literally saying like, I'm possessed by a demon. And you're like, what is going on? Why is it losing its mind here? And if you look inside again, these intermediate results, what's going on in this example, you'll see that over the prompts that just described the math problem, like before the model even did anything, there was a huge like intermediate results activation for 36. Like the model's already obsessed with the number 36 before it's even had a chance to like understand what the problem is.