Lenny's Podcast: Product | Career | Growth
00:57:01 · Cat Wu
[00:57:01] S0: Two people who I think are incredible at this are, one, Amanda, who molds Claude's character. It's just such a hard role because the task is so ambiguous. Even coding is easier because you can verify the success, whereas crafting the character requires a very strong sense of conviction in who Claude should be, and I think she has an incredible ability to not only mold the character, but also to articulate what the goals are, what the character, what's successful and what's not. The other group of people who I really trust is just the Claude Code team. So we often have team lunches and whenever there's a new model we're testing, one of the fastest ways for us to get feedback is to just at these team lunches, just go to every single person and just be like, "Hey, what is your vibe on the model?" And oftentimes we'll get feedback like, "Okay, this model is not fully explaining its thinking. It's too abrupt," or like, "Hey, this model just loves writing a ton of memories, but we're not sure if the memories are high quality or not," or some people will notice that, "Okay, this model loves to test itself, which is great, or this model isn't testing itself enough." So that informs what data we look at to verify, okay, is this a larger pattern? So we have a ton of data, but it is very hard to extract insights, and so the feedback from this group helps us inform, okay, what are the hypotheses we want to test, and then we're able to extract data to test that.
00:58:45 · Lenny Rachitsky
[00:58:45] S1: This point you made about the character of Claude, I had Ben Mann on the podcast, co-founder, and he talked about this, just the character, the constitution of Claude is such an important part of Claude. And I didn't realize until afterwards just with OpenClaw actually, one of the reasons people are sad is the personality of your Claude is... Because Claude's personality is so good and fun and interesting unlike other models. And the way he put it is the personality is what makes Claude so good at so many things. It feels like this trivial side thing. Okay, it's going to be funny and interesting and talk in a fun way, but it's so core to the success of Claude. Is there anything get shared there about just what people may not understand about why the character, as you described, and the personality is so key?
00:59:34 · Cat Wu, Lenny Rachitsky
[00:59:34] S0: When you reflect on everyone you've worked with, there's just some people where you're like, "I really like their energy. I really like their vibe," and when people think about Claude and Claude Code, this is one of the things that people bring up the most where they just really love that Claude, it's lighthearted and fun, but it also is extremely confident at your task. People really like that Claude's low ego, and so if you tell it, "Hey, you did this thing wrong." It's truly sorry. It's like, "Oh, shoot. Thanks for telling me. Let me fix it. Let's work together." It's also very positive. So if you're feeling like, "Oh, this is an insurmountable task. I don't know how to get started," Claude is like, "Okay, it's okay. These are the steps that I think we should take. Do you want me to get started on it for you?" I think part of what makes a great Coworker is this positivity, this bias towards action, this ability to give you earnest feedback, not just agreeing with every single thing that you say, and so we try to imbue this into Claude because we think it makes it a lot more enjoyable to work with. [01:00:45] S1: There's something I want to come back to. You talked about how when new models come out, you often have to revisit things you've built. That's so interesting and so frustrating maybe, just like, "Oh, Goddammit it. Reshipped this thing, now I have to rethink it." Talk about just how often you have to come back with a new model and they're like, "Okay, we have to redo this product that we launched a few months ago."
01:01:03 · Cat Wu
[01:01:03] S0: A lot of the changes that we make with a new model is removing features that are no longer needed. So a lot of times we add features to the product as a crutch for the model because it's not naturally doing itself. So the classic example for this is the to do list. When we first launched Claude Code, people would ask it to do these larger factors and Claude Code would say, "Okay, cool. I need to change these 20 call sites," and it would go and change five of them and then stop. And then we were like, "Okay, how do we force it to remember to get every single one of these 20?" And so Sid on our team was like, "Okay, what if we just think about what a human would do? A human would make a list of everything that they need to change. Similar to how in VS Code, you would look up all the call sites and it would be a list on the left side and you would go through them one by one and replace all. How do we give this kind of a tool to Claude?" And so he added the to do list and we found that with that, Claude was actually able to fix all these 20 call sites. But then with Opus 4 and later models, we realized that we didn't need to force it to use this to do list, it would naturally use itself. For the earlier models, we had to keep reminding it, "Hey, did you finish everything on the to do list? You can't finish until you're done with everything on the to do list." And for the later models, without prompting, it's just naturally thinks to do everything on the to do list. These days, the to do list is still nice to have as a user because then you can more clearly see what Claude is working on. But honestly, it's such a de-emphasized part of the product right now that the model may use it, the model may not use it. It's really not necessary for it to make thorough changes anymore.
01:02:44 · Lenny Rachitsky
[01:02:44] S1: I forget who said this on the podcast, that the model will eat your harness for breakfast, and what I'm hearing here is essentially you remove things over time that you've had to add on top of the model where it was not operating the way you wanted. And essentially, as the models get smarter, it becomes simpler and simpler for it just to do the thing you want it to do.
01:03:03 · Cat Wu
[01:03:03] S0: Yeah. We can remove a lot of prompting interventions every time the model gets smarter. And we actually do this every time we launch a model, we read through the entire system prompt and we reflect on, okay, for each of these sections, does the model really need this reminder anymore, and if not, we'll remove it. The most exciting thing that new models unlocks though is just entirely new features. So there's a lot of features that we've been testing out with prior models and the accuracy wasn't high enough for us to want to launch them, and so one example of this is code review. We tried to build a code review product a few times and we've launched simpler versions of code review, which is the /codereview command in the past, and it was only with the most recent models that we felt like, okay, this code review is so good that our engineering team relies on this code review to pass before we merge PRs. And we found that this was... We've always dreamed of Claude being able to be a reliable code reviewer that we can confidently feel catches the majority of bugs, and it was only with Opus 4.5 and 4.6, and Sonnet 4.6 that we felt like, okay, we are now able to run multiple code review agents simultaneously to traverse the entirety of the code base and to synthesize a set of real issues that an engineer needs to address before merge. And so this is a new capability that the newest models have unlocked.