Lenny's Podcast: Product | Career | Growth
00:53:19 · Lenny Rachitsky
[00:53:19] S1: And how do you build that skill? Is it just using each... Basically understanding the limits of each model? You talked about taste, understanding, having taste into what the model maybe is capable of, what it's great and not great to at, where it's changed?
00:53:32 · Cat Wu
[00:53:32] S0: Think it's spending a ton of time talking and using the model. One of the things I really like to do is to ask the model to introspect on its own behaviors. So sometimes when I notice that the model does something unexpected, like for example, there's situations where the model will make a front end change and run tests, but not actually use the UI. It's actually pretty useful to ask the model to reflect on why I did this, and sometimes they'll say that, "Hey, there was something confusing in the system prompt," or, "I didn't realize that the front end verification was part of this task," Or, "Hey, I delegated the verification to this subagent and the subagent didn't do the test and I didn't check its work." A lot of times just being very curious about why the model made the decision that it did will show you what misled it so that you can fix the harness in order to close this gap. The other thing that helps is to figure out who are the users who you trust the most to give you accurate feedback about the model. Usually there's a handful of people who are much better than others at articulating what makes a specific model or model harness combination good, and there's a lot of people who will give you feedback, but not everyone's feedback is as qualified. And so finding a group of those five people you trust is really important for getting very fast feedback. I think the third thing that is useful, but not everyone loves doing is building evals. You don't need to build hundreds of evals for them to be useful. Just building 10 great evals is important for helping the team quantify what the goal is and what their progress towards it is and what they're missing, and so I think evals is this underappreciated thing that more PMs, more engineers should be working on.
00:55:33 · Lenny Rachitsky, Cat Wu
[00:55:33] S1: We've covered evals a bunch. There's this trend of just like, that is the future of product management's writing evals because essentially it's what does success look like. Okay, cool. Let me actually concretely define it and then we'll know. How much of your time are you spending writing evals, would you say? [00:55:46] S0: I think the importance of evals varies a bit based on the feature that you're working on or what the problem you're trying to solve is. So there are a lot of folks on our team who do spend a lot of time working on evals. We have a small pod of folks who collaborate very closely with research to more precisely understand our Claude Code behaviors and what the largest areas of improvement are and trying to measure those pretty concretely. I personally jump into evals when there's a feature that I think needs a bit more product definition, and often the output of this is, "Okay, here are five evals that I made. This is how you run them. These are the ones that succeed and these are the ones that don't, and this is the prompt that I've used to increase the success rate." It varies a lot though based on the exact feature. Not every feature needs it, but I think features such as memory benefit a lot from it. [00:56:46] S1: This point you made about people being very good at evaluating models so interesting. It's almost like a human eval of just like, okay, they understand where it's spiking, where it's maybe lacking. Is there anyone specific that you want to shout out that's very good at this?
00:57:01 · Cat Wu
[00:57:01] S0: Two people who I think are incredible at this are, one, Amanda, who molds Claude's character. It's just such a hard role because the task is so ambiguous. Even coding is easier because you can verify the success, whereas crafting the character requires a very strong sense of conviction in who Claude should be, and I think she has an incredible ability to not only mold the character, but also to articulate what the goals are, what the character, what's successful and what's not. The other group of people who I really trust is just the Claude Code team. So we often have team lunches and whenever there's a new model we're testing, one of the fastest ways for us to get feedback is to just at these team lunches, just go to every single person and just be like, "Hey, what is your vibe on the model?" And oftentimes we'll get feedback like, "Okay, this model is not fully explaining its thinking. It's too abrupt," or like, "Hey, this model just loves writing a ton of memories, but we're not sure if the memories are high quality or not," or some people will notice that, "Okay, this model loves to test itself, which is great, or this model isn't testing itself enough." So that informs what data we look at to verify, okay, is this a larger pattern? So we have a ton of data, but it is very hard to extract insights, and so the feedback from this group helps us inform, okay, what are the hypotheses we want to test, and then we're able to extract data to test that.
00:58:45 · Lenny Rachitsky
[00:58:45] S1: This point you made about the character of Claude, I had Ben Mann on the podcast, co-founder, and he talked about this, just the character, the constitution of Claude is such an important part of Claude. And I didn't realize until afterwards just with OpenClaw actually, one of the reasons people are sad is the personality of your Claude is... Because Claude's personality is so good and fun and interesting unlike other models. And the way he put it is the personality is what makes Claude so good at so many things. It feels like this trivial side thing. Okay, it's going to be funny and interesting and talk in a fun way, but it's so core to the success of Claude. Is there anything get shared there about just what people may not understand about why the character, as you described, and the personality is so key?
00:59:34 · Cat Wu, Lenny Rachitsky
[00:59:34] S0: When you reflect on everyone you've worked with, there's just some people where you're like, "I really like their energy. I really like their vibe," and when people think about Claude and Claude Code, this is one of the things that people bring up the most where they just really love that Claude, it's lighthearted and fun, but it also is extremely confident at your task. People really like that Claude's low ego, and so if you tell it, "Hey, you did this thing wrong." It's truly sorry. It's like, "Oh, shoot. Thanks for telling me. Let me fix it. Let's work together." It's also very positive. So if you're feeling like, "Oh, this is an insurmountable task. I don't know how to get started," Claude is like, "Okay, it's okay. These are the steps that I think we should take. Do you want me to get started on it for you?" I think part of what makes a great Coworker is this positivity, this bias towards action, this ability to give you earnest feedback, not just agreeing with every single thing that you say, and so we try to imbue this into Claude because we think it makes it a lot more enjoyable to work with. [01:00:45] S1: There's something I want to come back to. You talked about how when new models come out, you often have to revisit things you've built. That's so interesting and so frustrating maybe, just like, "Oh, Goddammit it. Reshipped this thing, now I have to rethink it." Talk about just how often you have to come back with a new model and they're like, "Okay, we have to redo this product that we launched a few months ago."