Lenny's Podcast: Product | Career | Growth
00:49:25 · Lenny Rachitsky, Cat Wu
[00:49:25] S1: So cool. Something that kind of this question, this trend, I don't know, question topic comes up a lot recently, which is tokens spend exceeding people's salary where people just use AI and it costs more than how much they're making. Are there any numbers floating around Anthropic of just how much tokens spend, say, engineers spend, I don't know, a month, a day or PMs, anything like that? [00:49:50] S0: It is clear to us that as the models get better, people delegate far more tasks to it and they spend a lot more hours in tools like Claude Code and Cowork, and so we do see the token cost per engineer or per any knowledge worker increase every time that there's a model jump or a substantial product improvement. I think it's still much lower than what the average engineer salary is, but we see the percentage increasing over time. [00:50:22] S1: It's such an interesting... We talked about how you have access to the most cutting-edge models and other advantage of working Anthropic, I believe you guys have basically unlimited tokens. You can use as much as you want. Is that right? [00:50:33] S0: We can use a lot of tokens. Some people do run into limits, so- [00:50:37] S1: Okay, there's a limit. Okay. Boris, shut it down. Okay. It's so interesting how many advantages come from having the most advanced model. It's such an interesting flywheel that starts to kick in.
00:50:37 · Lenny Rachitsky, Cat Wu
[00:50:37] S1: Okay, there's a limit. Okay. Boris, shut it down. Okay. It's so interesting how many advantages come from having the most advanced model. It's such an interesting flywheel that starts to kick in. [00:50:49] S0: I think we also believe a lot in empowering our internal teams to build as fast as possible, and we also trust that everyone understands how much capacity that serving these models truly costs, and we trust our team to use the tokens responsibly. So it's very frowned upon to waste tokens, but we do trust individuals to make that judgment call. [00:51:14] S1: Awesome. Coming back to the PM role, we talked a little bit about this, but I think this will be really interesting for people to hear. Just what I want to understand is what do you think are the emerging skills that PMs need to develop/you most look for, AI companies most look for when they're hiring PMs these days?
00:51:35 · Cat Wu
[00:51:35] S0: I think the hardest skill is being able to define what the product should look like a month from now. I think there's a lot of ambiguity in what models are capable of in that timeline and how user behavior will change, but I think there are patterns that the best PMs can see based on how users are abusing the limits of the existing product and the best PMs can sense that, can set a direction and can steadily execute towards it and change the path if the model capabilities are much better than or worse than what they'd originally expected. I think it is very hard to be the right amount of AGI-pilled because I think everyone can see this future where the models are extremely smart and can do almost everything, in which case you actually don't need that complicated a product, you can actually just have a text box again where you tell the model what you want and it's so smart that it can add any tool or add any integration that it needs to get the job done. It knows when it's uncertain, it can ask clarifying questions. It's kind of very easy to build the product for the super AGI strong model. I think the hard thing is figuring out for the current model, how do you elicit the maximum capability? How do you help users go get onto the golden path? How do you guide users to interact with the model's strengths and patch its weaknesses? This skill is pretty rare.
00:53:19 · Lenny Rachitsky
[00:53:19] S1: And how do you build that skill? Is it just using each... Basically understanding the limits of each model? You talked about taste, understanding, having taste into what the model maybe is capable of, what it's great and not great to at, where it's changed?
00:53:32 · Cat Wu
[00:53:32] S0: Think it's spending a ton of time talking and using the model. One of the things I really like to do is to ask the model to introspect on its own behaviors. So sometimes when I notice that the model does something unexpected, like for example, there's situations where the model will make a front end change and run tests, but not actually use the UI. It's actually pretty useful to ask the model to reflect on why I did this, and sometimes they'll say that, "Hey, there was something confusing in the system prompt," or, "I didn't realize that the front end verification was part of this task," Or, "Hey, I delegated the verification to this subagent and the subagent didn't do the test and I didn't check its work." A lot of times just being very curious about why the model made the decision that it did will show you what misled it so that you can fix the harness in order to close this gap. The other thing that helps is to figure out who are the users who you trust the most to give you accurate feedback about the model. Usually there's a handful of people who are much better than others at articulating what makes a specific model or model harness combination good, and there's a lot of people who will give you feedback, but not everyone's feedback is as qualified. And so finding a group of those five people you trust is really important for getting very fast feedback. I think the third thing that is useful, but not everyone loves doing is building evals. You don't need to build hundreds of evals for them to be useful. Just building 10 great evals is important for helping the team quantify what the goal is and what their progress towards it is and what they're missing, and so I think evals is this underappreciated thing that more PMs, more engineers should be working on.
00:55:33 · Lenny Rachitsky, Cat Wu
[00:55:33] S1: We've covered evals a bunch. There's this trend of just like, that is the future of product management's writing evals because essentially it's what does success look like. Okay, cool. Let me actually concretely define it and then we'll know. How much of your time are you spending writing evals, would you say? [00:55:46] S0: I think the importance of evals varies a bit based on the feature that you're working on or what the problem you're trying to solve is. So there are a lot of folks on our team who do spend a lot of time working on evals. We have a small pod of folks who collaborate very closely with research to more precisely understand our Claude Code behaviors and what the largest areas of improvement are and trying to measure those pretty concretely. I personally jump into evals when there's a feature that I think needs a bit more product definition, and often the output of this is, "Okay, here are five evals that I made. This is how you run them. These are the ones that succeed and these are the ones that don't, and this is the prompt that I've used to increase the success rate." It varies a lot though based on the exact feature. Not every feature needs it, but I think features such as memory benefit a lot from it. [00:56:46] S1: This point you made about people being very good at evaluating models so interesting. It's almost like a human eval of just like, okay, they understand where it's spiking, where it's maybe lacking. Is there anyone specific that you want to shout out that's very good at this?