Lenny's Podcast: Product | Career | Growth
00:40:50 · Lenny Rachitsky, Dianne Penn
[00:40:50] S1: Is there an example of that? A lot of people hear first principles thinking, they're like, "Yes, I got it." I'm good at this. What's an example of someone having really demonstrated really good first principles thinking? [00:41:00] S0: I think one example is I think you think of a product manager as I own product strategy and delivering user value, but I demonstrate day-to-day by writing a PRD or writing a product vision doc. And for my team as research product managers, the way to drive user value is to figure out the right user feedback, the evals that then can be a personification of that user need. So we do write some product documents and PRDs, but we actually have a saying on the team of evals are the new PRDs. Because in order to deliver that user value, it's not that exact artifact that people used to write in the last one to two decades. It's a new way of working. And so the first principles thinking would be, let me figure out what is the thing I should do to achieve my goals, rather than here is a set of activities that I've done and therefore I will continue to do. [00:42:10] S1: So the idea here used to be have an idea, create a PRD, talk to people about it, align on the plan, design it, build it, ship it, see how it goes, iterate. What I'm hearing here is it's like, okay, here's some feedback about something that's wrong or an opportunity. Step one is the eval is now how you define what the work is versus a PRD.
00:42:32 · Dianne Penn, Lenny Rachitsky
[00:42:32] S0: Maybe step one would be understanding the user pain point. And so the way to even access a user pain point is different. In the past, we might do a user interview. I think if you go deep enough, you might have the user walk you through their user flow, the pixels. Here, you have to sweat the tokens as much as you sweat the pixels, and so one activity we have on the team is reading the transcripts and understanding what was the trajectories of failed very deeply to then say, was this a hallucination? Was this Claude being overconfident? So the theme of the failure actually has a lot of nuance, and then that allows you to build a description, a sustained description of that pain point. So that could be essentially in a new eval. And is the eval on distribution? Is it capturing both the positive situations where this is failing and also areas when it should actually not fail? And then bring that back to, let's say, research so that we can make the improvements and actually measure the quality of, okay, when we have Opus 5.5, is this area improving or not? Is Claude now able to identify the right places in the document and pull the right synthesis out? So it's just the actionability and shorten the distance to actionability for our stakeholders and partner teams like researchers to take action on. [00:44:16] S1: Is there an example of something like this where you found an issue or opportunity and then wrote the eval? And what is the eval looking like in most cases? When people want to picture an eval, what do they picture?
00:44:29 · Dianne Penn
[00:44:29] S0: We actually pioneered this concept within Anthropic. So one of the early examples is the early Claude models were not very good at following specific schemas. So things like outputs in JSON. And now that is fundamental to Claude being able to be a good agent. If you can't output a certain format, you don't know how to access APIs, you can call tools, et cetera. And so the initial end-to-end was I was hearing feedback around Claude two days. Claude was not very good at following instructions. So then digging in with the users, what do you mean by Claude is not good at following instructions? Give me what situations this was happening. What's the exact paragraph? What did you ask? What was Claude's response? Going to that level of detail. And what I saw was something like 80% of what people meant in the early days for this failure was Claude would not write the right JSON. And so then, okay, let's generate maybe to start, just 30 to 40 examples of when Claude was not doing this thing correctly. And then that actually is your eval set. And you could have essentially a prompt and a response. And if that is not working in the right golden answer that you might have, then that means that the eval essentially is beneficial because it's identifying a pain point consistently. And so then we added that to our repositories for evals, and when we have versions of Claude, we actually run that eval and just check. I think at this point, it's always 100% or 99.9 and so it's no longer a pain point. But in the early days, it was taking the user feedback, figuring out actually what they mean. Can we reproduce it? Is it consistent? Is it a big issue? And then figuring out how to standardize it in a way that can be consumable for researchers.
00:46:42 · Lenny Rachitsky, Dianne Penn
[00:46:42] S1: It's basically test-driven development for PMs is the world we're living now where you write the test first. So is this just a core part of the product management job now at Anthropic writing evals? [00:46:54] S0: I think so. I also think it's something I've talked to other PMs at other companies about, and I think it's also more and more of the skillset more broadly because a lot of the products that we're building is at the intersection of models with harnesses, with a set of contacts for a set of users. And so having things like evals actually is a way, not just for folks working on models, but generally within product to get to better user experiences because you can't improve what you can't measure. And a lot of this is very still tactile-based. It's still very judgment-based. And so you have to stay close to the details. [00:47:38] S1: And also very non-deterministic, which is a big part of this just like it's not going to give you the same answer every time so you got to describe it more broadly, it's not going to be an exact match. So this is a really interesting change in the way product happens and will happen is evals, writing evals versus PRDs is a big part of this. Do you guys still do PRDs? Is there still a one-pager describing a problem or is it replay? Okay, now you're shaking your head yes. [00:48:00] S0: We are. We do. [00:48:01] S1: Wait a minute. Okay. Now you're shaking your head.
00:48:01 · Lenny Rachitsky, Dianne Penn
[00:48:01] S1: Wait a minute. Okay. Now you're shaking your head. [00:48:03] S0: Yes. We are. We do. I think when there's a very defined problem, I think things like evals might be almost a shorthand. I think there's other cases where PRDs are really valuable. PRDs are great vehicles for getting a very large group of people aligned on a set of sources of truth about experience and set of goals. So when we do have a model, we actually, for every model, we do have a PRD, less necessarily for our researchers, but more for our growing product surfaces, for our engineering teams, for our stakeholders like legal and safety and others as just a source of truth of putting together what we're aiming to achieve so that a big group of people can row in the same direction. The other place where I do think PRDs are valuable are on the more ambiguous problems and opportunities. So if we haven't shipped a thing like computer use, we don't necessarily have a set of user specific pain points always. And I think there's value in the product vision portions of a PRD to explore what could, even if a technology is not yet ready to work for everyone, how do you get it to work well for some group so you can explore the value? You can actually bring something that is coherent to a user group. So we do have PRDs. I think the application's a little different now.
00:49:39 · Lenny Rachitsky
[00:49:39] S1: Okay, this is great. I just had Andrew, he's the head of the Codex app at OpenAI, and you guys are aligned. PRD's not dead. Still very useful for specific projects and ideas. Great. Okay. We've closed the book on... PRD is still kicking. Okay. So we've been talking a bit about just what kind of skills are emerging for product people. Is there anything else that you find has shifted in what patterns are common across people that are doing well in this new AI world in terms of product managers and folks on the product teams? Is there anything else that you're like, okay, this is something you got to shift or something you look for more people?