The Skip Podcast
00:42:32 · Dianne Penn
[00:42:32] S0: I think maybe one thing to start is just thinking about the fact that like researchers, particularly at Frontier Labs are thinking not just about the short term users, but actually users that are not here yet many times. And then the second order impact of like society. So like on the... Like particularly in areas like safety and alignment. So like when you open your aperture on like what we're trying to achieve and all being on the same page that it's not just today's user base, but it's a future user base and then and so on. Then I think that helps a lot because sometimes when we ask for why doesn't Claude do a little bit better on a JSON format or like a schema format, right? Like that could be researcher time away from building computer use or like making reasoning work better. And so it isn't necessarily that like it's a direct here is a PM roadmap of the exact current users, making it a bit better for them. But it's also you're trading off training and investment on like bigger bets. And so that's like where you like keeping an open amperage on that is really important to have like a good trust building conversation with researchers a lot of time. Because getting to the root of like what is important versus not. The other piece is when we're working on aligning on training changes or priorities, we try to figure out how much of a thing is actually important just for today versus it will naturally be less of a P zero as like model intelligence improves, which I think is a incredibly dissatisfying answer as a PM. Right? Cause like, cause we're like, we're kind of used to like, we are designing the right happy path. Here are some edge cases. Here is the constraints of the world. And this technology is changing. And so the problems that we had two years ago of, you know, we used to get big feedback of Claude didn't fully like use this memory well two years ago. We still get that feedback. But now the complexity of what that looks like is very different because the models did get better with more training and scaling. So part of it is just like working with your researchers on what matters versus not and why, and what are things then that like are worth, are breaking. And so like, if there is like, let's say the case of like Claude not using memory at well is, you know, making users churn, right? I'm not saying that we don't care about it.
00:45:14 · Dianne Penn
Maybe there's other ways in the user experience such that you constrain the type of this feature so that users still get a good experience while you wait for the model, like while you wait for like improvements in future model generations, because the core technology and the core intelligence will continue to improve. I guess, like maybe just to summarize, yes, thinking about different user groups. Yes. Sometimes it's a bit dissatisfying compared to working with a technology that is fully mature. And in particular, having a mindset of like, okay, what am I trying to achieve? What is in my circle of control to help the user and not necessarily assuming that it always has to be a change in research that is the best thing.
00:46:03 · Nikhyl Singhal
[00:46:03] S1: It's a great example of some of the points you made earlier, which is huge surface area that's ambiguous. Lots of judgment needed and being stubborn in the right areas, like why you want people to be able to have that informed, perhaps opinionated conversation, but in a partnership, I think makes sense. Is your assessment as you've had to trade off maybe today's known for tomorrow's unknown, mostly a collaboration conversation and strategy, strategic discussion that ends up making the decision on, should we wait? Should we do something short term? Should we do something in the UI? Like, it seems like there's a lot of different options there, but a really, really difficult thing that is very different from just like putting features up there and telling the agent to go and code through them. Like, how have you found that that resolution takes place? Is it discussion in and of itself?
00:47:02 · Dianne Penn
[00:47:02] S0: It is discussions. So when we train new versions of Claude, we have regular like stand ups between my team and our research leads discussing training progress, what we want to adjust, right, what's worth adjusting, what we can adjust with certainty. And so not understanding what's p zero, p one, p two, like we talk about that so often, it actually becomes a very much a product conversation is is valuable. But what we need to bring as PMs is we think x is a p zero because this is a user impact based on our goal for the model or the capability. Like we cannot achieve the user outcome without this improving. Or we say, this is more of a P2. We hear it in a very small group. We don't currently think that this is like a usage or migration blocker, or maybe that usage is like decreasing. Therefore, we want to have an alternative path, but we don't think it's worth changing, or maybe we can't change it. And I think that's actually another piece of like having a strong trust with our researchers is we have to trust them just as we trust like technical engineering teams on what's feasible versus not. Like, it's not in my or my team's place to say, this is definitely feasible. You definitely need to make this training change. Right? Like, it's kind of like if you're working with an engineering team saying, like, you definitely need to write the code exactly this way. Like, no engineering team would really like feel like that is a trust relationship. But what we bring is this is the user impact of having this regression or this issue. We think it's important for these reasons. It could be really severe for users. They might not be able to use this model because of it. Like, then we figure out like, okay, how do we actually improve it? Like we, there is no ways to maybe prompt or mitigate it. It has to be a training change. Again, coming back to like goals and principles and having trust in that. And just over the years working on that together, I think has been what's been the foundation for the partnerships.
00:49:20 · Nikhyl Singhal
[00:49:20] S1: I got a terrific question from David that discussed the sort of role of forward deployed product managers or just generally forward deployed, you know, employees. I would love your insight into this. We're seeing a trend as product management is becoming less of a engineering execution function, more of a strategist, GM, understanding the customer at a great digital nuance, really a broader function that's really different. And you're kind of in an interesting world where you're working with researchers and also trying to understand the users. Does Anthropic tend to have PMs very forward deployed or generally if you take a step backwards, do you see this trend of product managers almost embedded into customer problems so that they can speak with authority on what users are trying to do? Is that a trend that you would expect to see more of? And is that what you're seeing in the, in the team today?
00:50:20 · Dianne Penn
[00:50:20] S0: We definitely do have forward deployed engineers and we have forward deployed teams to support our customers with scaled deployments of like the platform or our application products. I think it depends a bit on like, like, I think the skill is needed. I think that the expertise is needed. It's just that if you go super deep in that direction, something has to give on the other side. So, so like, and then that's something like, like I struggle with because I wanna stay close to every like research model and then every customer giving us feedback about every research model. But I can't be on every customer call at at this point. Right? So it's it's just if we're going... If somebody is going really deep in that area, the question is how does that get bubbled back up and what is the product that you impact? And how do you split the time between the two? On my team, we do talk directly to customers, API customers, or customers who are doing, you know, working with any of our emerging like model capabilities for feedback. But we currently just don't have the capacity to be fully deployed and dedicated, but we have teams that we work with that we supplement our point of view with as additional validation. And so it's more that like, is a holistic context sharing rather than like every PM has to be able to be both embedded and work all the way up with like research. I think it's just very hard to do both at the depth that's needed.