The Skip Podcast
00:47:02 · Dianne Penn
[00:47:02] S0: It is discussions. So when we train new versions of Claude, we have regular like stand ups between my team and our research leads discussing training progress, what we want to adjust, right, what's worth adjusting, what we can adjust with certainty. And so not understanding what's p zero, p one, p two, like we talk about that so often, it actually becomes a very much a product conversation is is valuable. But what we need to bring as PMs is we think x is a p zero because this is a user impact based on our goal for the model or the capability. Like we cannot achieve the user outcome without this improving. Or we say, this is more of a P2. We hear it in a very small group. We don't currently think that this is like a usage or migration blocker, or maybe that usage is like decreasing. Therefore, we want to have an alternative path, but we don't think it's worth changing, or maybe we can't change it. And I think that's actually another piece of like having a strong trust with our researchers is we have to trust them just as we trust like technical engineering teams on what's feasible versus not. Like, it's not in my or my team's place to say, this is definitely feasible. You definitely need to make this training change. Right? Like, it's kind of like if you're working with an engineering team saying, like, you definitely need to write the code exactly this way. Like, no engineering team would really like feel like that is a trust relationship. But what we bring is this is the user impact of having this regression or this issue. We think it's important for these reasons. It could be really severe for users. They might not be able to use this model because of it. Like, then we figure out like, okay, how do we actually improve it? Like we, there is no ways to maybe prompt or mitigate it. It has to be a training change. Again, coming back to like goals and principles and having trust in that. And just over the years working on that together, I think has been what's been the foundation for the partnerships.
00:49:20 · Nikhyl Singhal
[00:49:20] S1: I got a terrific question from David that discussed the sort of role of forward deployed product managers or just generally forward deployed, you know, employees. I would love your insight into this. We're seeing a trend as product management is becoming less of a engineering execution function, more of a strategist, GM, understanding the customer at a great digital nuance, really a broader function that's really different. And you're kind of in an interesting world where you're working with researchers and also trying to understand the users. Does Anthropic tend to have PMs very forward deployed or generally if you take a step backwards, do you see this trend of product managers almost embedded into customer problems so that they can speak with authority on what users are trying to do? Is that a trend that you would expect to see more of? And is that what you're seeing in the, in the team today?
00:50:20 · Dianne Penn
[00:50:20] S0: We definitely do have forward deployed engineers and we have forward deployed teams to support our customers with scaled deployments of like the platform or our application products. I think it depends a bit on like, like, I think the skill is needed. I think that the expertise is needed. It's just that if you go super deep in that direction, something has to give on the other side. So, so like, and then that's something like, like I struggle with because I wanna stay close to every like research model and then every customer giving us feedback about every research model. But I can't be on every customer call at at this point. Right? So it's it's just if we're going... If somebody is going really deep in that area, the question is how does that get bubbled back up and what is the product that you impact? And how do you split the time between the two? On my team, we do talk directly to customers, API customers, or customers who are doing, you know, working with any of our emerging like model capabilities for feedback. But we currently just don't have the capacity to be fully deployed and dedicated, but we have teams that we work with that we supplement our point of view with as additional validation. And so it's more that like, is a holistic context sharing rather than like every PM has to be able to be both embedded and work all the way up with like research. I think it's just very hard to do both at the depth that's needed.
00:51:52 · Nikhyl Singhal, Dianne Penn
[00:51:52] S1: Yeah. You're describing something that's not super different from where we may have been even five years ago, where it was hard to scale. Your note around this forward deployed concept does tend to both have a scaling challenge, but also you only get one data point in your node is you need to not only know what the broad industry is using, but also where things are heading. So it's a lot of information to keep and you don't want to sway yourself in one data point. [00:52:17] S0: One thing that is helpful though, is we've actually had folks from our apply, like forward deployed engineering teams, then transfer into my team as PMs. And they've been exceptional because they understand they have spent time being really deep with users and understanding harnesses, understanding user applications, understanding user motivations. And so then they are... They can bring it back into research. It's just very hard to do that, for example, at the same time. And they actually bring us such a diverse point of view to the team because they've been so embedded in the past. So maybe that's just the one adjustment. [00:52:56] S1: Well, and maybe that's the modern day APM program. The modern day APM program could be the forward deployed engineer that in two years we say that's what, you know, if you couldn't found a company, you know, that perhaps was the best way for you to get your feet wet and exactly how the products would be used and on the latest and most cutting edge kind of constraints.
00:52:56 · Nikhyl Singhal, Dianne Penn
[00:52:56] S1: Well, and maybe that's the modern day APM program. The modern day APM program could be the forward deployed engineer that in two years we say that's what, you know, if you couldn't found a company, you know, that perhaps was the best way for you to get your feet wet and exactly how the products would be used and on the latest and most cutting edge kind of constraints. [00:53:16] S0: I mean, I went in 2023 on Claude dot ai, which was our only product at the time and it was in beta. Like, I think I personally wrote the system prompt for a while before somebody else on my team came who was a PM, then also wrote the system prompt. Right? Like, so so that's something you do as like a forward deployed engineer. You're like helping adjust harnesses. And so you have to be hands on and it feels potentially too in the weeds potentially like compared to a traditional product management, like career path. But I actually think you could be right that you just have to go into the depth, but then going to the depth is very vague and making it very grounded as maybe be a forward deployed engineer, sit really close to the customer. That's the depth you need to go to anyway. So maybe that's the right way to frame the career path.
00:54:06 · Nikhyl Singhal
[00:54:06] S1: Maybe last question as we round up, It's a little different than others that we've discussed. You've mentioned the importance of keeping the bar really high and keeping the pace really high because it is a very intense environment both in our industry as a whole, but definitely in the labs and the speed of AI has propelled everyone going forward. As your team has grown and you obviously have people at different phases of life, different phases of career, how one sustains the pace must be a conversation you have, I would argue, maybe every week with your team. And you also must have developed a little bit of a playbook of what has helped maintain boundaries, how it's helped people that last. I mean, you've been there for quite some time. How have you been able to sustain the pace and how have you been able to coach others to kind of manage their energy, their time, and kind of prepare for the marathon that's ahead of us here?