The Skip Podcast
00:37:41 · Dianne Penn
Like, I lean more on having actually smaller teams, but giving people space such that, like, they have maybe two to three areas that they're looking after. And so as part of that, like they're naturally stretched to go talk to different people to then come up with new ideas, like read up on what's happening. And so whatever they're working on is like very high priority and therefore they have more like urgency to like deliver and just bring them on pace, getting stuff done like that, I think has been important. We also have a spin up buddy for everybody on the team. So folks who are more tenured on the team always is a buddy for somebody new. We have a lot of products like Claude Tag, which helps folks with onboarding. And I think the culture of working in public and like public channels, part of it is just like I tell folks like, go back, scroll back two months, read what's happened because you're kind of joining a conversation when you're joining a company. So when everybody's talking, it's okay to say like, hey, can everybody stop and tell me what's going on? It's better to do a bit of your homework and then be like, hey, I understand this seems to be X, Y, and Z. Here's what I think. What do you all think? Right? So it's a bit of coming to the company rather than waiting for it like Anthropic to come to you as a new hire. But those tend to be some of like the initial ways of working to help people onboard.
00:39:13 · Nikhyl Singhal
[00:39:13] S1: Tell me more about the bootcamp. And the part of the reason I'm asking is that a lot of the things you mentioned are replicable in someone else's company as well. And I'm wondering this bootcamp, is it how structured is it? How much of it is like coursework versus skilled versus case studies? And it might, you know, I'm sure it's quite popular. So I would love to know how it sort of evolved over the years.
00:39:38 · Dianne Penn
[00:39:38] S0: It's very grassroots y. We thought at first we were starting in 2024 when we did the first year, myself and the team at the time was pretty small team. There was four or five of us. We thought like what we did day to day as product managers were starting to diverge as research PMs compared to other PMs. But then some of the things we were starting to do, you know, reading user consented feedback data or writing evals, those seem to be things that help our features too. So like, shouldn't more people do that? And so we just said, okay, let's write down how we do them. What like, what we look for, what we don't think is as valuable and see if people would, other parts of the team would want to come and join and listen. And so like a small group first did. And I do think a lot of it is replicable. Sometimes it's actually more to make it an open forum for people to ask questions because there's one piece about here's ways to improve Claude. But actually it's not that you have to just wait for the model. You could also prompt the model to get to the outcome you want as a PM. Like you have the power to go try to prompt, like to change the instructions And just doing it live reduces and demystifies it in a way where, you know, even our sales team would come, and they're like, oh, that's what an eval is. That's what prompting is. And so like, okay, I can do that. Like anyone, anyone can go, go ahead and learn. So it's more of like making it accessible. And I think it is something that like any company can, can do. Like there is enough, like knowledge is great, like things on the web. A lot of people have written about things like emails and prompting, obviously at this point. So I think they're all, they're all things that can be replicated and shared more scalably.
00:41:30 · Nikhyl Singhal
[00:41:30] S1: One area that I think we've talked a lot about is the challenge that large tech product managers face because they're in very structured environments that are moving information from one layer to another, and you're describing a pretty different job. I think a lot of other product managers listening to this, you know, work in, you know, SaaS or enterprise settings where there's direct user feedback that directly implements what the roadmap is. And, you know, there's a thoughtful question that came in through the live channel, which was around how do you manage the balance between research priorities and user feedback? And this is a conversation I had when I was talking to Tulsee, who's doing the sort of Gemini lab model as well, is that it's not quite the same thing as being able to say user wants X, telling the researcher, this is what we need to do next. Tell us a little bit about that finesse because that's a big part of the distinction between a traditional PM job and a research PM role.
00:42:32 · Dianne Penn
[00:42:32] S0: I think maybe one thing to start is just thinking about the fact that like researchers, particularly at Frontier Labs are thinking not just about the short term users, but actually users that are not here yet many times. And then the second order impact of like society. So like on the... Like particularly in areas like safety and alignment. So like when you open your aperture on like what we're trying to achieve and all being on the same page that it's not just today's user base, but it's a future user base and then and so on. Then I think that helps a lot because sometimes when we ask for why doesn't Claude do a little bit better on a JSON format or like a schema format, right? Like that could be researcher time away from building computer use or like making reasoning work better. And so it isn't necessarily that like it's a direct here is a PM roadmap of the exact current users, making it a bit better for them. But it's also you're trading off training and investment on like bigger bets. And so that's like where you like keeping an open amperage on that is really important to have like a good trust building conversation with researchers a lot of time. Because getting to the root of like what is important versus not. The other piece is when we're working on aligning on training changes or priorities, we try to figure out how much of a thing is actually important just for today versus it will naturally be less of a P zero as like model intelligence improves, which I think is a incredibly dissatisfying answer as a PM. Right? Cause like, cause we're like, we're kind of used to like, we are designing the right happy path. Here are some edge cases. Here is the constraints of the world. And this technology is changing. And so the problems that we had two years ago of, you know, we used to get big feedback of Claude didn't fully like use this memory well two years ago. We still get that feedback. But now the complexity of what that looks like is very different because the models did get better with more training and scaling. So part of it is just like working with your researchers on what matters versus not and why, and what are things then that like are worth, are breaking. And so like, if there is like, let's say the case of like Claude not using memory at well is, you know, making users churn, right? I'm not saying that we don't care about it.
00:45:14 · Dianne Penn
Maybe there's other ways in the user experience such that you constrain the type of this feature so that users still get a good experience while you wait for the model, like while you wait for like improvements in future model generations, because the core technology and the core intelligence will continue to improve. I guess, like maybe just to summarize, yes, thinking about different user groups. Yes. Sometimes it's a bit dissatisfying compared to working with a technology that is fully mature. And in particular, having a mindset of like, okay, what am I trying to achieve? What is in my circle of control to help the user and not necessarily assuming that it always has to be a change in research that is the best thing.