Lenny's Podcast: Product | Career | Growth
00:24:26 · Lenny Rachitsky, Dianne Penn
[00:24:26] S1: I've heard of it. [00:24:29] S0: Things like Claude Code, things like skills, and most recently Claude Design, MCP. The thing that we really try to emphasize within the teams is, especially right now, there are so many things that could be built. What does it mean then to have a discontinuous bet? And I think one approach that we're taking this year is you can be very strongly held opinion about the theme or the area, and then more weakly held about the exact prototype. And so there's a culture of experimentation. There's a lot of the bottoms up engineers on the team are very self-enabled, self-driven to test out different ideas. And sometimes, we have a thesis, and it might not work yet, and so we then might revisit it in one to two model generations. And so this idea of these prototypes that actually end up just helping us learn, that's also valuable even if it doesn't lead to something immediately shipping. And so I think that allows the incubation and the charter of Labs to really accelerate and see around corners more broadly for Anthropic. [00:25:44] S1: It's so funny to think about Labs within an Anthropic, which was already so innovative and creative and just shipping like crazy, that there's value to still creating a Labs team within Anthropic. What enables Labs to work as well as it has? Because you listed all these products and it's like, what else has Anthropic shipped? It feels like all the biggest wins almost. I'm sure there are many that I'm not thinking about right now. What's core to creating a successful Labs org within a larger company?
00:25:44 · Lenny Rachitsky, Dianne Penn
[00:25:44] S1: It's so funny to think about Labs within an Anthropic, which was already so innovative and creative and just shipping like crazy, that there's value to still creating a Labs team within Anthropic. What enables Labs to work as well as it has? Because you listed all these products and it's like, what else has Anthropic shipped? It feels like all the biggest wins almost. I'm sure there are many that I'm not thinking about right now. What's core to creating a successful Labs org within a larger company? [00:26:12] S0: I think the team culture, similar to broadly at Anthropic, I think the team culture is very valuable. I think Ben sets an incredible vision and pushes people to think about the 10X, 100X of the idea. And the teams, the pods within Labs is small, sometimes these ideas start with one engineer. And I think sometimes when there's almost really large teams pursuing very ambiguous large ideas, you end up actually being slowed down because of that. So I think it's culture. I think we actually also select for folks who actually want to do that zero to one experimentation. And it's not easy. There's a lot of bets that we end up turning down or turning off. And maybe we revisit them in the future, but that's hard. That's hard when you pour your heart and soul, you're acting as a founder for a bet and it's not working yet. So I think it's like that type, it's selecting for that type of personality, folks who are really passionate and deep about the zero to one.
00:27:30 · Lenny Rachitsky
[00:27:30] S1: So you lead product for the research team, you work with the researchers at Anthropic. A lot of people get a sense of what is research. What do researchers do? I think a lot of people don't totally understand these very valuable people at all the AI labs. The way I think about it, and I want to help people understand, help me understand just what are researchers doing all day. What I imagine is they have a hypothesis for how to improve the model. They find data, they tweak some algorithms, they adjust how it's trained, and they test it, see how it did, keep iterating and keep trying to find ways to improve the model. Is that roughly right? Slash help us understand what researchers are doing all day.
00:28:09 · Dianne Penn
[00:28:09] S0: I think that's a lot of maybe the more day-to-day. I think one piece around researchers and research organizations like at Anthropic is there's also a vision of the future more broadly. So for example, things like, I think even at the founding of the company, researchers were talking about how do we get Claude to use a computer? How do we get AI to navigate a screen? So there's a lot of actually very founder-like energy is how I describe it within researchers are really bold and ambitious researchers, and we have a ton of those at Anthropic. So there's one layer of vision of what this technology can go. And then I think on this other side of the loop, there's also, now that this technology or Claude is in people's hands, how do we make it better today? So it's a medium and long-term, and a lot of energy thinking about that lens of the future. And also, in the immediate and short term, what are the improvement areas we can make? And so I think you're describing a really good sense of how do we make iterative improvements on different versions of Claude? The way that my team works with researchers is being very integrated and embedded in those loops, particularly areas where there's a lot of impact on users. So this is things like vision, computer use, coding, agentic coding, tool use, test time compute, things where there's a direct user impact. And then figuring out what are the ways to bring the user feedback, and ground it in a level that is understandable for researchers, and also actionable for researchers. And I think that's the second piece is actually a big part of the job and sometimes a hard part of the job. So for example, we might get feedback on claude.ai. "Claude hallucinated." It's very vague. If you bring that to a researcher and you say, "Please fix Claude from being hallucinated," it's not very actionable. And so part of the time of the team is understanding, okay, what's the trajectory of why that user gave that feedback? And it's consented. And so we look at, okay, what should Claude have called tools in that moment? Or from its current knowledge or it looked at the right document, but it looked at the wrong facts. In the first case, that would have been a failure on tool use.
00:30:54 · Dianne Penn, Lenny Rachitsky
On the second case, it would've been a failure on, let's say, search or knowledge insertion, search synthesis, or it could be something around alignment. And so bring that level of detail to researchers coming up with, is this a big enough problem? Figure out things like evals to then describe what we've improved it. Those are the levels of actionability, and it's the day-to-day language of their researchers. And so we try to stay very close to how to bring that in an actionable manner between users to the core model training and the research development loop. [00:31:36] S1: I was talking to someone the other day about how it feels like AI research is the place to be now if you want to be very successful in life. What does it take to become a really successful researcher from which you can tell? Not everyone's brain is going to work this way, but just say people are like, "Hey, I want to explore this career path." From what you've seen, what does it take to make it there? [00:31:59] S0: Researchers generally, or research and product managers working with research, or both? [00:32:04] S1: Let's do both. [00:32:04] S0: Yeah. [00:32:05] S1: But the researchers, PMs working with researchers also going to be very successful, but it feels like everyone's trying to poach all the top researchers across every company. So I know you're not an AI researcher, but just from what you've seen, just what does it take to make it in that career path?
00:32:05 · Lenny Rachitsky, Dianne Penn
[00:32:05] S1: But the researchers, PMs working with researchers also going to be very successful, but it feels like everyone's trying to poach all the top researchers across every company. So I know you're not an AI researcher, but just from what you've seen, just what does it take to make it in that career path? [00:32:22] S0: Yeah. I think a lot of the most successful researchers and research leadership at Anthropic are folks who are really strong first principles thinkers about problems. They reason through problems really well, who are just passionate about their research area and have a bold description of what that could look like. And then who are actually close to the details. And so our leadership, our chief scientists, our heads of fine-tuning and RL, folks are actually really close to the training runs and actually look at things like how the training run is going, evals, looking at the underlying data. So actually staying really close and be excited to be in the details, I think have been a sign of really strong researchers and developing taste. And I think another piece is just their ability to think big over time and be very ambitious. Like the Dario, like we can transform software engineering. And I think going in that direction, you learn so much. You have to shoot for the stars in many ways across your ideas, I think, in order to be a successful researcher.