Lenny's Podcast: Product | Career | Growth
00:32:22 · Dianne Penn, Lenny Rachitsky
[00:32:22] S0: Yeah. I think a lot of the most successful researchers and research leadership at Anthropic are folks who are really strong first principles thinkers about problems. They reason through problems really well, who are just passionate about their research area and have a bold description of what that could look like. And then who are actually close to the details. And so our leadership, our chief scientists, our heads of fine-tuning and RL, folks are actually really close to the training runs and actually look at things like how the training run is going, evals, looking at the underlying data. So actually staying really close and be excited to be in the details, I think have been a sign of really strong researchers and developing taste. And I think another piece is just their ability to think big over time and be very ambitious. Like the Dario, like we can transform software engineering. And I think going in that direction, you learn so much. You have to shoot for the stars in many ways across your ideas, I think, in order to be a successful researcher. [00:33:49] S1: I love just this meme of just be more ambitious comes up so often now, which is so hard. It's easy to say that it's hard to actually just like how big can you think, and how that's so much of what AI now unlocks, just be more ambitious. [00:34:01] S0: Yeah. [00:34:02] S1: Yeah.
00:34:02 · Lenny Rachitsky, Dianne Penn
[00:34:02] S1: Yeah. [00:34:03] S0: I think it's thinking through it once or twice end-to-end. And then being, I think, stubborn about the area and maybe more loose around the exact approach. It is a question we challenge ourselves with, but the technology is moving so quickly, and so how do you make sure what you're building is actually forward compatible? And so it's also actually part of, I think, the core product development loop to think bigger. One thing I ask the team frequently or how I think about when we're building a product is let's say Claude 8 comes around. What changes in what users do? And then what does that mean for how you're building today? Is it going to be forward compatible to that experience? So just grounding, I think being ambitious is very broad. And so trying to ground it in some ways of describing that.
00:35:13 · Lenny Rachitsky
[00:35:13] S1: And also, yeah, everything heading in a direction that all is cohesive and makes sense versus just ambitious in a completely different direction. Speaking of ambition and Claude 8, Fable/Mythos recently feels like hit this very new tipping point with models where it used to be you have an awesome model, release it. Hey everyone, welcome. Opus 4.5 is out. Everyone can use it. Mythos went in a very different direction. We got blocked. There was a lot of scrutiny, a lot of concern about what it was capable of. All the companies had to go make sure it wasn't going to hack into all their systems. And it feels like now every model, because they continue to get better, will now have a lot more scrutiny and there'll be more restrictions on who can use them, which feels like a big deal. How do you think about that? How does that change the way you operate?
00:36:02 · Dianne Penn
[00:36:02] S0: I'm going to maybe leave the policy and the export control side to folks on that and work on that. I think the product question and how we interact with these internally is, I think as you mentioned, as frontier models become more capable, the safeguards and the ways of red teaming and testing and the pre-release process also needs to evolve and adapt quickly to address that. And so one example is before Fable models, we didn't have as strong of, let's say, fallback UXs and systems because our goal was to make sure that there is asymmetrical benefit for this technology and to minimize the downside or a severe risk of it. And so we ended up building fallback systems so that users will still get a great response from Opus 4.A immediately. And so I think there's a piece around as we evolve and improve safety systems, how do we continue to develop and deliver great user experiences? I think there's more that we can do on both sides. And so you'll see us innovating, improving on what we call now the model safeguards package more and more in the coming weeks and month.
00:37:29 · Lenny Rachitsky, Dianne Penn
[00:37:29] S1: What's really interesting and just unexpected here is it creates this really interesting advantage for Anthropic where you have access to the latest stuff. And this is going to happen at every lab. Everyone's going to keep improving and it creates this unfair advantage within the Labs to have access to the best stuff that other people can't yet outside of your control. You'd prefer everyone use it. So it's a really interesting, this new feedback loop that's going to start where models that are so advanced are only accessible to certain companies, and that's going to be a whole new unexpected, it's like a second order effect of all these restrictions. [00:37:59] S0: Our goal is to be to develop these systems in the models to be as inclusive as possible. I think our goal is to not have that happen for the general purpose, general use technologies and to make it more accessible. I think this is one of our top priorities right now to reduce what we're seeing there. [00:38:21] S1: Yeah, that makes sense. I would imagine you'd want as many customers if people are using this thing as possible.
00:38:21 · Lenny Rachitsky, Dianne Penn
[00:38:21] S1: Yeah, that makes sense. I would imagine you'd want as many customers if people are using this thing as possible. [00:40:04] S0: We actually, on my team, have not changed our hiring loop for three years now. So what we actually look for and the traits and how we evaluate generalists like PMs, generalists like research, product managers have actually been the same. So I think some of those traits, number one is first principles thinking. And this is really rather than pattern matching what you used to do in, let's say, consumer product or B2B SaaS, but actually figuring out in this moment for this user group with this technology, what is the user value? [00:40:50] S1: Is there an example of that? A lot of people hear first principles thinking, they're like, "Yes, I got it." I'm good at this. What's an example of someone having really demonstrated really good first principles thinking?