Lenny's Podcast: Product | Career | Growth
00:00:00 · Dianne Penn, Lenny Rachitsky
[00:00:00] S0: In 2023 when I started, nobody said Anthropic and Claude and coding in the same sentence. [00:00:06] S1: I want to go back to the beginning of Anthropic. I remember feeling, "Man, these guys have no chance. OpenAI is so far ahead." [00:00:14] S0: By the time I saw people were starting to use these models, not just for code autocomplete, but actually writing long-form code. Is that an opportunity for us to train Opus 3 to be better at? That was the inflection. [00:00:26] S1: I always think about Opus 4.5 a year later during winter break when everyone was home able to code. [00:00:31] S0: What was magical about Opus 4.5 is we also now not just had a model, but a vehicle, a great product experience like Claude Code. Opus 4.5 wouldn't have had that moment without a product like Claude Code and Claude Code wouldn't have had that type of adoption accelerated without Opus 4.5. [00:00:50] S1: I want to talk about how the product role is changing. [00:00:53] S0: For my team, the way to drive user value is to figure out the right user feedback. The evals, we actually have a saying on the team of evals are the new PRDs. [00:01:02] S1: Something Garry Tan's been talking about. If you are willing to spend $100,000 a year right now in tokens, you are living the way somebody in 2028 is going to live. [00:01:10] S0: You have to sweat the tokens as much as you sweat the pixels. You have to be using the models to come up with good and great and better ideas, and there's no substitute for that.
00:01:10 · Dianne Penn, Lenny Rachitsky
[00:01:10] S0: You have to sweat the tokens as much as you sweat the pixels. You have to be using the models to come up with good and great and better ideas, and there's no substitute for that. [00:01:20] S1: People need to be more ambitious with AI tools these days because they're just capable of so much. [00:01:26] S0: One thing I ask the team is let's say Claude 8 comes around. What changes in what users do? What does that mean for how you're building today? [00:01:36] S1: Today, my guest is Dianne Penn, head of product for the AI research and Labs teams at Anthropic. She joined Anthropic as the first technical product manager over three years ago, which is a lifetime in AI time. When the product team was just five engineers, she's helped ship every model at Anthropic from Claude 2 through Fable. She's also helped incubate and launch Claude Code, MCP, skills, Claude Design, and also core capabilities like computer use, tool use, and reasoning. [00:02:41] S0: Thank you, Lenny. It's so nice to see you again. [00:02:43] S1: I want to go back to the beginning of Anthropic, the early days. I remember when Anthropic first launched, this was, I don't know, the first model when it launched years ago, three years ago, something like that. [00:02:56] S0: It works.
00:02:56 · Dianne Penn, Lenny Rachitsky
[00:02:56] S0: It works. [00:02:56] S1: Three years. I remember just feeling that, man, these guys have no chance. OpenAI is so far ahead. They're just like, what are they thinking? How is this possible? OpenAI is one. It's too late. Things are very different now. The latest number I saw was Anthropic was making, I don't know, $50 billion in ARR. That's what companies used to go public at. Very successful companies went public at 50 billion in valuation. Anthropic reportedly is making that every single year. You joined as one of the earliest PMs. There were something like five engineers when you joined. The model hadn't even launched when you joined. What was it like in those early days of Anthropic? What's something that might surprise people about what it was like at the beginning?
00:03:44 · Dianne Penn, Lenny Rachitsky
[00:03:44] S0: I think a big part of what's made Anthropic today actually has been very much the core of even the early days. So I joined in 2023. Like you said, we had five product engineers. There was one engineer for the entirety of our API business, if you believe. And I think a big portion of it was the culture was really strong. And I think this is something I emphasize for folks who are interested in the company, really do walk the walk of the mission and the culture and the values. And the energy was very much like a startup. I think you're right, we were very much trying to find our identity in the early years. I think there's one piece around the technology, but how does that technology bring value to users, bring value to society, and what could it possibly be? And I think the early years were us exploring that in different ways. We did start with Claude.ai, another chatbot, chat assistant, and evolving into things like tool use. I think one of the moments where really we started to get into our groove was shipping things like Golden Gate Claude. I don't know if you remember that. [00:05:02] S1: No.
00:05:03 · Dianne Penn
[00:05:03] S0: So this was actually up for about 24 hours or so. We had just published one of our early interpretability research in early 2024. And one of the examples was essentially you could have what's called features of the model within the layers, which express certain types of thematics. So one of the themes that the researchers was able to identify was, let's say, bullet point writing. Another one was people and places. And one that really came up frequently that resonated was the Golden Gate Bridge. And so when you actually essentially dialed up that feature, Claude would obsess about the Golden Gate Bridge. So meaning in every one of its responses, it would come back and talk about the Golden Gate Bridge. So if you said like, "Give me a recipe for making spaghetti," it would say, " Here is a recipe and the orange color is just like international red that the Golden Gate Bridge looked like." And so it was really quirky. And we very much wanted to, in that situation, just bring it to the masses and bring it to people who were starting to use Claude. And so the entire experience actually we spun up on our Claude.ai website within 24 hours. And that took engineering, product, design, our research teams all working together. And we were really, really proud of it. I think it maybe reached only 2,000 people, to be honest. But it made us feel like, oh, we can actually bring new user experiences, showcase our research in a way that's different and authentic to us. And in a very startup be like pace. That to me was one of those maybe hidden inflection points of we were starting to find our identity, that we could build products, build experiences that were different from what our competitors had seen, what was already out there. And I think that obviously Labs, Claude Code, et cetera, we then started to identify ourselves as would we actually think the world, how to think about AI, how to bring that closer to the public. But it was a very bottoms-up culture. And so that entire experience was very bottoms up. I see engineers, I see designers donating time to work on. And so I like to always use that as an example of what the early days were like. But the culture and the values have very much, I think, stayed the same since those early days. Definitely when we were training and testing Opus 3, I think that was the moment when the company...
00:08:38 · Dianne Penn
I think we were less than 200 people still at that point. And it was very clear that we needed and wanted to create a frontier model. And that was very important in terms of our ability to reach users, consumers, and to showcase our research. And we were looking for ways for also why should somebody choose Claude? And that was a core question. And that was a core question we were getting asked in the early days. And I think with Opus 3, it launched, I think, early March 2024, but there was many, many months of various teens across inference, across research, fine-tuning, pre-training that rallied at different points and towards a common goal. And I think everybody that was involved was really proud. I remember being the PM, us, the research leads, myself, we were all in our... This was around December, so we were all at home in our various parents' homes and seeing everybody's background of their childhood room. And everybody was working really hard to figure out what are we training the model for? Is it showing up the right way? So I think that was really powerful in terms of just building a lot of trust. And a lot of our research leads have actually, from that time, are now leading reinforcement learning, leading our character work and lineman work. So that foundational trust I think also helped us work well now with any of our production models across product and research because we were working just so much in the trenches together in the early days. And then I think there were things like identifying that coding was important. In 2023 when I started, nobody said Anthropic and Claude and coding in the same sentence. I think competitor models like GPT-4 at the time was used a bit for coding, but it was one of many use cases. And one thing that, for example, I saw was people are starting to use these models, not just for code, not just code autocomplete, but actually writing long-form code. And is that an opportunity for us to train Opus 3 to be better at? And it ended up being a relatively smaller change from a training perspective, but it ended up helping us differentiate in the early days competitively for users and actually bring a lot of the very early Claude enthusiasts and developers because we were providing a value that they didn't really think was possible at the time.