Lenny's Podcast: Product | Career | Growth
00:08:38 · Dianne Penn
I think we were less than 200 people still at that point. And it was very clear that we needed and wanted to create a frontier model. And that was very important in terms of our ability to reach users, consumers, and to showcase our research. And we were looking for ways for also why should somebody choose Claude? And that was a core question. And that was a core question we were getting asked in the early days. And I think with Opus 3, it launched, I think, early March 2024, but there was many, many months of various teens across inference, across research, fine-tuning, pre-training that rallied at different points and towards a common goal. And I think everybody that was involved was really proud. I remember being the PM, us, the research leads, myself, we were all in our... This was around December, so we were all at home in our various parents' homes and seeing everybody's background of their childhood room. And everybody was working really hard to figure out what are we training the model for? Is it showing up the right way? So I think that was really powerful in terms of just building a lot of trust. And a lot of our research leads have actually, from that time, are now leading reinforcement learning, leading our character work and lineman work. So that foundational trust I think also helped us work well now with any of our production models across product and research because we were working just so much in the trenches together in the early days. And then I think there were things like identifying that coding was important. In 2023 when I started, nobody said Anthropic and Claude and coding in the same sentence. I think competitor models like GPT-4 at the time was used a bit for coding, but it was one of many use cases. And one thing that, for example, I saw was people are starting to use these models, not just for code, not just code autocomplete, but actually writing long-form code. And is that an opportunity for us to train Opus 3 to be better at? And it ended up being a relatively smaller change from a training perspective, but it ended up helping us differentiate in the early days competitively for users and actually bring a lot of the very early Claude enthusiasts and developers because we were providing a value that they didn't really think was possible at the time.
00:12:10 · Lenny Rachitsky, Dianne Penn
[00:12:10] S1: It's so interesting you talk about Opus 3 like that's so long ago and just it's hard to think that was a big inflection. And so this is really interesting to hear that that was internally a big milestone. It almost feels like this confidence y'all built that, wow, we could really ship a frontier model, which is now today so not great if you compare it to what we've got today. What I always think about is Opus 4.5, which was an interestingly a year later also during winter break when everyone was home able to code. Was that another big milestone? [00:12:38] S0: Yeah. Opus 4.5 was definitely another large moment. I think what was magical about Opus 4.5 is we also now not just had a model, but a vehicle, which is a great product experience like Claude Code. One thing we say a lot on the team is you need frontier products in order to have frontier models and for people to feel the magic of frontier models. And I think we felt the magic of Claude Code for many months before that. But the fact that the model essentially got to a level of intelligence where at a very broad level, users can experience both frontier intelligence in new use cases, allow it to run things end to end in agentic manner, I think that was the inflection. It was actually both. I think Opus 4.5 wouldn't have had that moment without a product like Claude Code. And Claude Code, I think, wouldn't have had that type of adoption accelerated without Opus 4.5.
00:13:50 · Lenny Rachitsky
[00:13:50] S1: So speaking on this thread, Dario, interestingly, if you look back at all his predictions, he's just like, "Okay, coding's going to be solved 100% in a year." Something like that. He kept talking about how AI is going to do all our code. And I remember everyone being like, "There's no way. This is way too complicated. How is AI ever going to get really good at this very complex thing that humans do? No, this is going to be humans for a long time." He was completely right. Something else that he talks a lot about is this exponential. Now that we're on, that's the way he describes it, now we're like, we're on the exponential curve. I remember not long ago, new models were being released and everybody was like, "Okay, we're done. There's no more upside. It's plateauing. It's over. There's no more room to grow." And now it's the opposite. Now we're inside. If you think about the curve of the exponential, we're inside of the exponential now, which by definition means every improvement is a massive jump because we're on that hockey stick part. What's it like just being on the inside of this crazy historic moment when AI is improving so fast, so much is being unlocked? What is it like and how should people prepare for the coming acceleration of more and more improvement from AI?
00:15:05 · Dianne Penn
[00:15:05] S0: One thing I like to say on the team is most of us weren't actively working yet when the internet transitioned from this novelty to something that everyone can use. And it feels like that's just taking humans... I think analogies are helpful. And so the analogy of that is I think a couple of things. Number one is adaptability becomes very important. I think we have evals, we have on the safety side, safety testing, red teaming on the capabilities and product side, new prototypes, products like Claude Code, Tag and others. But it's very hard to predict the exact moment or the exact model. And so the adaptability of when you're faced with new information, how do you then make better decisions versus keeping the same plan? And so that agility is really important. I think another piece is with that, how do you actually be thinking very first principles and reason through what's next? What's the so what? How do we invest in new products? How do we invest in explaining the differences to users? So a lot of the experiences I think of being in that exponential is that pace, understanding how you operate and make better decisions, and then applying that first principle's thinking to then do something that maybe we pull up a plan that we're expecting a few months from now, but now the model can actually do and work on and actually bring that to users. So this is things like co-work, skills, tag. It's a very positive self-enforcing loop. And I think a big part of it also is just having the trust in each other, making sure we're thinking through the right decision making. We're bringing folks along. Some teams might see the exponential feel it faster than others. So how do we have the grace to bring the organization, the growing organization and company along on that?
00:17:23 · Lenny Rachitsky
[00:17:23] S1: So what I'm hearing here is you almost don't know what will be possible with every model release. And so the important things to focus on is being adaptable as things emerge. To your point, the product itself has to catch up to what is possible. To your point again, just like it can do so much, but people may not understand how to do it and may not be able to do it. So the product making it easy and even just telling you, here's something you could do feels like an important part. Is that roughly what you're describing?
00:17:53 · Dianne Penn, Lenny Rachitsky
[00:17:53] S0: I think so. I think there's some really interesting graphs in the original scaling law papers. And I think folks are very familiar with the scaling loss in the lens of as you add in more compute and data, what's called loss, AKA the loss from next token prediction goes down. And so it's a very smooth linear curve of the models get more intelligent as you scale them up. What's actually also interesting in that paper is there are these very different emerging capability graphs. And so for example, as you add in more data and you train the models with more compute, you essentially see these actually discontinuous emerging capabilities jump. So the models go from one plus one being a thing that it can't calculate to a thing that it could reliably calculate. And so these emerging capabilities, this some nature of predictability is not necessarily everyone knows the exact moment. You need the evals to be able to assess that has actually always been a part of how this technology works. And also what makes things like safety harder. Because unless you have the evals, unless you have the systems to test, these jumps might actually happen and you don't know. [00:19:16] S1: That's so interesting that you may have developed this AI brain that can do something you're not even aware of. And so part of the job is just uncovering, "Wow, we just got really good at this thing. What can we do with that?"