Lenny's Podcast: Product | Career | Growth
00:20:25 · Lenny Rachitsky
[00:20:25] S1: Have you noticed that it is getting much better at this? Because this is the holy grail. Right now it's, cool, building solved. Code review became the next bottleneck with all these PRs. Who is going to review them all? The next big open question is just, like, "Okay. Now humans are necessary for figuring out what to build, what to prioritize," and you're saying that's where Claude Code is starting to help you. Has it gotten a lot better with, like, Opus 4.6, or what's been the trajectory there?
00:20:50 · Boris Cherny
[00:20:50] S0: Yeah. Yeah. It's improved a lot. I think some of it is training that we do specific to coding. So, obviously, the best coding model in the world, and it's getting better and better. Like, 4.6 is just incredible. But also actually a lot of the training that we do outside of coding translates pretty well too. So, there is this transfer where you teach the model to do X, and it gets better at Y. Yeah. And the gains have just been insane. Like, at Anthropic, over the last year, like, since we introduced Claude Code, we probably ... I don't know the exact number. We probably 4X-ed the engineering team, or something like this. But productivity per engineer has increased 200% in terms of pull requests. And this number is just crazy for anyone that actually works in the space and works on dev productivity. Because back in a previous life, I was at Meta, and one of my responsibilities was code quality for the company. So, this is all of our code bases, that was my responsibility. Like, Facebook, Instagram, WhatsApp, all this stuff. And a lot of that was about productivity, because if you make the code higher quality, then engineers are more productive. And things that we saw is in a year with hundreds of engineers working on it, you would see a gain of a few percentage points of productivity, something like this. And so, nowadays, seeing these gains of just hundreds of percentage points is just absolutely insane.
00:22:06 · Lenny Rachitsky
[00:22:06] S1: What's also insane is just how normalized this has all been. Like, we hear these numbers. Like, of course, AI is doing this to us. It's so unprecedented, the amount of change that is happening to software development, to building products, to just the world of tech. It's just so easy to get used to it, but it's important to recognize this is crazy.
00:22:25 · Boris Cherny
[00:22:25] S0: This is something I have to remind myself once in a while. There's a downside of this, because the model changes so ... Well, there's many downsides that we could talk about, but I think one of them on a personal level is the model changes so often that I sometimes get stuck in this old way of thinking about it. And I even find that new people on the team, or even new grads that join do stuff in a more AGI-forward way than I do. So, sometimes, for example, I had this case a couple of months ago where there was a [inaudible 00:22:57] week. And so, what this is is Claude Code, the memory usage is going up, and, at some point, it crashes. This is a very common engineering problem that every engineer has debugged 1000 times. And, traditionally, the way that you do it is you take a heap snapshot, you put it into a special debugger, and figure out what's going on. You use these special tools to see what's happening. And I was doing this, and I was looking through these traces, and trying to figure out what was going on, and the engineer that was newer on the team just had Claude Code do it. And was like, "Hey, Claude. It sounds like there's a leak. Can you figure it out?" And so, Claude Code did exactly the same thing that I was doing. It took the heap snapshot, it wrote a little tool for itself, so, it can analyze it itself. It was a just in time program. And it found the issue, and put up a pull request faster than I could. So, it's something where for those of us that have been using the model for a long time, you still have to transport yourself to the current moment, and not get stuck back in old model, because it's not Sonnet 3.5 anymore. The new models are just completely, completely different. And just this mindset shift is very different.
00:24:03 · Lenny Rachitsky
[00:24:03] S1: I hear you have these very specific principles that you've codified for your team that when people join you walk them through them. I believe one of them is, "What's better than doing something? Having Claude do it." And it feels like that's exactly what you describe with this memory leak is you almost forgot that principle of like, "Okay. Let me see if Claude can solve this for me."
00:24:21 · Boris Cherny
[00:24:21] S0: There's an interesting thing that happens also when you under-fund everything a little bit, because then people are forced to Claude-ify. And this is something that we see so ... For work, where sometimes we just put one engineer on a project, and the way that they're able to ship really quickly ... Because they want to ship quickly. This is an intrinsic motivation that comes from within. It's just wanting to do a good job. If you have a good idea, you just really want to get it out there. No one has to force you to do that. That comes from you. And so, if you have Claude, you can just use that to automate a lot of work, and that's what we see over and over. So, I think that's one principle is under-funding things a little bit. I think another principle is just encouraging people to go faster. So, if you can do something today, you should just do it today. And this is something we really, really encourage on the team. Early on, it was really important, because it was just me. And so, our only advantage was speed. That's the only way that we could ship a product that would compete in this very crowded coding market. But nowadays, it's still very much a principle we have on the team, and if you want to go faster, a really good way to do that is to just have Claude do more stuff. So, it just very much encourages that.