Lenny's Podcast: Product | Career | Growth
01:01:22 · Lenny Rachitsky, Boris Cherny
[01:01:22] S1: Isn't that great? Yeah. [01:01:24] S0: I remember reading something, this was maybe very early versions of ACM magazine, or something, where people were saying, "No. It's not the same thing." Like, "This isn't really coding." And they called it a programming. I think coding is a new word. But I think about this. Back in the ... My family is from the Soviet Union. I was born in Ukraine. And my grandpa was actually one of the first programmers in the Soviet Union, and he programmed using punch cards. My mom growing up told these stories of ... Or she told these stories of when she was growing up, he would bring these punch cards home. And there was these big stacks of punch cards. And for her she would draw all over them with crayons, and that was her childhood memory. But for him that was, like, his experience of programming, and he actually never saw the software transition. But, at some point, it did transition to software. And I think there was probably this older generation of programmers that just didn't take software very seriously. And they would have been like, "Well, it's not really coding." But I think this is a field that just has always been changing in this way. [01:02:27] S1: I don't think you know this, but I was born in Ukraine also. [01:02:30] S0: Oh, I don't [inaudible 01:02:31]. Yeah. [01:02:31] S1: Yes. [01:02:32] S0: Which town? [01:02:32] S1: I'm from Odessa. [01:02:34] S0: Oh, me too. [01:02:35] S1: What? [01:02:36] S0: Yeah. That's crazy. [01:02:39] S1: Wow. Incredible. What a moment. Maybe related in some small way.
01:02:39 · Lenny Rachitsky, Boris Cherny
[01:02:39] S1: Wow. Incredible. What a moment. Maybe related in some small way. [01:02:44] S0: Yeah. [01:02:44] S1: What year did you leave? And your family leave. [01:02:48] S0: We came in '95. [01:02:50] S1: Okay. We left in '88. A little earlier. Yeah. What a different life that would have been to not leave home. [01:02:57] S0: Yeah. I feel so lucky every day that I got to grow up here. [01:03:02] S1: Yeah. My family, any time there's a toast or a meal, they're just like, "To America." [inaudible 01:03:07]. It's, like, "Okay. Enough about that. We get it." Once you start really thinking about what life could have been. [01:03:12] S0: Yeah. Yeah. Exactly. Yeah. We do the same toast, but it's still vodka. [01:03:16] S1: It's still vodka. Absolut. Oh, man. Okay. Let me ask you a couple more things here. You shared some really cool tips for how to get the most out of AI, how to build on AI, how to build great products in AI. One tip you shared is give your team as many tokens as they want. Just let them experiment. You also shared just advice, generally, of just build towards where the model is going, not to where it is today. What other advice do you have for folks that are trying to build AI products?
01:03:43 · Boris Cherny
[01:03:43] S0: I'd probably share a few more things. So, one is don't try to box the model in. I think a lot of people's instinct when they build on the model is they try to make it behave a very particular way. They're like, "This is a component of a bigger system." I think some examples of this are people layering very strict workflows on a model, for example, to say, like, "You must do step one, and then step two, then step three." And you have this very fancy orchestrator doing this. But actually almost always you get better results if you just give the model tools, you give it a goal, and you let it figure it out. I think a year ago you actually needed a lot of the scaffolding, but nowadays you don't really need it. So, I don't know what to call this principle, but it's, like, ask not what the model can do for you. Maybe it's something like this. Just think about how do you give the model the tools to do things. Don't try to over-curate it. Don't try to put it into a box. Don't try to give it a bunch of context upfront. Give it a tool, so, that it can get the context it needs. You're just going to get better results. I think a second one is maybe actually even a more general version of this principle is just the bitter lesson. And actually for the Claude Code team we have a ... Hopefully, listeners have read this, but [inaudible 01:04:52] this blog post maybe 10 years ago called The Bitter Lesson. And it's actually a really simple idea. His idea was that the more general model will always out-perform the more specific model. And I think for him he was talking about self-driving cars and other domains like this. But actually there's just so many corollaries to the bitter lesson, and, for me, the biggest one is just always bet on the more general model. And over the long-term. Like, don't try to use tiny models for stuff. Don't try to fine-tune. Don't try to do any of this stuff. There's some applications. There's some reasons to do this, but almost always try to bet on the more general model, if you can, if you have that flexibility. And so, these workflows are, essentially, a way that ... It's not a general model. It's putting the scaffolding around it. And, in general, what we see is maybe scaffolding can improve performance maybe 10%, 20%, something like this, but often these gains just get wiped out with the next model.
01:05:50 · Boris Cherny
So, it's almost better to just wait for the next one. And I think maybe this is a final principle, and something that Claude Code I think got right in hindsight, from the very beginning, we bet on building for the model six months from now. Not for the model of today. And for the very early versions of the product, it just wrote so little of my code, because I didn't trust it. Because it was, like, Sonnet 3.5. Then it was, like, 3.6, or ... I forget. 3.5 New, whatever name we gave it. These models just weren't very good at coding yet. They were getting there, but it was still pretty early. So, back then the model did [inaudible 01:06:26] it automated some things, but it really wasn't doing a huge amount of my coding. And so, the bet with Claude Code was, at some point, the model gets good enough that it can just write a lot of the code. And this is a thing that we first started seeing with Opus 4 and Sonnet 4, and Opus 4 was our first ASL3 class model that we released back in May. And we just saw this inflection, because everyone started to use Claude Code for the first time. And that was when our growth really went exponential. And, like I said, it stayed there. So, I think this is advice that I actually gave to a lot of folks, especially, people building startups. It's going to be uncomfortable, because your product market fit won't be very good for the first six months. But if you build for the model six months out, when that model comes out, you're just going to hit the ground running, and the product is going to click, and start to work.
01:07:15 · Lenny Rachitsky, Boris Cherny
[01:07:15] S1: And when you say build for the model six months out, what is it that you think people can assume will happen? Is it just generally it will get better at things? Is it just, like, "Okay. It's almost good enough, and that's a sign that it'll probably get better at that thing"? Is there any advice there? [01:07:30] S0: I think that's a good way to do it. Obviously, within an AI lab, we get to see the specific ways that it gets better. [01:07:36] S1: Yeah.
01:07:36 · Lenny Rachitsky, Boris Cherny
[01:07:36] S1: Yeah. [01:07:37] S0: So, it's a little unfair, but also, we try to talk about this. So, one of the ways that it's going to get better is it's going to get better and better at using tools, and using computers. This is a bet that I would make. Another one is it's going to get better and better for running it for long periods of time. And this is a place ... Like, there's all sorts of studies about this, but if you just trace the trajectory, or maybe even for my own experience when I use Sonnet 3.5 back a year ago, it could run for maybe 15 or 30 seconds before it started going off the rails, and you just really had to hold its hand through any kind of complicated task. But nowadays with Opus 4.6, on average it'll run maybe 10, 20, 30 minutes unattended, and I'll just start another Claude, and have it do something else. And, like I said, I always have a bunch of Claudes running. And they can also run for hours, or even days at a time. I think there were some examples where they ran for many weeks. And so, I think over time this is going to become more and more normal where the models are running for a very, very long period of time, and you don't have to sit there and babysit them anymore. [01:08:39] S1: So, you just talked about tips for building AI products. Any tips for someone just using Claude Code, say, for the first time or just someone already using Claude Code that wants to get better. What are a couple pro-tips that you could share?