Lenny's Podcast: Product | Career | Growth
00:49:43 · Boris Cherny
Maybe we should just build a product for them. And we saw this actually pretty early. Back in maybe May of last year, I remember walking into the office and our data scientist Brendan had a Claude Code on his computer. He just had a terminal up. And I was shocked. I was like, "Brendan, what are you doing?" He figured out how to open the terminal, which is ... It's a very engineer-y product. Even a lot of engineers don't want to use a terminal. It's just the lowest level way to do your work. Just really, really in the weeds of the computer. And so, he figured out how to use the terminal. He downloaded [inaudible 00:50:28]. He downloaded Claude Code. And he was doing SQL analysis in the terminal. It was crazy. And then the next week all of the data scientists were doing the same thing. So, when you see people abusing the product in this way, using it in a way that it wasn't designed in order to do something that is useful for them, it's just such a strong indicator that you should just build a product and people are going to like that. It's something that's special purpose for that. I think now there's also this interesting second dimension to latent demand. This is the traditional framing is look at what people are doing, make that a little bit easier, empower them. The modern framing that I've been seeing in the last six months is a little bit different. And it's look at what the model is trying to do, and make that a little bit easier. And so, when we first started building Claude Code, I think a lot of the way that people approached designing things with LLMs is they put the model in a box, and they were like, "Here's this application that I want to build. Here's the thing that I wanted to do, a model, you're going to do this one component of it. Here's the way that you're going to interact with these tools and APIs," and whatever. And for Claude Code, we inverted that. We said, "The product is the model. We want to expose it. We want to put the minimal scaffolding around it. Give it the minimal set of tools. So, it can do the things, it can decide which tools to run. It can decide in what order to run them in," and so on. And I think a lot of this was just based on latent demand of what the model wanted to do. And so, in research we call this being on distribution. You want to see what the model is trying to do.
00:51:50 · Boris Cherny, Lenny Rachitsky
In product terms, latent demand is just the same exact concept but applied to a model. [00:51:55] S1: You talked about Cowork. Something that I saw you talk about when you launched that initially is your team built that in 10 days. That's insane. [00:52:02] S0: Yeah. [00:52:02] S1: It came out, I think it was used by millions of people pretty quickly, something like that being built in 10 days. Anything there? Any stories there other than it was just, "We used a lot of code to build it and that's it"?
00:52:14 · Boris Cherny
[00:52:14] S0: Yeah. It's funny. Claude Code, like I said, when we released it it was not immediately a hit. It became a hit over time, and there was a few inflection points. So, one was, like, Opus 4, it just really, really inflected, and then in November it inflected, and it just keeps inflecting. The growth just keeps getting steeper and steeper and steeper every day. But for the first few months it wasn't a hit. People used it, but a lot of people couldn't figure out how to use it. They didn't know what it was for. The model still wasn't very good. Cowork when we released it it was just immediately a hit. Much more so than Claude Code was early on. I think a lot of the credit, honestly, just goes to Felix and Sam and Jenny, and the team that built this. It's just an incredibly strong team. And, again, the place Cowork came from is just this latent demand. Like, we saw people using Claude Code for these non-technical things. And we're trying to figure out, "What do we do?" And so, for a few months the team was exploring, they were trying all sorts of different options. And, in the end, someone was just like, "Okay. What if we just take Claude Code and put it in the desktop app?" And that's, essentially, the thing that worked. And so, over 10 days they just completely used Claude Code to build it. And Cowork is actually ... There's this very sophisticated security system that's built in. And, essentially, these guardrails to make sure that the model does the right thing, it doesn't go off the rails. So, for example, we ship an entire virtual machine with it, and Claude Code just wrote all of this code. So, we just had to think about, "All right. How do we make this a little bit safer? A little more self-guided for people that are not engineers." It was fully implemented with Claude Code. It took about 10 days. We launched it early. It was still pretty rough, and it's still pretty rough around the edges. But this is the way that we learn, both on the product side and on the safety side is we have to release things a little bit earlier than we think, so, that we can get the feedback, so, that we can talk to users. We can understand what people want, and that'll shape where the product goes in the future.
00:54:05 · Lenny Rachitsky
[00:54:05] S1: Yeah. I think that point is so interesting, and it's so unique. There's always been this idea, release early, learn from users, get feedback, iterate. The fact that it's hard to even know what the AI is capable of, and how people will try to use it is a unique reason to start releasing things early. So, that'll help you as you exactly describe this idea of what is a latent demand in this thing that we didn't really know? Let's put it out there and see what people do with it.
00:54:30 · Boris Cherny
[00:54:30] S0: Yeah. And for Anthropic as a safety lab, the other dimension of that is safety, because when you think about model safety there's a bunch of different ways to study it. The lowest level is alignment, and mechanistic interpretability. So, this is when we train the model we want to make sure that it's safe. We, at this point, have pretty sophisticated technology to understand what's happening in the neurons to trace it. And so, for example, if there's a neuron related to deception, we're starting to get to the point where we can monitor it and understand that it's activating. And so, this is alignment, this is mechanistic interpretability, it's the lowest layer. The second layer is evals, and this is, essentially, a laboratory setting, the model is in a Petri dish, and you study it. And you put in the synthetic situation and just say, "Okay. Model, what do you do?" And, "Are you doing the right thing? Is it aligned? Is it safe?" And then the third layer is seeing how the model behaves in the wild. And as the model gets more sophisticated, this becomes so important, because it might look very good on these first two layers, but not great on the third one. We released Claude Code really early, because we wanted to study safety. And we actually used it within Anthropic for I think four or five months, or something before we released it, because we weren't really sure. Like, this is the first big agent that I think folks had released at that point. It was definitely the first coding agent that became broadly used. And so, we weren't sure if it was safe. And so, we actually had to study it internally for a long time before we felt good about that. And even since there's a lot that we've learned about alignment. There's a lot that we've learned about safety, that we've been able to put back into the model, back into the product. And for Cowork, it's pretty similar. The model is in this new setting. It's doing these tasks that are not engineering tasks. It's an agent that's acting on your behalf. It looks good on alignment, it looks good on evals, we tried it internally, it looks good. We tried it with a few customers, it looks good. Now we have to make sure it's safe in the real world. And so, that's why we release a little early. That's why we call it a research preview. But, yeah. It's constantly improving.
00:56:26 · Boris Cherny, Lenny Rachitsky
And this is really the only way to make sure that over the long-term the model is aligned, and it's doing the right things. [00:56:33] S1: It's such a wild space that you work in where there's this insane competition and pace. At the same time, there's this fear that if the God can escape and cause damage, and just finding that balance must be so challenging. What I'm hearing is there's these three layers, and I know there's ... This could be a whole podcast conversation is how you all think about the safety piece, but just what I'm hearing is there's these three layers you work with. There's observing the model thinking and operating. There's tests, evals that tell you this is doing bad things. And then releasing it early. I haven't actually heard a ton about that first piece. That is so cool. So, you guys can ... There's an observability tool that can let you peak inside the model's brain and see how it's thinking and where it's heading.