Lenny's Podcast: Product | Career | Growth
00:20:41 · Lenny Rachitsky
[00:20:41] S1: There's going to be a lot to talk about on the security side. You've done a lot of thinking and writing about the dangers there, but I want to follow this thread. So, in terms of what AI's been doing for teams. If you think about it, it's kind of going on the middle and expanding, so it's taking on more and more of the building components. It's doing code reviews now with QA, as you've been describing, constantly building, and it feels like the front of that is the big now gap in opportunity, which is coming up with the idea. What the heck should we build? 'Cause then once you tell the AI, build this thing, as you're describing, it's getting better and better at building something great. Have you had any luck yet with using AI there? And do you think it starts to eat that and just becomes the strategy PM basically?
00:21:27 · Simon Willison
[00:21:27] S0: So, one of the most interesting problems we're having with all of this is we've taken the writing code bit and we've massively accelerated that, now the bottlenecks are everywhere else, right? How do we redesign our processes? Now that the bit that used to take the longest, it used to be you'd come up with a spec and you handed it to your engineering team, and three weeks later, if you're lucky, they'd come back with an implementation for you to then start. Now maybe that takes three hours, depending on how well-established the coding agents are for that kind of thing. So, now what, right? Now, where else are the bottlenecks? I mean, there's coming up with the initial ideas. Anyone who's done any product work knows that your initial ideas are always wrong. What matters is proving them, it's testing them. We can test things so much faster now, because we can build workable prototypes so much quicker. So there's an interesting thing I've been doing in my own work where any feature that I want to design, I'll often prototype three different ways it could work, 'cause that takes very little time, and then I can start experimenting them and trying them and seeing which ones I like. That feels to me like the really transformational step here is that when you get AI involved in your ideation phase, it's much more about the prototypes. It's about, okay, we can see a UI prototype is free. Now, ChatGPT and Claude will just build you a very convincing UI for anything that you describe, and that's how you should be working. I think anyone who's doing product design isn't vibe coding little prototypes is missing out on the most powerful boost that we get in that step. But then what do you do, right? Given your three options now that you have instead of one option, how do you prove to yourself which one of those is the best? I don't have a confident answer to that. I expect this is where the good old-fashioned usability testing comes in. Get somebody on Zoom, screen shared using your software, see what happens. You can tell the AI to do it and you can simulate your users with the AI. I don't think that's credible. I don't think you're going to get as good results from ChatGPT pretending to click around on your prototype than you would from an actual human being.
00:23:37 · Lenny Rachitsky
[00:23:37] S1: This is so interesting. A question I've been tackling is just where are human brains going to continue to be valuable? What I'm hearing here is there's the initial idea. You made such a good point here. It's like the initial idea's often not the actual winning idea, it's just the beginning of an idea. So there's the idea for the feature, then there's the try it out, prototype it, help you narrow on the direction, build it, make it awesome, get it out into the world. It feels to me like AI's going to be really good at suggesting ideas and coming up with initial ideas, and I wonder if the human brain ... it's not like maybe someday we don't need human brains at all, and that's a whole other discussion, but maybe the next phase is AI will help us come up with great ideas.
00:24:17 · Simon Willison
[00:24:17] S0: I mean, that's been the case for probably a couple of years now, they've been strong enough to do really good brainstorming. I like to compare it to the thing where when you've got a group brainstorming exercise, you book a meeting room for an hour, you've got a whiteboard, you get a dozen people in, and the first two-thirds of that brainstorming session, honestly it's kind of just everyone going through the most obvious basic ideas, right? You get them all out on the whiteboard, you get them all up, and then things get interesting when you start saying, "Okay, well, let's talk about these, let's start combining them." The AI is so good at that first two-thirds of the ideas. I brainstorm with them all the time where I just get them to spit out all of the obvious stuff, and they'll come up with 20 things and they'll all be kind of dumb, they just won't be very interesting. What gets interesting is when if you ask them for 20 more and now- What gets interesting is when if you ask them for 20 more and now by the end of that list, you're beginning to get things which are not good ideas, but they point you in interesting directions. There are so many other tricks like this. You can tell AI to combine weird fields. You can say, "Okay, I want ideas for marketing my new SaaS platform inspired by marine biology," and you see what happens. And most of it will be complete junk, but there might be a spark that gets you to the good idea. So I love them as brainstorming companions on that front.
00:25:33 · Lenny Rachitsky
[00:25:33] S1: That reminds me of a chat I had with David Placek. He's a expert naming person. He helps companies come up with names for product. And one of the things that he does at his company is he creates three teams to come to brainstorm names. So for example, let's say Windsurf was a product they named. So the first team is, "Okay, this is an AI IDE thing." That's exactly what it is. Second team is, "Okay, this is a boat. You're naming a boat and here's the constraints. And then here, this is a spaceship. So name it from that perspective." And he finds the best names come from those other directions where it's a different metaphor with the same sort of benefits. Okay. So what I'm hearing here is this is good. This is good for humans right now, that there's still opportunity for us to contribute to the process.
00:26:17 · Simon Willison
[00:26:17] S0: And actually, I want to stand in defense of software engineers for a bit because on the one hand, these things can write code. That used to be our thing. I'm finding that using coding agents well is taking every inch of my 25 years of experience as a software engineer, and it is mentally exhausting. This is something which people are talking a lot more about now. I can fire up four agents in parallel and have them work on four different problems, and by 11:00 AM, I am wiped out for the day, because there is a limit on human cognition in how much, even if you're not reviewing everything you're doing, just how much you can hold in your head at one time. And it's very easy to pop that stack at the moment. There's a sort of personal skill that we have to learn, which is finding our new limits. What is a responsible way for us to not burn out and for us to use the time that we have? And I've talked to a lot of people who are losing sleep because they're like, "My agents could be doing work for me. I'm just going to stay up an extra half hour and set off a bunch of extra things and I'm waking up at 4:00 in the morning." That's obviously unsustainable. I hope that that's a novelty thing. The agents only really got good in the past four to five months. We're all learning what that looks like and what that lets us do. But it's concerning. There's an element of gambling and addiction to how we're using some of these tools. But to stand in defense of software engineers, I get great results out of these things because they are amplifiers of existing skills and experience. And I have 25 years of existing pre-AI experience, which I can now amplify because I can talk to the agent at a very high level. I can use sophisticated engineering language that I've mastered over the years, which they appear to know as well, and we can collaborate incredibly effectively. And it means I can look at a problem and I can say, "This problem is a one-sentence prompt and I know it'll find that bug and fix that bug," as opposed to this other problem, which is who knows how big a problem. There is a flip side to this, which is that I've got 25 years of experience in how long it takes to build something, and that's all completely gone. That doesn't work anymore because I can look at a problem and say, "Okay, well, this is going to take two weeks.