Lenny's Podcast: Product | Career | Growth
00:43:31 · Lenny Rachitsky, Zevi Arnovitz
[00:43:31] S0: That is such an incredible rant/way to understand what's going on. Okay, awesome. So we just ran the review. So show us what we saw there and let's actually try this peer review. I'm really excited to see what you learned there. [00:43:45] S1: Yeah. So basically Claude has reviewed its code and it's found a bunch of bugs, a critical bug it found in the prompt, some high bugs, some medium bugs, and now what I'll do is I'll do the same thing with the other models. So Codex has a built-in code review that you can do, or I just like to say review all the code in this branch. Of course, branch is referring to the GitHub branch that we're working on. We're not working on the live code base. And then I'll do this with Composer say with, let's do it with Composer One. So I'll do /review here as well. And basically these are both going to run and do a in-depth review similar to what Claude does. But again, because of the differences between the models, they're all going to catch different things and they're all going to look differently, and this is a really cool way to work. It's basically if you had other team leads within the company review the code. Here, you can see how fast Composer is. I think GPT probably will take a bunch of time. Like I said, it's in its own dark room right now reviewing code and we'll come back in a few minutes.
00:43:45 · Zevi Arnovitz, Lenny Rachitsky
[00:43:45] S1: Yeah. So basically Claude has reviewed its code and it's found a bunch of bugs, a critical bug it found in the prompt, some high bugs, some medium bugs, and now what I'll do is I'll do the same thing with the other models. So Codex has a built-in code review that you can do, or I just like to say review all the code in this branch. Of course, branch is referring to the GitHub branch that we're working on. We're not working on the live code base. And then I'll do this with Composer say with, let's do it with Composer One. So I'll do /review here as well. And basically these are both going to run and do a in-depth review similar to what Claude does. But again, because of the differences between the models, they're all going to catch different things and they're all going to look differently, and this is a really cool way to work. It's basically if you had other team leads within the company review the code. Here, you can see how fast Composer is. I think GPT probably will take a bunch of time. Like I said, it's in its own dark room right now reviewing code and we'll come back in a few minutes. [00:44:56] S0: Okay, cool. So we can let these run in. We don't actually have to go through the whole process, but is the idea once you get these results, you run peer review and you copy and paste kind of these results. Is that the idea?
00:44:56 · Lenny Rachitsky, Zevi Arnovitz
[00:44:56] S0: Okay, cool. So we can let these run in. We don't actually have to go through the whole process, but is the idea once you get these results, you run peer review and you copy and paste kind of these results. Is that the idea? [00:45:04] S1: Exactly. I'll copy and paste the results. I'll do peer review and then I'll say dev lead one and then paste from one of the models. And then I'll say dev lead two and paste from the other model and basically have them fight it out until I feel like we have no more issues. [00:45:17] S0: Incredible. [00:45:18] S1: For me, this is super important because I'm not technical and I'm not a developer. And I'll also use /learning opportunity a bunch during this to learn about stuff that I don't understand or don't fully grasp. [00:45:31] S0: Incredible. What a clever solution to solving this code review problem where it's like, I don't know what you ... I don't know how to recode, so what am I going to even ... Yeah. [00:45:39] S1: Yeah. [00:45:40] S0: Okay. Incredible. Let's wrap up this workflow. Is there anything else that's important in this workflow? And again, all this stuff is going to be available where people can just plug this stuff into their Cursor account and use it themselves.
00:45:52 · Zevi Arnovitz
[00:45:52] S1: 100%. The one thing I'll say is that I think just like working in general with AI and even just like working on any product, doing constant postmortems is critical. So a lot of times we'll find all these kind of bugs or maybe Claude will fail to execute something correctly. And at the beginning when I started vibe coding, I would basically just keep running at it like running at the wall and until it worked. And once it worked, I was like, "All right, awesome. This works. Let's keep going." But I've learned over time that updating documentation and tooling is one of the biggest hacks for productivity. So when Claude will fail to do something or I'll see this really bad bug that shows that Claude really didn't understand something, I'll ask it, "What in your system prompt or tooling made you make this mistake?" And Claude will kind of like go introspective and think of what made it create that mistake. And then I'll say, "Okay, great. Let's update your tooling and documentation so that this mistake never occurs again." And I do this every time I'm either building an internal tool or anything. And I think this is just like working. If you end up doing a bunch of mistakes and then end up releasing the feature to users, so you're like, "All right, it's a big success." But going back and even when you've succeeded, looking and understanding what you did and what you could have done better is critical. And also using AI, this is probably one of the biggest unlocks. Going back to your prompts, understanding what was not good enough, iterating on them and then seeing how AI's responses get better, I think that's probably one of the most important things and one of the things that divides between people who are okay with using AI and the people who actually know how to use it.
00:47:38 · Lenny Rachitsky, Zevi Arnovitz
[00:47:38] S0: That is such good advice. So what I'm hearing is when the models do something dumb, make a mistake, you ask it to reflect on what the mistake it made was, and then you update the /command prompts with that knowledge so that in the future, it's not making that same mistake, and it just keeps getting better. [00:47:38] S1: Exactly. [00:47:56] S0: These things just keep getting smarter and smarter. So you're building up this really incredible prompt that just gets better and better. [00:48:01] S1: Exactly. Not always the /commands. It will sometimes update different documentation or its tooling, but basically it's understanding what the root cause of the mistake that the AI made and fixing it. [00:48:15] S0: Awesome. So the models are getting smarter and then there's also the other parts of your workflow can get smarter as you find flaws in the way it does stuff. [00:48:25] S1: 100%. Yep. [00:48:26] S0: Amazing. Okay. Is there anything else there before I move in a couple other directions?
00:48:26 · Lenny Rachitsky, Zevi Arnovitz
[00:48:26] S0: Amazing. Okay. Is there anything else there before I move in a couple other directions? [00:48:31] S1: I think that's it. I think we covered pretty much everything. Basically, just to wrap this up, what I do is I do a bunch of code review and then update the documentation so that everything is documented. So the next time I try to build a feature in this area, there won't be any mistakes. And then I'll do a bunch of testing. I'll do some user testing as well before I release this to general availability. Obviously we're not going to release this. This was just a show, but hopefully maybe by the time the podcast comes out, I'll have done this correctly and release the feature. [00:48:59] S0: It's incredible that this was not possible, I don't know, two years ago, maybe a year ago, you are a product manager shipping a product without knowing how to write code, barely knowing how to review code. You said you're afraid of looking at code. As a product manager, you're building a product in Cursor using all of these different AI models. You're making money with this product. We're so used to this now, but it's insane what is now possible.