Lenny's Podcast: Product | Career | Growth
00:40:48 · Lazar Jovanovic
In the next request, instead of focusing on reading, it spends another 30% of tokens trying to come up with an apology. Again, I'm not educated, but if you ever read a stream of ChatGPT's thinking in thinking models, you see exactly what I mean. When I insult it, I see that the first message says, "Okay, the user is mad, so I need to think of ways how to reduce their anxiety or whatever." I'm like, "Oh man, I just fell for the worst trick in the book. I made it spend the most scarce resource, which is those tokens on thinking how it should address my anxiety versus focusing on the actual problem." So my advice for people is, yes, vibe your way for fun and vibe your way while you're prototyping because that's the exploration part. I love that part. But when exploration is done, please, please, please use referencing, documentation. Use all the agent files that you can because that token allocation is so scarce. It's going to get expanded over time. Things are going to get cheaper, faster, but right now it's still so valuable and precious, you really need to make sure that they are allocated in the right direction.
00:42:04 · Lenny Rachitsky
[00:42:04] S1: This is hilarious. I think the genie metaphor is so good here. Just thinking about this genie, is you're trying to be clear about what it is you want. And if you're just vibe wishing, it'll do the wrong thing. So the advice here is give it as much context about what you want it to do as possible. And these files, we'll talk about right after this. But the idea here is just point the laser at where you want it to fix the problem. Don't just assume it'll go figure out because it will, and it'll try really hard to, and it'll waste all your tokens. It'll fill the context window. And I remember at one point you mentioned before this recording that because it starts to run out of space in the context window, it just like, the solution ends up. It doesn't actually work that hard on figuring it out in the end because it spent all this energy on reading and thinking. And then it's like, "Okay, here," at the last second, "Here's a solution."
00:42:59 · Lazar Jovanovic
[00:42:59] S0: I think it just picks the first thing it thinks is broken. Again, this is me completely uneducated, coming into the conversation and just thinking out loud. That's just my gut feeling and the way I think logically about it, which is, "Hey, if it consumes most of its window and knows that it's running out of it, maybe it's aware that it's running out, maybe it isn't." But either way, I've had the experience, anecdotally to where my request is unclear. I feel it takes the easiest fix in the book, just the easiest versus the other way around where I'm spending so much time finding the right file, referencing that file, really putting in the effort of handholding it in dark, maybe giving it a flashlight and then saying, "Here's the problem. I think that this is the problematic file." And then it's like, "Oh yeah, you're right. And now I'm going to actually fix over and over and over." And I've seen that because again, all I do is read the output. Agent makes me learn how to use it. So people read, I don't know what people read, but all I read is the output. I don't read the code, and it's later down the road because I know that it can do that much better than I can. Again, I feel if ... There's a good quote I've read. I apologize to the author because I can't attribute it off the top of my head, but it's like, "The ceiling on the AI isn't the model intelligence, it's what the model sees before it acts." So that's the ceiling right now. What are you exposing? We talk about exposure time for humans. What are you exposing your agents to, as well is as important, if not even more important, before it makes code edits. Yeah.
00:44:44 · Lenny Rachitsky, Lazar Jovanovic
[00:44:44] S1: Coming back to these files, I think this is really important. So let's think about just what's the MVP for someone that wants to do this better? You listed all these MD files essentially, that you're building over the course of a day before you start actually building the thing. You had design guidelines, the user journey, tasks, agents.md, rules.md. Say you wanted to just move one step forward and be better at this stuff, what are the files you'd create and then what do they roughly look like? What's inside these files? [00:45:12] S0: Yeah. So the master plan is the first one, which is like, it's a 10,000-foot overview, right? It really, high-level explains the intent that I have with this app. [00:45:22] S1: And this is masterplan.md? Is that what you call it, or ... [00:45:25] S0: Yes. Yeah, masterplan.md. And it's really just like, "Hey, this is why I'm doing this. This is who I'm doing it for. This is how I want them to feel." And a lot of times in the master plan, I will reference the other PRDs. I'll be like, "The design needs to feel modern and slick, but for exact parameters, consult and read design guidelines.md." So I'm using just the master plan as this high-level overview to get the agent into, "Oh, okay. Yeah, we are building X, Y, Z." Then there's the implementation plan because there needs to be some order. If you just dump stuff on top of each other without any order, you're never going to get to the finish line. [00:46:11] S1: And this is tasks.md? Is that what you call this? [00:46:13] S0: No, that's the implementation plan.
00:46:13 · Lazar Jovanovic, Lenny Rachitsky
[00:46:13] S0: No, that's the implementation plan. [00:46:15] S1: Implementation plan. [00:46:15] S0: I call it implementation plan. Yeah. [00:46:15] S1: Okay.
00:46:17 · Lazar Jovanovic
[00:46:17] S0: And implementation plan is kind of in service of the future tasks.md, if that make ... All of these files are in service of building tasks.md. When you build tasks.md, then the rest is almost irrelevant. It's just the basis for you to build tasks to execute. The implementation plan is kind of the first layer, which is again, higher-level overview. It doesn't go into the depth of how to get there. It just goes into the explaining of, "Oh well, if we're building this, I think we should start with the backend, and we should start with tables and then later authentication. And then after that, we're going to bring in the API. And then after that, we're going to do this." It's like, again, just think of it as having ... I'm an ideas guy. I'm sitting with a technical guy. It's me and you. We're building our startup. I know you're a software engineer by background and I'm telling you my idea. I'm giving you the master plan. And you come to me back and you're like, "Okay, if you want to do this, it's doable. Here's how I would order it." You didn't have a roadmap, you didn't open your Linear and started writing features and RFCs and whatever. You're just high-level talking about the order of things. And then me and you, again, as two co-founders, we talk and say, "Okay, well, if we agree on this, how should this look like? How should this feel? Let's describe it high-level," but now because I use AI, I can go a little bit deeper. And that's where I like to see Lovable or any other tool. ChatGPT is good at it. I even have my ... I've built custom GPTs. So if people want to start somewhere before they even get into any tool, they can go to ChatGPT store for GPTs and just type Lovable base prompt generator or Lovable PRD generator and find those that I built and just brain dump in them and then get these files as output. Right? So I like to see some elements of CSS in design guidelines because with design, it's a little bit tricky. AI is sometimes overcreative. So that's where I'm doing a little bit more technical steering. And then finally, it's just the user journeys, just like if we know how things look like, if we know how they feel, if we know what we're building, high-level. High-level, just very high-level again, how do people navigate? What are some of the features in there and stuff like that?