Lenny's Podcast: Product | Career | Growth
00:38:12 · Lazar Jovanovic
[00:38:12] S0: Yes, yes, because again, let's say you didn't do this. Let's talk about you ignoring this and you're like, "I just want to vibe my way." Okay, great. No problem. You work, you work, you work. At one point, something breaks, right? You haven't documented anything. There's no reference points. You report a problem. You're not referencing files or architecture at all, you're just describing the issue. Here's what's going to happen. Any tool, Lovable or Cursor or Claude, whatever tool you talk about is going to do this. It's going to be like, "Okay, let me start investigating." And then your code base gets bigger and bigger and bigger and bigger and bigger. When you first start, you have 20 files. It can read 20 files. But what happens when you have ... I'm just building a project right now that has 60, 70 edge functions. What happens then when I say, "This broke and there's no reference which edge function does what?" Guess what? Lovable's going to read all of those and it's going to consume 80% of the token allocation on reading to get clarity, leaving only the final 20% for thinking and executing. What I'm guessing, and I can't prove this, an LLM expert in the comments may say that I'm wrong, but this is my best guess as a non-educated person. These tools are very obedient and very agreeable. They're going to lie to you. They're going to tell you that they fixed the problem, even though they didn't. They're just going to try to make you feel happy and say, "Yes, I found what the problem is and I fixed it," when a lot of times when they don't, people blame the machine. And to an extent, I will say that's true. It's your fault, my friend. You did not provide any clarity or context to this tool. You just used its raw power and dug a deeper hole with spinning your wheels into the mud. And obviously, I think we're heading into a world where AI is more honest than obedient in saying, "Hey, I only partially fixed this. You did not give me enough of a context." The bigger mistake that people make then is they trust the tool fixed it. They test, they see it didn't, then they get mad at it, start cursing and yelling, as we say, and then it gets even worse because guess what? Another bad trait of AI, is it does its best not to hurt your feelings and never say, "You're the dumb one." It says, "No, I'm the dumb one." So it focuses ...
00:40:48 · Lazar Jovanovic
In the next request, instead of focusing on reading, it spends another 30% of tokens trying to come up with an apology. Again, I'm not educated, but if you ever read a stream of ChatGPT's thinking in thinking models, you see exactly what I mean. When I insult it, I see that the first message says, "Okay, the user is mad, so I need to think of ways how to reduce their anxiety or whatever." I'm like, "Oh man, I just fell for the worst trick in the book. I made it spend the most scarce resource, which is those tokens on thinking how it should address my anxiety versus focusing on the actual problem." So my advice for people is, yes, vibe your way for fun and vibe your way while you're prototyping because that's the exploration part. I love that part. But when exploration is done, please, please, please use referencing, documentation. Use all the agent files that you can because that token allocation is so scarce. It's going to get expanded over time. Things are going to get cheaper, faster, but right now it's still so valuable and precious, you really need to make sure that they are allocated in the right direction.
00:42:04 · Lenny Rachitsky
[00:42:04] S1: This is hilarious. I think the genie metaphor is so good here. Just thinking about this genie, is you're trying to be clear about what it is you want. And if you're just vibe wishing, it'll do the wrong thing. So the advice here is give it as much context about what you want it to do as possible. And these files, we'll talk about right after this. But the idea here is just point the laser at where you want it to fix the problem. Don't just assume it'll go figure out because it will, and it'll try really hard to, and it'll waste all your tokens. It'll fill the context window. And I remember at one point you mentioned before this recording that because it starts to run out of space in the context window, it just like, the solution ends up. It doesn't actually work that hard on figuring it out in the end because it spent all this energy on reading and thinking. And then it's like, "Okay, here," at the last second, "Here's a solution."
00:42:59 · Lazar Jovanovic
[00:42:59] S0: I think it just picks the first thing it thinks is broken. Again, this is me completely uneducated, coming into the conversation and just thinking out loud. That's just my gut feeling and the way I think logically about it, which is, "Hey, if it consumes most of its window and knows that it's running out of it, maybe it's aware that it's running out, maybe it isn't." But either way, I've had the experience, anecdotally to where my request is unclear. I feel it takes the easiest fix in the book, just the easiest versus the other way around where I'm spending so much time finding the right file, referencing that file, really putting in the effort of handholding it in dark, maybe giving it a flashlight and then saying, "Here's the problem. I think that this is the problematic file." And then it's like, "Oh yeah, you're right. And now I'm going to actually fix over and over and over." And I've seen that because again, all I do is read the output. Agent makes me learn how to use it. So people read, I don't know what people read, but all I read is the output. I don't read the code, and it's later down the road because I know that it can do that much better than I can. Again, I feel if ... There's a good quote I've read. I apologize to the author because I can't attribute it off the top of my head, but it's like, "The ceiling on the AI isn't the model intelligence, it's what the model sees before it acts." So that's the ceiling right now. What are you exposing? We talk about exposure time for humans. What are you exposing your agents to, as well is as important, if not even more important, before it makes code edits. Yeah.
00:44:44 · Lenny Rachitsky, Lazar Jovanovic
[00:44:44] S1: Coming back to these files, I think this is really important. So let's think about just what's the MVP for someone that wants to do this better? You listed all these MD files essentially, that you're building over the course of a day before you start actually building the thing. You had design guidelines, the user journey, tasks, agents.md, rules.md. Say you wanted to just move one step forward and be better at this stuff, what are the files you'd create and then what do they roughly look like? What's inside these files? [00:45:12] S0: Yeah. So the master plan is the first one, which is like, it's a 10,000-foot overview, right? It really, high-level explains the intent that I have with this app. [00:45:22] S1: And this is masterplan.md? Is that what you call it, or ... [00:45:25] S0: Yes. Yeah, masterplan.md. And it's really just like, "Hey, this is why I'm doing this. This is who I'm doing it for. This is how I want them to feel." And a lot of times in the master plan, I will reference the other PRDs. I'll be like, "The design needs to feel modern and slick, but for exact parameters, consult and read design guidelines.md." So I'm using just the master plan as this high-level overview to get the agent into, "Oh, okay. Yeah, we are building X, Y, Z." Then there's the implementation plan because there needs to be some order. If you just dump stuff on top of each other without any order, you're never going to get to the finish line. [00:46:11] S1: And this is tasks.md? Is that what you call this? [00:46:13] S0: No, that's the implementation plan.
00:46:13 · Lazar Jovanovic, Lenny Rachitsky
[00:46:13] S0: No, that's the implementation plan. [00:46:15] S1: Implementation plan. [00:46:15] S0: I call it implementation plan. Yeah. [00:46:15] S1: Okay.