Lenny's Podcast: Product | Career | Growth
00:35:46 · Lazar Jovanovic
Yes, I'll go in, test, maybe put a prompt in here or there, but that's how I can build five projects simultaneously and never lose the productivity part, which is again, as I said, I do this today, manually. Call me to talk three months from now, an agent will do this for me. I'll be out of job, pretty much. That's why I don't optimize for this skill at all. I'm using it today, to bypass the shortcomings of human nature and LLMs, but I'm optimizing 100% of my time today on good judgment, clarity, quality, taste, good copy, good fonts. People don't talk about fonts at all that work with AI. They're 60% in my mind, maybe even more in how your output's going to look like. That's my obsession. I don't obsess over these things that I'm talking today because I know what's coming. The agents are going to get better, the models are going to get better. They're not going to need me to extend the context. They're going to do it themselves. So for me, the skill that I optimize for is the one that requires better decision-making rather than better output or better alignment.
00:36:57 · Lenny Rachitsky
[00:36:57] S1: Oh, my God. There's so much here. This is so awesome. Okay. So essentially, what's happening here is you start a project, try a bunch of stuff, pick a direction that feels most correct. And once you have a set direction, you spend essentially a day, not building, but working with this AI agent to plan. And then, and well, I want to talk about that. And once you have the plan, then it's ... And it's amazing that you could do stuff like this with what some people may feel are not sophisticated tools that can build incredibly powerful things. You can do a lot of this with tools like Lovable, like have plans and rules and MD files. A lot of people may not know that. And so the idea is, okay, spend all this time planning because again, that'll save you a lot of time down the road. And then only once you have a plan, you get it going. And a key part of this, this three-wishes rule is really important. The reason you're doing this in large part beyond just being really clear about the plan is this idea of one task at a time keeps the agent's context window small so that it doesn't lose track of where it's at. That part seems important. It's like, "Do this thing." And then, "Okay, cool. Now do the next thing." Right?
00:38:12 · Lazar Jovanovic
[00:38:12] S0: Yes, yes, because again, let's say you didn't do this. Let's talk about you ignoring this and you're like, "I just want to vibe my way." Okay, great. No problem. You work, you work, you work. At one point, something breaks, right? You haven't documented anything. There's no reference points. You report a problem. You're not referencing files or architecture at all, you're just describing the issue. Here's what's going to happen. Any tool, Lovable or Cursor or Claude, whatever tool you talk about is going to do this. It's going to be like, "Okay, let me start investigating." And then your code base gets bigger and bigger and bigger and bigger and bigger. When you first start, you have 20 files. It can read 20 files. But what happens when you have ... I'm just building a project right now that has 60, 70 edge functions. What happens then when I say, "This broke and there's no reference which edge function does what?" Guess what? Lovable's going to read all of those and it's going to consume 80% of the token allocation on reading to get clarity, leaving only the final 20% for thinking and executing. What I'm guessing, and I can't prove this, an LLM expert in the comments may say that I'm wrong, but this is my best guess as a non-educated person. These tools are very obedient and very agreeable. They're going to lie to you. They're going to tell you that they fixed the problem, even though they didn't. They're just going to try to make you feel happy and say, "Yes, I found what the problem is and I fixed it," when a lot of times when they don't, people blame the machine. And to an extent, I will say that's true. It's your fault, my friend. You did not provide any clarity or context to this tool. You just used its raw power and dug a deeper hole with spinning your wheels into the mud. And obviously, I think we're heading into a world where AI is more honest than obedient in saying, "Hey, I only partially fixed this. You did not give me enough of a context." The bigger mistake that people make then is they trust the tool fixed it. They test, they see it didn't, then they get mad at it, start cursing and yelling, as we say, and then it gets even worse because guess what? Another bad trait of AI, is it does its best not to hurt your feelings and never say, "You're the dumb one." It says, "No, I'm the dumb one." So it focuses ...
00:40:48 · Lazar Jovanovic
In the next request, instead of focusing on reading, it spends another 30% of tokens trying to come up with an apology. Again, I'm not educated, but if you ever read a stream of ChatGPT's thinking in thinking models, you see exactly what I mean. When I insult it, I see that the first message says, "Okay, the user is mad, so I need to think of ways how to reduce their anxiety or whatever." I'm like, "Oh man, I just fell for the worst trick in the book. I made it spend the most scarce resource, which is those tokens on thinking how it should address my anxiety versus focusing on the actual problem." So my advice for people is, yes, vibe your way for fun and vibe your way while you're prototyping because that's the exploration part. I love that part. But when exploration is done, please, please, please use referencing, documentation. Use all the agent files that you can because that token allocation is so scarce. It's going to get expanded over time. Things are going to get cheaper, faster, but right now it's still so valuable and precious, you really need to make sure that they are allocated in the right direction.
00:42:04 · Lenny Rachitsky
[00:42:04] S1: This is hilarious. I think the genie metaphor is so good here. Just thinking about this genie, is you're trying to be clear about what it is you want. And if you're just vibe wishing, it'll do the wrong thing. So the advice here is give it as much context about what you want it to do as possible. And these files, we'll talk about right after this. But the idea here is just point the laser at where you want it to fix the problem. Don't just assume it'll go figure out because it will, and it'll try really hard to, and it'll waste all your tokens. It'll fill the context window. And I remember at one point you mentioned before this recording that because it starts to run out of space in the context window, it just like, the solution ends up. It doesn't actually work that hard on figuring it out in the end because it spent all this energy on reading and thinking. And then it's like, "Okay, here," at the last second, "Here's a solution."
00:42:59 · Lazar Jovanovic
[00:42:59] S0: I think it just picks the first thing it thinks is broken. Again, this is me completely uneducated, coming into the conversation and just thinking out loud. That's just my gut feeling and the way I think logically about it, which is, "Hey, if it consumes most of its window and knows that it's running out of it, maybe it's aware that it's running out, maybe it isn't." But either way, I've had the experience, anecdotally to where my request is unclear. I feel it takes the easiest fix in the book, just the easiest versus the other way around where I'm spending so much time finding the right file, referencing that file, really putting in the effort of handholding it in dark, maybe giving it a flashlight and then saying, "Here's the problem. I think that this is the problematic file." And then it's like, "Oh yeah, you're right. And now I'm going to actually fix over and over and over." And I've seen that because again, all I do is read the output. Agent makes me learn how to use it. So people read, I don't know what people read, but all I read is the output. I don't read the code, and it's later down the road because I know that it can do that much better than I can. Again, I feel if ... There's a good quote I've read. I apologize to the author because I can't attribute it off the top of my head, but it's like, "The ceiling on the AI isn't the model intelligence, it's what the model sees before it acts." So that's the ceiling right now. What are you exposing? We talk about exposure time for humans. What are you exposing your agents to, as well is as important, if not even more important, before it makes code edits. Yeah.