Lenny's Podcast: Product | Career | Growth
01:08:22 · Lenny Rachitsky
[01:08:22] S1: Okay. Amazing. And I love that you share this with people. I know you're not sharing at all, but this just empowers everyone else to piggyback off the work that you've already done over the past. Okay, So another agenetic pattern is Red/Green, test room development, and then this idea of first run the test. Talk about that.
01:08:37 · Simon Willison
[01:08:37] S0: This is the most important thing when you're working with coding agents is they have to test the code. That's the whole point of a coding agent is if they haven't run the code, you're back to copy and pasting out of ChatGPT and crossing your fingers and hoping that it got things right. So, how do you get them to run the code? The best way to do that is to use a programming technique that we've been using for decades called test-driven development, where you have automated tests, you have code that tests your other code, and we call those the tests. Agents will write tests the moment you even hint at them that they should write a test, they'll write a test, which is great because I try to make it so pretty much every line of code that I release into the world, there's an automated test that has at least made sure that that works. The reason these tests are so valuable, there's two things. Firstly, it means that the agent has at least run the code. So if there are syntax errors and things, it'll have found those. And it gives you that significant boost in confidence that it actually works. And then the test, because they go into the repository, they add up over time. And that's what gives you the confidence that when you tell your agent to build a new feature, it won't break old features. This is exactly the same thing for human software engineering teams. The reason I like having automated tests is that I can build new features and I don't then have to manually test every single other feature to make sure it didn't break because the test automate that process works great with agents. If your coding agent has a repository with a good set of tests, you can tell it to change something and it'll change that thing and it won't break anything else, or at least it won't break the things that the tests are covering. So occasionally, I run into people who are using AI for coding and they're like, and we don't even have to test it anymore. We've stopped doing tests because it's so quick that it's faster for us to not use the test. I think those people are wrong. I think it's a huge mistake if you drop tests in exchange for speed of development, because very quickly when you're working with tests, you find your development speed goes up.
01:10:40 · Simon Willison
The existence of the test lets you move faster, because you don't have to constantly worry that you're breaking older things. So that's test-driven development. I think that's absolutely crucial for getting the most out of coding agents. The other thing you mentioned was Red/Green TDD. And I like this one as an example of a sort of miniature prompt that you can use. So, when you are doing test-driven development, one of the ways you can do this as a human programmer is this thing where you first write the test, which won't work because you haven't written the code, and then you run it and you watch it fail. And that gives you confidence that the test, because if it passes, something's gone wrong. So you want to see the test fail, and then you go and implement whatever needs to be done to make the test pass, and then you run the test again and you watch it pass. And I hate doing this. There are a lot of programmers believe that this is the one true way to write software. I tried it for a couple of years. It just slowed me down and frustrated me. I did not enjoy the intellectual challenge of, okay, and the discipline of write the tests first and then watch them part fail because I like to explore by writing a bunch of code and then add the tests later on. Coding agents, I don't care if they're bored. I couldn't care, that's what their opinions on test-driven development are. If you get them to write the tests first, you do get better results because they're much less likely to forget to test something or to add bits of code that aren't necessary. And so you could tell them, write this using test, make sure that you write the test first, then watch the tests fail, then write the information, then watch them pass again. That's a lot of typing. If you use the term red/green TDD, that's programming jargon, which I didn't used to use, but it is jargon for run the test and watch it fail. The agents know what that means. So now we've reduced that sort of lengthy paragraph about how to test to Red/Green TDD enter, you're done. So there are two ideas that that illustrates. Firstly, the importance of that technique of having them run the test and watch them fail. And secondly, the fact that sometimes you do find something you can type in five seconds that has a material impact on how these things are working.
01:12:48 · Lenny Rachitsky, Simon Willison
[01:12:48] S1: Amazing. And on your site, you have the actual markdown. You can just copy and paste. [01:12:52] S0: Yeah, click copy. [01:12:53] S1: That one is really simple. And I love that this is an example of people here, okay, engineers are not even looking at their code anymore and they assumes this is terrible slop, knowing it's going to break, but these sorts of practices is what allows this to happen where- [01:13:08] S0: Exactly. [01:13:09] S1: ... you can trust that the tests are running and passing and that it's not building a bunch of stuff that's really brittle. [01:13:15] S0: It's also an interesting example of how my idea of quality code has changed, because the challenge with tests is that you can test absolutely everything and you might end up with thousands of lines of tests for a hundred lines of code. And sometimes that's good, but usually that's bad. It's a bad design pattern. If you look at a repo and there's huge amounts of tests that aren't really doing anything interesting, that's really expensive, because now when you change the code, you've got to update a thousand lines of tests and all of that. It turns out I don't care anymore because updating a thousand lines of tests is now the job of the coding agent. So I'm much more tolerant of very lengthy verbose test suites. A lot of my small libraries now have over a hundred tests. Normally that would be over testing. Now it's fine. As long as the tests are good tests and I can have the agents throw them away later if it needs to, the code is cheap now.
01:13:15 · Simon Willison, Lenny Rachitsky
[01:13:15] S0: It's also an interesting example of how my idea of quality code has changed, because the challenge with tests is that you can test absolutely everything and you might end up with thousands of lines of tests for a hundred lines of code. And sometimes that's good, but usually that's bad. It's a bad design pattern. If you look at a repo and there's huge amounts of tests that aren't really doing anything interesting, that's really expensive, because now when you change the code, you've got to update a thousand lines of tests and all of that. It turns out I don't care anymore because updating a thousand lines of tests is now the job of the coding agent. So I'm much more tolerant of very lengthy verbose test suites. A lot of my small libraries now have over a hundred tests. Normally that would be over testing. Now it's fine. As long as the tests are good tests and I can have the agents throw them away later if it needs to, the code is cheap now. [01:14:04] S1: Amazing. So, the advice here is when you're building something, have the AI build the test first, just ask it and the phrasing is use Red/Green TDD. [01:14:15] S0: I think so, yeah. [01:14:16] S1: It just makes it so easy to ... I used to be an engineer and many people don't know this and I did not enjoy writing tests before I wrote the code. And I love that AI could just-
01:14:16 · Lenny Rachitsky, Simon Willison
[01:14:16] S1: It just makes it so easy to ... I used to be an engineer and many people don't know this and I did not enjoy writing tests before I wrote the code. And I love that AI could just- [01:14:27] S0: Running tests is boring. It's really boring. And it used to be I would force myself to do it because I knew that I'd seen the value, but it wasn't the bit that I enjoyed. Agents are so good at writing tests. They can test anything and they can write lots and lots of very boring boilerplate code and it just works. [01:14:44] S1: Is there any other design pattern, agenetic engineering pattern that you think is important to share before we move on to a final topic?