Lenny's Podcast: Product | Career | Growth
01:04:13 · Lenny Rachitsky, Simon Willison
[01:04:13] S1: And then the other is just like questions that you wanted answers to. And then here's the answer so that you could just say, "Hey, use this research we've done previously to help us solve this problem." [01:04:22] S0: But the key thing about that is this isn't research in this traditional sense of go and search the web and do me a deep research report. These are all coding agent research tasks where actually written code and run it. Because that's what makes them ... If I published a GitHub repository full of unverified deep research reports, that's very little value to anyone. But the moment the coding agent has written the code, run the code, plotted a graph of how it worked or whatever, that's what turns it into not just LLM vomit. It becomes something that's at least slightly actionable. [01:04:54] S1: Yeah. And I love that you use the term hoard, which comes across as keep it secret, but you make it publicly available in open source, which is recording. [01:04:54] S0: For the most part yeah. [01:05:07] S1: For the most ... Yeah, because I'm grassing it in, it's all here. But I guess is there some stuff that you hoard, hoard for real? [01:05:08] S0: I mean, I've got 10,000 Apple notes as well that I just constantly add new things to. But generally I default to putting this stuff in public, because it benefits me more that way. It's easier for me to find later on. It's like I use GitHub as a backup system and it's great for my credibility as a programmer that I've got all of this stuff out there.
01:05:08 · Simon Willison, Lenny Rachitsky
[01:05:08] S0: I mean, I've got 10,000 Apple notes as well that I just constantly add new things to. But generally I default to putting this stuff in public, because it benefits me more that way. It's easier for me to find later on. It's like I use GitHub as a backup system and it's great for my credibility as a programmer that I've got all of this stuff out there. [01:05:29] S1: So for people that want to do this, what's the advice here? Is it just keep notes at the start of things you've learned is possible and works? [01:05:37] S0: Yes, but find a note system that you trust and that you're not going to lose. So the easiest one would be like a folder synced to Dropbox or something like that. I really like GitHub. I've got lots of private GitHub repositories. My public research one has 75 projects in it. I've got a private research one with another 50 that are things that just didn't ... They're tied to my personal projects or whatever it is. So I have a whole bunch of things like that as well. GitHub is free for private repositories somehow. So I'm doing all of this stuff in GitHub. And when you put something on GitHub, they back it up to three continents. Your chances of losing something on GitHub are very, very slim. Occasionally they'll go and stick it in a vault in the Arctic as well. So I feel pretty good about them as a place to keep that data. [01:06:27] S1: And then how do you actually use this? Is this feeded into the LLM when you're building or is it on occasion, go look at this, go look at that? Is that in memory or not?
01:06:36 · Simon Willison
[01:06:36] S0: It's definitely both. But the key trick that I've been using a lot is, especially for my little HTML and JavaScript tools, you can tell an LLM to consult them and combine them. So, a very early example of that is I'd written some code pre-LLMs, which used a PDF library from Mozilla. So it's in JavaScript, but it can open up a PDF and show you that PDF on the page. And I'd also written some code that used Tesseract, which is an OCR library that can run in your browser and do actually really good OCR all in JavaScript. And I just realized I wanted to do OCR against PDF files. So I told Claude Opus 3, I think back then I said, "Here's the code for the PDF thing I did. Here's the code for the OCR thing, build a new thing that can open a PDF file and OCR every page." And it did it. And these days, I'll often just tell Claude Code, " Paste in the URL to this thing, this thing here, here's another thing, go and read the source code and then solve this new problem. And it works so, so well. My research repository, I'll say things like, check out SimonW/research from GitHub and look at the ones in there that deal with web assembly and Rust and then use that to feed into solving this new task in Web Assembly and Rust. Because it's hard to overstate how good these things are without reusing context that you can make available to them. It used to be that you had to think really carefully about the length limits because they could only handle 100,000 or 200,000 tokens at a time. Coding agents can do searches. So, you can give them access to an entire hard drive full of stuff and tell them what you need to solve. And they will run search tools to find just the examples that they need to piece things together. It's incredibly powerful.
01:08:22 · Lenny Rachitsky
[01:08:22] S1: Okay. Amazing. And I love that you share this with people. I know you're not sharing at all, but this just empowers everyone else to piggyback off the work that you've already done over the past. Okay, So another agenetic pattern is Red/Green, test room development, and then this idea of first run the test. Talk about that.
01:08:37 · Simon Willison
[01:08:37] S0: This is the most important thing when you're working with coding agents is they have to test the code. That's the whole point of a coding agent is if they haven't run the code, you're back to copy and pasting out of ChatGPT and crossing your fingers and hoping that it got things right. So, how do you get them to run the code? The best way to do that is to use a programming technique that we've been using for decades called test-driven development, where you have automated tests, you have code that tests your other code, and we call those the tests. Agents will write tests the moment you even hint at them that they should write a test, they'll write a test, which is great because I try to make it so pretty much every line of code that I release into the world, there's an automated test that has at least made sure that that works. The reason these tests are so valuable, there's two things. Firstly, it means that the agent has at least run the code. So if there are syntax errors and things, it'll have found those. And it gives you that significant boost in confidence that it actually works. And then the test, because they go into the repository, they add up over time. And that's what gives you the confidence that when you tell your agent to build a new feature, it won't break old features. This is exactly the same thing for human software engineering teams. The reason I like having automated tests is that I can build new features and I don't then have to manually test every single other feature to make sure it didn't break because the test automate that process works great with agents. If your coding agent has a repository with a good set of tests, you can tell it to change something and it'll change that thing and it won't break anything else, or at least it won't break the things that the tests are covering. So occasionally, I run into people who are using AI for coding and they're like, and we don't even have to test it anymore. We've stopped doing tests because it's so quick that it's faster for us to not use the test. I think those people are wrong. I think it's a huge mistake if you drop tests in exchange for speed of development, because very quickly when you're working with tests, you find your development speed goes up.
01:10:40 · Simon Willison
The existence of the test lets you move faster, because you don't have to constantly worry that you're breaking older things. So that's test-driven development. I think that's absolutely crucial for getting the most out of coding agents. The other thing you mentioned was Red/Green TDD. And I like this one as an example of a sort of miniature prompt that you can use. So, when you are doing test-driven development, one of the ways you can do this as a human programmer is this thing where you first write the test, which won't work because you haven't written the code, and then you run it and you watch it fail. And that gives you confidence that the test, because if it passes, something's gone wrong. So you want to see the test fail, and then you go and implement whatever needs to be done to make the test pass, and then you run the test again and you watch it pass. And I hate doing this. There are a lot of programmers believe that this is the one true way to write software. I tried it for a couple of years. It just slowed me down and frustrated me. I did not enjoy the intellectual challenge of, okay, and the discipline of write the tests first and then watch them part fail because I like to explore by writing a bunch of code and then add the tests later on. Coding agents, I don't care if they're bored. I couldn't care, that's what their opinions on test-driven development are. If you get them to write the tests first, you do get better results because they're much less likely to forget to test something or to add bits of code that aren't necessary. And so you could tell them, write this using test, make sure that you write the test first, then watch the tests fail, then write the information, then watch them pass again. That's a lot of typing. If you use the term red/green TDD, that's programming jargon, which I didn't used to use, but it is jargon for run the test and watch it fail. The agents know what that means. So now we've reduced that sort of lengthy paragraph about how to test to Red/Green TDD enter, you're done. So there are two ideas that that illustrates. Firstly, the importance of that technique of having them run the test and watch them fail. And secondly, the fact that sometimes you do find something you can type in five seconds that has a material impact on how these things are working.