Lenny's Podcast: Product | Career | Growth
01:00:47 · Simon Willison, Lenny Rachitsky
[01:00:47] S0: Because remembering the triangles of the frame is actually really difficult. Most people can't draw bicycles. [01:00:52] S1: Okay. I'm going to get us back on track. I want to talk through a couple other agenetic engineering patterns you recommend. Another is hoarding things you know how to do. What's that all about?
01:01:03 · Simon Willison
[01:01:03] S0: Yeah, again, this is sort of a lifelong piece of career advice. Something that I'm enjoying with the book that I'm writing is most of the things that make agents write better code work for humans too. I'm basically just writing a book about software engineering and what works well, and pretending it's about agents, but it's not. So yeah, the hoarding things you know what to do is a piece of career advice where the way you build value as a software engineer or pretty much any other profession is you build a really big backlog of things that you've tried in the past that worked or didn't work, such that when a new problem comes along, you can think, "Okay, well, in 2015, I built a system that used Redis to do an activity inbox. And then in 2017, I did rate limiting with Node.js. I can combine those two things right now and that will solve this new problem. And so having that backlog of things you've solved in the past of techniques that you know to work, that's what gives you enormous value because you can face it, you can see a new problem and maybe you're the only person in the world who's tried technology X and technology Y and technique B, and spots that this new problem can be solved by combining those things. I've spent my career hoarding all of these different bits and pieces that I've got just a little bit of experience with. And AI makes that so much easier because now I can knock out a very quick prototype that tries out this new No SQL database or whatever it is, costs me nothing to do. I've now got a markdown file somewhere with the output of the document. I have a couple of GitHub repositories that I specifically use for this. I've got one called Tools, SimonW/Tools, and that's little HTML and JavaScript tools that I've built or that I've got Claude to build for me. And there's like 193 of those now. And a lot of them are very simple things. Some of them are a little bit more complicated. Every single one of them captures an idea or a thing that I now know is possible to do. I don't know how to do it off the top of my head, but I can go and look at the code or I can have Claude look at the code and combine that with other things to solve new problems. Then the other one I have is SimonW/research on GitHub, which are AI-driven research projects.
01:03:13 · Simon Willison, Lenny Rachitsky
So I will say to Claude Code, usually Claude Code on my phone, "Here's a new piece of software, go and download it, look at how it works, write the report what it can do and try it against this problem." And the output will be a markdown file that then sits in GitHub, and that's it. That's the whole thing. But these research projects are a really quick way for me to try porting something from JavaScript to Python or see, or I'll run little benchmarks and see how performant a new thing is. And each one of those just gets added into that backlog of things that I've tried or things that I've got a starting point figuring out how effective they are. [01:03:51] S1: So interesting. So, essentially you collect learnings in these various formats, you're doing it in GitHub. So, the two kind of buckets here is one is specific little features and tools you've built that kind of plug in to help solve problems in projects you're working on. [01:04:09] S0: Yeah. And then all little client side web applications, it's just HTML and JavaScript. That's the whole thing. Yeah. [01:04:13] S1: And then the other is just like questions that you wanted answers to. And then here's the answer so that you could just say, "Hey, use this research we've done previously to help us solve this problem."
01:04:13 · Lenny Rachitsky, Simon Willison
[01:04:13] S1: And then the other is just like questions that you wanted answers to. And then here's the answer so that you could just say, "Hey, use this research we've done previously to help us solve this problem." [01:04:22] S0: But the key thing about that is this isn't research in this traditional sense of go and search the web and do me a deep research report. These are all coding agent research tasks where actually written code and run it. Because that's what makes them ... If I published a GitHub repository full of unverified deep research reports, that's very little value to anyone. But the moment the coding agent has written the code, run the code, plotted a graph of how it worked or whatever, that's what turns it into not just LLM vomit. It becomes something that's at least slightly actionable. [01:04:54] S1: Yeah. And I love that you use the term hoard, which comes across as keep it secret, but you make it publicly available in open source, which is recording. [01:04:54] S0: For the most part yeah. [01:05:07] S1: For the most ... Yeah, because I'm grassing it in, it's all here. But I guess is there some stuff that you hoard, hoard for real? [01:05:08] S0: I mean, I've got 10,000 Apple notes as well that I just constantly add new things to. But generally I default to putting this stuff in public, because it benefits me more that way. It's easier for me to find later on. It's like I use GitHub as a backup system and it's great for my credibility as a programmer that I've got all of this stuff out there.
01:05:08 · Simon Willison, Lenny Rachitsky
[01:05:08] S0: I mean, I've got 10,000 Apple notes as well that I just constantly add new things to. But generally I default to putting this stuff in public, because it benefits me more that way. It's easier for me to find later on. It's like I use GitHub as a backup system and it's great for my credibility as a programmer that I've got all of this stuff out there. [01:05:29] S1: So for people that want to do this, what's the advice here? Is it just keep notes at the start of things you've learned is possible and works? [01:05:37] S0: Yes, but find a note system that you trust and that you're not going to lose. So the easiest one would be like a folder synced to Dropbox or something like that. I really like GitHub. I've got lots of private GitHub repositories. My public research one has 75 projects in it. I've got a private research one with another 50 that are things that just didn't ... They're tied to my personal projects or whatever it is. So I have a whole bunch of things like that as well. GitHub is free for private repositories somehow. So I'm doing all of this stuff in GitHub. And when you put something on GitHub, they back it up to three continents. Your chances of losing something on GitHub are very, very slim. Occasionally they'll go and stick it in a vault in the Arctic as well. So I feel pretty good about them as a place to keep that data. [01:06:27] S1: And then how do you actually use this? Is this feeded into the LLM when you're building or is it on occasion, go look at this, go look at that? Is that in memory or not?
01:06:36 · Simon Willison
[01:06:36] S0: It's definitely both. But the key trick that I've been using a lot is, especially for my little HTML and JavaScript tools, you can tell an LLM to consult them and combine them. So, a very early example of that is I'd written some code pre-LLMs, which used a PDF library from Mozilla. So it's in JavaScript, but it can open up a PDF and show you that PDF on the page. And I'd also written some code that used Tesseract, which is an OCR library that can run in your browser and do actually really good OCR all in JavaScript. And I just realized I wanted to do OCR against PDF files. So I told Claude Opus 3, I think back then I said, "Here's the code for the PDF thing I did. Here's the code for the OCR thing, build a new thing that can open a PDF file and OCR every page." And it did it. And these days, I'll often just tell Claude Code, " Paste in the URL to this thing, this thing here, here's another thing, go and read the source code and then solve this new problem. And it works so, so well. My research repository, I'll say things like, check out SimonW/research from GitHub and look at the ones in there that deal with web assembly and Rust and then use that to feed into solving this new task in Web Assembly and Rust. Because it's hard to overstate how good these things are without reusing context that you can make available to them. It used to be that you had to think really carefully about the length limits because they could only handle 100,000 or 200,000 tokens at a time. Coding agents can do searches. So, you can give them access to an entire hard drive full of stuff and tell them what you need to solve. And they will run search tools to find just the examples that they need to piece things together. It's incredibly powerful.