Lenny's Podcast: Product | Career | Growth
00:58:40 · Lenny Rachitsky, Simon Willison
[00:58:40] S1: Oh, that's so funny. [00:58:43] S0: People kept on asking me for the past year, they've been saying, "What if the labs cheat on the benchmark?" And my answer has always been, really, all I want from life is a really good picture of a Pelican riding a bicycle. And if I can trick every AI lab into the world into cheating on benchmarks to get it, then that just achieves my goal. [00:59:02] S1: Why do you want this? What's the drive here? [00:59:05] S0: I live in Half Moon Bay. We have the world's second-largest mega rooster of the California Brown Pelican is like 15 minutes walk down the hill and they're really cool. I just like Pelicans. When I moved to California from England, one of the convinces was I was up on the cliffs in Marin and a Pelican flew by at eye level and I'm like, "That's a Pelican like in the books." And the Americans were like, "What it's a Pelican? We see them all the time." But yeah, I like Pelicans. [00:59:32] S1: I think this is a bigger point that you've been an engineer for a long time. You've embraced this big shift in the role. And I think a big ... Because I'm wondering just because a lot of people are scared, freaked out, like, "I hate this, my job's changing." And you've been the opposite. You're having so much fun. And I feel like this kind of whimsy joy that you bring to it is a key part of being successful in this transition.
00:59:32 · Lenny Rachitsky, Simon Willison
[00:59:32] S1: I think this is a bigger point that you've been an engineer for a long time. You've embraced this big shift in the role. And I think a big ... Because I'm wondering just because a lot of people are scared, freaked out, like, "I hate this, my job's changing." And you've been the opposite. You're having so much fun. And I feel like this kind of whimsy joy that you bring to it is a key part of being successful in this transition. [00:59:56] S0: I think something people often miss is that this space is inherently funny. It is ridiculous. The fact that you could trick ChatGPT into telling you how to make Napalm by saying that your grandmother worked at the Napalm factory and you missed her and all of that kind of ... It's so silly. And I like leaning into that, the fact that we have these incredibly expensive, power hungry, supposedly the most advanced computers of all time. And if you ask them to draw a Pelican on a bicycle, it looks like a five-year-old drew it. That's really funny to me. And I am enjoying that. I'm enjoying embracing the inherent ridiculousness of what we're trying to achieve with these things. [01:00:33] S1: I love that. And honest YouTube will show the Pelicans because the progress is made, by the way, is just absurd. It started so bad and that's really good. And it's shockingly hard to make a bicycle, turns out. [01:00:43] S0: I mean, if you try and draw a bicycle right now- [01:00:46] S1: I have no idea. [01:00:47] S0: Because remembering the triangles of the frame is actually really difficult. Most people can't draw bicycles.
01:00:47 · Simon Willison, Lenny Rachitsky
[01:00:47] S0: Because remembering the triangles of the frame is actually really difficult. Most people can't draw bicycles. [01:00:52] S1: Okay. I'm going to get us back on track. I want to talk through a couple other agenetic engineering patterns you recommend. Another is hoarding things you know how to do. What's that all about?
01:01:03 · Simon Willison
[01:01:03] S0: Yeah, again, this is sort of a lifelong piece of career advice. Something that I'm enjoying with the book that I'm writing is most of the things that make agents write better code work for humans too. I'm basically just writing a book about software engineering and what works well, and pretending it's about agents, but it's not. So yeah, the hoarding things you know what to do is a piece of career advice where the way you build value as a software engineer or pretty much any other profession is you build a really big backlog of things that you've tried in the past that worked or didn't work, such that when a new problem comes along, you can think, "Okay, well, in 2015, I built a system that used Redis to do an activity inbox. And then in 2017, I did rate limiting with Node.js. I can combine those two things right now and that will solve this new problem. And so having that backlog of things you've solved in the past of techniques that you know to work, that's what gives you enormous value because you can face it, you can see a new problem and maybe you're the only person in the world who's tried technology X and technology Y and technique B, and spots that this new problem can be solved by combining those things. I've spent my career hoarding all of these different bits and pieces that I've got just a little bit of experience with. And AI makes that so much easier because now I can knock out a very quick prototype that tries out this new No SQL database or whatever it is, costs me nothing to do. I've now got a markdown file somewhere with the output of the document. I have a couple of GitHub repositories that I specifically use for this. I've got one called Tools, SimonW/Tools, and that's little HTML and JavaScript tools that I've built or that I've got Claude to build for me. And there's like 193 of those now. And a lot of them are very simple things. Some of them are a little bit more complicated. Every single one of them captures an idea or a thing that I now know is possible to do. I don't know how to do it off the top of my head, but I can go and look at the code or I can have Claude look at the code and combine that with other things to solve new problems. Then the other one I have is SimonW/research on GitHub, which are AI-driven research projects.
01:03:13 · Simon Willison, Lenny Rachitsky
So I will say to Claude Code, usually Claude Code on my phone, "Here's a new piece of software, go and download it, look at how it works, write the report what it can do and try it against this problem." And the output will be a markdown file that then sits in GitHub, and that's it. That's the whole thing. But these research projects are a really quick way for me to try porting something from JavaScript to Python or see, or I'll run little benchmarks and see how performant a new thing is. And each one of those just gets added into that backlog of things that I've tried or things that I've got a starting point figuring out how effective they are. [01:03:51] S1: So interesting. So, essentially you collect learnings in these various formats, you're doing it in GitHub. So, the two kind of buckets here is one is specific little features and tools you've built that kind of plug in to help solve problems in projects you're working on. [01:04:09] S0: Yeah. And then all little client side web applications, it's just HTML and JavaScript. That's the whole thing. Yeah. [01:04:13] S1: And then the other is just like questions that you wanted answers to. And then here's the answer so that you could just say, "Hey, use this research we've done previously to help us solve this problem."
01:04:13 · Lenny Rachitsky, Simon Willison
[01:04:13] S1: And then the other is just like questions that you wanted answers to. And then here's the answer so that you could just say, "Hey, use this research we've done previously to help us solve this problem." [01:04:22] S0: But the key thing about that is this isn't research in this traditional sense of go and search the web and do me a deep research report. These are all coding agent research tasks where actually written code and run it. Because that's what makes them ... If I published a GitHub repository full of unverified deep research reports, that's very little value to anyone. But the moment the coding agent has written the code, run the code, plotted a graph of how it worked or whatever, that's what turns it into not just LLM vomit. It becomes something that's at least slightly actionable. [01:04:54] S1: Yeah. And I love that you use the term hoard, which comes across as keep it secret, but you make it publicly available in open source, which is recording. [01:04:54] S0: For the most part yeah. [01:05:07] S1: For the most ... Yeah, because I'm grassing it in, it's all here. But I guess is there some stuff that you hoard, hoard for real? [01:05:08] S0: I mean, I've got 10,000 Apple notes as well that I just constantly add new things to. But generally I default to putting this stuff in public, because it benefits me more that way. It's easier for me to find later on. It's like I use GitHub as a backup system and it's great for my credibility as a programmer that I've got all of this stuff out there.