Lenny's Podcast: Product | Career | Growth
00:57:46 · Lenny Rachitsky, Simon Willison
[00:57:46] S1: I think with Gemini when they released one of their models, Inc., that was like their tweet was the image of the [inaudible 00:57:50]. [00:57:50] S0: 3.1. Gemini 3.1 just a few weeks ago, they had a video which featured a Pelican riding a bicycle like animated. And I'm like, "Oh my God, it's my Pelican." But I thought it's okay, because the way my benchmark works is I've actually got a bunch of secret alternatives in my pocket, because obviously what happens if the AI labs train them to draw really good Pelicans riding bicycles? And I'm like, "Well, then I'll get into an Ocelot on a moped." And if the Ocelot on the moped sucks, but the Pelicans are really good, I can prove that they cheated on the benchmark. And that would be amazing. That would be a great thing to be able to say, "Hey, look, they cheated." Except that when Gemini 3.1 came out, they did all of the other combinations. They were like, "And here's a giraffe and a little tiny car." And so on. I'm like, "wow, they've beat me. They're doing all of the animals and all of the modes of transport." [00:58:37] S1: And they didn't know that you had this in your back pocket to test. [00:58:39] S0: I don't know if they knew or not. [00:58:40] S1: Oh, that's so funny.
00:58:40 · Lenny Rachitsky, Simon Willison
[00:58:40] S1: Oh, that's so funny. [00:58:43] S0: People kept on asking me for the past year, they've been saying, "What if the labs cheat on the benchmark?" And my answer has always been, really, all I want from life is a really good picture of a Pelican riding a bicycle. And if I can trick every AI lab into the world into cheating on benchmarks to get it, then that just achieves my goal. [00:59:02] S1: Why do you want this? What's the drive here? [00:59:05] S0: I live in Half Moon Bay. We have the world's second-largest mega rooster of the California Brown Pelican is like 15 minutes walk down the hill and they're really cool. I just like Pelicans. When I moved to California from England, one of the convinces was I was up on the cliffs in Marin and a Pelican flew by at eye level and I'm like, "That's a Pelican like in the books." And the Americans were like, "What it's a Pelican? We see them all the time." But yeah, I like Pelicans. [00:59:32] S1: I think this is a bigger point that you've been an engineer for a long time. You've embraced this big shift in the role. And I think a big ... Because I'm wondering just because a lot of people are scared, freaked out, like, "I hate this, my job's changing." And you've been the opposite. You're having so much fun. And I feel like this kind of whimsy joy that you bring to it is a key part of being successful in this transition.
00:59:32 · Lenny Rachitsky, Simon Willison
[00:59:32] S1: I think this is a bigger point that you've been an engineer for a long time. You've embraced this big shift in the role. And I think a big ... Because I'm wondering just because a lot of people are scared, freaked out, like, "I hate this, my job's changing." And you've been the opposite. You're having so much fun. And I feel like this kind of whimsy joy that you bring to it is a key part of being successful in this transition. [00:59:56] S0: I think something people often miss is that this space is inherently funny. It is ridiculous. The fact that you could trick ChatGPT into telling you how to make Napalm by saying that your grandmother worked at the Napalm factory and you missed her and all of that kind of ... It's so silly. And I like leaning into that, the fact that we have these incredibly expensive, power hungry, supposedly the most advanced computers of all time. And if you ask them to draw a Pelican on a bicycle, it looks like a five-year-old drew it. That's really funny to me. And I am enjoying that. I'm enjoying embracing the inherent ridiculousness of what we're trying to achieve with these things. [01:00:33] S1: I love that. And honest YouTube will show the Pelicans because the progress is made, by the way, is just absurd. It started so bad and that's really good. And it's shockingly hard to make a bicycle, turns out. [01:00:43] S0: I mean, if you try and draw a bicycle right now- [01:00:46] S1: I have no idea. [01:00:47] S0: Because remembering the triangles of the frame is actually really difficult. Most people can't draw bicycles.
01:00:47 · Simon Willison, Lenny Rachitsky
[01:00:47] S0: Because remembering the triangles of the frame is actually really difficult. Most people can't draw bicycles. [01:00:52] S1: Okay. I'm going to get us back on track. I want to talk through a couple other agenetic engineering patterns you recommend. Another is hoarding things you know how to do. What's that all about?
01:01:03 · Simon Willison
[01:01:03] S0: Yeah, again, this is sort of a lifelong piece of career advice. Something that I'm enjoying with the book that I'm writing is most of the things that make agents write better code work for humans too. I'm basically just writing a book about software engineering and what works well, and pretending it's about agents, but it's not. So yeah, the hoarding things you know what to do is a piece of career advice where the way you build value as a software engineer or pretty much any other profession is you build a really big backlog of things that you've tried in the past that worked or didn't work, such that when a new problem comes along, you can think, "Okay, well, in 2015, I built a system that used Redis to do an activity inbox. And then in 2017, I did rate limiting with Node.js. I can combine those two things right now and that will solve this new problem. And so having that backlog of things you've solved in the past of techniques that you know to work, that's what gives you enormous value because you can face it, you can see a new problem and maybe you're the only person in the world who's tried technology X and technology Y and technique B, and spots that this new problem can be solved by combining those things. I've spent my career hoarding all of these different bits and pieces that I've got just a little bit of experience with. And AI makes that so much easier because now I can knock out a very quick prototype that tries out this new No SQL database or whatever it is, costs me nothing to do. I've now got a markdown file somewhere with the output of the document. I have a couple of GitHub repositories that I specifically use for this. I've got one called Tools, SimonW/Tools, and that's little HTML and JavaScript tools that I've built or that I've got Claude to build for me. And there's like 193 of those now. And a lot of them are very simple things. Some of them are a little bit more complicated. Every single one of them captures an idea or a thing that I now know is possible to do. I don't know how to do it off the top of my head, but I can go and look at the code or I can have Claude look at the code and combine that with other things to solve new problems. Then the other one I have is SimonW/research on GitHub, which are AI-driven research projects.
01:03:13 · Simon Willison, Lenny Rachitsky
So I will say to Claude Code, usually Claude Code on my phone, "Here's a new piece of software, go and download it, look at how it works, write the report what it can do and try it against this problem." And the output will be a markdown file that then sits in GitHub, and that's it. That's the whole thing. But these research projects are a really quick way for me to try porting something from JavaScript to Python or see, or I'll run little benchmarks and see how performant a new thing is. And each one of those just gets added into that backlog of things that I've tried or things that I've got a starting point figuring out how effective they are. [01:03:51] S1: So interesting. So, essentially you collect learnings in these various formats, you're doing it in GitHub. So, the two kind of buckets here is one is specific little features and tools you've built that kind of plug in to help solve problems in projects you're working on. [01:04:09] S0: Yeah. And then all little client side web applications, it's just HTML and JavaScript. That's the whole thing. Yeah. [01:04:13] S1: And then the other is just like questions that you wanted answers to. And then here's the answer so that you could just say, "Hey, use this research we've done previously to help us solve this problem."