Lenny's Podcast: Product | Career | Growth
00:54:10 · Simon Willison, Lenny Rachitsky
[00:54:10] S0: I use Claude for the code stuff. The other thing that I use a lot of is for research. And this is this thing where a couple of years ago, if you told me that you were replacing user Google with ChatGPT, I'd assume that you just didn't understand how this technology worked in its limitations, because that was a terrible idea. Now that all of the major models have really good search integration, they're just better at searching than I am. I can ask them a question and watch them fire off five searches in parallel for aspects of answering that question, pull the data back. And if it's something I'm going to publish, I always double check, make sure it didn't hallucinate a detail, because that would be embarrassing. But honestly, I hardly use Google search directly at all. I'm always using it via ... I'm doing searches via Claude or via ChatGPT, or sometimes via the Gemini app, that's a good option as well. And then I mean, for image generation, I'm using Gemini because of Nano Banana, but I only use that for fun. I don't publish images I generate, I use them for pranks, and that's great. That's deeply entertaining. [00:55:13] S1: I wasn't planning to go here, but you famously created the Pelican riding a bike benchmark for the quality of imagery. [00:55:19] S0: Yes. [00:55:20] S1: Anything there that might be worth sharing?
00:55:22 · Simon Willison
[00:55:22] S0: So this one's fascinating. So, about a year and a half ago, I started benchmark. So there were lots of benchmarks these models and there were all these numeric things like it scored 72% on terminal bench or whatever. And those always frustrated me, because they don't really tell you anything interesting. If this one got 74 and this one got 72, does that actually mean that one of them is better at something than the other? And so, basically to make fun of the benchmarks, I started my own benchmark, which was generate an SVG of a Pelican riding a bicycle. And it's an SVG. This isn't a test of the image models. This is a test of the text models, because they can all output SVG code. And if you ask them to draw you an SVG of something, they're almost universally terrible, because they don't have good spatial reasoning. And drawing things by plotting out vectors is difficult anyway. So, I started getting the models to generate an SVG of a Pelican and a bicycle, because then you can look at them. You can say, "Here's one, here's one model, here's the other, which is best?" And the weirdest thing happened where there appears to be a very strong correlation between how good their drawing of a Pelican riding a bicycle is and how good they are at everything else. And nobody can explain to me why that is. But as I started looking at these things, I realized, "Wow, the better models really do draw better Pelicans riding a bicycle." Because the point now, it's a meme. The AI labs are all very aware of this, and they relish in how good their Pelicans riding a bicycle are. The other day OpenAI released GPT 5.4 mini and nano at five different thinking levels that you could have them do low thinking, medium thinking, high thinking. So, I did a grid of 15 Pelicans riding bicycles for the three GPT 5.4 models across the things. And sure enough, GPT 5.4 running at X high did draw the best Pelican. Why? I don't know. I don't know why that was, but it did.
00:57:11 · Lenny Rachitsky, Simon Willison
[00:57:11] S1: First of all, I didn't realize this was a test of the LLM, because you'd think an image would be a test of the imaging model, but now it makes sense. [00:57:19] S0: It's all about the code generation. [00:57:20] S1: That is so funny. [00:57:21] S0: The other thing is they're generating SVG and it has comments in. So, you can see little code comments that say things like making sure the Pelican's legs are hitting the pedals, and added a fish for whimsy. And that's really fun. The Chinese AI models, I love playing with the Chinese open-weight models. Some of those have drawn quite good Pelicans, and they've run on my laptop. So, I have my laptop drawing these pictures of Pelicans with these little comments about what it's trying to do. [00:57:46] S1: I think with Gemini when they released one of their models, Inc., that was like their tweet was the image of the [inaudible 00:57:50].
00:57:46 · Lenny Rachitsky, Simon Willison
[00:57:46] S1: I think with Gemini when they released one of their models, Inc., that was like their tweet was the image of the [inaudible 00:57:50]. [00:57:50] S0: 3.1. Gemini 3.1 just a few weeks ago, they had a video which featured a Pelican riding a bicycle like animated. And I'm like, "Oh my God, it's my Pelican." But I thought it's okay, because the way my benchmark works is I've actually got a bunch of secret alternatives in my pocket, because obviously what happens if the AI labs train them to draw really good Pelicans riding bicycles? And I'm like, "Well, then I'll get into an Ocelot on a moped." And if the Ocelot on the moped sucks, but the Pelicans are really good, I can prove that they cheated on the benchmark. And that would be amazing. That would be a great thing to be able to say, "Hey, look, they cheated." Except that when Gemini 3.1 came out, they did all of the other combinations. They were like, "And here's a giraffe and a little tiny car." And so on. I'm like, "wow, they've beat me. They're doing all of the animals and all of the modes of transport." [00:58:37] S1: And they didn't know that you had this in your back pocket to test. [00:58:39] S0: I don't know if they knew or not. [00:58:40] S1: Oh, that's so funny.
00:58:40 · Lenny Rachitsky, Simon Willison
[00:58:40] S1: Oh, that's so funny. [00:58:43] S0: People kept on asking me for the past year, they've been saying, "What if the labs cheat on the benchmark?" And my answer has always been, really, all I want from life is a really good picture of a Pelican riding a bicycle. And if I can trick every AI lab into the world into cheating on benchmarks to get it, then that just achieves my goal. [00:59:02] S1: Why do you want this? What's the drive here? [00:59:05] S0: I live in Half Moon Bay. We have the world's second-largest mega rooster of the California Brown Pelican is like 15 minutes walk down the hill and they're really cool. I just like Pelicans. When I moved to California from England, one of the convinces was I was up on the cliffs in Marin and a Pelican flew by at eye level and I'm like, "That's a Pelican like in the books." And the Americans were like, "What it's a Pelican? We see them all the time." But yeah, I like Pelicans. [00:59:32] S1: I think this is a bigger point that you've been an engineer for a long time. You've embraced this big shift in the role. And I think a big ... Because I'm wondering just because a lot of people are scared, freaked out, like, "I hate this, my job's changing." And you've been the opposite. You're having so much fun. And I feel like this kind of whimsy joy that you bring to it is a key part of being successful in this transition.
00:59:32 · Lenny Rachitsky, Simon Willison
[00:59:32] S1: I think this is a bigger point that you've been an engineer for a long time. You've embraced this big shift in the role. And I think a big ... Because I'm wondering just because a lot of people are scared, freaked out, like, "I hate this, my job's changing." And you've been the opposite. You're having so much fun. And I feel like this kind of whimsy joy that you bring to it is a key part of being successful in this transition. [00:59:56] S0: I think something people often miss is that this space is inherently funny. It is ridiculous. The fact that you could trick ChatGPT into telling you how to make Napalm by saying that your grandmother worked at the Napalm factory and you missed her and all of that kind of ... It's so silly. And I like leaning into that, the fact that we have these incredibly expensive, power hungry, supposedly the most advanced computers of all time. And if you ask them to draw a Pelican on a bicycle, it looks like a five-year-old drew it. That's really funny to me. And I am enjoying that. I'm enjoying embracing the inherent ridiculousness of what we're trying to achieve with these things. [01:00:33] S1: I love that. And honest YouTube will show the Pelicans because the progress is made, by the way, is just absurd. It started so bad and that's really good. And it's shockingly hard to make a bicycle, turns out. [01:00:43] S0: I mean, if you try and draw a bicycle right now- [01:00:46] S1: I have no idea. [01:00:47] S0: Because remembering the triangles of the frame is actually really difficult. Most people can't draw bicycles.