Lenny's Podcast: Product | Career | Growth
00:52:19 · Lenny Rachitsky, Simon Willison
[00:52:19] S1: Yeah. Because I was thinking as you're talking, what is the thing that will get someone to stick with a model? And it could be what you're describing, the way it writes code. It could be the UX, it could be the conversation vibes. [00:52:33] S0: Well, the stickiest thing is meant to be memory. They all have these features where they will remember things about you. And I hate those features and I turn them off wherever I can, because mainly because as an AI researcher, I need to see what everyone else sees when I'm prompting. I don't want to say to the world, "Oh, my goodness, look, this thing works now." And it sounds like it only works for me, because it's based on previous conversations that I've had. And maybe I'm missing out on something really important there. But the memory feature is that thing that all of the labs are trying to be more [inaudible 00:53:08]. That said, when the OpenAI military stuff happened a few weeks ago, Anthropic took advantage by saying, "Hey, why don't you move to Claude?" And the way they did that is they had a Claude onboarding page that said, "Transfer your memories from ChatGPT by clicking this button and then pasting it into ChatGPT." And it was just a prompt. They had a prompt, which was, "Hey, ChatGPT, tell me everything that you've remembered about me." And so you paste that prompt into ChatGPT and it gives you all of their memories, and then you paste them into Claude. And I thought that was hilarious. A whole export move from one to the other just by prompting it to give you the information you needed.
00:53:48 · Lenny Rachitsky, Simon Willison
[00:53:48] S1: Yeah. It always felt like that was hard to extract, and they made it so easy. And that was such a moment for Anthropic. They were like the number one app in the App Store, such a interesting, not what you'd expect when they were being banned by the government, essentially. Is there any other AI tools that you find really useful just along the side like Wispr Flow, anything along those lines? [00:54:10] S0: I use Claude for the code stuff. The other thing that I use a lot of is for research. And this is this thing where a couple of years ago, if you told me that you were replacing user Google with ChatGPT, I'd assume that you just didn't understand how this technology worked in its limitations, because that was a terrible idea. Now that all of the major models have really good search integration, they're just better at searching than I am. I can ask them a question and watch them fire off five searches in parallel for aspects of answering that question, pull the data back. And if it's something I'm going to publish, I always double check, make sure it didn't hallucinate a detail, because that would be embarrassing. But honestly, I hardly use Google search directly at all. I'm always using it via ... I'm doing searches via Claude or via ChatGPT, or sometimes via the Gemini app, that's a good option as well. And then I mean, for image generation, I'm using Gemini because of Nano Banana, but I only use that for fun. I don't publish images I generate, I use them for pranks, and that's great. That's deeply entertaining.
00:54:10 · Simon Willison, Lenny Rachitsky
[00:54:10] S0: I use Claude for the code stuff. The other thing that I use a lot of is for research. And this is this thing where a couple of years ago, if you told me that you were replacing user Google with ChatGPT, I'd assume that you just didn't understand how this technology worked in its limitations, because that was a terrible idea. Now that all of the major models have really good search integration, they're just better at searching than I am. I can ask them a question and watch them fire off five searches in parallel for aspects of answering that question, pull the data back. And if it's something I'm going to publish, I always double check, make sure it didn't hallucinate a detail, because that would be embarrassing. But honestly, I hardly use Google search directly at all. I'm always using it via ... I'm doing searches via Claude or via ChatGPT, or sometimes via the Gemini app, that's a good option as well. And then I mean, for image generation, I'm using Gemini because of Nano Banana, but I only use that for fun. I don't publish images I generate, I use them for pranks, and that's great. That's deeply entertaining. [00:55:13] S1: I wasn't planning to go here, but you famously created the Pelican riding a bike benchmark for the quality of imagery. [00:55:19] S0: Yes. [00:55:20] S1: Anything there that might be worth sharing?
00:55:22 · Simon Willison
[00:55:22] S0: So this one's fascinating. So, about a year and a half ago, I started benchmark. So there were lots of benchmarks these models and there were all these numeric things like it scored 72% on terminal bench or whatever. And those always frustrated me, because they don't really tell you anything interesting. If this one got 74 and this one got 72, does that actually mean that one of them is better at something than the other? And so, basically to make fun of the benchmarks, I started my own benchmark, which was generate an SVG of a Pelican riding a bicycle. And it's an SVG. This isn't a test of the image models. This is a test of the text models, because they can all output SVG code. And if you ask them to draw you an SVG of something, they're almost universally terrible, because they don't have good spatial reasoning. And drawing things by plotting out vectors is difficult anyway. So, I started getting the models to generate an SVG of a Pelican and a bicycle, because then you can look at them. You can say, "Here's one, here's one model, here's the other, which is best?" And the weirdest thing happened where there appears to be a very strong correlation between how good their drawing of a Pelican riding a bicycle is and how good they are at everything else. And nobody can explain to me why that is. But as I started looking at these things, I realized, "Wow, the better models really do draw better Pelicans riding a bicycle." Because the point now, it's a meme. The AI labs are all very aware of this, and they relish in how good their Pelicans riding a bicycle are. The other day OpenAI released GPT 5.4 mini and nano at five different thinking levels that you could have them do low thinking, medium thinking, high thinking. So, I did a grid of 15 Pelicans riding bicycles for the three GPT 5.4 models across the things. And sure enough, GPT 5.4 running at X high did draw the best Pelican. Why? I don't know. I don't know why that was, but it did.
00:57:11 · Lenny Rachitsky, Simon Willison
[00:57:11] S1: First of all, I didn't realize this was a test of the LLM, because you'd think an image would be a test of the imaging model, but now it makes sense. [00:57:19] S0: It's all about the code generation. [00:57:20] S1: That is so funny. [00:57:21] S0: The other thing is they're generating SVG and it has comments in. So, you can see little code comments that say things like making sure the Pelican's legs are hitting the pedals, and added a fish for whimsy. And that's really fun. The Chinese AI models, I love playing with the Chinese open-weight models. Some of those have drawn quite good Pelicans, and they've run on my laptop. So, I have my laptop drawing these pictures of Pelicans with these little comments about what it's trying to do. [00:57:46] S1: I think with Gemini when they released one of their models, Inc., that was like their tweet was the image of the [inaudible 00:57:50].
00:57:46 · Lenny Rachitsky, Simon Willison
[00:57:46] S1: I think with Gemini when they released one of their models, Inc., that was like their tweet was the image of the [inaudible 00:57:50]. [00:57:50] S0: 3.1. Gemini 3.1 just a few weeks ago, they had a video which featured a Pelican riding a bicycle like animated. And I'm like, "Oh my God, it's my Pelican." But I thought it's okay, because the way my benchmark works is I've actually got a bunch of secret alternatives in my pocket, because obviously what happens if the AI labs train them to draw really good Pelicans riding bicycles? And I'm like, "Well, then I'll get into an Ocelot on a moped." And if the Ocelot on the moped sucks, but the Pelicans are really good, I can prove that they cheated on the benchmark. And that would be amazing. That would be a great thing to be able to say, "Hey, look, they cheated." Except that when Gemini 3.1 came out, they did all of the other combinations. They were like, "And here's a giraffe and a little tiny car." And so on. I'm like, "wow, they've beat me. They're doing all of the animals and all of the modes of transport." [00:58:37] S1: And they didn't know that you had this in your back pocket to test. [00:58:39] S0: I don't know if they knew or not. [00:58:40] S1: Oh, that's so funny.