Lenny's Podcast: Product | Career | Growth
00:50:38 · Simon Willison
If it's security adjacent or super important, I might pull it down to my laptop to do a thorough review later on. But most of the review you can do through GitHub. These things will file pull requests and then you use the same tools you'd use to review code from other people to review the code from the agents. That said, OpenAI came out with GPT 5.4 about three weeks ago. It's very, very, very good. I think it's on par with Claude Opus 4.6, and possibly even better. These companies are constantly leapfrogging each other. So I have been leaning ... It's also cheaper. So I've been leaning on GPT 5.4 a lot more this month, and OpenAI Codex, and OpenAI Codex and Claude Code are almost, almost indistinguishable from each other now. They're both very, very good pieces of software. And I kind of expect this to happen. The next Gemini model comes out might become the best coding model for a couple of months, in which case I might switch myself into that ecosystem. Partly because I write about this stuff as well. I like to stay familiar with as many of the offerings as possible. But I keep on coming back to Claude Code, mainly because it fits my taste. There's this weird thing where I've got a very specific taste in how I like code to work, which coincidentally happens to map to how Claude Code likes to work, which is kind of interesting. And GPT 5.4, it almost matches my taste, but not quite. And maybe that's because I've just spent more time with Claude, so my prompting style has evolved more to fit the Claude way of thinking. I don't know. This stuff's all so weird. It's vibes all the way down.
00:52:07 · Lenny Rachitsky, Simon Willison
[00:52:07] S1: That is so interesting. So, the taste is the code, the quality of the code it puts out is what you're talking about, not like the conversation and the UX. [00:52:14] S0: Absolutely. Don't care about how they talk to me. I'm using them to get stuff done. Yeah. [00:52:19] S1: Yeah. Because I was thinking as you're talking, what is the thing that will get someone to stick with a model? And it could be what you're describing, the way it writes code. It could be the UX, it could be the conversation vibes.
00:52:19 · Lenny Rachitsky, Simon Willison
[00:52:19] S1: Yeah. Because I was thinking as you're talking, what is the thing that will get someone to stick with a model? And it could be what you're describing, the way it writes code. It could be the UX, it could be the conversation vibes. [00:52:33] S0: Well, the stickiest thing is meant to be memory. They all have these features where they will remember things about you. And I hate those features and I turn them off wherever I can, because mainly because as an AI researcher, I need to see what everyone else sees when I'm prompting. I don't want to say to the world, "Oh, my goodness, look, this thing works now." And it sounds like it only works for me, because it's based on previous conversations that I've had. And maybe I'm missing out on something really important there. But the memory feature is that thing that all of the labs are trying to be more [inaudible 00:53:08]. That said, when the OpenAI military stuff happened a few weeks ago, Anthropic took advantage by saying, "Hey, why don't you move to Claude?" And the way they did that is they had a Claude onboarding page that said, "Transfer your memories from ChatGPT by clicking this button and then pasting it into ChatGPT." And it was just a prompt. They had a prompt, which was, "Hey, ChatGPT, tell me everything that you've remembered about me." And so you paste that prompt into ChatGPT and it gives you all of their memories, and then you paste them into Claude. And I thought that was hilarious. A whole export move from one to the other just by prompting it to give you the information you needed.
00:53:48 · Lenny Rachitsky, Simon Willison
[00:53:48] S1: Yeah. It always felt like that was hard to extract, and they made it so easy. And that was such a moment for Anthropic. They were like the number one app in the App Store, such a interesting, not what you'd expect when they were being banned by the government, essentially. Is there any other AI tools that you find really useful just along the side like Wispr Flow, anything along those lines? [00:54:10] S0: I use Claude for the code stuff. The other thing that I use a lot of is for research. And this is this thing where a couple of years ago, if you told me that you were replacing user Google with ChatGPT, I'd assume that you just didn't understand how this technology worked in its limitations, because that was a terrible idea. Now that all of the major models have really good search integration, they're just better at searching than I am. I can ask them a question and watch them fire off five searches in parallel for aspects of answering that question, pull the data back. And if it's something I'm going to publish, I always double check, make sure it didn't hallucinate a detail, because that would be embarrassing. But honestly, I hardly use Google search directly at all. I'm always using it via ... I'm doing searches via Claude or via ChatGPT, or sometimes via the Gemini app, that's a good option as well. And then I mean, for image generation, I'm using Gemini because of Nano Banana, but I only use that for fun. I don't publish images I generate, I use them for pranks, and that's great. That's deeply entertaining.
00:54:10 · Simon Willison, Lenny Rachitsky
[00:54:10] S0: I use Claude for the code stuff. The other thing that I use a lot of is for research. And this is this thing where a couple of years ago, if you told me that you were replacing user Google with ChatGPT, I'd assume that you just didn't understand how this technology worked in its limitations, because that was a terrible idea. Now that all of the major models have really good search integration, they're just better at searching than I am. I can ask them a question and watch them fire off five searches in parallel for aspects of answering that question, pull the data back. And if it's something I'm going to publish, I always double check, make sure it didn't hallucinate a detail, because that would be embarrassing. But honestly, I hardly use Google search directly at all. I'm always using it via ... I'm doing searches via Claude or via ChatGPT, or sometimes via the Gemini app, that's a good option as well. And then I mean, for image generation, I'm using Gemini because of Nano Banana, but I only use that for fun. I don't publish images I generate, I use them for pranks, and that's great. That's deeply entertaining. [00:55:13] S1: I wasn't planning to go here, but you famously created the Pelican riding a bike benchmark for the quality of imagery. [00:55:19] S0: Yes. [00:55:20] S1: Anything there that might be worth sharing?
00:55:22 · Simon Willison
[00:55:22] S0: So this one's fascinating. So, about a year and a half ago, I started benchmark. So there were lots of benchmarks these models and there were all these numeric things like it scored 72% on terminal bench or whatever. And those always frustrated me, because they don't really tell you anything interesting. If this one got 74 and this one got 72, does that actually mean that one of them is better at something than the other? And so, basically to make fun of the benchmarks, I started my own benchmark, which was generate an SVG of a Pelican riding a bicycle. And it's an SVG. This isn't a test of the image models. This is a test of the text models, because they can all output SVG code. And if you ask them to draw you an SVG of something, they're almost universally terrible, because they don't have good spatial reasoning. And drawing things by plotting out vectors is difficult anyway. So, I started getting the models to generate an SVG of a Pelican and a bicycle, because then you can look at them. You can say, "Here's one, here's one model, here's the other, which is best?" And the weirdest thing happened where there appears to be a very strong correlation between how good their drawing of a Pelican riding a bicycle is and how good they are at everything else. And nobody can explain to me why that is. But as I started looking at these things, I realized, "Wow, the better models really do draw better Pelicans riding a bicycle." Because the point now, it's a meme. The AI labs are all very aware of this, and they relish in how good their Pelicans riding a bicycle are. The other day OpenAI released GPT 5.4 mini and nano at five different thinking levels that you could have them do low thinking, medium thinking, high thinking. So, I did a grid of 15 Pelicans riding bicycles for the three GPT 5.4 models across the things. And sure enough, GPT 5.4 running at X high did draw the best Pelican. Why? I don't know. I don't know why that was, but it did.