Lenny's Podcast: Product | Career | Growth
00:39:15 · Dan Shipper
[00:39:15] S1: Automation is a lie, in the sense that every time you automate something, in order to make sure the automation is working well you need a human on top of it making sure that it's working well. And so, I wrote this piece a couple years ago about the allocation economy, the idea that the way that humans are going to work with AI is going to be like being a manager. And the thing that you have to remember about managers is like managers actually spend a lot of time working. Most managers are not on the beach, they're checking in with their employees all the time and trying to figure out, "Okay, how do we make this work good? How do we make it better? How's it doing? How's this person doing?" All that kind of stuff. And I think there's some differences between being a human manager and being a model manager, but fundamentally it still requires a lot of time and attention. And I think that we miss that in the model discourse. And one of the reasons is, benchmarks make it look like AI is more autonomous than it is. And by autonomy, I mean something specific by autonomy, and I'm going to try to express this. It's a little hard to express, but I learned this for myself because I've been feeling this paradox a little bit. I've been feeling that we have so much automation, so much AI, and I also work way more. And I think part of the paradox started to resolve for me a little bit when I made my own benchmark. So I made this senior, it's called the senior engineer benchmark, and it's like how good is AI versus a human engineer? And the way that I built it is again, have this app, Proof. I just vibe coded it on the side while running the rest of Every. And when we launched it, because it was completely vibe coded, it just started going down and I couldn't fix it and it was very embarrassing. I had a lot of egg on my face. And the product worked, we tested it internally. We had a lot of beta testers, but the day after launch it was like just every 10 minutes the servers would go down and people were looking at me and I'd be like, "I don't know what's going on. Codex, fix it." And Codex is like, "I don't know what's going on." Or really Codex was like, "I do know what's going on, I fixed it." And then it would cause four other errors and then you're just going around in a circle, and I wasn't sleeping, and I vibe coded so hard I got bursitis on my elbow.
00:41:39 · Dan Shipper, Lenny Rachitsky
So, there's a life lesson in there. [00:41:42] S0: Vibe coder elbow.
00:41:46 · Dan Shipper
[00:41:46] S1: So anyway, I got actually two different senior engineers to fix it independently. So, I have two different rewrites of the code base that tells me how they did it. And so, what I get to do is when we get new models, I just give the new model a prompt. I say like, "This is vibe coded slop. If you wanted to rewrite it from first principles, how would you rewrite it? Go do it." And all the models until GPT 5.5 got like a 30 out of 100, and a human senior engineer gets like high 80s, low 90s out of 100. So, there's a lot to go. And then I tried GPT 5.5 and it got like a 62. And mind you, the 60 score was GPT 5.5 using an Opus 4.7 plan. Opus 4.7 plans are very good, GPT 5.5 is the only model though that has the sense of agency and confidence to just like rip out old code and just actually rewrite from first principles. Other coding models, they end up papering over the edges or around the edges, and they're like, "Oh, this is a big job. I'll just do a little patch." And you're like, "No, I specifically told you not to." So GPT 5.5, there's like a 30 point bump in the score, 60 out of 100. It's very clear that in a year or less it's going to be senior engineer level, and that gives you a certain picture in your mind, especially based on how I named the benchmark, which I think a lot of benchmarks do. And I can tell you that when we get to that point, it will be very easy for me to change the benchmark to zero out the current model, so that gets a zero out of 100. And so for example, it seems like there's no skill or no thought into the prompt, which is this is vibe coder slop. Fix it from first principles. But actually it took me a while to get to a prompt that didn't give away the answer, but got the model to reveal what it's capable of. And the original prompt I gave it was the original prompt that I gave it when I was trying to fix the issue and production was going down, which is like I had woken up in the morning and I was like, "Okay, we had four or five reported issues yesterday. I want you to go through all the issues and then make a plan for how to resolve all of them, and go do it." And every coding model on the market, and I'm pretty sure this ... Here's a prediction, I'm pretty sure every coding model on the market will still do this in a year. Every coding model in the market will take that instruction seriously.
00:44:23 · Dan Shipper
And if I tell it, "Here's a bunch of issues, go fix it," they will just go try to fix the issues. What an actual human senior engineer does is they go look at the code base and they're like, "This is a piece of shit. This guy doesn't know what he's doing." And then they say, "We're going to have to actually rewrite a lot of this and it's going to be hard and risky. I know you all want to hear that, but we're going to have to do that." And if you asked the model, "Hey, should we do that?" It'll probably get there, but it's not going to do it on its own. And there's a lot of incentives pushing against it doing that. And even if it does that, there's always a higher frame for us to go. And so, I think it's really important when we think about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that it can't be scored until you write it down, but the act of thinking to prompt it or write it down is something that you can't measure but means that even if the benchmarks get saturated, it doesn't mean the same thing as you totally replace all senior engineers. And I think it's why even though the models are getting better at automation, I still hire engineers.
00:45:52 · Lenny Rachitsky, Dan Shipper
[00:45:52] S0: I am so excited to tell you about this season's supporting sponsor, Vanta. Vanta helps over 15,000 companies like Cursor, Ramp, Duolingo, Snowflake, and Atlassian earn and prove trust with their customers. Teams are building and shipping products faster than ever thanks to AI, but as a result, the amount of risk being introduced into your product and your business is higher than it's ever been. Every security leader that I talk to is feeling the increasing weight of protecting their organization, their business, and not to mention their customer data. Because things are moving so fast, they are constantly reacting, having to guess at priorities, and having to make do with outdated solutions. Vanta automates compliance and risk management with over 35 security and privacy frameworks, including SOC 2, ISO 27001, and HIPAA. This helps companies get compliant fast and stay compliant. More than ever before, trust has the power to make or break your business. Learn more at vanta.com/lenny. And as a listener of this podcast, you get $1,000 off Vanta. That's vanta.com/ lenny. One thing I mentioned recently on the podcast, I heard that speaking of the code that you have of humans writing code, data labeling companies are buying code that was written before 2021, 2022, before AI became a thing is like very valuable data [inaudible 00:47:16] [00:47:15] S1: Artisanal human code. [00:47:17] S0: Yeah, exactly. That's exactly right. And it's so interesting that that's exactly the kind of code used to build this model.
00:47:17 · Lenny Rachitsky, Dan Shipper
[00:47:17] S0: Yeah, exactly. That's exactly right. And it's so interesting that that's exactly the kind of code used to build this model. [00:47:23] S1: Well, what's interesting, so I want to clarify there. So, I did not have a human write the code all by hand, because I actually think that it feels silly to me. I don't really care, because I know if an engineer's not using AI I'm not going to work with them. I don't really care. It's like am I going to race a human against a car? I probably wouldn't do that. But I would race a human in a car versus another human in a car and say, "Which one's better?" And in this case, the way the benchmark is structured is, yeah, these human engineers used AI, but they used it in a way that I could not because I didn't understand it and I didn't have time- They used it in a way that I could not because I didn't understand it and I didn't have time and I didn't really want to go in and try to understand the code base, to be honest. And I think that's a really important thing when we think about benchmarks is AI is a broadly distributed technology that any human can use. And when we are benchmarking against humans, AI against humans, we're actually really always talking about one human using AI versus another human using AI 'cause AI doesn't use itself. It may be able to in this slightly somewhat recursive way, but in any real use case, there's always a human pretty close to it making sure that it's working.