Lenny's Podcast: Product | Career | Growth
00:41:39 · Dan Shipper, Lenny Rachitsky
So, there's a life lesson in there. [00:41:42] S0: Vibe coder elbow.
00:41:46 · Dan Shipper
[00:41:46] S1: So anyway, I got actually two different senior engineers to fix it independently. So, I have two different rewrites of the code base that tells me how they did it. And so, what I get to do is when we get new models, I just give the new model a prompt. I say like, "This is vibe coded slop. If you wanted to rewrite it from first principles, how would you rewrite it? Go do it." And all the models until GPT 5.5 got like a 30 out of 100, and a human senior engineer gets like high 80s, low 90s out of 100. So, there's a lot to go. And then I tried GPT 5.5 and it got like a 62. And mind you, the 60 score was GPT 5.5 using an Opus 4.7 plan. Opus 4.7 plans are very good, GPT 5.5 is the only model though that has the sense of agency and confidence to just like rip out old code and just actually rewrite from first principles. Other coding models, they end up papering over the edges or around the edges, and they're like, "Oh, this is a big job. I'll just do a little patch." And you're like, "No, I specifically told you not to." So GPT 5.5, there's like a 30 point bump in the score, 60 out of 100. It's very clear that in a year or less it's going to be senior engineer level, and that gives you a certain picture in your mind, especially based on how I named the benchmark, which I think a lot of benchmarks do. And I can tell you that when we get to that point, it will be very easy for me to change the benchmark to zero out the current model, so that gets a zero out of 100. And so for example, it seems like there's no skill or no thought into the prompt, which is this is vibe coder slop. Fix it from first principles. But actually it took me a while to get to a prompt that didn't give away the answer, but got the model to reveal what it's capable of. And the original prompt I gave it was the original prompt that I gave it when I was trying to fix the issue and production was going down, which is like I had woken up in the morning and I was like, "Okay, we had four or five reported issues yesterday. I want you to go through all the issues and then make a plan for how to resolve all of them, and go do it." And every coding model on the market, and I'm pretty sure this ... Here's a prediction, I'm pretty sure every coding model on the market will still do this in a year. Every coding model in the market will take that instruction seriously.
00:44:23 · Dan Shipper
And if I tell it, "Here's a bunch of issues, go fix it," they will just go try to fix the issues. What an actual human senior engineer does is they go look at the code base and they're like, "This is a piece of shit. This guy doesn't know what he's doing." And then they say, "We're going to have to actually rewrite a lot of this and it's going to be hard and risky. I know you all want to hear that, but we're going to have to do that." And if you asked the model, "Hey, should we do that?" It'll probably get there, but it's not going to do it on its own. And there's a lot of incentives pushing against it doing that. And even if it does that, there's always a higher frame for us to go. And so, I think it's really important when we think about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that it can't be scored until you write it down, but the act of thinking to prompt it or write it down is something that you can't measure but means that even if the benchmarks get saturated, it doesn't mean the same thing as you totally replace all senior engineers. And I think it's why even though the models are getting better at automation, I still hire engineers.
00:45:52 · Lenny Rachitsky, Dan Shipper
[00:45:52] S0: I am so excited to tell you about this season's supporting sponsor, Vanta. Vanta helps over 15,000 companies like Cursor, Ramp, Duolingo, Snowflake, and Atlassian earn and prove trust with their customers. Teams are building and shipping products faster than ever thanks to AI, but as a result, the amount of risk being introduced into your product and your business is higher than it's ever been. Every security leader that I talk to is feeling the increasing weight of protecting their organization, their business, and not to mention their customer data. Because things are moving so fast, they are constantly reacting, having to guess at priorities, and having to make do with outdated solutions. Vanta automates compliance and risk management with over 35 security and privacy frameworks, including SOC 2, ISO 27001, and HIPAA. This helps companies get compliant fast and stay compliant. More than ever before, trust has the power to make or break your business. Learn more at vanta.com/lenny. And as a listener of this podcast, you get $1,000 off Vanta. That's vanta.com/ lenny. One thing I mentioned recently on the podcast, I heard that speaking of the code that you have of humans writing code, data labeling companies are buying code that was written before 2021, 2022, before AI became a thing is like very valuable data [inaudible 00:47:16] [00:47:15] S1: Artisanal human code. [00:47:17] S0: Yeah, exactly. That's exactly right. And it's so interesting that that's exactly the kind of code used to build this model.
00:47:17 · Lenny Rachitsky, Dan Shipper
[00:47:17] S0: Yeah, exactly. That's exactly right. And it's so interesting that that's exactly the kind of code used to build this model. [00:47:23] S1: Well, what's interesting, so I want to clarify there. So, I did not have a human write the code all by hand, because I actually think that it feels silly to me. I don't really care, because I know if an engineer's not using AI I'm not going to work with them. I don't really care. It's like am I going to race a human against a car? I probably wouldn't do that. But I would race a human in a car versus another human in a car and say, "Which one's better?" And in this case, the way the benchmark is structured is, yeah, these human engineers used AI, but they used it in a way that I could not because I didn't understand it and I didn't have time- They used it in a way that I could not because I didn't understand it and I didn't have time and I didn't really want to go in and try to understand the code base, to be honest. And I think that's a really important thing when we think about benchmarks is AI is a broadly distributed technology that any human can use. And when we are benchmarking against humans, AI against humans, we're actually really always talking about one human using AI versus another human using AI 'cause AI doesn't use itself. It may be able to in this slightly somewhat recursive way, but in any real use case, there's always a human pretty close to it making sure that it's working.
00:48:37 · Lenny Rachitsky, Dan Shipper
[00:48:37] S0: Okay. I want to try to wrap up our first bucket. There's so much to talk about. I made a little list of things that I think people should do based on your predictions to be successful. We'll talk about this at the end too, but just a few things. One is start using Codex or Claude Code more and more for the work you're doing and especially the browser, use tools inside of it. Two is allow agents to use your products. If you're building a SaaS tool, make it easy for agents to be a user essentially. Three is start thinking about some Slack bot that you can work with, like try out tools. I know Slack has their own Slack bot that I think is really good too and I haven't played with it, but people really like it. So look for, I guess, a tool that could become the AI agent within your company. Buy SaaS stock ASAP, not investment advice. [00:49:32] S1: I think that's totally right. My slight tweak is when you're thinking about building your software for agents, the current model is I'm building a CLI that an agent uses, but they're using it in a sort of like they're being... I delegated to task the agent and the agent's using the CLI. And where I think it's going is you and the agent are using the app together. The agent's probably using the CLI, but you're using the web interface and they both need to be in sync. And that is, I think, a new challenge that's really interesting. [00:50:06] S0: Awesome. Anything else before we get to our next category? [00:50:10] S1: Buy SaaS.