Lenny's Podcast: Product | Career | Growth
00:36:41 · Dan Shipper
[00:36:41] S1: Well, I think it actually may save their margins, because right now all these companies are rushing to add a agent to their offering and thinking, "Oh, the agent is going to be the main way that people interact with me." And that cost tokens, obviously. And I actually think once I have Codex or Cowork as my main work surface, I still want to use SaaS. So, this is another good prediction. I would buy SaaS stocks right now. I think the SaaS apocalypse is dumb, and SaaS stocks will be up majorly in the next couple years. Not investment advice, but I would buy SaaS stocks. So, I think it saves your margin because now the way that you're thinking then is not I have to build AI into this, it's more like I have to make a piece of software that humans and AI want to collaborate on together. And that's hard, but once you build it, it's a lot cheaper than assuming everyone's spending tokens, and I think it's a good business. And part of the reason I'm so bullish on SaaS is A, everybody internally here is, like I said, we have all got agents and we're all using Codex and whatever, and we still pay for a ton of SaaS and our SaaS spend is up year over year, and we're not vibe coding every single little thing. And I think that what agents do is increase the number of users of SaaS, not get rid of it. And so, I think SaaS companies are going to see an insane spike in the amount of demand that they have, because there's going to be tons of agents using these products at a very high volume. And like I said, that's a huge infrastructure challenge. There's a lot of interesting pricing challenges, but it makes me very bullish on SaaS.
00:38:47 · Lenny Rachitsky, Dan Shipper
[00:38:47] S0: I love that. If anything else comes out of this conversation, Dan Shipper, SaaS is the future of AI. B2B SaaS. [00:38:56] S1: Hashtag send tweet. [00:38:59] S0: Yeah, this is quite contrarian. And the other interesting piece is that the fact that you guys are hiring, that you doubled in people in the past year, which is not what people would have expected from a company that is so AI forward. Talk about your experience there of just, okay, we still actually need humans.
00:39:15 · Dan Shipper
[00:39:15] S1: Automation is a lie, in the sense that every time you automate something, in order to make sure the automation is working well you need a human on top of it making sure that it's working well. And so, I wrote this piece a couple years ago about the allocation economy, the idea that the way that humans are going to work with AI is going to be like being a manager. And the thing that you have to remember about managers is like managers actually spend a lot of time working. Most managers are not on the beach, they're checking in with their employees all the time and trying to figure out, "Okay, how do we make this work good? How do we make it better? How's it doing? How's this person doing?" All that kind of stuff. And I think there's some differences between being a human manager and being a model manager, but fundamentally it still requires a lot of time and attention. And I think that we miss that in the model discourse. And one of the reasons is, benchmarks make it look like AI is more autonomous than it is. And by autonomy, I mean something specific by autonomy, and I'm going to try to express this. It's a little hard to express, but I learned this for myself because I've been feeling this paradox a little bit. I've been feeling that we have so much automation, so much AI, and I also work way more. And I think part of the paradox started to resolve for me a little bit when I made my own benchmark. So I made this senior, it's called the senior engineer benchmark, and it's like how good is AI versus a human engineer? And the way that I built it is again, have this app, Proof. I just vibe coded it on the side while running the rest of Every. And when we launched it, because it was completely vibe coded, it just started going down and I couldn't fix it and it was very embarrassing. I had a lot of egg on my face. And the product worked, we tested it internally. We had a lot of beta testers, but the day after launch it was like just every 10 minutes the servers would go down and people were looking at me and I'd be like, "I don't know what's going on. Codex, fix it." And Codex is like, "I don't know what's going on." Or really Codex was like, "I do know what's going on, I fixed it." And then it would cause four other errors and then you're just going around in a circle, and I wasn't sleeping, and I vibe coded so hard I got bursitis on my elbow.
00:41:39 · Dan Shipper, Lenny Rachitsky
So, there's a life lesson in there. [00:41:42] S0: Vibe coder elbow.
00:41:46 · Dan Shipper
[00:41:46] S1: So anyway, I got actually two different senior engineers to fix it independently. So, I have two different rewrites of the code base that tells me how they did it. And so, what I get to do is when we get new models, I just give the new model a prompt. I say like, "This is vibe coded slop. If you wanted to rewrite it from first principles, how would you rewrite it? Go do it." And all the models until GPT 5.5 got like a 30 out of 100, and a human senior engineer gets like high 80s, low 90s out of 100. So, there's a lot to go. And then I tried GPT 5.5 and it got like a 62. And mind you, the 60 score was GPT 5.5 using an Opus 4.7 plan. Opus 4.7 plans are very good, GPT 5.5 is the only model though that has the sense of agency and confidence to just like rip out old code and just actually rewrite from first principles. Other coding models, they end up papering over the edges or around the edges, and they're like, "Oh, this is a big job. I'll just do a little patch." And you're like, "No, I specifically told you not to." So GPT 5.5, there's like a 30 point bump in the score, 60 out of 100. It's very clear that in a year or less it's going to be senior engineer level, and that gives you a certain picture in your mind, especially based on how I named the benchmark, which I think a lot of benchmarks do. And I can tell you that when we get to that point, it will be very easy for me to change the benchmark to zero out the current model, so that gets a zero out of 100. And so for example, it seems like there's no skill or no thought into the prompt, which is this is vibe coder slop. Fix it from first principles. But actually it took me a while to get to a prompt that didn't give away the answer, but got the model to reveal what it's capable of. And the original prompt I gave it was the original prompt that I gave it when I was trying to fix the issue and production was going down, which is like I had woken up in the morning and I was like, "Okay, we had four or five reported issues yesterday. I want you to go through all the issues and then make a plan for how to resolve all of them, and go do it." And every coding model on the market, and I'm pretty sure this ... Here's a prediction, I'm pretty sure every coding model on the market will still do this in a year. Every coding model in the market will take that instruction seriously.
00:44:23 · Dan Shipper
And if I tell it, "Here's a bunch of issues, go fix it," they will just go try to fix the issues. What an actual human senior engineer does is they go look at the code base and they're like, "This is a piece of shit. This guy doesn't know what he's doing." And then they say, "We're going to have to actually rewrite a lot of this and it's going to be hard and risky. I know you all want to hear that, but we're going to have to do that." And if you asked the model, "Hey, should we do that?" It'll probably get there, but it's not going to do it on its own. And there's a lot of incentives pushing against it doing that. And even if it does that, there's always a higher frame for us to go. And so, I think it's really important when we think about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that it can't be scored until you write it down, but the act of thinking to prompt it or write it down is something that you can't measure but means that even if the benchmarks get saturated, it doesn't mean the same thing as you totally replace all senior engineers. And I think it's why even though the models are getting better at automation, I still hire engineers.