Lenny's Podcast: Product | Career | Growth
00:44:23 · Dan Shipper
And if I tell it, "Here's a bunch of issues, go fix it," they will just go try to fix the issues. What an actual human senior engineer does is they go look at the code base and they're like, "This is a piece of shit. This guy doesn't know what he's doing." And then they say, "We're going to have to actually rewrite a lot of this and it's going to be hard and risky. I know you all want to hear that, but we're going to have to do that." And if you asked the model, "Hey, should we do that?" It'll probably get there, but it's not going to do it on its own. And there's a lot of incentives pushing against it doing that. And even if it does that, there's always a higher frame for us to go. And so, I think it's really important when we think about benchmark progress to think about it from that perspective, which is benchmarks rise on problems that we've framed that we can articulate, that we can score. And there's a lot of work that's human work that it can't be scored until you write it down, but the act of thinking to prompt it or write it down is something that you can't measure but means that even if the benchmarks get saturated, it doesn't mean the same thing as you totally replace all senior engineers. And I think it's why even though the models are getting better at automation, I still hire engineers.
00:45:52 · Lenny Rachitsky, Dan Shipper
[00:45:52] S0: I am so excited to tell you about this season's supporting sponsor, Vanta. Vanta helps over 15,000 companies like Cursor, Ramp, Duolingo, Snowflake, and Atlassian earn and prove trust with their customers. Teams are building and shipping products faster than ever thanks to AI, but as a result, the amount of risk being introduced into your product and your business is higher than it's ever been. Every security leader that I talk to is feeling the increasing weight of protecting their organization, their business, and not to mention their customer data. Because things are moving so fast, they are constantly reacting, having to guess at priorities, and having to make do with outdated solutions. Vanta automates compliance and risk management with over 35 security and privacy frameworks, including SOC 2, ISO 27001, and HIPAA. This helps companies get compliant fast and stay compliant. More than ever before, trust has the power to make or break your business. Learn more at vanta.com/lenny. And as a listener of this podcast, you get $1,000 off Vanta. That's vanta.com/ lenny. One thing I mentioned recently on the podcast, I heard that speaking of the code that you have of humans writing code, data labeling companies are buying code that was written before 2021, 2022, before AI became a thing is like very valuable data [inaudible 00:47:16] [00:47:15] S1: Artisanal human code. [00:47:17] S0: Yeah, exactly. That's exactly right. And it's so interesting that that's exactly the kind of code used to build this model.
00:47:17 · Lenny Rachitsky, Dan Shipper
[00:47:17] S0: Yeah, exactly. That's exactly right. And it's so interesting that that's exactly the kind of code used to build this model. [00:47:23] S1: Well, what's interesting, so I want to clarify there. So, I did not have a human write the code all by hand, because I actually think that it feels silly to me. I don't really care, because I know if an engineer's not using AI I'm not going to work with them. I don't really care. It's like am I going to race a human against a car? I probably wouldn't do that. But I would race a human in a car versus another human in a car and say, "Which one's better?" And in this case, the way the benchmark is structured is, yeah, these human engineers used AI, but they used it in a way that I could not because I didn't understand it and I didn't have time- They used it in a way that I could not because I didn't understand it and I didn't have time and I didn't really want to go in and try to understand the code base, to be honest. And I think that's a really important thing when we think about benchmarks is AI is a broadly distributed technology that any human can use. And when we are benchmarking against humans, AI against humans, we're actually really always talking about one human using AI versus another human using AI 'cause AI doesn't use itself. It may be able to in this slightly somewhat recursive way, but in any real use case, there's always a human pretty close to it making sure that it's working.
00:48:37 · Lenny Rachitsky, Dan Shipper
[00:48:37] S0: Okay. I want to try to wrap up our first bucket. There's so much to talk about. I made a little list of things that I think people should do based on your predictions to be successful. We'll talk about this at the end too, but just a few things. One is start using Codex or Claude Code more and more for the work you're doing and especially the browser, use tools inside of it. Two is allow agents to use your products. If you're building a SaaS tool, make it easy for agents to be a user essentially. Three is start thinking about some Slack bot that you can work with, like try out tools. I know Slack has their own Slack bot that I think is really good too and I haven't played with it, but people really like it. So look for, I guess, a tool that could become the AI agent within your company. Buy SaaS stock ASAP, not investment advice. [00:49:32] S1: I think that's totally right. My slight tweak is when you're thinking about building your software for agents, the current model is I'm building a CLI that an agent uses, but they're using it in a sort of like they're being... I delegated to task the agent and the agent's using the CLI. And where I think it's going is you and the agent are using the app together. The agent's probably using the CLI, but you're using the web interface and they both need to be in sync. And that is, I think, a new challenge that's really interesting. [00:50:06] S0: Awesome. Anything else before we get to our next category? [00:50:10] S1: Buy SaaS.
00:50:10 · Dan Shipper, Lenny Rachitsky
[00:50:10] S1: Buy SaaS. [00:50:12] S0: That's the title. Oh, man. Okay. So the second category of predictions is around just the shape of the work that we're going to be doing is going to change. What do you predict?
00:50:25 · Dan Shipper
[00:50:25] S1: There's all this interesting stuff in terms of the shape of work. Once you're in this land where you've got async agents off that you delegate work to, then you've got your Codex Claude Code work surface that starts to happen. So one thing that we see a lot internally, and you also see this in the big model companies, is the number of pull requests that you get is like skyrockets. We have people in consulting or in ops roles or whatever or editors just making pull requests. And A, that's really cool and that's a very different shape of work where you can expect that a higher percentage of your company or your users are going to be doing things that previously only technical users can do. And what that does is it creates all this pressure on the other end for the people who have to deal with all of the new code for how to deal with that. And so I think there's a lot of interesting things that happen with that. So for example, like OpenClaw, I mentioned that earlier. Pete gets thousands of poll requests a day on OpenClaw and then he just spins up 50,000 Codex instances and then sorts through them and then merges a thousand of them. It's really crazy. I actually think that that's going to be more and more common. It brings up a lot of really interesting questions around which poll requests should you merge. And whenever you add capacity in one part of your process, it breaks things. It used to be really hard to build things and now it's very easy. So the point is not, can we build it? It's like, would it make sense with the rest of what we've built and how do we keep a sense of a coherent whole. And also, what do we delete? I think Anthropic does this really well. They delete a lot of stuff from Claude Code to make sure that it's not bloated. So I think there's a lot of that going to happen on one side. There's a lot of non-technical people can do technical work and then technical people are in charge of making sure that that work gets into a product or into a process in a cohesive, coherent way. And also their product people are going to be doing that too. And I think that's quite cool.