Lenny's Podcast: Product | Career | Growth
00:30:28 · Andrew Ambrosino, S1
[00:30:28] S0: Yeah. [00:30:28] S1: Just shepherding throughout, this obsession with making it awesome. This kind of high agency, high taste person exactly as you described. Is that kind of the way you think about here's who we're hiring, here's who's going to do really well in this new world? [00:30:40] S0: Yeah, I think that's the core piece right now. And it also speaks to how I sort of see IC versus management, which is that it's not that management is going away. It's not that everyone's an IC, but everyone's kind of both now. If you are an IC, you're not typing code out character by character. You are managing something. You're managing agents. You are managing work that is happening that comes together to do a certain thing. If you're a manager of teams, you're doing the same thing just at a different granularity. I generally look for obviously command over the discipline, but then the taste to say, "Hey, you're going to have unlimited tokens and we can't just be doing slop." You need to be able to determine what signal, what's noise in a world of just infinite content. [00:31:38] S1: Mention planning. At the pace things are moving, it's become very hard to plan roadmaps. I imagine, especially in your world. [00:31:47] S0: People are very frustrated with me all the time on this. Yes. [00:31:50] S1: Because things are just constantly shifting, things are changing. How do you plan on your team? How far ahead are you thinking and what does a plan look like? Is it a spreadsheet? Is it MD file? What's the output of a plan?
00:32:00 · Andrew Ambrosino
[00:32:00] S0: Yeah, I don't think we do anything revolutionary on that. We're not clever about planning. I think the basic gist is, the shorter term something is, the more detail it needs. And then it's not that we don't plan for nine months out. It's that that just has to stay very hazy. Because any amount of precision that you add to a nine-month plan right now is false precision and you're just going to waste time. You can say stuff, but nothing that we planned. I think research is different. So I'm not speaking for research here, but on the applied side when we do product, anything that you could have planned in November may have been true for December, but isn't what happened. So it's hard. It is really hard to do planning. We generally need to know what do we think models are able to do on what timeline? And my last company, I kind of saw this shift where we were starting to use the models to drive features. And the product process fell down. It basically had to be like, "Let's list out all of the things that we think we are interested in doing for the next year or two." Let's prototype all of them, decide which things are ready now, and then just let the others sit and bake. And then every time there's like a new leap in models, let's try that thing again with it swapped out. Because the whole premise of whether features were good or not were based on whether they were smart enough, not the shape of them. So this is a great story about the Codex app. I am very confident that the Codex app that we released in February, if that had been ready in November, it would've absolutely failed in the market. And the only difference was the models between November and February. And I think there's a lot to that, that this product with the exact same shape, I think its outcomes were totally different depending on just a few months of timing. Yeah, I think we have a lot of that. I think sometimes the challenge is you have to be very clear again about what stage of the design process that's in. People still have this muscle memory of like, "Oh, I wrote the code for this thing, therefore we should put it out there." It's like, "No, no, no." That means you have an artifact now that we can test against for into future models. This happens with the in-app browser in the app that we have. We had a kind of a working version.
00:35:19 · Andrew Ambrosino
I mean, go back to Atlas, we had agent working inside of Atlas and that was pretty cool. We had Operator before that in ChatGPT. That didn't work out. Very cool idea. There's some thread that you can draw between Operator, Atlas, Codex, ChatGPT that's fundamentally the same feature. But the re-releasing of it with different intelligence totally changes the outcome here. And so I push people not to be stubborn about like, "No, this isn't working so it's a bad feature." It's like, "No, it might not be ready yet." There's also this aspect of, especially in research, there's always a desire to be the most ambitious. And to say, "Okay, but at the limit the model can just do this." And that just doesn't work on the product side. If you go back to the original Codex release, basically what it was is it said, it was Codex web and it wasn't good for interacting with. It was like, you give the model a task and it's going to go off, do the task, come back to you with it finished. It doesn't sound that radical. The problem is it didn't do the task that well. It wrote code, it was good, but it was like that form factor was too early. And then the Claude Code comes out. Totally local, not hooked up to the cloud, doesn't pretend to be as... It's not as AGI-pilled. It's going to ask you questions, it's going to sit there. You can't just delegate your life to it. That worked way better because that's the point that the models were out there. So we were too AGI-pilled for the moment. And I think about that lesson a lot on this stuff. It used to be that failing in market told you all these things about the shape of the product, about the communication of the product. And now it's like, "No, you might need to release this thing six different times before it works." And the shape might not change at all.
00:37:55 · S1, Andrew Ambrosino
[00:37:55] S1: It's so interesting to hear about all the variables you have to think about building product now. There's the timeline for the models and the research and how smart it gets. There's people's ability to even understand this is how you could build software in the cloud and this is the future. Get people prepared for this new feature and then just what you can build as a team. And I love that Codex example because it comes back to this idea of ambition. And I want to hear if there's anything there for you, of just this thread of just be more ambitious because these models can do so much more than you even imagine. And sometimes it's too ambitious for the market and they're not ready for it. But you think about that at all, just pushing your team to be more ambitious because it's so much easier to just do things that maybe felt crazy hard in the past. [00:38:34] S0: Yeah, this is a core challenge. Once there's a product that exists or a feature that exists, it's really easy for people to find paper cuts and- [00:38:43] S1: Micro-optimize.
00:38:43 · S1, Andrew Ambrosino
[00:38:43] S1: Micro-optimize. [00:38:44] S0: And they should. And people on Twitter like to remind us of this and I thank them for that. People should be focused on the features that exist and make them more reliable and better. But this is why we also have a culture of bottoms up exploration here. Because sometimes in the same way that Codex app came and disrupted ChatGPT in some way, this thing will get disrupted by a future effort. And that's part of the design is that. You can't always as one team be good at both the disruptive piece instead of maintaining a product and its quality piece. At some point, you have to design a process that allows for both. [00:39:18] S1: Kind of zooming out a little bit. If you think about the progression that we've been on of AI impacting how we build product, it's insane how far we've come from. As you said, we used to write all our code by hand, artisanally created human code. To AI writing 100% of our code, to you actually put it this way that now coding is steering the AI. And when you think about what percentage of my code is written by AI, it's almost like how many times did I have to steer it in the right direction as the AU version of coding? And now that there's agents and loops and all these things, what's kind of the latest frontier from what you've seen of how people are building? Is it loops? Is there something else of just the most AI-pilled, AI forward teams, here's how they operate now that people may not be aware of.
00:40:07 · Andrew Ambrosino
[00:40:07] S0: Yeah. I mean, loops are so last week, man. I mean, we talked about this. One of the big questions is always, "Well, how much of the product is AI written?" And it's always hard to answer that question because if you're using the goalposts from last year, it's like, "Well, 100% of our product right now is AI written code." So the question is more like, "Well, okay, fine. Is the code written supervised versus unsupervised?" And that's a totally different thing. I welcome the moving of goalposts because that means we're making product progress. There have been a lot of explorations here around autonomously developed software. A lot of harness engineering stuff, a lot of different explorations on like, "Okay, well, what if you came in overnight and did garbage collection of the code base to clean it up?" One thing that I think all models suffer with right now is just they usually increase complexity. If research is listening at any company, please make the models better at deleting code. But that becomes a problem right now when you try to put development completely on autopilot and it's both on the human side and the code base side. So feature requests. How do you teach a model which features to build, which ones to ignore, which ones to group together and reframe a little bit? How do you teach to model how to build the right abstractions? All of this is getting better. I don't think we're at place yet where we're like, "We're just going to set up a loop that's like improve the app." And listens to Twitter and listens to Slack and listens to email. We're not there yet, but we're trying to make it happen.