Lenny's Podcast: Product | Career | Growth
00:35:19 · Andrew Ambrosino
I mean, go back to Atlas, we had agent working inside of Atlas and that was pretty cool. We had Operator before that in ChatGPT. That didn't work out. Very cool idea. There's some thread that you can draw between Operator, Atlas, Codex, ChatGPT that's fundamentally the same feature. But the re-releasing of it with different intelligence totally changes the outcome here. And so I push people not to be stubborn about like, "No, this isn't working so it's a bad feature." It's like, "No, it might not be ready yet." There's also this aspect of, especially in research, there's always a desire to be the most ambitious. And to say, "Okay, but at the limit the model can just do this." And that just doesn't work on the product side. If you go back to the original Codex release, basically what it was is it said, it was Codex web and it wasn't good for interacting with. It was like, you give the model a task and it's going to go off, do the task, come back to you with it finished. It doesn't sound that radical. The problem is it didn't do the task that well. It wrote code, it was good, but it was like that form factor was too early. And then the Claude Code comes out. Totally local, not hooked up to the cloud, doesn't pretend to be as... It's not as AGI-pilled. It's going to ask you questions, it's going to sit there. You can't just delegate your life to it. That worked way better because that's the point that the models were out there. So we were too AGI-pilled for the moment. And I think about that lesson a lot on this stuff. It used to be that failing in market told you all these things about the shape of the product, about the communication of the product. And now it's like, "No, you might need to release this thing six different times before it works." And the shape might not change at all.
00:37:55 · S1, Andrew Ambrosino
[00:37:55] S1: It's so interesting to hear about all the variables you have to think about building product now. There's the timeline for the models and the research and how smart it gets. There's people's ability to even understand this is how you could build software in the cloud and this is the future. Get people prepared for this new feature and then just what you can build as a team. And I love that Codex example because it comes back to this idea of ambition. And I want to hear if there's anything there for you, of just this thread of just be more ambitious because these models can do so much more than you even imagine. And sometimes it's too ambitious for the market and they're not ready for it. But you think about that at all, just pushing your team to be more ambitious because it's so much easier to just do things that maybe felt crazy hard in the past. [00:38:34] S0: Yeah, this is a core challenge. Once there's a product that exists or a feature that exists, it's really easy for people to find paper cuts and- [00:38:43] S1: Micro-optimize.
00:38:43 · S1, Andrew Ambrosino
[00:38:43] S1: Micro-optimize. [00:38:44] S0: And they should. And people on Twitter like to remind us of this and I thank them for that. People should be focused on the features that exist and make them more reliable and better. But this is why we also have a culture of bottoms up exploration here. Because sometimes in the same way that Codex app came and disrupted ChatGPT in some way, this thing will get disrupted by a future effort. And that's part of the design is that. You can't always as one team be good at both the disruptive piece instead of maintaining a product and its quality piece. At some point, you have to design a process that allows for both. [00:39:18] S1: Kind of zooming out a little bit. If you think about the progression that we've been on of AI impacting how we build product, it's insane how far we've come from. As you said, we used to write all our code by hand, artisanally created human code. To AI writing 100% of our code, to you actually put it this way that now coding is steering the AI. And when you think about what percentage of my code is written by AI, it's almost like how many times did I have to steer it in the right direction as the AU version of coding? And now that there's agents and loops and all these things, what's kind of the latest frontier from what you've seen of how people are building? Is it loops? Is there something else of just the most AI-pilled, AI forward teams, here's how they operate now that people may not be aware of.
00:40:07 · Andrew Ambrosino
[00:40:07] S0: Yeah. I mean, loops are so last week, man. I mean, we talked about this. One of the big questions is always, "Well, how much of the product is AI written?" And it's always hard to answer that question because if you're using the goalposts from last year, it's like, "Well, 100% of our product right now is AI written code." So the question is more like, "Well, okay, fine. Is the code written supervised versus unsupervised?" And that's a totally different thing. I welcome the moving of goalposts because that means we're making product progress. There have been a lot of explorations here around autonomously developed software. A lot of harness engineering stuff, a lot of different explorations on like, "Okay, well, what if you came in overnight and did garbage collection of the code base to clean it up?" One thing that I think all models suffer with right now is just they usually increase complexity. If research is listening at any company, please make the models better at deleting code. But that becomes a problem right now when you try to put development completely on autopilot and it's both on the human side and the code base side. So feature requests. How do you teach a model which features to build, which ones to ignore, which ones to group together and reframe a little bit? How do you teach to model how to build the right abstractions? All of this is getting better. I don't think we're at place yet where we're like, "We're just going to set up a loop that's like improve the app." And listens to Twitter and listens to Slack and listens to email. We're not there yet, but we're trying to make it happen.
00:41:49 · S1, Andrew Ambrosino
[00:41:49] S1: Do you think we'll get there? Do you think we'll get to a place where it's just like grow, win? [00:41:54] S0: Slash goal, make money, make me a billion dollars. [00:41:58] S1: Win the market. [00:41:58] S0: I don't know, man. I am not in the business of saying never or always or whatever. [00:42:05] S1: Yeah. How are you using AI in your work as product leader, eng leader? [00:42:10] S0: Yeah. [00:42:11] S1: What are some ways that you use it that maybe people may not be aware they can use AI for?
00:42:15 · Andrew Ambrosino
[00:42:15] S0: Yeah, I think I have the best job in the world right now. But one of the things that makes it very fun is that when we were developing the original Codex app, the goal for me personally was to make it the thing that I wrote the code with. I was like, "I need to make this so good at development that I can build the Codex app with this." And the Codex app at that time was a development tool. And we did that super quick dog fooding loop because you've got your personal dog fooding loop. Where you're like, "Oh, I can't do this thing. I should fix that so that I can do the thing. Now I can do the thing, now I can do more things." We released that. And then the next challenge was, "Hey, people are starting to do some different shaped things with this. And now I need to grow this and so I need to hire a few people and help." So then my role changed at the same time that the role of the app needed to change. I'm like, "Okay, I need to do more product discovery here. I need to figure out the right loops for seeing what everybody's working on and steering things that are off track." And so all of a sudden that's what I started using the Codex app for. I did still write code. I've tried to align my own usage of it with the problem that we're trying to solve. And now I'm like, "I need to build a spreadsheet that models this out." I need to do internal deep research on all of the efforts that have gone into this area of research for the next version of this. There was a release or a series of releases in May-ish that introduced the in-app browser, computer use, and artifact creation to the Codex app. That was, I think our Codex is for almost everyone release. And everybody knows the term vibe coding. I think that was our first vibe coordinated release. Where I had a Notion Doc somewhere with everything that needed to happen. And I was automating, going out to gather updates from pull requests, from Slack channels, and updating the status tracker. And now this is pretty commonplace, but at the time I felt like I was at the bleeding edge of how to manage a product release. In short, the way that I use the Codex app is basically like, what has my job grown into and how do I make this thing able to do everything I need to do? I will get up in the morning, I will see the daily brief that I have from everything from the 3,000 Slack channels that I'm in.