Lenny's Podcast: Product | Career | Growth
00:36:37 · Fiona Fung, Lenny Rachitsky
[00:36:37] S1: To say hey. That's right. But then it's able to actually go and kick off agents on your behalf because even before it would be, oh, you could, yeah, do a chronic job to automate, but now it's, "Hey, look at these feedback." And then if you're seeing some of these bugs, that what are some polish fixes that you might be able to knock out? And then it would go kickoff and then I wake up and I end up having PRs that I could review versus before, which is still more of AI can have different agents. And then I'm still thinking about, okay, now what do I do with this information? So it's like that higher level abstraction of, okay, now how can I actually write a routine that also basically does prompts for me for spawning different agents? So I think we'll move more towards that kind of asynchronous style of working. [00:37:23] S0: So interesting. So in a sense as a manager, you have these kind of things you do daily and what you're saying here is you can set up these prompts that kind of check in every day on the things you would be doing. Like how are the projects going? What's falling behind? What should I unblock? Who's struggling? And how do I improve some polish? And so the idea here, what you're describing here is write these things that you almost do as a manager daily and have Claude essentially do them for you and then show you, here's what I'm doing, here's what there is to review. [00:37:54] S1: And then it's even like, then giving even more autonomy at some point of, you know what, go for it. For example, a verification-
00:37:54 · Fiona Fung, Lenny Rachitsky
[00:37:54] S1: And then it's even like, then giving even more autonomy at some point of, you know what, go for it. For example, a verification- [00:38:01] S0: [inaudible 00:38:01]. [00:38:00] S1: Is really good. [00:38:01] S0: Yeah. [00:38:02] S1: Go for it. [00:38:03] S0: I see. So you give it a little bit of freedom to do more of this? [00:38:05] S1: Yeah. [00:38:06] S0: Okay. That is so interesting. It reminds me, speaking on this idea of go for it, like I wasn't planning to talk about this, but I just watched Tyler Cowen had this awesome talk about just like what's happening. I don't know if you know Tyler Cowen, this really smart dude, economist, has a podcast. There's so much talk of the people that are doing best in this world. His term is initiative. They have initiative. Another term people use, agency. Is that something just like, people hear this a lot, just like the people that seem to be doing best are taking the most initiative, getting the most proactive, have the most agency. Does that kind of spark any thoughts to you, just what people may need to, I don't know, think about or improve on?
00:38:06 · Lenny Rachitsky, Fiona Fung
[00:38:06] S0: Okay. That is so interesting. It reminds me, speaking on this idea of go for it, like I wasn't planning to talk about this, but I just watched Tyler Cowen had this awesome talk about just like what's happening. I don't know if you know Tyler Cowen, this really smart dude, economist, has a podcast. There's so much talk of the people that are doing best in this world. His term is initiative. They have initiative. Another term people use, agency. Is that something just like, people hear this a lot, just like the people that seem to be doing best are taking the most initiative, getting the most proactive, have the most agency. Does that kind of spark any thoughts to you, just what people may need to, I don't know, think about or improve on? [00:38:47] S1: Actually, I love that word: agency. That's what we really hold important on the Claude Code and Cowork team. But it's interesting, I say along with high, because we really, it's about like, "Hey, here's a problem." And then it's really everybody on the team has ideas for how to address the problem. So it's really high agency. And then we say with high agency, it's also high accountability. So it's all about making sure folks have that freedom to cook, but then it's also like, okay, what's the accountability for it as well? And again, it goes back to what's a hypothesis of what you're trying to solve. So I think, yeah, the balance like are almost like two sides are the same kind of agency and then accountability I think has served our team really well.
00:39:29 · Lenny Rachitsky
[00:39:29] S0: Such an important element of agency. Okay, cool. Everyone's off doing stuff, but okay, what have you busted it up? What have you done? So kind of along those lines, it feels like there's this vibe shift recently from token maxing, just go crazy, spend as much as possible, just see what's possible to like, wait, what are we actually getting out of this? How much is it costing? What's the ROI? Interestingly, Boris was actually head of eng productivity at Meta. That was his job, is measuring eng productivity. And then also just this tweet of lines per code, like code shift. There's this interesting discussion of just how do you actually measure productivity increase and ROI of AI tools in spend? You guys have the unfair advantage of getting free tokens working in Anthropic. With that in mind, what have you learned about just how to measure say ROI of engineers in today's world?
00:40:27 · Fiona Fung
[00:40:27] S1: This eng productivity is such a fascinating topic. Yeah, because I remember even Boris mentioned like, "We'll first start with lines of code." And then I remembered once then there was a debate of, "Well, lines of code, this engineer had this crazy lines of code, but they just took some library and then just was porting it." So they checked it in and then I was like, "Well, maybe it's significant lines of code." And then it's, "Well, but what if we're updating our frameworks and now we're generating less code, but the output's still the same?" So now I was like, "Okay, maybe it's time to land PR." But it's very interesting of it's always been whichever metric, like if you're really focused on the out... My advice here is one is output, like is the output really going towards the outcome? Because yeah, the token maxing, it's almost like the lines that we code that we used to have, but I'm really much more about a, what is it that we're trying to do? All of this... There was another saying I really like, "Don't forsake motion for progress." Because if you're measuring tool user usage, then you're measuring the action, but is it really making whatever the end outcome of yours important? And so I really try to zoom out and focus on what is the problem we're trying to solve, what's a good way to measure that. And then that's what we focus mostly on versus the productivity measurement. I would say though, outside of metrics, especially on teams as you're having more people adopt AI tooling, I would say definitely go on a listening tour for the leaders listening to a podcast. Especially, I love to focus on the senior engineers as well, like hear from them on what's working, what's not, how can we actually make it better because they will also help you multiply and scale that up across a full engineering team. And sometimes it's those conversations where you might spark an idea that comes up and also really good shared learning versus metric dashboards. It's very interesting. I've learned some good lessons from metrics. I'm all about, like it's really great when you have a metric that you can actually help climb on, but always keep in mind of is, kind of my whole growth mindset of, is it still serving you? Always keep in mind like, is that metric really still serving the outcome that you were aiming for?
00:42:42 · Fiona Fung
A fun example I have of this is in the early Facebook Marketplace days, we were kind of launching by region and we really wanted to make sure we're building a delightful product before we expand. And I remember in the early days, one thing we would keep an eye on is number of sellers. And I remembered after launching to our first region, I'm like, "Oh, in this area the number of sellers is low, but actually people are finding items that they're looking for," which is what we're aiming for or like helping people find items that they need. And then I realized in that region it wasn't a large number of sellers, but there were power sellers. But our first gate before we expand would've just been factoring heavily number of sellers. And I remember that quick conversation of, hey, and this goes back to that whole people will use things in ways you may not expect and shift, iterate, learn. And so then we updated the metric to go, oh, it's not a number of seller because it didn't factor in power sellers. And so that's the advice I have too. Whatever metric, whether for productivity or even for product, always keep an eye and make sure that you're not just having blinders on that's blindly following a metric that used to make sense because sometimes the landscape can change so fast, even the metrics themselves might need to be adjusted.