Lenny's Podcast: Product | Career | Growth
00:37:54 · Fiona Fung, Lenny Rachitsky
[00:37:54] S1: And then it's even like, then giving even more autonomy at some point of, you know what, go for it. For example, a verification- [00:38:01] S0: [inaudible 00:38:01]. [00:38:00] S1: Is really good. [00:38:01] S0: Yeah. [00:38:02] S1: Go for it. [00:38:03] S0: I see. So you give it a little bit of freedom to do more of this? [00:38:05] S1: Yeah. [00:38:06] S0: Okay. That is so interesting. It reminds me, speaking on this idea of go for it, like I wasn't planning to talk about this, but I just watched Tyler Cowen had this awesome talk about just like what's happening. I don't know if you know Tyler Cowen, this really smart dude, economist, has a podcast. There's so much talk of the people that are doing best in this world. His term is initiative. They have initiative. Another term people use, agency. Is that something just like, people hear this a lot, just like the people that seem to be doing best are taking the most initiative, getting the most proactive, have the most agency. Does that kind of spark any thoughts to you, just what people may need to, I don't know, think about or improve on?
00:38:06 · Lenny Rachitsky, Fiona Fung
[00:38:06] S0: Okay. That is so interesting. It reminds me, speaking on this idea of go for it, like I wasn't planning to talk about this, but I just watched Tyler Cowen had this awesome talk about just like what's happening. I don't know if you know Tyler Cowen, this really smart dude, economist, has a podcast. There's so much talk of the people that are doing best in this world. His term is initiative. They have initiative. Another term people use, agency. Is that something just like, people hear this a lot, just like the people that seem to be doing best are taking the most initiative, getting the most proactive, have the most agency. Does that kind of spark any thoughts to you, just what people may need to, I don't know, think about or improve on? [00:38:47] S1: Actually, I love that word: agency. That's what we really hold important on the Claude Code and Cowork team. But it's interesting, I say along with high, because we really, it's about like, "Hey, here's a problem." And then it's really everybody on the team has ideas for how to address the problem. So it's really high agency. And then we say with high agency, it's also high accountability. So it's all about making sure folks have that freedom to cook, but then it's also like, okay, what's the accountability for it as well? And again, it goes back to what's a hypothesis of what you're trying to solve. So I think, yeah, the balance like are almost like two sides are the same kind of agency and then accountability I think has served our team really well.
00:39:29 · Lenny Rachitsky
[00:39:29] S0: Such an important element of agency. Okay, cool. Everyone's off doing stuff, but okay, what have you busted it up? What have you done? So kind of along those lines, it feels like there's this vibe shift recently from token maxing, just go crazy, spend as much as possible, just see what's possible to like, wait, what are we actually getting out of this? How much is it costing? What's the ROI? Interestingly, Boris was actually head of eng productivity at Meta. That was his job, is measuring eng productivity. And then also just this tweet of lines per code, like code shift. There's this interesting discussion of just how do you actually measure productivity increase and ROI of AI tools in spend? You guys have the unfair advantage of getting free tokens working in Anthropic. With that in mind, what have you learned about just how to measure say ROI of engineers in today's world?
00:40:27 · Fiona Fung
[00:40:27] S1: This eng productivity is such a fascinating topic. Yeah, because I remember even Boris mentioned like, "We'll first start with lines of code." And then I remembered once then there was a debate of, "Well, lines of code, this engineer had this crazy lines of code, but they just took some library and then just was porting it." So they checked it in and then I was like, "Well, maybe it's significant lines of code." And then it's, "Well, but what if we're updating our frameworks and now we're generating less code, but the output's still the same?" So now I was like, "Okay, maybe it's time to land PR." But it's very interesting of it's always been whichever metric, like if you're really focused on the out... My advice here is one is output, like is the output really going towards the outcome? Because yeah, the token maxing, it's almost like the lines that we code that we used to have, but I'm really much more about a, what is it that we're trying to do? All of this... There was another saying I really like, "Don't forsake motion for progress." Because if you're measuring tool user usage, then you're measuring the action, but is it really making whatever the end outcome of yours important? And so I really try to zoom out and focus on what is the problem we're trying to solve, what's a good way to measure that. And then that's what we focus mostly on versus the productivity measurement. I would say though, outside of metrics, especially on teams as you're having more people adopt AI tooling, I would say definitely go on a listening tour for the leaders listening to a podcast. Especially, I love to focus on the senior engineers as well, like hear from them on what's working, what's not, how can we actually make it better because they will also help you multiply and scale that up across a full engineering team. And sometimes it's those conversations where you might spark an idea that comes up and also really good shared learning versus metric dashboards. It's very interesting. I've learned some good lessons from metrics. I'm all about, like it's really great when you have a metric that you can actually help climb on, but always keep in mind of is, kind of my whole growth mindset of, is it still serving you? Always keep in mind like, is that metric really still serving the outcome that you were aiming for?
00:42:42 · Fiona Fung
A fun example I have of this is in the early Facebook Marketplace days, we were kind of launching by region and we really wanted to make sure we're building a delightful product before we expand. And I remember in the early days, one thing we would keep an eye on is number of sellers. And I remembered after launching to our first region, I'm like, "Oh, in this area the number of sellers is low, but actually people are finding items that they're looking for," which is what we're aiming for or like helping people find items that they need. And then I realized in that region it wasn't a large number of sellers, but there were power sellers. But our first gate before we expand would've just been factoring heavily number of sellers. And I remember that quick conversation of, hey, and this goes back to that whole people will use things in ways you may not expect and shift, iterate, learn. And so then we updated the metric to go, oh, it's not a number of seller because it didn't factor in power sellers. And so that's the advice I have too. Whatever metric, whether for productivity or even for product, always keep an eye and make sure that you're not just having blinders on that's blindly following a metric that used to make sense because sometimes the landscape can change so fast, even the metrics themselves might need to be adjusted.
00:44:01 · Lenny Rachitsky, Fiona Fung
[00:44:01] S0: And this comes back to your process that you shared of having Claude watch all the PRs shipping per engineer, let's say, and then not focusing on the metrics, but focusing on that leading to a conversation about what impact does this have, what was maybe some bugs that happened and that being a really powerful way of understanding what's going on with this engineer. Awesome. [00:44:24] S1: That's right. [00:44:25] S0: Let me come back to this, just this question of speed and quality and just impact. Is there anything else you've learned about just how to balance those things, just like this crazy velocity of code that's being generated and just staying on top of quality and impact. Is there anything else there that might be helpful to other teams that are trying to wrangle all these PRs being shipped every day?