Lenny's Podcast: Product | Career | Growth
00:42:42 · Fiona Fung
A fun example I have of this is in the early Facebook Marketplace days, we were kind of launching by region and we really wanted to make sure we're building a delightful product before we expand. And I remember in the early days, one thing we would keep an eye on is number of sellers. And I remembered after launching to our first region, I'm like, "Oh, in this area the number of sellers is low, but actually people are finding items that they're looking for," which is what we're aiming for or like helping people find items that they need. And then I realized in that region it wasn't a large number of sellers, but there were power sellers. But our first gate before we expand would've just been factoring heavily number of sellers. And I remember that quick conversation of, hey, and this goes back to that whole people will use things in ways you may not expect and shift, iterate, learn. And so then we updated the metric to go, oh, it's not a number of seller because it didn't factor in power sellers. And so that's the advice I have too. Whatever metric, whether for productivity or even for product, always keep an eye and make sure that you're not just having blinders on that's blindly following a metric that used to make sense because sometimes the landscape can change so fast, even the metrics themselves might need to be adjusted.
00:44:01 · Lenny Rachitsky, Fiona Fung
[00:44:01] S0: And this comes back to your process that you shared of having Claude watch all the PRs shipping per engineer, let's say, and then not focusing on the metrics, but focusing on that leading to a conversation about what impact does this have, what was maybe some bugs that happened and that being a really powerful way of understanding what's going on with this engineer. Awesome. [00:44:24] S1: That's right. [00:44:25] S0: Let me come back to this, just this question of speed and quality and just impact. Is there anything else you've learned about just how to balance those things, just like this crazy velocity of code that's being generated and just staying on top of quality and impact. Is there anything else there that might be helpful to other teams that are trying to wrangle all these PRs being shipped every day?
00:44:47 · Fiona Fung, Lenny Rachitsky
[00:44:47] S1: I would say, and this is what honestly we want to keep doing more of and being better at it too. The proactive quality, so especially for quality, making sure that what are the experiences that are key and making sure you actually is, actually speaking of metrics, those are really good things that make sure you can see trends over time. And so on the quality front we found, and this is like the more proactive we can be of making sure we can get an earlier detection into quality. And so that's been one thing that we've been paying a lot of attention to. So I started this, "Hey, let's have a concept of what's bad versus what's sad." And bad is like a very bad irrecoverable error and sad is something that's kind of like a pain point, recoverable, but it's interesting when you stack up sads, it could generally go to bad. But even starting with a high level framework like that and because if not, I think sometimes with dashboards you can have time to load or all these other, but when you're dealing with a lot of different product surfaces, it's harder to go, "Wait, is that a good number or not a good number?" And so one thing that's helped us is versus just raw performance or reliability numbers, also having some framework of what we think is a bad experience and making sure we're focused on addressing those and then also keeping an eye on where we're seeing in terms of the sad. [00:46:12] S0: I like that. [00:46:12] S1: That helped us a little bit.
00:46:12 · Fiona Fung, Lenny Rachitsky
[00:46:12] S1: That helped us a little bit. [00:46:14] S0: Yeah, I like the bad and sad. So these are like thresholds of this is bad. Okay, this is serious. And this is in terms of performance or failures or what sort of things are you measuring here? [00:46:24] S1: So for example, we allow each team, like speaking of agency, so knowing that bad is like a really bad irrecoverable error, we enable each team of further surface areas or it could be services that they lead. So for example, on CLI could be crash rates, like a crash is pretty bad, you lost work. And for example, sad might be, "Hey, is it flickering?" It might be recoverable. But we have each team, and that's why it's been interesting because surface areas are different, we would have all these dashboards, but it's harder to zoom out and go, "Okay, what's the overall theme of the experience?" So to your point, we give high agency to each team of what we think constitutes a bad and what's a sad, and then what's the goal that each team wants to take.
00:46:24 · Fiona Fung, Lenny Rachitsky
[00:46:24] S1: So for example, we allow each team, like speaking of agency, so knowing that bad is like a really bad irrecoverable error, we enable each team of further surface areas or it could be services that they lead. So for example, on CLI could be crash rates, like a crash is pretty bad, you lost work. And for example, sad might be, "Hey, is it flickering?" It might be recoverable. But we have each team, and that's why it's been interesting because surface areas are different, we would have all these dashboards, but it's harder to zoom out and go, "Okay, what's the overall theme of the experience?" So to your point, we give high agency to each team of what we think constitutes a bad and what's a sad, and then what's the goal that each team wants to take. [00:47:07] S0: The main takeaway I'm hearing here is one of the best tools for staying on top of quality is just monitoring and tests versus spending more time reviewing, which makes sense because the speed is just impossible to stay on top of. And it almost speaks to this idea of closing the loop for agents to be able to figure things out themselves. They know what success looks like, cool, we'll fix it ourselves. So that's a really interesting takeaway, is just invest more in the tests. Evals I imagine is part of this and then just monitoring failures and speed and things like that. [00:47:38] S1: That's right.
00:47:38 · Fiona Fung, Lenny Rachitsky
[00:47:38] S1: That's right. [00:47:39] S0: Something, I think this is public. I know that you guys have this dashboard that tracks just like F-words, how often people are like, because they're so and frustrated. There's like a funny term for it, I think. I forget what it's called.