Lenny's Podcast: Product | Career | Growth
00:44:01 · Lenny Rachitsky, Fiona Fung
[00:44:01] S0: And this comes back to your process that you shared of having Claude watch all the PRs shipping per engineer, let's say, and then not focusing on the metrics, but focusing on that leading to a conversation about what impact does this have, what was maybe some bugs that happened and that being a really powerful way of understanding what's going on with this engineer. Awesome. [00:44:24] S1: That's right. [00:44:25] S0: Let me come back to this, just this question of speed and quality and just impact. Is there anything else you've learned about just how to balance those things, just like this crazy velocity of code that's being generated and just staying on top of quality and impact. Is there anything else there that might be helpful to other teams that are trying to wrangle all these PRs being shipped every day?
00:44:47 · Fiona Fung, Lenny Rachitsky
[00:44:47] S1: I would say, and this is what honestly we want to keep doing more of and being better at it too. The proactive quality, so especially for quality, making sure that what are the experiences that are key and making sure you actually is, actually speaking of metrics, those are really good things that make sure you can see trends over time. And so on the quality front we found, and this is like the more proactive we can be of making sure we can get an earlier detection into quality. And so that's been one thing that we've been paying a lot of attention to. So I started this, "Hey, let's have a concept of what's bad versus what's sad." And bad is like a very bad irrecoverable error and sad is something that's kind of like a pain point, recoverable, but it's interesting when you stack up sads, it could generally go to bad. But even starting with a high level framework like that and because if not, I think sometimes with dashboards you can have time to load or all these other, but when you're dealing with a lot of different product surfaces, it's harder to go, "Wait, is that a good number or not a good number?" And so one thing that's helped us is versus just raw performance or reliability numbers, also having some framework of what we think is a bad experience and making sure we're focused on addressing those and then also keeping an eye on where we're seeing in terms of the sad. [00:46:12] S0: I like that. [00:46:12] S1: That helped us a little bit.
00:46:12 · Fiona Fung, Lenny Rachitsky
[00:46:12] S1: That helped us a little bit. [00:46:14] S0: Yeah, I like the bad and sad. So these are like thresholds of this is bad. Okay, this is serious. And this is in terms of performance or failures or what sort of things are you measuring here? [00:46:24] S1: So for example, we allow each team, like speaking of agency, so knowing that bad is like a really bad irrecoverable error, we enable each team of further surface areas or it could be services that they lead. So for example, on CLI could be crash rates, like a crash is pretty bad, you lost work. And for example, sad might be, "Hey, is it flickering?" It might be recoverable. But we have each team, and that's why it's been interesting because surface areas are different, we would have all these dashboards, but it's harder to zoom out and go, "Okay, what's the overall theme of the experience?" So to your point, we give high agency to each team of what we think constitutes a bad and what's a sad, and then what's the goal that each team wants to take.
00:46:24 · Fiona Fung, Lenny Rachitsky
[00:46:24] S1: So for example, we allow each team, like speaking of agency, so knowing that bad is like a really bad irrecoverable error, we enable each team of further surface areas or it could be services that they lead. So for example, on CLI could be crash rates, like a crash is pretty bad, you lost work. And for example, sad might be, "Hey, is it flickering?" It might be recoverable. But we have each team, and that's why it's been interesting because surface areas are different, we would have all these dashboards, but it's harder to zoom out and go, "Okay, what's the overall theme of the experience?" So to your point, we give high agency to each team of what we think constitutes a bad and what's a sad, and then what's the goal that each team wants to take. [00:47:07] S0: The main takeaway I'm hearing here is one of the best tools for staying on top of quality is just monitoring and tests versus spending more time reviewing, which makes sense because the speed is just impossible to stay on top of. And it almost speaks to this idea of closing the loop for agents to be able to figure things out themselves. They know what success looks like, cool, we'll fix it ourselves. So that's a really interesting takeaway, is just invest more in the tests. Evals I imagine is part of this and then just monitoring failures and speed and things like that. [00:47:38] S1: That's right.
00:47:38 · Fiona Fung, Lenny Rachitsky
[00:47:38] S1: That's right. [00:47:39] S0: Something, I think this is public. I know that you guys have this dashboard that tracks just like F-words, how often people are like, because they're so and frustrated. There's like a funny term for it, I think. I forget what it's called.
00:47:51 · Fiona Fung
[00:47:51] S1: Yes. Actually, I remember. Yeah, that was last September because we were all seeing some frustrations and yeah, that was an engineer on the team of, "Hey, we should maybe track swear words." I'm like, "Oh, that's a great idea." I remember it. I had just joined when we were having that really fun conversation. Yeah. So it's again, like it's very interesting to look at... And that's why evals is hard too because it's going back to that user experience and how we can make sure it's a delightful experience and less frustrating. But yeah, the swear word dashboard is a fun one. I love it. It's funny, when I first join... I love it. It's funny when I first joined, and I have amazing recruiting partners and I noticed this whole theme of growth mindset, it's just because the landscape is changing so fast. What worked well before may not make sense and even what makes sense today may need to change tomorrow. That's what I have to keep reminding myself. So yeah, when I first joined recruiters like, okay, yeah, we have these couple of manager postings and because actually how it came from a listening tour I did with all the members on the team, and I heard a lot of these, "Hey, I really appreciate the agency, but how can I make sure prioritization?" And so through that, and then there was some really good feedback too of making sure that it's not too many layers of reviews. There was good feedback. Some folks might have joined from other companies. And I actually think as a leader, if you actually start as IC first without the worry of supporting people, because that's a very heavy responsibility, but before you have to take on that full responsibility, give yourself that maker time to actually dive deep into the code and learn the code base or the product, whatever it is. Honestly, the PRs I do, but it's more about for me, it's important because it keeps me in the flow because we're making so many changes to Cowork and code. So even me doing PRs, it's less about what it is I'm fixing. It's more about me using the product every day just to keep that touch, because as amazing as metrics and everything are, and I do look at those dashboards, as a leader, if you're not living and breathing your product every day, you sometimes lose the touch and feel of the product.