Lenny's Podcast: Product | Career | Growth
00:44:47 · Fiona Fung, Lenny Rachitsky
[00:44:47] S1: I would say, and this is what honestly we want to keep doing more of and being better at it too. The proactive quality, so especially for quality, making sure that what are the experiences that are key and making sure you actually is, actually speaking of metrics, those are really good things that make sure you can see trends over time. And so on the quality front we found, and this is like the more proactive we can be of making sure we can get an earlier detection into quality. And so that's been one thing that we've been paying a lot of attention to. So I started this, "Hey, let's have a concept of what's bad versus what's sad." And bad is like a very bad irrecoverable error and sad is something that's kind of like a pain point, recoverable, but it's interesting when you stack up sads, it could generally go to bad. But even starting with a high level framework like that and because if not, I think sometimes with dashboards you can have time to load or all these other, but when you're dealing with a lot of different product surfaces, it's harder to go, "Wait, is that a good number or not a good number?" And so one thing that's helped us is versus just raw performance or reliability numbers, also having some framework of what we think is a bad experience and making sure we're focused on addressing those and then also keeping an eye on where we're seeing in terms of the sad. [00:46:12] S0: I like that. [00:46:12] S1: That helped us a little bit.
00:46:12 · Fiona Fung, Lenny Rachitsky
[00:46:12] S1: That helped us a little bit. [00:46:14] S0: Yeah, I like the bad and sad. So these are like thresholds of this is bad. Okay, this is serious. And this is in terms of performance or failures or what sort of things are you measuring here? [00:46:24] S1: So for example, we allow each team, like speaking of agency, so knowing that bad is like a really bad irrecoverable error, we enable each team of further surface areas or it could be services that they lead. So for example, on CLI could be crash rates, like a crash is pretty bad, you lost work. And for example, sad might be, "Hey, is it flickering?" It might be recoverable. But we have each team, and that's why it's been interesting because surface areas are different, we would have all these dashboards, but it's harder to zoom out and go, "Okay, what's the overall theme of the experience?" So to your point, we give high agency to each team of what we think constitutes a bad and what's a sad, and then what's the goal that each team wants to take.
00:46:24 · Fiona Fung, Lenny Rachitsky
[00:46:24] S1: So for example, we allow each team, like speaking of agency, so knowing that bad is like a really bad irrecoverable error, we enable each team of further surface areas or it could be services that they lead. So for example, on CLI could be crash rates, like a crash is pretty bad, you lost work. And for example, sad might be, "Hey, is it flickering?" It might be recoverable. But we have each team, and that's why it's been interesting because surface areas are different, we would have all these dashboards, but it's harder to zoom out and go, "Okay, what's the overall theme of the experience?" So to your point, we give high agency to each team of what we think constitutes a bad and what's a sad, and then what's the goal that each team wants to take. [00:47:07] S0: The main takeaway I'm hearing here is one of the best tools for staying on top of quality is just monitoring and tests versus spending more time reviewing, which makes sense because the speed is just impossible to stay on top of. And it almost speaks to this idea of closing the loop for agents to be able to figure things out themselves. They know what success looks like, cool, we'll fix it ourselves. So that's a really interesting takeaway, is just invest more in the tests. Evals I imagine is part of this and then just monitoring failures and speed and things like that. [00:47:38] S1: That's right.
00:47:38 · Fiona Fung, Lenny Rachitsky
[00:47:38] S1: That's right. [00:47:39] S0: Something, I think this is public. I know that you guys have this dashboard that tracks just like F-words, how often people are like, because they're so and frustrated. There's like a funny term for it, I think. I forget what it's called.
00:47:51 · Fiona Fung
[00:47:51] S1: Yes. Actually, I remember. Yeah, that was last September because we were all seeing some frustrations and yeah, that was an engineer on the team of, "Hey, we should maybe track swear words." I'm like, "Oh, that's a great idea." I remember it. I had just joined when we were having that really fun conversation. Yeah. So it's again, like it's very interesting to look at... And that's why evals is hard too because it's going back to that user experience and how we can make sure it's a delightful experience and less frustrating. But yeah, the swear word dashboard is a fun one. I love it. It's funny, when I first join... I love it. It's funny when I first joined, and I have amazing recruiting partners and I noticed this whole theme of growth mindset, it's just because the landscape is changing so fast. What worked well before may not make sense and even what makes sense today may need to change tomorrow. That's what I have to keep reminding myself. So yeah, when I first joined recruiters like, okay, yeah, we have these couple of manager postings and because actually how it came from a listening tour I did with all the members on the team, and I heard a lot of these, "Hey, I really appreciate the agency, but how can I make sure prioritization?" And so through that, and then there was some really good feedback too of making sure that it's not too many layers of reviews. There was good feedback. Some folks might have joined from other companies. And I actually think as a leader, if you actually start as IC first without the worry of supporting people, because that's a very heavy responsibility, but before you have to take on that full responsibility, give yourself that maker time to actually dive deep into the code and learn the code base or the product, whatever it is. Honestly, the PRs I do, but it's more about for me, it's important because it keeps me in the flow because we're making so many changes to Cowork and code. So even me doing PRs, it's less about what it is I'm fixing. It's more about me using the product every day just to keep that touch, because as amazing as metrics and everything are, and I do look at those dashboards, as a leader, if you're not living and breathing your product every day, you sometimes lose the touch and feel of the product.
00:51:11 · Fiona Fung, Lenny Rachitsky
But anyways, on the manager front, I think giving time for managers that join to be able to go deep into do that before supporting people, and then they actually end up building really great rapport with the team, because if not, I think sometimes as managers, you might come and join a new team and you instantly think, "I have to manage. Let me dig into my manager toolbox and do manager-y things." But if you actually give yourself time to not have to worry about that first and actually learn what it's like to be an engineer and teammate on the team, that also goes really far to building rapport. In terms of using the product, I think that's important. It's interesting, every team I've joined, one of the first actually across all the different products, it could be VR, it could be smart glasses, it was like Instagram. Usually when I first joined, one comment that comes back to me is, "Hey, you know what? I really love how you're actually using what we build." It's refreshing to see you giving user feedback, and so I think as leaders too, it's also a way for us to experience the work personally that the team does. [00:52:50] S0: Something that people may not realize you were overseeing an organization of like 500 people at Meta before you moved to Anthropic and you moved to IC engineer basically and Anthropic from that.