Lenny's Podcast: Product | Career | Growth
00:46:24 · Fiona Fung, Lenny Rachitsky
[00:46:24] S1: So for example, we allow each team, like speaking of agency, so knowing that bad is like a really bad irrecoverable error, we enable each team of further surface areas or it could be services that they lead. So for example, on CLI could be crash rates, like a crash is pretty bad, you lost work. And for example, sad might be, "Hey, is it flickering?" It might be recoverable. But we have each team, and that's why it's been interesting because surface areas are different, we would have all these dashboards, but it's harder to zoom out and go, "Okay, what's the overall theme of the experience?" So to your point, we give high agency to each team of what we think constitutes a bad and what's a sad, and then what's the goal that each team wants to take. [00:47:07] S0: The main takeaway I'm hearing here is one of the best tools for staying on top of quality is just monitoring and tests versus spending more time reviewing, which makes sense because the speed is just impossible to stay on top of. And it almost speaks to this idea of closing the loop for agents to be able to figure things out themselves. They know what success looks like, cool, we'll fix it ourselves. So that's a really interesting takeaway, is just invest more in the tests. Evals I imagine is part of this and then just monitoring failures and speed and things like that. [00:47:38] S1: That's right.
00:47:38 · Fiona Fung, Lenny Rachitsky
[00:47:38] S1: That's right. [00:47:39] S0: Something, I think this is public. I know that you guys have this dashboard that tracks just like F-words, how often people are like, because they're so and frustrated. There's like a funny term for it, I think. I forget what it's called.
00:47:51 · Fiona Fung
[00:47:51] S1: Yes. Actually, I remember. Yeah, that was last September because we were all seeing some frustrations and yeah, that was an engineer on the team of, "Hey, we should maybe track swear words." I'm like, "Oh, that's a great idea." I remember it. I had just joined when we were having that really fun conversation. Yeah. So it's again, like it's very interesting to look at... And that's why evals is hard too because it's going back to that user experience and how we can make sure it's a delightful experience and less frustrating. But yeah, the swear word dashboard is a fun one. I love it. It's funny, when I first join... I love it. It's funny when I first joined, and I have amazing recruiting partners and I noticed this whole theme of growth mindset, it's just because the landscape is changing so fast. What worked well before may not make sense and even what makes sense today may need to change tomorrow. That's what I have to keep reminding myself. So yeah, when I first joined recruiters like, okay, yeah, we have these couple of manager postings and because actually how it came from a listening tour I did with all the members on the team, and I heard a lot of these, "Hey, I really appreciate the agency, but how can I make sure prioritization?" And so through that, and then there was some really good feedback too of making sure that it's not too many layers of reviews. There was good feedback. Some folks might have joined from other companies. And I actually think as a leader, if you actually start as IC first without the worry of supporting people, because that's a very heavy responsibility, but before you have to take on that full responsibility, give yourself that maker time to actually dive deep into the code and learn the code base or the product, whatever it is. Honestly, the PRs I do, but it's more about for me, it's important because it keeps me in the flow because we're making so many changes to Cowork and code. So even me doing PRs, it's less about what it is I'm fixing. It's more about me using the product every day just to keep that touch, because as amazing as metrics and everything are, and I do look at those dashboards, as a leader, if you're not living and breathing your product every day, you sometimes lose the touch and feel of the product.
00:51:11 · Fiona Fung, Lenny Rachitsky
But anyways, on the manager front, I think giving time for managers that join to be able to go deep into do that before supporting people, and then they actually end up building really great rapport with the team, because if not, I think sometimes as managers, you might come and join a new team and you instantly think, "I have to manage. Let me dig into my manager toolbox and do manager-y things." But if you actually give yourself time to not have to worry about that first and actually learn what it's like to be an engineer and teammate on the team, that also goes really far to building rapport. In terms of using the product, I think that's important. It's interesting, every team I've joined, one of the first actually across all the different products, it could be VR, it could be smart glasses, it was like Instagram. Usually when I first joined, one comment that comes back to me is, "Hey, you know what? I really love how you're actually using what we build." It's refreshing to see you giving user feedback, and so I think as leaders too, it's also a way for us to experience the work personally that the team does. [00:52:50] S0: Something that people may not realize you were overseeing an organization of like 500 people at Meta before you moved to Anthropic and you moved to IC engineer basically and Anthropic from that.
00:53:02 · Fiona Fung
[00:53:02] S1: I started out, it was for a very short amount of time, but actually this was my journey between Microsoft and Meta. So at Meta, I interviewed as a manager, but I think at least for the first quarter, I was also an IC because I really wanted to learn what it's like to be a Meta engineer, because before then I was at Microsoft, cut my teeth on engineering at Microsoft. So I knew all the code bases, the tools, the languages, but it was so valuable to have those first months actually learn what it's like to ship as a Meta engineer. Plus it was also fun. I think sometimes we forget, but that's by the way, what I love about Claude Code because the last time I shipped production software at Meta was probably 2017, and every year, I do a lot of dogfooding and I also use dogfooding to help me vet the quality of product as well. But it's been a very long time since I shipped production software, it's because part of it is I don't want to screw something up. I'm always so scared of what if I do something and then I cause a bug and then am I verifying everything properly, or am I wasting someone's time? Because also the tool flows would change, but I remember that first week on Claude, at first I almost again did my usual, let me go meet all the engineers and treat them to coffee, and then I'm like, "Oh wait, let me ask Claude." And so Claude was this really good onboarding buddy of mine because I was curious about the copays asking questions, and then it also really helped me do the automated test, but I also want to still do some manual testing, and I asked Claude, "Hey, help me come up with what's the way for me to manually test this to make sure I cover all the case." But all that then gives me confidence that okay, I can ship here again. And then I got kind of more comfortable with it again, and so I've actually had a lot of friends reach out to me that have been managed for a while that's like, "Hey, I'm shipping code again, thanks to Claude." But in general, it's just important to me as leaders to make sure you're using the product that your team builds.
00:55:09 · Lenny Rachitsky
[00:55:09] S0: It's so interesting that you are overseeing the team as an engine leader that is most changing the role of an engineer. It's such a meta role. The work of an engineer is transforming because of the software that you're building. One question along these lines is just do you worry about engineers' skills to code atrophying from not actually writing code anymore and does it even matter? Is this something you think about? Is it a big deal?