Lenny's Podcast: Product | Career | Growth
00:31:49 · Tom Verrilli, Lenny Rachitsky
I can sit there and go through and be like, it feels like there's some car crash of modals that must be hitting new users as they open up the app. And you can literally just go through and be like, let's load up the feed and talk to me about the logic of who sees what in what order and when does this thing fire? And you can get answers really quickly. And having, as I said before, folks with enough tenure and enough reps that they can see that and say, "You know what? That's bad. Let's just make a good decision and change the ordering of those." I've saved now a kickoff meeting, an alignment meeting, a week writing PRDs, experiment time, all of that by just having somebody who's kind of empowered to go and make a decision. [00:32:28] S1: So coming back to people that are trying to get a job as a PM, whether you're new, let's actually hold off on new people, but people that are say managers, senior folks that are just like, wow, the market has really shifted. A big thing we're hearing right now is you need to be comfortable with moving back into IC, giving up your fancy title. Is there anything more along those lines of just people looking for a job, struggling to find a job? Any other advice for them?
00:32:53 · Tom Verrilli
[00:32:53] S0: Start doing IC work in the role you're in would be my push. Get back to the basics of make sure that you're taking on practical work. Because I just think it's good to make sure that you are keeping those muscles well honed. I think it also starts with pushing internally for those things. I would bet that if you started bringing that level of productivity back into the role that you're in, it probably helps you where you are in addition to help you where you might move to. But I think there's a lot of chatter online obviously about PMs or engineers now. And I think that's all well and good. It's a great muscle to go and go and hone. As I said, I've pushed some production code because I wanted to go through the exercise of understanding it. But I think there's also, before you get to building things, it's like how quickly can you get back into the muscle of scoping the correct thing, understanding the problem, being able to define what good looks like? And take advantage of the scale and the scope that you've got that you can see more and bring more to things. I think most folks who've sat in a director plus role will know the pain of sitting there and watching a junior PM yo-yo back and forth on the same PRD back to review where you kind of know what the answer is. Somewhere along the way we decided that lead a horse to water as opposed to help them understand the answer and then keep moving. And I think there's something in our coaching styles that we can get back to of help somebody understand what good looks like relatively quickly as opposed to just endless review yo-yo.
00:34:19 · Lenny Rachitsky, Tom Verrilli
[00:34:19] S1: This point you make about PMs not shipping to production, I so agree with. Because my mind changed on this recently with a previous podcast guest with this point that PMs have so much more leverage if they're not sitting there trying to ship to production. Enabling the team to ship better and faster and making sure the things that are shipping are better is a much better use of PM's time than sitting there shipping stuff. [00:34:41] S0: I mean, this sounds really silly, mate, but it takes me substantially longer to go through the minutia of getting, get, commit, and all of the pieces that are second eng to engineer aligned than it does to actually work out what the problem is and be able to describe it. So yes, good practice to try and make sure you're doing something. For example, I don't want to judge if our dev tools have gotten easier or not based on what someone tells me. I'm going to go and try it and be like, yep, that was easier than last time I did it. But I do think you can get an awful long way understanding the code base and then talking to somebody who can actually execute well. Otherwise you're just going to get fall afoul of a thousand classic traps that every other engineer learned how not to do when they were an L4.
00:34:41 · Tom Verrilli, Lenny Rachitsky
[00:34:41] S0: I mean, this sounds really silly, mate, but it takes me substantially longer to go through the minutia of getting, get, commit, and all of the pieces that are second eng to engineer aligned than it does to actually work out what the problem is and be able to describe it. So yes, good practice to try and make sure you're doing something. For example, I don't want to judge if our dev tools have gotten easier or not based on what someone tells me. I'm going to go and try it and be like, yep, that was easier than last time I did it. But I do think you can get an awful long way understanding the code base and then talking to somebody who can actually execute well. Otherwise you're just going to get fall afoul of a thousand classic traps that every other engineer learned how not to do when they were an L4. [00:35:22] S1: So following that thread a little bit, what are some of the ways that AI has enabled you and/or your team to move faster and be more productive other than prototyping, which is the very clear benefit of AI for PMs and product teams? What else? What are maybe in the top three of, wow, this has really unlocked our productivity and the quality of what we do?
00:35:43 · Tom Verrilli
[00:35:43] S0: I mean, the first one I think by a country mile is data science. We use Hex threads internally at Whatnot. I'm sure there are other comparable products, but I think it's almost hard to remember time as a PM before you had tooling like that where you could genuinely start pulling very nuanced cohorts of data where you could grab an individual user where you've heard a report. Actually understand, let's pull logs, help me understand exactly what this user did and saw. How many other users look like this? What would impact be? And then all of a sudden you can build pretty meaningful sensitivity models or forecasts of what might happen, regression models, et cetera, really, really, really quickly, which is incredibly powerful. The other thing that we've found is it helps us move much more quickly with shipping things because you can spot regressions and weird knock-on effects of two products into mixing more quickly than you used to be able to. And I think in really large complicated systems, that's always one of the things that ends up slowing you down of release trends and all of that versus if you build the right AI tool, you can spot regressions really quickly, which basically lets people just go. So I don't know what the future of data science looks like. But I think as a product manager, I've spent less time in the last year talking to a data scientist than I ever have in my career, even though I've probably spent 10 times more time in data and understanding actually how the product's working than I ever have in my career. So that one I think is really powerful. Second one I've already mentioned, which is stop bothering engineers with how does the code base work? And actually just go and talk to Claude and understand it, which is really helpful. I used to say early on in my career that the goal was always to be understand your systems at the boxes and lines level of which system drives which thing. And now there's no excuse not to understand that or a nuanced layer. But the other one, and this might be very specific to Whatnot, so I don't know that this will help everyone, but one of the things that I've been lucky to do in my career is basically worked on live products for a decade now. And so it's always been really cool to be able to ship a product and then watch a customer use it and watch them figure it out.
00:37:48 · Tom Verrilli, Lenny Rachitsky
So I think people have just gotten this experience with Listen Labs and others in that cohort of watching people use your product. But I've always been able to sit and watch people use a thing for the first time and go through that new user comprehension gap. What's really cool with a bunch of the AI tooling right now is as they're describing, oh, I'm having a problem, you can literally be watching the code base live and work out, is that actually a bug that's happening right now right now? Or is that a comprehension gap where it doesn't work as expected? And suddenly you've got this video artifact of someone using your product, you can be analyzing the code base in real time, and you can be just talking through AI to the code base to understand what's actually happening. And it's like a feedback loop on steroids because all of a sudden you know exactly what's going on, customer side, code side, and observe side as a viewer in real time, which is really cool. [00:38:40] S1: That sounds both awesome and very stressful to be building products that are that live in real time. I think about Netflix where they invest in Live Now, and that's just all you've done for 10 years and how big a deal that was for them. I know the scale is different.