Lenny's Podcast: Product | Career | Growth
00:34:19 · Lenny Rachitsky, Tom Verrilli
[00:34:19] S1: This point you make about PMs not shipping to production, I so agree with. Because my mind changed on this recently with a previous podcast guest with this point that PMs have so much more leverage if they're not sitting there trying to ship to production. Enabling the team to ship better and faster and making sure the things that are shipping are better is a much better use of PM's time than sitting there shipping stuff. [00:34:41] S0: I mean, this sounds really silly, mate, but it takes me substantially longer to go through the minutia of getting, get, commit, and all of the pieces that are second eng to engineer aligned than it does to actually work out what the problem is and be able to describe it. So yes, good practice to try and make sure you're doing something. For example, I don't want to judge if our dev tools have gotten easier or not based on what someone tells me. I'm going to go and try it and be like, yep, that was easier than last time I did it. But I do think you can get an awful long way understanding the code base and then talking to somebody who can actually execute well. Otherwise you're just going to get fall afoul of a thousand classic traps that every other engineer learned how not to do when they were an L4.
00:34:41 · Tom Verrilli, Lenny Rachitsky
[00:34:41] S0: I mean, this sounds really silly, mate, but it takes me substantially longer to go through the minutia of getting, get, commit, and all of the pieces that are second eng to engineer aligned than it does to actually work out what the problem is and be able to describe it. So yes, good practice to try and make sure you're doing something. For example, I don't want to judge if our dev tools have gotten easier or not based on what someone tells me. I'm going to go and try it and be like, yep, that was easier than last time I did it. But I do think you can get an awful long way understanding the code base and then talking to somebody who can actually execute well. Otherwise you're just going to get fall afoul of a thousand classic traps that every other engineer learned how not to do when they were an L4. [00:35:22] S1: So following that thread a little bit, what are some of the ways that AI has enabled you and/or your team to move faster and be more productive other than prototyping, which is the very clear benefit of AI for PMs and product teams? What else? What are maybe in the top three of, wow, this has really unlocked our productivity and the quality of what we do?
00:35:43 · Tom Verrilli
[00:35:43] S0: I mean, the first one I think by a country mile is data science. We use Hex threads internally at Whatnot. I'm sure there are other comparable products, but I think it's almost hard to remember time as a PM before you had tooling like that where you could genuinely start pulling very nuanced cohorts of data where you could grab an individual user where you've heard a report. Actually understand, let's pull logs, help me understand exactly what this user did and saw. How many other users look like this? What would impact be? And then all of a sudden you can build pretty meaningful sensitivity models or forecasts of what might happen, regression models, et cetera, really, really, really quickly, which is incredibly powerful. The other thing that we've found is it helps us move much more quickly with shipping things because you can spot regressions and weird knock-on effects of two products into mixing more quickly than you used to be able to. And I think in really large complicated systems, that's always one of the things that ends up slowing you down of release trends and all of that versus if you build the right AI tool, you can spot regressions really quickly, which basically lets people just go. So I don't know what the future of data science looks like. But I think as a product manager, I've spent less time in the last year talking to a data scientist than I ever have in my career, even though I've probably spent 10 times more time in data and understanding actually how the product's working than I ever have in my career. So that one I think is really powerful. Second one I've already mentioned, which is stop bothering engineers with how does the code base work? And actually just go and talk to Claude and understand it, which is really helpful. I used to say early on in my career that the goal was always to be understand your systems at the boxes and lines level of which system drives which thing. And now there's no excuse not to understand that or a nuanced layer. But the other one, and this might be very specific to Whatnot, so I don't know that this will help everyone, but one of the things that I've been lucky to do in my career is basically worked on live products for a decade now. And so it's always been really cool to be able to ship a product and then watch a customer use it and watch them figure it out.
00:37:48 · Tom Verrilli, Lenny Rachitsky
So I think people have just gotten this experience with Listen Labs and others in that cohort of watching people use your product. But I've always been able to sit and watch people use a thing for the first time and go through that new user comprehension gap. What's really cool with a bunch of the AI tooling right now is as they're describing, oh, I'm having a problem, you can literally be watching the code base live and work out, is that actually a bug that's happening right now right now? Or is that a comprehension gap where it doesn't work as expected? And suddenly you've got this video artifact of someone using your product, you can be analyzing the code base in real time, and you can be just talking through AI to the code base to understand what's actually happening. And it's like a feedback loop on steroids because all of a sudden you know exactly what's going on, customer side, code side, and observe side as a viewer in real time, which is really cool. [00:38:40] S1: That sounds both awesome and very stressful to be building products that are that live in real time. I think about Netflix where they invest in Live Now, and that's just all you've done for 10 years and how big a deal that was for them. I know the scale is different.
00:38:55 · Tom Verrilli
[00:38:55] S0: Honestly, my second week at Whatnot, I remember sitting in a room watching, so one of those not familiar kind of commerce, largely auction platform, live commerce platform. And there's varieties of different ways to run an auction, but one of them is what's called sudden death, which is like when the timer ends, it ends. Otherwise, the classic auction environment, somebody bids in the last five seconds, it adds 10 seconds back on the clock, and a seller can decide what auction model they want. And I was sitting there watching a seller who was like, "These are taking too long. I wish this seven-second timer was actually three seconds because I want to move more product." And I watched two engineers in the office look at each other, and be like, "That's a [inaudible 00:39:33]. We could totally do that." And so they went and updated it in real time and then jumped in the chat of a show, and just said, "Refresh your app." And then all of a sudden, bang, it was operating that way. And I was like, "Cool. I'm with my people. I'm in the right place." Because that's the level of responsiveness that you can get in a live environment. And obviously AI means that that's really easy for lots of people to take on. Not that I would touch production code in that kind of way because that's a disastrous idea. But for qualified humans, it's a wonderful one. Yeah, listen, I have a lot of empathy for that because I've certainly seen it. Another plug for why having fewer more senior PMs is helpful because those are folks who've seen more of those reps and understand. But I also think a lot of the time that lack of clarity comes from the other thing, which is organizations historically underinvested in data engineering and data structures and good data labeling, and whether or not you've got, not just your taxonomies right, but whether or not you really understand whether or not your data systems and structures are set up well. And so what we've found is a lot of our best data scientists are pushing in that direction of are we actually correctly updating all of the ways in which our tracking and attribution works so that it's less easy for people to misunderstand those? And then yeah, using an AI tool to find a piece of data, much like using an AI tool to write code, doesn't absolve you of responsibility to make sure that that was good analysis, good code.
00:42:40 · Tom Verrilli, Lenny Rachitsky
It just tends to be leveraged for those people who are naturally inclined that way. [00:42:48] S1: So thinking about these different functions, data science, user research, design, engineering, PM, what I'm hearing so far is we'll need probably fewer PMs, we'll need fewer data scientists. Are there any other roles that you think they're trending down in terms of we'll need... Are there any roles trending up like, wow, we're going to need a lot more of this kind of person?