Lenny's Podcast: Product | Career | Growth
00:35:43 · Tom Verrilli
[00:35:43] S0: I mean, the first one I think by a country mile is data science. We use Hex threads internally at Whatnot. I'm sure there are other comparable products, but I think it's almost hard to remember time as a PM before you had tooling like that where you could genuinely start pulling very nuanced cohorts of data where you could grab an individual user where you've heard a report. Actually understand, let's pull logs, help me understand exactly what this user did and saw. How many other users look like this? What would impact be? And then all of a sudden you can build pretty meaningful sensitivity models or forecasts of what might happen, regression models, et cetera, really, really, really quickly, which is incredibly powerful. The other thing that we've found is it helps us move much more quickly with shipping things because you can spot regressions and weird knock-on effects of two products into mixing more quickly than you used to be able to. And I think in really large complicated systems, that's always one of the things that ends up slowing you down of release trends and all of that versus if you build the right AI tool, you can spot regressions really quickly, which basically lets people just go. So I don't know what the future of data science looks like. But I think as a product manager, I've spent less time in the last year talking to a data scientist than I ever have in my career, even though I've probably spent 10 times more time in data and understanding actually how the product's working than I ever have in my career. So that one I think is really powerful. Second one I've already mentioned, which is stop bothering engineers with how does the code base work? And actually just go and talk to Claude and understand it, which is really helpful. I used to say early on in my career that the goal was always to be understand your systems at the boxes and lines level of which system drives which thing. And now there's no excuse not to understand that or a nuanced layer. But the other one, and this might be very specific to Whatnot, so I don't know that this will help everyone, but one of the things that I've been lucky to do in my career is basically worked on live products for a decade now. And so it's always been really cool to be able to ship a product and then watch a customer use it and watch them figure it out.
00:37:48 · Tom Verrilli, Lenny Rachitsky
So I think people have just gotten this experience with Listen Labs and others in that cohort of watching people use your product. But I've always been able to sit and watch people use a thing for the first time and go through that new user comprehension gap. What's really cool with a bunch of the AI tooling right now is as they're describing, oh, I'm having a problem, you can literally be watching the code base live and work out, is that actually a bug that's happening right now right now? Or is that a comprehension gap where it doesn't work as expected? And suddenly you've got this video artifact of someone using your product, you can be analyzing the code base in real time, and you can be just talking through AI to the code base to understand what's actually happening. And it's like a feedback loop on steroids because all of a sudden you know exactly what's going on, customer side, code side, and observe side as a viewer in real time, which is really cool. [00:38:40] S1: That sounds both awesome and very stressful to be building products that are that live in real time. I think about Netflix where they invest in Live Now, and that's just all you've done for 10 years and how big a deal that was for them. I know the scale is different.
00:38:55 · Tom Verrilli
[00:38:55] S0: Honestly, my second week at Whatnot, I remember sitting in a room watching, so one of those not familiar kind of commerce, largely auction platform, live commerce platform. And there's varieties of different ways to run an auction, but one of them is what's called sudden death, which is like when the timer ends, it ends. Otherwise, the classic auction environment, somebody bids in the last five seconds, it adds 10 seconds back on the clock, and a seller can decide what auction model they want. And I was sitting there watching a seller who was like, "These are taking too long. I wish this seven-second timer was actually three seconds because I want to move more product." And I watched two engineers in the office look at each other, and be like, "That's a [inaudible 00:39:33]. We could totally do that." And so they went and updated it in real time and then jumped in the chat of a show, and just said, "Refresh your app." And then all of a sudden, bang, it was operating that way. And I was like, "Cool. I'm with my people. I'm in the right place." Because that's the level of responsiveness that you can get in a live environment. And obviously AI means that that's really easy for lots of people to take on. Not that I would touch production code in that kind of way because that's a disastrous idea. But for qualified humans, it's a wonderful one. Yeah, listen, I have a lot of empathy for that because I've certainly seen it. Another plug for why having fewer more senior PMs is helpful because those are folks who've seen more of those reps and understand. But I also think a lot of the time that lack of clarity comes from the other thing, which is organizations historically underinvested in data engineering and data structures and good data labeling, and whether or not you've got, not just your taxonomies right, but whether or not you really understand whether or not your data systems and structures are set up well. And so what we've found is a lot of our best data scientists are pushing in that direction of are we actually correctly updating all of the ways in which our tracking and attribution works so that it's less easy for people to misunderstand those? And then yeah, using an AI tool to find a piece of data, much like using an AI tool to write code, doesn't absolve you of responsibility to make sure that that was good analysis, good code.
00:42:40 · Tom Verrilli, Lenny Rachitsky
It just tends to be leveraged for those people who are naturally inclined that way. [00:42:48] S1: So thinking about these different functions, data science, user research, design, engineering, PM, what I'm hearing so far is we'll need probably fewer PMs, we'll need fewer data scientists. Are there any other roles that you think they're trending down in terms of we'll need... Are there any roles trending up like, wow, we're going to need a lot more of this kind of person?
00:43:07 · Tom Verrilli
[00:43:07] S0: Well, funnily enough, as I say, I haven't spoken to a data scientist in a while. We've certainly still hired plenty because I do think all that tracking and attribution and measurement, really, really powerful. I also think one of the flip sides, if we say we need fewer, it's like fewer of for the same output doesn't necessarily mean fewer of in macro. Because if you are using these systems right, you can just grow more quickly. You can build more things, you can take on more stuff. So I would be surprised actually if we ended up with net fewer. I think it's more like net fewer vis-a-vis customer impact for both. Certainly I think trending up over time within these roles are these kind of, and I don't remember the exact label that people used to use, but this idea of tech leads that are kind of like a hybrid EM where you're running a very small team, running at things. Because same way that we'd say the cost of trying something has come down, the idea that says you can have core focus, which is a thing we're very big on. We've also found incubating a lot of smaller teams to go over and sit in the corner and just try and build this thing. It's not quite prototyping, but it's like go and take a swing at a thing that we've historically thought was too hard, is getting cheaper and increasingly has very high leverage. So the idea of engineering manager light, for want of a better term, is the thing that I'm definitely think that we'll see more and more of. It's not quite the technical member of staff, although that must be lovely, but certainly not the kind of full-blown EM role I think is one that definitely will come about more often.
00:44:49 · Lenny Rachitsky
[00:44:49] S1: So maybe just to close the loop on this part of the conversation, there's this trend towards everyone's kind of a builder, PMs are shipping a little bit, being a little more engineer. Engineers are taking on more of the PM work. How do you think this just maybe plays out over the next couple of years in terms of what product teams look like broadly? Is it still PM engineers, a designer, data scientist somewhere? How do you envision the canonical product team over the coming years?