Lenny's Podcast: Product | Career | Growth
00:27:42 · Sean Ellis
[00:27:42] S1: Yeah. I hadn't really thought thought too much on that. Again, for for me, generally I'm trying to just figure out, is this a product that can grow? So when I'm using it, I'm trying to just... So if I got a 37%, am I going to be like, Oh, no, this would be impossible? Or if I had a 70%, does that mean I guarantee, say, Oh, yeah, I want to jump in and work with this company. It's more nuanced than that. Obviously, if it's a 70% but I have no idea how I grow the business, I'm going to be stuck there. I do think he brought up a really good point that culturally some people are going to be more optimistic or pessimistic. Interestingly, when I came up with a question, I used to just use kind of a normal satisfaction question. When I was working at Xobni, I'm just like an intensely curious person anyway, so I'm just trying to dig in and understand the customers. And so I've always done lots of surveying, but at Xobni, I was going to ask it as, like use my filter as a satisfaction question. So, you know, how satisfied are you with this? I'm very satisfied. I'm somewhat satisfied. And our main customers were actually senior management. And so I thought, senior management's never satisfied. I'm going to get always this super lukewarm thing. How can I change this question more me a more kind of real answer from these guys? Well, if I flip it and say, How would you feel if you could no longer use this product? I'll probably get a more honest answer back from them. And of course, they're very disappointed if they can't get what they want. And so initially it was just for the case of Xobni, but then I went to Dropbox right after Xobni and I'll try the question again. And the insights I got back were really useful. And so each company I went to, I kept using the question. I'm like, this works way better than your typical satisfaction question. But initially, it was more about thinking just senior management to get a more honest answer out of them.
00:29:54 · Lenny Rachitsky, Sean Ellis
[00:29:54] S0: So that's the origin story right there. [00:29:56] S1: Yeah. [00:29:57] S0: Wow. That senior managers are just very harsh and they... They're not... They don't need anything. Yeah. And you have to flip it. That is so interesting. Just like that question is such a good reminder of how hard it is to build anything people really would be disappointed not to have. Like, that's why this works so well. People are like, I don't need this app. Who cares? Right? Like, that's the core of this. It's just... That is hard.
00:30:19 · Sean Ellis
[00:30:19] S1: I mean, especially when I first moved to Silicon Valley. So first, like, fifteen years of my career were not in Silicon Valley. And so that was in Eastern Europe and then New York and then Boston. You moved to Silicon Valley and you have you have people who get really excited about technology for technology's sake. And so, know, just something being cool is isn't it cool that we can actually do this? Drives a lot of people. And so to me, I'm very practical. If it's not something that is really bringing value to people, the likelihood that that product's successful long term is going be pretty low. And so even interestingly at Dropbox, through the six months I was there, I'd ask one question multiple times a month. I broke the early beta users into a bunch of different lists and I'd ask, Which best describes you? I like to be among the first to try cool new technology or I only try things that I think will be useful for me. And over the six months, it flipped from 90% being people who try things that they want to try cool new technology to six months later, it was people who only are going to try something that they feel like is useful. But what's kind of cool is just because what motivates you to try something is like you're an early adopter and you want to try something cool. If you're going to keep using it, it's because it's giving you some utility. So I can still use those early adopters to, help me figure out where's the value inside the product.
00:31:53 · Lenny Rachitsky, Sean Ellis
[00:31:53] S0: Awesome. So actually two questions along those lines. How durable do you find this percentage being? Say you hit 40%, how often does that fade and go away versus stay there or go higher? [00:32:07] S1: Yeah, I haven't seen it really like fade back down, but I've seen companies fail, you know, despite having it. And I think a lot of times then it becomes like an execution challenge. Once you have product market fit. Not everyone's going to be a good executor, but before that, I think getting to product market fit, obviously there's a lot of methodology for doing it today that might make it a bit easier for people, but I still think it's fairly random and pretty dang hard. And so ultimately, the risk factor of creating something that people care about is really difficult. So if you can get to the point where you have 40% of the people who are using it saying they'd be very disappointed and you have a reasonable sample size, let's say, you you got 10 people and four of them said they'd be very disappointed without it, you're still going to get something useful from those four. But I wouldn't say that's a sample size that you can really go to market on. Yeah. [00:33:07] S0: What's a good sample size you look for just like, okay, is actually good data I wanna rely on?
00:33:07 · Lenny Rachitsky, Sean Ellis
[00:33:07] S0: What's a good sample size you look for just like, okay, is actually good data I wanna rely on? [00:33:11] S1: It's really funny. So much of the stuff I kind of self learned, but I basically at one point said I need at least 30 responses. And I just thought I just kind of randomly made up a number then I had people telling me, Yeah, 30 is kind of the minimum that you want on stuff. Was like, Okay. And even when I first created the survey, I remember showing it to the co founder of SlideShare and she was like, her PhD was in survey related stuff like cognitive psychology, but she basically said it was really about surveying. She's like, This methodology is amazing. How did you come up with this? And so having some of that validation around these things helped, but a lot of it was just, again, driven by my own curiosity and also just knowing that failure is such a likely outcome that trying to reverse engineer that failure and the number one reason for failure would be that people don't actually care about the product. And so when I find that, that's a really good sign that we're now down to an execution challenge. [00:34:15] S0: And there's this obvious element of you may have product market fit with people, but that group might end up being very small. And the business you build around it could actually be cool, but it's not gonna be a massive business. Is there anything there you can share just like... It's hard to know the size opportunity even though you know some people really, really like it.
00:34:15 · Lenny Rachitsky, Sean Ellis
[00:34:15] S0: And there's this obvious element of you may have product market fit with people, but that group might end up being very small. And the business you build around it could actually be cool, but it's not gonna be a massive business. Is there anything there you can share just like... It's hard to know the size opportunity even though you know some people really, really like it. [00:34:35] S1: Yeah. I talked about I I go to a multiple choice after I initially use open ended to open ended questions to sort of crowdsource the different use cases. But then I try to force people in a bucket. And then I can run filters on each of those buckets and I'll be like, Oh, people who use it this way are like 60% likely to be very disappointed without the product, but people who use it this way are 35% likely to be very disappointed, but way more people use it the 35% way. And so then having to like, do you want that intensely loyal group or the much broader group that's maybe a bit less, but almost there? I think that becomes a bit of a strategic conversation of like, we want to have a better chance of surviving going after a niche that we know we can serve well? Or have we raised so much money that we have to go after a really big market that one's not going to be long term, but maybe then you're like, Okay, once I have traction in that market, I can start to try to appeal to some other markets. But I think that's where kind of some strategic decisions come in. [00:35:47] S0: Do you have a heuristic of which you often recommend, is it very dependent on the situation?