Lenny's Podcast: Product | Career | Growth
00:30:19 · Sean Ellis
[00:30:19] S1: I mean, especially when I first moved to Silicon Valley. So first, like, fifteen years of my career were not in Silicon Valley. And so that was in Eastern Europe and then New York and then Boston. You moved to Silicon Valley and you have you have people who get really excited about technology for technology's sake. And so, know, just something being cool is isn't it cool that we can actually do this? Drives a lot of people. And so to me, I'm very practical. If it's not something that is really bringing value to people, the likelihood that that product's successful long term is going be pretty low. And so even interestingly at Dropbox, through the six months I was there, I'd ask one question multiple times a month. I broke the early beta users into a bunch of different lists and I'd ask, Which best describes you? I like to be among the first to try cool new technology or I only try things that I think will be useful for me. And over the six months, it flipped from 90% being people who try things that they want to try cool new technology to six months later, it was people who only are going to try something that they feel like is useful. But what's kind of cool is just because what motivates you to try something is like you're an early adopter and you want to try something cool. If you're going to keep using it, it's because it's giving you some utility. So I can still use those early adopters to, help me figure out where's the value inside the product.
00:31:53 · Lenny Rachitsky, Sean Ellis
[00:31:53] S0: Awesome. So actually two questions along those lines. How durable do you find this percentage being? Say you hit 40%, how often does that fade and go away versus stay there or go higher? [00:32:07] S1: Yeah, I haven't seen it really like fade back down, but I've seen companies fail, you know, despite having it. And I think a lot of times then it becomes like an execution challenge. Once you have product market fit. Not everyone's going to be a good executor, but before that, I think getting to product market fit, obviously there's a lot of methodology for doing it today that might make it a bit easier for people, but I still think it's fairly random and pretty dang hard. And so ultimately, the risk factor of creating something that people care about is really difficult. So if you can get to the point where you have 40% of the people who are using it saying they'd be very disappointed and you have a reasonable sample size, let's say, you you got 10 people and four of them said they'd be very disappointed without it, you're still going to get something useful from those four. But I wouldn't say that's a sample size that you can really go to market on. Yeah. [00:33:07] S0: What's a good sample size you look for just like, okay, is actually good data I wanna rely on?
00:33:07 · Lenny Rachitsky, Sean Ellis
[00:33:07] S0: What's a good sample size you look for just like, okay, is actually good data I wanna rely on? [00:33:11] S1: It's really funny. So much of the stuff I kind of self learned, but I basically at one point said I need at least 30 responses. And I just thought I just kind of randomly made up a number then I had people telling me, Yeah, 30 is kind of the minimum that you want on stuff. Was like, Okay. And even when I first created the survey, I remember showing it to the co founder of SlideShare and she was like, her PhD was in survey related stuff like cognitive psychology, but she basically said it was really about surveying. She's like, This methodology is amazing. How did you come up with this? And so having some of that validation around these things helped, but a lot of it was just, again, driven by my own curiosity and also just knowing that failure is such a likely outcome that trying to reverse engineer that failure and the number one reason for failure would be that people don't actually care about the product. And so when I find that, that's a really good sign that we're now down to an execution challenge. [00:34:15] S0: And there's this obvious element of you may have product market fit with people, but that group might end up being very small. And the business you build around it could actually be cool, but it's not gonna be a massive business. Is there anything there you can share just like... It's hard to know the size opportunity even though you know some people really, really like it.
00:34:15 · Lenny Rachitsky, Sean Ellis
[00:34:15] S0: And there's this obvious element of you may have product market fit with people, but that group might end up being very small. And the business you build around it could actually be cool, but it's not gonna be a massive business. Is there anything there you can share just like... It's hard to know the size opportunity even though you know some people really, really like it. [00:34:35] S1: Yeah. I talked about I I go to a multiple choice after I initially use open ended to open ended questions to sort of crowdsource the different use cases. But then I try to force people in a bucket. And then I can run filters on each of those buckets and I'll be like, Oh, people who use it this way are like 60% likely to be very disappointed without the product, but people who use it this way are 35% likely to be very disappointed, but way more people use it the 35% way. And so then having to like, do you want that intensely loyal group or the much broader group that's maybe a bit less, but almost there? I think that becomes a bit of a strategic conversation of like, we want to have a better chance of surviving going after a niche that we know we can serve well? Or have we raised so much money that we have to go after a really big market that one's not going to be long term, but maybe then you're like, Okay, once I have traction in that market, I can start to try to appeal to some other markets. But I think that's where kind of some strategic decisions come in. [00:35:47] S0: Do you have a heuristic of which you often recommend, is it very dependent on the situation?
00:35:47 · Lenny Rachitsky, Sean Ellis
[00:35:47] S0: Do you have a heuristic of which you often recommend, is it very dependent on the situation? [00:35:52] S1: I prefer kind of a more passionate customer base and work from there just because I think your biggest competition when you're really innovating is just being irrelevant. And so if you're deeply relevant to anyone, I think that gives you a much better chance of long term success. [00:36:11] S0: Awesome. That's a really good insight. Okay. Two more questions along this line, and then I wanna talk about growth strategy. One very tactical question. Is there a tool you recommend for doing this sort of survey? Like, you recommend Inline, In the Product, In email, something else? [00:36:25] S1: I've used a lot of different tools. I actually had a survey business that I sold to private equity years ago and that was a on product. It's called Qualaroo that's kind of an inflow survey tool. I think just using SurveyMonkey with emailed surveys works fine. And for me, it's a lot more of like, what's pleasant for the customer to fill out and then what's going to give me something where I can work really easily with the data? So at Bounce, for example, they had already Intercom in place that had just introduced surveying, but it was kind of a really crappy customer experience on at least at that time. That's been almost a year now, or actually a little over a year. And so I'm really sensitive to like, is it a good survey experience for the consumer itself? And then, but yeah, I don't think I'm stuck to any one platform there.
00:37:30 · Lenny Rachitsky
[00:37:30] S0: Such an important topic. Yeah. Just like, again, to remind people why this is so important. One of the most common questions founders ask is, do I have product market fit? Have I built something people want? Like, that's just an endless series of I don't know. How do I know? When do I know? And this is telling you in a really interesting way. So your advice is this is a leading indicator. You don't actually know until people actually start using it and whether they retain and continue using it. Is there just, like, advice on the shift you make from relying on the survey to actually looking at retention cohorts? Is it just once you have enough data, once you have a couple cohorts, then start looking at that, forget about the survey?