Lenny's Podcast: Product | Career | Growth
01:07:52 · Elena Verna
[01:07:52] S0: First of all, I think that people should trust their intuition a little bit more. Data is good, but data is only good if you have enough of it. So if you have low volume real estate that is gonna take you eight months to reach some sort of answer, do you really wanna test it for eight months? What's the point of it? It's my rule of thumb, if we cannot collect the sample size in a month, we shouldn't test it, period. Because it's then it's not fast enough. We should just go and do pre versus post. Pre versus post is pretty powerful. It's also a way to assess your impact, and you can still roll back if doesn't work, but don't think that everything needs a scientific explanation to whether it needs to be moved forward or not. Also, experimentation cannot be the way that people make decisions in the company. There's still so much about knowing your user, understanding the market, for your brain to connect all of those dots and to know what needs to be an experience that your customers are going to want. That, yeah, if scientific data, like a very tight determination of the probability that this is a success is important, absolutely do it because it might be a really big strategical pivot that you're planning to do. So, it's a data point to validate that all of this extra work will be needed. It might be a very high trafficked real estate that even like 0.1% difference will mean millions of dollars for you. But other than that, it should be just go, go, go. Do pre versus post, by the way. I'm like, I'm not saying just, like, release and release and move on. Release, do seven day, twenty four hour readout, seven day readout, twenty eight day readout, even come back to it a year later and measure some of the retention or expansion data that is associated with it. But to test everything is debilitating to growth teams, and it paralyzes them in its place. So, kind of look at your initiatives and say, Where do I need precision? And it's important, and I can get it fast enough, versus where we should just go for it. And yeah, we will fail there too. And that's okay. And we can roll back and we can figure out how to make it better. But failure is going to happen regardless. And statistics is tricky. Many people take it for face value versus just like a directional data point to say, There is likely a new distribution that has maybe a different mean.
01:10:22 · Elena Verna, Lenny Rachitsky
And 5% of the chances, if you're measuring 95% statistical significance, you might not even be there. And yet, we'll, like, take it for some granted. Oh, it's gonna drive this much lift. So I just think that people stop, like, in this age of data, don't always rely enough on their own intuition. [01:10:42] S1: A lot of contrarian takes here. I love this. Elena, we've reached number 10. [01:10:47] S0: Okay. [01:10:47] S1: And I know number 10 is like a special one where it's more than one quick
01:10:51 · Elena Verna
[01:10:51] S0: So number 10 is gonna be my little fire round. My fire round on, like, little things that I just see people spending way too much time on and, like, it hurts my heart because it's not going to drive any results on them. So number one, color optimizations. For the love of God, a blue is a blue is a blue. As long as it's accessible and as long as it's bright enough, off you go. You do not need to test the shades of blue or test it against green or so on. Pick a color, move on. Please don't spend time on it. That's a early 2000s tactic. It doesn't work anymore. We passed that in technology sector. Number two, third party signups. A lot of times we think, Oh, we're going get so much more acquisition if we add Google Auth, or if we add Facebook Auth, or Slack Auth, or Microsoft Auth, whichever Auth you want to add to it. In some cases, it's very important to have third party auth. So, for example, if you're a developer product, please have a GitHub auth. It's kind of unnecessary. The developers already have the account there, have them connect with it as opposed to create a new one. However, if you're a productivity product, email is fine for the longest time, Gmail is nice, it's not gonna drive more acquisition for you. You're just gonna do a mix shift into more people using it, it's not gonna create incrementality, it's not gonna improve your activation, it's not gonna improve your retention. It's not a growth tactic, it's part of your customer experience that you want to invest into. Number three, on the fire round. One email wonders. We stress too much about this one email that we're going to send to this one customer group, and we're like, Oh my gosh, how much lift is it gonna cost? It will never cause any lift. It will never work as a one off email. Too few people open it, on average 25% open rates. At best, you'll get 40 to 50% open rate. Too few people click it. One email will never do anything. If you're gonna go into email, please think about it as a series about communication, about how it interacts with product communication. It's a whole strategy. It's a whole thing. Please don't stress about one off email. Never test a change in one email. It'll never work. You're gonna have to do the whole goddamn thing and see how it's gonna do, not just one email. Just never, gonna work.
01:13:03 · Lenny Rachitsky
[01:13:03] S1: I'm definitely guilty of this one. That is a really good one for sure. But it's so obvious in hindsight, so I love that you shared that.
01:13:09 · Elena Verna
[01:13:09] S0: Yes. And then the last one. Growth teams are often too obsessed about removing friction. Because if we remove the steps, then more people are gonna get to the destination. And to an extent, if you cause horrible friction in your product where it's just confusing about what the next step is, I agree. Go fix the friction of the cognitive load that it takes to complete a step. That is the friction that you should be working on. However, just removing steps or yanking or simplifying things to an oblivion where you lose an identity of what you even do or what you're capable of doing is a completely failed growth tactic. So, simplifying may be an initiative of a different problem that you're solving, but if you ever have a line item on your roadmap that says simplified onboarding, please cross it out. It's not going to work because simplifying onboarding is an action. What is the problem that you're solving? You're never trying to solve a problem of simplifying. You always have a problem of people are confused in it, where people don't know where to go or they get lost in it, or they're not educated enough. That is the problem. And simplifying might be a solution, but it can never be a problem on its own. And too many growth teams are just obsessed with this notion that came out, I think in early 2000s, like, Oh, simplifying is like the biggest growth hack that you can do. Please don't do it. It's only a solution to a very specific problem set of too much complexity.
01:14:48 · Lenny Rachitsky, Elena Verna
[01:14:48] S1: Wow. That was a big one to end on that I think will help a lot of people avoid wasting time. Again, a whole other podcast conversation probably focused just on onboarding advice that you probably have... That I know you have. We've reached the last item. Is there anything else you wanna share about this list before we go on? [01:15:07] S0: I think we spent so much time on the list. We're ready to go. [01:15:10] S1: Much. Just enough time. And kind of on that note, wanna call an audible. And so we were gonna talk through all of your frame... Your favorite frameworks. I like the way this conversation has gone. I don't want people to get overwhelmed with information. So one idea is just just list maybe your favorite growth frameworks just for folks to go check these out and not spend too much time on each one.