Lenny's Podcast: Product | Career | Growth
01:05:48 · Lenny Rachitsky, Elena Verna
[01:05:48] S1: That was an awesome tip. And I could see why people would be like, Goddamn, Elena, don't say that. Makes so much sense. Okay. Here we go. Number nine. Here we go. [01:05:59] S0: My last one is also that I see something way too often nowadays on growth teams specifically. This is very, very growth problem, and that is too much risk averseness on growth, where you're starting to test everything. If every single one of your initiatives that you're doing on growth is an experiment, that's a problem. And that is something that you should take a look at and say, What am I doing? And why do I need a precise scientific measurement for every single thing that I touch in order for it to go to production or like to hit the market. It's almost like a disease, like a paralyzing disease that slows down progress, that slows down velocity, that slows down learnings, that creates very terrible consequences to the output that growth teams produce. And it's a little bit counterintuitive because experimentation is a way to do growth for growth teams, it's their process. Growth teams are meant to experiment. But I also think that experimenting on everything is something that is quite terrible once the team starts to get locked into that state. And it's really hard to get out of because they're then afraid to do any change until or unless that they test.
01:05:59 · Elena Verna, Lenny Rachitsky
[01:05:59] S0: My last one is also that I see something way too often nowadays on growth teams specifically. This is very, very growth problem, and that is too much risk averseness on growth, where you're starting to test everything. If every single one of your initiatives that you're doing on growth is an experiment, that's a problem. And that is something that you should take a look at and say, What am I doing? And why do I need a precise scientific measurement for every single thing that I touch in order for it to go to production or like to hit the market. It's almost like a disease, like a paralyzing disease that slows down progress, that slows down velocity, that slows down learnings, that creates very terrible consequences to the output that growth teams produce. And it's a little bit counterintuitive because experimentation is a way to do growth for growth teams, it's their process. Growth teams are meant to experiment. But I also think that experimenting on everything is something that is quite terrible once the team starts to get locked into that state. And it's really hard to get out of because they're then afraid to do any change until or unless that they test. [01:07:39] S1: So where do you find that balance? I imagine people hearing this are like, oh, yes, this makes sense to me. And then they continue to test basically everything. What's kind of some heuristics you'd recommend for knowing, okay, just don't test that. Don't worry about it. You don't need credit for that win.
01:07:52 · Elena Verna
[01:07:52] S0: First of all, I think that people should trust their intuition a little bit more. Data is good, but data is only good if you have enough of it. So if you have low volume real estate that is gonna take you eight months to reach some sort of answer, do you really wanna test it for eight months? What's the point of it? It's my rule of thumb, if we cannot collect the sample size in a month, we shouldn't test it, period. Because it's then it's not fast enough. We should just go and do pre versus post. Pre versus post is pretty powerful. It's also a way to assess your impact, and you can still roll back if doesn't work, but don't think that everything needs a scientific explanation to whether it needs to be moved forward or not. Also, experimentation cannot be the way that people make decisions in the company. There's still so much about knowing your user, understanding the market, for your brain to connect all of those dots and to know what needs to be an experience that your customers are going to want. That, yeah, if scientific data, like a very tight determination of the probability that this is a success is important, absolutely do it because it might be a really big strategical pivot that you're planning to do. So, it's a data point to validate that all of this extra work will be needed. It might be a very high trafficked real estate that even like 0.1% difference will mean millions of dollars for you. But other than that, it should be just go, go, go. Do pre versus post, by the way. I'm like, I'm not saying just, like, release and release and move on. Release, do seven day, twenty four hour readout, seven day readout, twenty eight day readout, even come back to it a year later and measure some of the retention or expansion data that is associated with it. But to test everything is debilitating to growth teams, and it paralyzes them in its place. So, kind of look at your initiatives and say, Where do I need precision? And it's important, and I can get it fast enough, versus where we should just go for it. And yeah, we will fail there too. And that's okay. And we can roll back and we can figure out how to make it better. But failure is going to happen regardless. And statistics is tricky. Many people take it for face value versus just like a directional data point to say, There is likely a new distribution that has maybe a different mean.
01:10:22 · Elena Verna, Lenny Rachitsky
And 5% of the chances, if you're measuring 95% statistical significance, you might not even be there. And yet, we'll, like, take it for some granted. Oh, it's gonna drive this much lift. So I just think that people stop, like, in this age of data, don't always rely enough on their own intuition. [01:10:42] S1: A lot of contrarian takes here. I love this. Elena, we've reached number 10. [01:10:47] S0: Okay. [01:10:47] S1: And I know number 10 is like a special one where it's more than one quick
01:10:51 · Elena Verna
[01:10:51] S0: So number 10 is gonna be my little fire round. My fire round on, like, little things that I just see people spending way too much time on and, like, it hurts my heart because it's not going to drive any results on them. So number one, color optimizations. For the love of God, a blue is a blue is a blue. As long as it's accessible and as long as it's bright enough, off you go. You do not need to test the shades of blue or test it against green or so on. Pick a color, move on. Please don't spend time on it. That's a early 2000s tactic. It doesn't work anymore. We passed that in technology sector. Number two, third party signups. A lot of times we think, Oh, we're going get so much more acquisition if we add Google Auth, or if we add Facebook Auth, or Slack Auth, or Microsoft Auth, whichever Auth you want to add to it. In some cases, it's very important to have third party auth. So, for example, if you're a developer product, please have a GitHub auth. It's kind of unnecessary. The developers already have the account there, have them connect with it as opposed to create a new one. However, if you're a productivity product, email is fine for the longest time, Gmail is nice, it's not gonna drive more acquisition for you. You're just gonna do a mix shift into more people using it, it's not gonna create incrementality, it's not gonna improve your activation, it's not gonna improve your retention. It's not a growth tactic, it's part of your customer experience that you want to invest into. Number three, on the fire round. One email wonders. We stress too much about this one email that we're going to send to this one customer group, and we're like, Oh my gosh, how much lift is it gonna cost? It will never cause any lift. It will never work as a one off email. Too few people open it, on average 25% open rates. At best, you'll get 40 to 50% open rate. Too few people click it. One email will never do anything. If you're gonna go into email, please think about it as a series about communication, about how it interacts with product communication. It's a whole strategy. It's a whole thing. Please don't stress about one off email. Never test a change in one email. It'll never work. You're gonna have to do the whole goddamn thing and see how it's gonna do, not just one email. Just never, gonna work.
01:13:03 · Lenny Rachitsky
[01:13:03] S1: I'm definitely guilty of this one. That is a really good one for sure. But it's so obvious in hindsight, so I love that you shared that.