Databricks: From Data to Decisions - [Business Breakdowns, EP.238]

· Alan Tu, Matt Reustle

[00:00:00] S1: I think it's a very unique part of the Databricks story that to this day, the culture and the DNA of the organization, you can trace a lot of the decisions back to this founding story. And so, it's seven founders that came out of Berkeley. They were all working in what's called the AMPLab at Berkeley around the 2009 time period. And if you roll back the clock to that time period, that was actually the beginning stages of the cloud. And what Ali, the CEO and one of the co-founders, would say is that the seven of them were working in this building together doing research. And actually, on the floor below was another team that was putting out some of the very early research around the data center being the next computer, basically the early concept around cloud computing. And so, you had in one part of the building a lot of innovation around just, cloud computing at the hardware layer, and then you had Ali and his colleagues that were thinking about, "What are some of the software opportunities in cloud?" They were so close to the research around the early ages. And unlike with AI today, where there's been a very clear recognition that AI is going to be a big deal, back then the idea of cloud was still somewhat controversial. [00:00:00] S0: Hard to imagine, but yes.

· Alan Tu

[00:00:00] S1: Totally. This group of folks that were rooted in research gained conviction around the idea of cloud. So then they thought about, "Well, what are the problems that we should work on within cloud?" They actually thought of a few different ideas, but where they ended up was around this idea that data is going to be a really big problem, data at scale. And that if you think about all the different use cases, going back to the beginning of the conversation, there's infinite number of applications around data. The other interesting thing that Ali will say is that during that time period, it was also when Twitter was becoming big and Airbnb was becoming big, Facebook was still becoming big. It was actually a very positive time in technology, and so there was a lot of optimism around entrepreneurship and starting startups. And so, that also fed into the energy of the group. The thought was, well, one of the co-founders was actually the creator of Apache Spark, and they believed that data was going to be an important thing, and so how do we create a business around that? And the other piece that they thought of was using open source as another key bet. And that really was very aligned with this idea of coming from academia and research. When you roll back the clock, there was really three major bets that they had a view on. It was: cloud was going to be big, data was going to be big, and open source was going to be a good way to build a business. In hindsight, it turned out all three of those bets were very good bets. But the idea that at the time was less clear, but because of that environment where there was a lot of optimism around technology, I think all of that coalesced together to be the beginnings of Databricks.

· Matt Reustle, Alan Tu

[00:00:00] S0: On the point of connecting cloud to data, is it fair to assume that by transitioning to cloud, there would actually be more capacity for data to be stored or to be used? Like, is that connected in the sense that prior to cloud, the data capabilities might have been constrained? [00:00:00] S1: Yeah. One of the paradigm shifts of cloud was generally this concept of scale-out architecture, which basically allowed the ability to use more commodity hardware to be able to address larger and larger amounts of both compute and storage and data processing in this case. So, that was an important underlying trend that enabled this proliferation and this idea of just data explosion that I think you're touching on. When you look at what Apache Spark was, it was leveraging this concept of distributed compute and applying that to data processing. That was very important.

· Matt Reustle

[00:00:00] S0: The reason I asked is because it's always interesting to think about second-order impacts or a certain market enabling another market on the back of it, particularly with AI today, everyone looking for second-order impacts. So, it's interesting to hear how they're connected. And to your point, those three ideas, those three bets they made, certainly together compounded in many ways. On that point of the open source, there are all different ways to approach open source, and we have walled gardens versus open source, and there's the very famous Apple versus Windows example, or Apple versus Microsoft. Can you talk about the commercialization and that how that played into things? Because I know that was a major stepping stone for the business in terms of evolving from this tool that gained usership. Oftentimes, offering things for free is a great way to get people in; it has to be great, so it certainly was. But that feels like such a meaningful point in time for this business, and I'm curious how you would describe what happened and how they really turned this into a great product and evolved it into a great business.

· Alan Tu

[00:00:00] S1: There are a lot of examples of successful open source projects, and that's one of the magical things of open source is that it does enable a level of adoption. In the consumer world, we talk about capturing magic in a bottle. In the enterprise world and with technology, open source is a really good way of getting just massive amounts of awareness, mind share, and adoption. There are a lot of examples of successful open source technologies, but there are actually not very many examples of successful businesses that have been built on top of open source. And a lot has been made about, Red Hat was one of the first companies that built a business on top of Linux, but the reality is that it's actually one of the hardest things to do. You have to hit two home runs. This is actually one of the ways that Ali explains it: is you need to hit the first home run, which is to develop an open source technology that gets its mainstream adoption. But what a lot of folks don't think about is the second home run, which is, how do you actually build a business on top of it? And part of the problem is that the open source technology ends up becoming one of the business's main competitors because anyone can get the product for free, anyone, including your customers, but also including competitors. They can also leverage that technology, and actually, those competitors that have more distribution, more customer relationships, can actually do a better job of monetizing that open source technology. This is where again, that coming from academia, I think really helped inform the strategy for Databricks, which is the benefit of not being immersed in the commercial market is that you don't have the pre-existing notions of what you should or should not do when it comes to building a business on top of open source. Because again, the biggest example before that was Red Hat, which basically provided services and support for Linux, but that was the main thing. But because the Databricks team wasn't really aware of the precedence, they thought about things in a very first principles way. And when you think about this challenge of, how do you monetize something where there's a free alternative? It's actually a simple answer, and the answer is: you need to create a better product that is worth paying for.

· Matt Reustle, Alan Tu

[00:00:00] S0: Simple answer, maybe not simple execution. [00:00:00] S1: Totally. And part of the reason why it's not a simple execution is that if you've built your brand and gotten a lot of positive feedback for successfully creating the open source technology, it can be weird to then say, "I'm actually going to create a competing product, and I'm not going to put all of the bells and whistles into that open source technology." And for a lot of folks, that creates a lot of tension. You've got a lot of folks in the community that are like, "How could you do that?" There's almost a feeling of betrayal. When you're successful with open source, it can almost be a curse because you become very popular, you're viewed by technologists as someone who's brought this great thing into the world, and then all of a sudden, you need to be willing to be a villain. I think a lot of people have trouble making that jump. But I think again, in the case of Databricks, they just very much realized that there's no way we can compete unless we have differentiation. So, creating that differentiation was just a very important concept. And so, one of the things they did with Databricks, the company, and we can also talk about how they chose the name Databricks, was that they created a new implementation of Spark that was completely proprietary that had a lot more performance and things that enterprises want: reliability, scalability, etc. And they were just very unabashed about the fact that if you want to use this new implementation, you would have to pay for it.