The Frontier with Brendan Foody
00:06:14 · Brendan Foody
[00:06:14] S1: Well, and I definitely feel like in the arc of the industry over the last few years, it felt like initially there was a big push towards custom models and efficiency. And then the frontier models improved so quickly that all of the inference really went towards the frontier models, I think, for most of the last 12 months. But it feels like as costs are ballooning so dramatically, we will naturally converge more towards more and more of those costs, going towards custom, smaller, more efficient models. And a lot of that uh work and the continuation of it that you're doing at Databricks will be really, really important and impactful as that transition accelerates. Um, so maybe share more about uh the acquisition and how you all got in touch with the Databricks team, what that was like and what you're now working on and spending your time on at Databricks.
00:07:00 · Jonathan Frankle, Brendan Foody
[00:07:00] S0: Yeah. I mean, Mosaic went on this crazy journey where we were trying to train custom vision models for a while because who didn't want ResNet 50 as an image classifier? That was, you know, such a hot thing back then. Um, and who didn't want a segmentation model that was, you know, a custom, like, I don't even remember the name of the model at this point, or an object detection model, like a mascar CNN, like all these things that sound like they're from a history book at this point because they're three whole years old or five years old. And we were doing a lot of that. And then someone on my team said to me at some point, maybe people are gonna want more than eight GPUs, and they're gonna want to train these things, these large language models, something that's like maybe a billion parameters. It's like, Obby, nobody's gonna need more than eight GPUs, and nobody's ever gonna need to train, you know, a model with a billion parameters. 25 million is already a lot. Like you're 25 million. [00:07:46] S1: Wow. [00:07:47] S0: Yeah, those. I mean, this is again, I've this was 2022. [00:07:50] S1: Yeah. [00:07:51] S0: This wasn't that long ago. This was four years. [00:07:53] S1: Yeah.
00:07:53 · Jonathan Frankle
[00:07:53] S0: And thank God that Obby didn't listen to me. Um, and one of my favorite things about running a research team is the people who fail to listen to me do something, get it to the point where there's some value there, and then I'm like, okay, we'll pivot the whole team. He went and built out a multi-node training stack on his own, built out the initial version of our what became our LLM pre-training stack on its own. And this was in like summer of 22. So pre-Chat GPT, he just he felt like something was coming. And it ended up working well enough that we pivoted the whole team, and that became our product, custom LLMs. Um, and you know, less than a year later, we had trained the first open source reproduction of Llama, which again sounds really quaint. 7 billion parameter model on 1 trillion tokens. Really hard thing to do back then. Now an intern project for many people. But you know, back then it was a full team ordeal on a really scary, this is gonna be sound like a huge cluster, 256 A100s. That was so expensive and so scary. Why did we get that? Why are we burning all this cash? Legend speaks that Naveen Rao and Ali God seemed that at the first Cerebral Valley. I have no idea if this is actually true or if this is just what legend speaks of, because these things tend to accelerate, but they met, they hit it off and became part of Databricks and got to go big. Um, it turns out that, you know, when you're a custom training company, you really need data. And when you're a data company, you really need custom training. And to Databricks' credit, the annual conference for Databricks, which is why I'm even out here, is called the Data and AI Summit. And it sounds like a very 2026 pilled name. I think the name was changed from the Spark Summit to the Data and AI Summit in like 2018. There was, I think the Databricks founders understood at that time that AI is what you do with data. Yeah. And, you know, AI doesn't work without data. Now we call these environments and agents, but the two go together.
00:09:36 · Brendan Foody
[00:09:36] S1: Talk a little bit more about what your data day looks like, what key initiatives that Databricks you're owning, especially within the enterprise, um, as we think through what are the bottlenecks to AI becoming useful and the economy and enterprise deployment.
00:09:50 · Jonathan Frankle
[00:09:50] S0: Yeah, I feel like running a research team is such a weird job, especially if you do it at a company where you're really thinking about the customers. On the one hand, my day job is to like I frame it lots of different ways to the team depending on my mood and what we're trying to do and what I want to sell them on. One version is that we're the risk budget for the company. If you're the CEO of a company like Databricks or, you know, any reasonably sized company, you need someone who's willing to go out there and just take a lot of risks, do the crazy things. And, you know, whenever you look at just your team, you say, I need a portfolio of risks, I need to like do some conservative things so I deliver value and do some high-risk things, but we aren't a portfolio as a team. We are literally the high risk budget for a massive company. And so if we're not taking risks and trying new things, we're not serving our purpose within the organization. And so our job is to fail. And that means failing on some research projects, trying to do some reinforcement learning at scale on a task that's too hard and failing at it. That means trying to develop some new products that just don't work because the technology is not there yet. And then trying again next year and the technology is there, and this has happened a couple of times where we've taken two or three cracks at the same problem over the course of two or three years. And like there's a thing this year that started working that someone had been trying, I think since 2022. And finally the models got good enough that this project actually started working in a really big way and has been game-changing for the team and the product. Well, the other way I look at this is that my job is to spend a lot of time with our customers. Just hang out with a lot of enterprises that have AI needs, figure out what problems they're getting stuck on, and build entirely new technology to solve those problems. It's not just giving them a product or an interface or a way to do a thing like the plumbing, it is inventing a new way of solving that problem through our expertise in machine learning. And that's a lot of fun. Like there aren't that many research teams in the world where you can say, I'm a scientist and I spend, you know, a quarter of my time with customers.
00:11:35 · Jonathan Frankle, Brendan Foody
The way I frame it to our team at least is the customers are the evals. Like the one true measure of our success in the world is is there somebody at one of our customers who actually adopted it and actually liked it and is paying to use it and maybe got promoted for using it and we made them look really good? That's the eval. Like Suitebench is great, Suitebench does not matter at all. Birdbench is great, Birdbench doesn't matter at all. Are people using our models? [00:11:57] S1: Yeah. And you have a very unique vantage point in both leading a research team as well as spending a lot of time with customers and understanding what things are actually working about the models in production and what things aren't. What do you see as the largest bottleneck to 10xing enterprise adoption and making these models actually useful on the true eval of the customer?