Unsupervised Learning with Jacob Effron
00:03:46 · Erik Bernhardsson, Patrick Chase
[00:03:46] S1: I I think a lot of this actually boils down to reliability. I I mean, first of all, it's like, you know, differences in like disk and networking, and of course CPU and memory and things like that. Like all these things vary and region or like location, like they're all over the world. Like we have GPUs in The US, of course, like all over The US, but we also have GPUs in like Indonesia, like, all over the year... All over Europe, etcetera. So so location is is one of those big factors. And then... But but also reliability. And and and that's another thing we do exceptionally well is, like, we aggregate all this capacity. And then when Neo Clouds die or, like, when they have failures, which they often have because some of these are, like, former crypto miners who haven't really thought about, you know, running a twenty four seven operation, we can just take it out of rotation or, like, when GPUs fall off the bus or whatever. Like, we... In many cases, we just file a ticket automatically. Someone goes and looks at it. [00:04:38] S2: Have you found the data providers, the companies that provide pricing for GPU and that sort of things, have you found them to be helpful?
00:04:38 · Patrick Chase, Erik Bernhardsson, Jacob Effron
[00:04:38] S2: Have you found the data providers, the companies that provide pricing for GPU and that sort of things, have you found them to be helpful? [00:04:45] S1: I I think moderately helpful. I I I think the pricing less so than just their ability to connect with a lot of providers. I I don't fully trust the pricing today, but I think these... Some of these providers are actually very good at, like, know, knowing all the neo clouds, and we work with them sometimes to get connections with with cloud providers. The pricing, I think, over time will get more accurate. Like, our internal pricing does not match what we see publicly when we buy capacity. But I think over time, hopefully, like, we will have more efficient, like, external indices. I I think that would actually be very useful. [00:05:16] S0: You've been on this, like, rocket ship journey these past years, and so lots of things we'd love to dig into for our listeners on the on the business side. Maybe starting... You know, I think one of the interesting things about the history of modal is, you know, for a while it felt like you, you know, publicly said, I think on on your first unsupervised learning episode that you were less focused, you know, on these LLM inference calls, specifically like a simple API, you know, call for for a model. And, you know, you kind of recently released a product there. Maybe talk a little bit about the journey and and kind of that initial reticence and what made you decide that like now is the right time to really, you know, double down on this.
00:05:16 · Jacob Effron, Erik Bernhardsson, Patrick Chase
[00:05:16] S0: You've been on this, like, rocket ship journey these past years, and so lots of things we'd love to dig into for our listeners on the on the business side. Maybe starting... You know, I think one of the interesting things about the history of modal is, you know, for a while it felt like you, you know, publicly said, I think on on your first unsupervised learning episode that you were less focused, you know, on these LLM inference calls, specifically like a simple API, you know, call for for a model. And, you know, you kind of recently released a product there. Maybe talk a little bit about the journey and and kind of that initial reticence and what made you decide that like now is the right time to really, you know, double down on this. [00:05:53] S1: Yeah. So we started building this in in late twenty twenty one basically or early somewhere around that time. And and and and the initial product was this serverless function. And we thought it'd be really useful for a wide range of stuff around, like, data AI machine learning, broadly speaking. Kind of, like, looked at a lot of different use cases and... But then, like, ChatGPT came late twenty twenty two, and StableEfusion also came around the the the same time. [00:06:18] S2: Right. Right.
00:06:18 · Erik Bernhardsson
[00:06:18] S1: It turned out, like, we were, like, super well positioned to to host a number of StableEfusion applications. So that provide... That was, like, the first year of growth. It was, like, all StableEfusion, and then we started, like, expanding into, like, adjacent use cases, like audio and video and stuff like that. And it was, like, early teams that, you know, they they were deploying JAN AI and friends. They didn't wanna deal with data lang infrastructure. They wanted to move fast. Like, we we we started aggregating a lot of different GPUs across a lot of different clouds even back then. We always knew we wanted to be multi product, but we started with the serverless, like this sort of compute primitives. We felt like it was so general purpose. And we focused a lot on like kind of what we... What I think was like high code use cases, like focus on like machine learning engineers and researchers who who wanna write code and and focus a lot on, like, being kinda low in the lower in the stack. Right? And pretty early, we built also a storage layer, a file system to to store data in in in particular container images. Then we launched a Sandbox product in 2023. And and and the the reason the why we launched it back then, we started seeing people, like, messing around with LMs to generate code. We figured actually, like, a lot of stuff we built on the infrastructure side is, like, really well suited to run that code, and and they're gonna need a place to run that in an isolated way. Didn't immediately... It wasn't like an overnight success, but, like, basically, 2025, it kinda blew up because I think models got really good at writing code back then. So about a year ago. So that was sort of our venture into, like, kind of becoming like a multiproduct vendor. Like, starting about twelve months ago, like, Sandboxes has been on a tier and and grown, like, I don't know, a thousand x almost or something like that. It's about a third of our revenue right now. Standalone is a 300,000,000 run rate business, sandboxes. So that's been incredible in terms of, you know, becoming a multi product vendor. We just launched also a training product. So sort of, you know, building... Trying to lay out like all the different building blocks and, you know, what does that suite of tool looks like, and what are the toolboxes in this like new world of like AI applications?
00:08:13 · Erik Bernhardsson
Like they're all gonna need a bunch of new primitives, and like we want to offer all those primitives, and and what does that what does that look like? So so LM inference was something... We were a little bit on the sidelines for a while until actually, I would say the DeepSeek moment. So that was about a year and a half ago, like early twenty twenty five when when DeepSeek came out, which was just like the first Chinese open source model. Was, like, actually really good. And then I was like, fuck it. Like, we gotta get in here. And, like, we we started building up a team to do a lot of inference optimization. Model has always been really good at LM inference, like... But we didn't necessarily have a product for it. We... In fact, we didn't even have an inference product until very recently. We're just like really good at hosting inference, including LM inference, but it was like kind of custom, and a lot of the teams that want to run LM inference a year or two ago, like they had to deploy it themselves or or like we would work with them. But we started seeing a lot of demand with the deep seek moment over the last eighteen months, started seeing a lot of growth. LM inferences now, but a third of our revenue too. So we've seen incredible growth with that. And a few months ago, we started... We we shipped a number of LLM specific features because we think, you know, we kinda laid this like foundational layers, the storage, compute, the serverless functions, the file system, etcetera. I think there's a lot of opportunity to move up in the stacks. We started moving up in the stack now and and launched dedicated inference product, so you can get like a single tenant models, like all the state of the art Chinese models or or a bunch of US ones too, like one single click or or CLI command you can just deploy on modal, as well as much as shared endpoints. Like, you can just call models off the shelf without having to deploy them fully token based pricing. And so I I I think those are just, you know, the first out of, like, many more products to come, like, slightly higher up in the in the layer. We think there's lot of opportunity. Because we're sitting on this, like, very solid foundation in the in the compute and the storage.
00:10:02 · Patrick Chase
[00:10:02] S2: Maybe, like, you guys will will start at the bottom of the stack and and work our way up. But going back in time, you touched on it, but you guys invested in some really hardcore technology in the early days. You built your own file system, container runtime, those types of things. How does that show up in the product today when the experienced developers can have?