Unsupervised Learning with Jacob Effron
00:06:18 · Erik Bernhardsson
[00:06:18] S1: It turned out, like, we were, like, super well positioned to to host a number of StableEfusion applications. So that provide... That was, like, the first year of growth. It was, like, all StableEfusion, and then we started, like, expanding into, like, adjacent use cases, like audio and video and stuff like that. And it was, like, early teams that, you know, they they were deploying JAN AI and friends. They didn't wanna deal with data lang infrastructure. They wanted to move fast. Like, we we we started aggregating a lot of different GPUs across a lot of different clouds even back then. We always knew we wanted to be multi product, but we started with the serverless, like this sort of compute primitives. We felt like it was so general purpose. And we focused a lot on like kind of what we... What I think was like high code use cases, like focus on like machine learning engineers and researchers who who wanna write code and and focus a lot on, like, being kinda low in the lower in the stack. Right? And pretty early, we built also a storage layer, a file system to to store data in in in particular container images. Then we launched a Sandbox product in 2023. And and and the the reason the why we launched it back then, we started seeing people, like, messing around with LMs to generate code. We figured actually, like, a lot of stuff we built on the infrastructure side is, like, really well suited to run that code, and and they're gonna need a place to run that in an isolated way. Didn't immediately... It wasn't like an overnight success, but, like, basically, 2025, it kinda blew up because I think models got really good at writing code back then. So about a year ago. So that was sort of our venture into, like, kind of becoming like a multiproduct vendor. Like, starting about twelve months ago, like, Sandboxes has been on a tier and and grown, like, I don't know, a thousand x almost or something like that. It's about a third of our revenue right now. Standalone is a 300,000,000 run rate business, sandboxes. So that's been incredible in terms of, you know, becoming a multi product vendor. We just launched also a training product. So sort of, you know, building... Trying to lay out like all the different building blocks and, you know, what does that suite of tool looks like, and what are the toolboxes in this like new world of like AI applications?
00:08:13 · Erik Bernhardsson
Like they're all gonna need a bunch of new primitives, and like we want to offer all those primitives, and and what does that what does that look like? So so LM inference was something... We were a little bit on the sidelines for a while until actually, I would say the DeepSeek moment. So that was about a year and a half ago, like early twenty twenty five when when DeepSeek came out, which was just like the first Chinese open source model. Was, like, actually really good. And then I was like, fuck it. Like, we gotta get in here. And, like, we we started building up a team to do a lot of inference optimization. Model has always been really good at LM inference, like... But we didn't necessarily have a product for it. We... In fact, we didn't even have an inference product until very recently. We're just like really good at hosting inference, including LM inference, but it was like kind of custom, and a lot of the teams that want to run LM inference a year or two ago, like they had to deploy it themselves or or like we would work with them. But we started seeing a lot of demand with the deep seek moment over the last eighteen months, started seeing a lot of growth. LM inferences now, but a third of our revenue too. So we've seen incredible growth with that. And a few months ago, we started... We we shipped a number of LLM specific features because we think, you know, we kinda laid this like foundational layers, the storage, compute, the serverless functions, the file system, etcetera. I think there's a lot of opportunity to move up in the stacks. We started moving up in the stack now and and launched dedicated inference product, so you can get like a single tenant models, like all the state of the art Chinese models or or a bunch of US ones too, like one single click or or CLI command you can just deploy on modal, as well as much as shared endpoints. Like, you can just call models off the shelf without having to deploy them fully token based pricing. And so I I I think those are just, you know, the first out of, like, many more products to come, like, slightly higher up in the in the layer. We think there's lot of opportunity. Because we're sitting on this, like, very solid foundation in the in the compute and the storage.
00:10:02 · Patrick Chase
[00:10:02] S2: Maybe, like, you guys will will start at the bottom of the stack and and work our way up. But going back in time, you touched on it, but you guys invested in some really hardcore technology in the early days. You built your own file system, container runtime, those types of things. How does that show up in the product today when the experienced developers can have?
00:10:23 · Erik Bernhardsson
[00:10:23] S1: Yeah. I think a massive source of alpha in this market is just, like, not being afraid to do hard things. So we went all in when we realized, like, in order to develop... To to to... In order to to to deliver on the developer experience we wanted to have in the very early days, we realized, like, kinda have to build our own file system to do container images really fast. So we we started building that in the very early days, and and that become, like, over the next few years, you know, still is, like, a very massive source of of alpha. It... It's been it's been a longer journey for sure. Like, we've had to go a much longer route, like building our own file system, building our own container runtime, our own schedule. Like, we don't use Kubernetes and Docker. Like, we we we use, like, very little off the shelf software. We use gVisor, which is a fantastic Google product for isolation. Recently started using also KVM for for certain tasks. But but like decided pretty pretty early on that like a lot of the stuff we're gonna have to build ourselves. And and I'll let us do some really cool stuff. I I think, know, one of the best examples of where that shows up is we can do cold starts extremely fast. Turned out like the the initial... The developer experience that we wanted, very fast feedback loop when you're launching stuff in the cloud. That is also very helpful when you do building auto scaling applications that need to scale up and down very fast. So so we we... So first of all, the the loading the container is very fast, but over time, we also launched a a really... A couple of really cool features around CPU snapshotting and GPU snapshotting. So we can, like, very fast load data from the disk or from network straight into the GPU and and fire up, like, even these, like, large models in a few seconds, which is really cool. And and over the last six months, we we built this incredibly complex system that basically uses RDMA over peer to peer between the nodes to copy data very fast. So I I I think a lot of this stuff is, like, kind of, you know, accumulating and and really starting to pay off. Like, we we built some crazy stuff, and and now that's, you know, we're we're really kind of, you know, reaping the devided... Dividends of of of that... Those sort of investments that we made for many, many years.
00:12:25 · Erik Bernhardsson, Patrick Chase
It was a longer route, but, you know, now we... I think we have some incredibly unique technology that's very hard [00:12:31] S2: to replicate. And and then kind of moving up to to inference, what do you view as... You guys have made, you know, tons of investments there as well and and optimizations. What do you view as the unsolved problems in in the inference world or in the inference stack?
00:12:45 · Erik Bernhardsson
[00:12:45] S1: I I think we're still pretty early with inference. I I think, you know, a lot of stuff is like... It's it's like one of those, like, very empirical sciences. Like, I think people are trying different things and seeing what works. I mean I mean, over time, like, people have kinda figured out a lot of stuff. You know? Take something like disaggregate it prefilled. Right? Like, it's a very nonobvious idea. Like, you you sort of, you know, do the prefill on on on a set of notes and then you copy the data over RDMA and do the decode and kinda swap. There's some, like, very non obvious things, but, we've invested a lot in those types of things. The KV caching, there's a lot of complexity around, like, KV caching. Of course, like running more efficient kernels, that's something people do a lot. There's a lot of these things that, you know, we we have a team of, I think, 20, 30 people right now who are extremely focused on just like LM performance optimization. And, you know, it takes a lot of work. You have to stay up with with research, you have to read papers, you have to look at all the software coming out all the time and try try and and see what works, what doesn't work. It's still kind of an empirical science. Over time, I think, you know, patterns will emerge. I think also the models themselves will change very quickly. Like, now we have, like, mixture of experts and more complex things over time. Like, who knows? Like, maybe we'll swap out the transformers for something else and, you know, and that would change how, like, kvcaching works. Like, there's all these things, like, that could, you know, change overnight if someone figures out, like, a very different paradigm for all these things. And maybe they will, maybe they won't. I don't know. There's also obviously a lot of stuff around quantization and and and other things. So so, yeah, it's it's it's it's a wild world out there. But but, yeah, we're we're we're we're playing around with all these things.