Unsupervised Learning with Jacob Effron
00:02:27 · Erik Bernhardsson, Jacob Effron
[00:02:27] S1: It... It's it's a lot of late night WhatsApp groups and, like, it's pretty unorganized. Like, I mean, I went to GTC last year or so earlier this year in San Jose, the NVIDIA, big conference they're doing. And it was like an open air drug market, but, like, GPUs. Like, it's just like walking around, like, talking to people, trying to get GPUs. And it it... You know, meeting people in hotel lobbies and, you know, exchanging numbers and, you know... I mean, at some point, we... I think we got connected with someone who had GPUs because I think our VP of sales cousin was golfing with someone, and and we got connected. Like, there's many such cases. Like like, it's very far from an efficient market. I mean, some people are working on this, and I think over time, it'll get a little bit more efficient. But it... It's very heterogeneous. Like, compute comes in many sort of different shapes or forms. So I also feel like structurally, like, it will not... Like, it will be a little bit, like, sort of unwieldy for a long time. Like, it's just not something you can, like, sort of buy and sell on an exchange because it's not like a fungible contract. Yeah. [00:03:29] S0: Mean, the key differences being, like, I guess, the the kind of actual powered shell or size of the cluster and, like, the chips themselves or... Because that... You know, to your point, there's lot of people that have been trying to build, hey. You know, compute is this fungible single asset that you could build a marketplace around. What what are some of the kind of the key differences you observe?
00:03:29 · Jacob Effron, Erik Bernhardsson
[00:03:29] S0: Mean, the key differences being, like, I guess, the the kind of actual powered shell or size of the cluster and, like, the chips themselves or... Because that... You know, to your point, there's lot of people that have been trying to build, hey. You know, compute is this fungible single asset that you could build a marketplace around. What what are some of the kind of the key differences you observe? [00:03:46] S1: I I think a lot of this actually boils down to reliability. I I mean, first of all, it's like, you know, differences in like disk and networking, and of course CPU and memory and things like that. Like all these things vary and region or like location, like they're all over the world. Like we have GPUs in The US, of course, like all over The US, but we also have GPUs in like Indonesia, like, all over the year... All over Europe, etcetera. So so location is is one of those big factors. And then... But but also reliability. And and and that's another thing we do exceptionally well is, like, we aggregate all this capacity. And then when Neo Clouds die or, like, when they have failures, which they often have because some of these are, like, former crypto miners who haven't really thought about, you know, running a twenty four seven operation, we can just take it out of rotation or, like, when GPUs fall off the bus or whatever. Like, we... In many cases, we just file a ticket automatically. Someone goes and looks at it.
00:03:46 · Erik Bernhardsson, Patrick Chase
[00:03:46] S1: I I think a lot of this actually boils down to reliability. I I mean, first of all, it's like, you know, differences in like disk and networking, and of course CPU and memory and things like that. Like all these things vary and region or like location, like they're all over the world. Like we have GPUs in The US, of course, like all over The US, but we also have GPUs in like Indonesia, like, all over the year... All over Europe, etcetera. So so location is is one of those big factors. And then... But but also reliability. And and and that's another thing we do exceptionally well is, like, we aggregate all this capacity. And then when Neo Clouds die or, like, when they have failures, which they often have because some of these are, like, former crypto miners who haven't really thought about, you know, running a twenty four seven operation, we can just take it out of rotation or, like, when GPUs fall off the bus or whatever. Like, we... In many cases, we just file a ticket automatically. Someone goes and looks at it. [00:04:38] S2: Have you found the data providers, the companies that provide pricing for GPU and that sort of things, have you found them to be helpful?
00:04:38 · Patrick Chase, Erik Bernhardsson, Jacob Effron
[00:04:38] S2: Have you found the data providers, the companies that provide pricing for GPU and that sort of things, have you found them to be helpful? [00:04:45] S1: I I think moderately helpful. I I I think the pricing less so than just their ability to connect with a lot of providers. I I don't fully trust the pricing today, but I think these... Some of these providers are actually very good at, like, know, knowing all the neo clouds, and we work with them sometimes to get connections with with cloud providers. The pricing, I think, over time will get more accurate. Like, our internal pricing does not match what we see publicly when we buy capacity. But I think over time, hopefully, like, we will have more efficient, like, external indices. I I think that would actually be very useful. [00:05:16] S0: You've been on this, like, rocket ship journey these past years, and so lots of things we'd love to dig into for our listeners on the on the business side. Maybe starting... You know, I think one of the interesting things about the history of modal is, you know, for a while it felt like you, you know, publicly said, I think on on your first unsupervised learning episode that you were less focused, you know, on these LLM inference calls, specifically like a simple API, you know, call for for a model. And, you know, you kind of recently released a product there. Maybe talk a little bit about the journey and and kind of that initial reticence and what made you decide that like now is the right time to really, you know, double down on this.
00:05:16 · Jacob Effron, Erik Bernhardsson, Patrick Chase
[00:05:16] S0: You've been on this, like, rocket ship journey these past years, and so lots of things we'd love to dig into for our listeners on the on the business side. Maybe starting... You know, I think one of the interesting things about the history of modal is, you know, for a while it felt like you, you know, publicly said, I think on on your first unsupervised learning episode that you were less focused, you know, on these LLM inference calls, specifically like a simple API, you know, call for for a model. And, you know, you kind of recently released a product there. Maybe talk a little bit about the journey and and kind of that initial reticence and what made you decide that like now is the right time to really, you know, double down on this. [00:05:53] S1: Yeah. So we started building this in in late twenty twenty one basically or early somewhere around that time. And and and and the initial product was this serverless function. And we thought it'd be really useful for a wide range of stuff around, like, data AI machine learning, broadly speaking. Kind of, like, looked at a lot of different use cases and... But then, like, ChatGPT came late twenty twenty two, and StableEfusion also came around the the the same time. [00:06:18] S2: Right. Right.
00:06:18 · Erik Bernhardsson
[00:06:18] S1: It turned out, like, we were, like, super well positioned to to host a number of StableEfusion applications. So that provide... That was, like, the first year of growth. It was, like, all StableEfusion, and then we started, like, expanding into, like, adjacent use cases, like audio and video and stuff like that. And it was, like, early teams that, you know, they they were deploying JAN AI and friends. They didn't wanna deal with data lang infrastructure. They wanted to move fast. Like, we we we started aggregating a lot of different GPUs across a lot of different clouds even back then. We always knew we wanted to be multi product, but we started with the serverless, like this sort of compute primitives. We felt like it was so general purpose. And we focused a lot on like kind of what we... What I think was like high code use cases, like focus on like machine learning engineers and researchers who who wanna write code and and focus a lot on, like, being kinda low in the lower in the stack. Right? And pretty early, we built also a storage layer, a file system to to store data in in in particular container images. Then we launched a Sandbox product in 2023. And and and the the reason the why we launched it back then, we started seeing people, like, messing around with LMs to generate code. We figured actually, like, a lot of stuff we built on the infrastructure side is, like, really well suited to run that code, and and they're gonna need a place to run that in an isolated way. Didn't immediately... It wasn't like an overnight success, but, like, basically, 2025, it kinda blew up because I think models got really good at writing code back then. So about a year ago. So that was sort of our venture into, like, kind of becoming like a multiproduct vendor. Like, starting about twelve months ago, like, Sandboxes has been on a tier and and grown, like, I don't know, a thousand x almost or something like that. It's about a third of our revenue right now. Standalone is a 300,000,000 run rate business, sandboxes. So that's been incredible in terms of, you know, becoming a multi product vendor. We just launched also a training product. So sort of, you know, building... Trying to lay out like all the different building blocks and, you know, what does that suite of tool looks like, and what are the toolboxes in this like new world of like AI applications?