Unsupervised Learning with Jacob Effron
00:02:18 · Jacob Effron, Erik Bernhardsson
[00:02:18] S0: How much of, like, an an organized marketplaces compute these days versus, like, the Wild West, like, random people that somehow have capacity and, you know, the late night text messages? [00:02:27] S1: It... It's it's a lot of late night WhatsApp groups and, like, it's pretty unorganized. Like, I mean, I went to GTC last year or so earlier this year in San Jose, the NVIDIA, big conference they're doing. And it was like an open air drug market, but, like, GPUs. Like, it's just like walking around, like, talking to people, trying to get GPUs. And it it... You know, meeting people in hotel lobbies and, you know, exchanging numbers and, you know... I mean, at some point, we... I think we got connected with someone who had GPUs because I think our VP of sales cousin was golfing with someone, and and we got connected. Like, there's many such cases. Like like, it's very far from an efficient market. I mean, some people are working on this, and I think over time, it'll get a little bit more efficient. But it... It's very heterogeneous. Like, compute comes in many sort of different shapes or forms. So I also feel like structurally, like, it will not... Like, it will be a little bit, like, sort of unwieldy for a long time. Like, it's just not something you can, like, sort of buy and sell on an exchange because it's not like a fungible contract. Yeah.
00:02:27 · Erik Bernhardsson, Jacob Effron
[00:02:27] S1: It... It's it's a lot of late night WhatsApp groups and, like, it's pretty unorganized. Like, I mean, I went to GTC last year or so earlier this year in San Jose, the NVIDIA, big conference they're doing. And it was like an open air drug market, but, like, GPUs. Like, it's just like walking around, like, talking to people, trying to get GPUs. And it it... You know, meeting people in hotel lobbies and, you know, exchanging numbers and, you know... I mean, at some point, we... I think we got connected with someone who had GPUs because I think our VP of sales cousin was golfing with someone, and and we got connected. Like, there's many such cases. Like like, it's very far from an efficient market. I mean, some people are working on this, and I think over time, it'll get a little bit more efficient. But it... It's very heterogeneous. Like, compute comes in many sort of different shapes or forms. So I also feel like structurally, like, it will not... Like, it will be a little bit, like, sort of unwieldy for a long time. Like, it's just not something you can, like, sort of buy and sell on an exchange because it's not like a fungible contract. Yeah. [00:03:29] S0: Mean, the key differences being, like, I guess, the the kind of actual powered shell or size of the cluster and, like, the chips themselves or... Because that... You know, to your point, there's lot of people that have been trying to build, hey. You know, compute is this fungible single asset that you could build a marketplace around. What what are some of the kind of the key differences you observe?
00:03:29 · Jacob Effron, Erik Bernhardsson
[00:03:29] S0: Mean, the key differences being, like, I guess, the the kind of actual powered shell or size of the cluster and, like, the chips themselves or... Because that... You know, to your point, there's lot of people that have been trying to build, hey. You know, compute is this fungible single asset that you could build a marketplace around. What what are some of the kind of the key differences you observe? [00:03:46] S1: I I think a lot of this actually boils down to reliability. I I mean, first of all, it's like, you know, differences in like disk and networking, and of course CPU and memory and things like that. Like all these things vary and region or like location, like they're all over the world. Like we have GPUs in The US, of course, like all over The US, but we also have GPUs in like Indonesia, like, all over the year... All over Europe, etcetera. So so location is is one of those big factors. And then... But but also reliability. And and and that's another thing we do exceptionally well is, like, we aggregate all this capacity. And then when Neo Clouds die or, like, when they have failures, which they often have because some of these are, like, former crypto miners who haven't really thought about, you know, running a twenty four seven operation, we can just take it out of rotation or, like, when GPUs fall off the bus or whatever. Like, we... In many cases, we just file a ticket automatically. Someone goes and looks at it.
00:03:46 · Erik Bernhardsson, Patrick Chase
[00:03:46] S1: I I think a lot of this actually boils down to reliability. I I mean, first of all, it's like, you know, differences in like disk and networking, and of course CPU and memory and things like that. Like all these things vary and region or like location, like they're all over the world. Like we have GPUs in The US, of course, like all over The US, but we also have GPUs in like Indonesia, like, all over the year... All over Europe, etcetera. So so location is is one of those big factors. And then... But but also reliability. And and and that's another thing we do exceptionally well is, like, we aggregate all this capacity. And then when Neo Clouds die or, like, when they have failures, which they often have because some of these are, like, former crypto miners who haven't really thought about, you know, running a twenty four seven operation, we can just take it out of rotation or, like, when GPUs fall off the bus or whatever. Like, we... In many cases, we just file a ticket automatically. Someone goes and looks at it. [00:04:38] S2: Have you found the data providers, the companies that provide pricing for GPU and that sort of things, have you found them to be helpful?
00:04:38 · Patrick Chase, Erik Bernhardsson, Jacob Effron
[00:04:38] S2: Have you found the data providers, the companies that provide pricing for GPU and that sort of things, have you found them to be helpful? [00:04:45] S1: I I think moderately helpful. I I I think the pricing less so than just their ability to connect with a lot of providers. I I don't fully trust the pricing today, but I think these... Some of these providers are actually very good at, like, know, knowing all the neo clouds, and we work with them sometimes to get connections with with cloud providers. The pricing, I think, over time will get more accurate. Like, our internal pricing does not match what we see publicly when we buy capacity. But I think over time, hopefully, like, we will have more efficient, like, external indices. I I think that would actually be very useful. [00:05:16] S0: You've been on this, like, rocket ship journey these past years, and so lots of things we'd love to dig into for our listeners on the on the business side. Maybe starting... You know, I think one of the interesting things about the history of modal is, you know, for a while it felt like you, you know, publicly said, I think on on your first unsupervised learning episode that you were less focused, you know, on these LLM inference calls, specifically like a simple API, you know, call for for a model. And, you know, you kind of recently released a product there. Maybe talk a little bit about the journey and and kind of that initial reticence and what made you decide that like now is the right time to really, you know, double down on this.
00:05:16 · Jacob Effron, Erik Bernhardsson, Patrick Chase
[00:05:16] S0: You've been on this, like, rocket ship journey these past years, and so lots of things we'd love to dig into for our listeners on the on the business side. Maybe starting... You know, I think one of the interesting things about the history of modal is, you know, for a while it felt like you, you know, publicly said, I think on on your first unsupervised learning episode that you were less focused, you know, on these LLM inference calls, specifically like a simple API, you know, call for for a model. And, you know, you kind of recently released a product there. Maybe talk a little bit about the journey and and kind of that initial reticence and what made you decide that like now is the right time to really, you know, double down on this. [00:05:53] S1: Yeah. So we started building this in in late twenty twenty one basically or early somewhere around that time. And and and and the initial product was this serverless function. And we thought it'd be really useful for a wide range of stuff around, like, data AI machine learning, broadly speaking. Kind of, like, looked at a lot of different use cases and... But then, like, ChatGPT came late twenty twenty two, and StableEfusion also came around the the the same time. [00:06:18] S2: Right. Right.