Unsupervised Learning with Jacob Effron
00:14:20 · Jacob Effron, Anne Maris
[00:14:20] S0: But it feels like there's so many folks that are working on, you know, different parts of of of improving inference. And, obviously, what's, I think, been really interesting in the last two, three years is a lot of those optimizations and improvements have been in open source, and for the broader community, maybe shared via research. As a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, hey, we're gonna benefit from just the entire community pushing this space Hi. [00:14:46] S3: I'm Anne Maris, and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership, and see you next episode. [00:15:02] S0: As like a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, we're gonna benefit from just the entire community pushing this space
00:15:02 · Jacob Effron, Erik Bernhardsson
[00:15:02] S0: As like a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, we're gonna benefit from just the entire community pushing this space [00:15:12] S1: forward? We actually love open source because I think a lot of our alpha is is in the proprietary side, in the infrastructure layer, like all this cool stuff around auto scaling for instance. I think unlike other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLang team. We contribute very actively to both those repos and do a lot of optimizations with them. And and so I I think this may be a little bit of a different approach than than maybe other inference providers. Of course, there's gonna be, you know, certain things that, you know, we we keep internal, but but almost everything we we push to to open source. And I and I think it's, you know, it's because, like, we have so much alpha in the infrastructure anyway. Like, we might as well just, you know, work with the amazing open source engines that are out there that maybe were... You know, they weren't state of the art, like, you know, a few years ago, but now they are. And and and that's an incredible thing that, like, we'd love to leverage, frankly.
00:15:12 · Erik Bernhardsson, Patrick Chase
[00:15:12] S1: forward? We actually love open source because I think a lot of our alpha is is in the proprietary side, in the infrastructure layer, like all this cool stuff around auto scaling for instance. I think unlike other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLang team. We contribute very actively to both those repos and do a lot of optimizations with them. And and so I I think this may be a little bit of a different approach than than maybe other inference providers. Of course, there's gonna be, you know, certain things that, you know, we we keep internal, but but almost everything we we push to to open source. And I and I think it's, you know, it's because, like, we have so much alpha in the infrastructure anyway. Like, we might as well just, you know, work with the amazing open source engines that are out there that maybe were... You know, they weren't state of the art, like, you know, a few years ago, but now they are. And and and that's an incredible thing that, like, we'd love to leverage, frankly. [00:16:07] S2: You know, you were talking about heterogeneous compute and the the different places where you're getting compute from and also how the, you know, kind of inference stack is changing. How important are, you know, the long tail of chips to you guys, the non NVIDIA chips and and things like that as you think about your supply or kind of what the... What your serving stack looks like over time?
00:16:29 · Erik Bernhardsson
[00:16:29] S1: I I mean, I think NVIDIA is still like the king here. Like, they they have the best chips. There's way more demand for NVIDIA versus anything else. Like, you know, we're we're, you know, curious about some of the other chips. Like, you know, we made some experiments, like, taking a look at it. But but NVIDIA definitely has the best chips. But, you know, like, looking at AMD, Tranium, TPUs, etcetera, like, starting to be, like, pretty good. So so we're always curious about that. And then then, of course, there's there's a lot of, like, older chips too. And and I I think the the challenge has been, like, for a lot of the LMs, like, you're kinda constantly pushing, like, you know, how many parameters can you fit. So so, typically, latest and greatest LMs, like, they will always require, you know, a a full node, maybe even in the future, even multiple nodes. Who knows? So, like, eight b three hundreds. Like, they will require the latest chip and, like, eight of them because that's how many you can fit on a node. But but there's plenty of models outside of that, like older LMs, smaller ones that have... You know, they're they're optimized for performance might be fewer parameters. And then outside language, there's there's a lot of stuff around, voice and and, you know, audio transcription or speech synthesis or or, like, you know, pharma, like, bio stuff. Like, a lot of that stuff actually... They don't require, like, the latest and greatest GPUs. Like, they actually get more cost efficient serving on on hoppers or or even m pairs or, like, you know, different types of, like, l 40 x or RTX Pro 6,000, stuff like that. So we actually... I mean, we are ramping up, like, very, very aggressively on on Blackwalls, primarily GBs now. But we we see actually, like, plenty of demand also in the older chips, which I think is quite exciting. There's just so much stuff going on. People forget about the hoppers, but, like, the... They... They're our most common type right now still at modal, like, h one hundreds. We run some very, like, large production use cases based on hoppers.
00:18:19 · Jacob Effron
[00:18:19] S0: How efficient, like, is model to chip matching today, like, know, in the industry at large? Do you feel like most people are using the, like, optimal, you know, chip or or, you know, from a price perspective for what they're what they're working, or is there still a lot of room for improvement there?
00:18:30 · Erik Bernhardsson
[00:18:30] S1: There's a little bit of room to improve. I I mean, one of the things that we wanna do over time is, like, kinda automate that for people. Think in practice, it's, like, less of a problem than it might seem like because because often, like, you're kinda memory constrained. So, like, knowing how much memory you need for a model, like, you basically pick the cheapest GPU that fits the model. That's usually, like, the the heuristic that works in almost all cases. So if you have... You know, if your model is 80 gigs, like, you can either fit it on a hopper or or an older a 180 gig. I I guess in that case, it depends a little bit on the flops. But but in many cases, it's like whatever whatever GPU, like, is the smallest that fits your model. I think over time, like, there's a bunch of stuff we've been thinking about that we're, like, basically, you deploy something a model, and then we manage, like, different chips and maybe automatically give you discounts because, like, we can... You know, the more options we can have to like run it on a lot of different chips, like that lets us aggregate capacity in a better way and run things at higher utilization because then we can switch between different GPU types as you know, there's like underutilization on those. So that's something actually we're we're we're quite interested in doing over the next six months. It's, like, giving people an option to to deploy things in a way where, like, we automatically switch types underneath in a way that's still consistent with your dollar per request or or dollar per token or whatever. I I think there's some incredible cool stuff to do there.