Unsupervised Learning with Jacob Effron
00:16:29 · Erik Bernhardsson
[00:16:29] S1: I I mean, I think NVIDIA is still like the king here. Like, they they have the best chips. There's way more demand for NVIDIA versus anything else. Like, you know, we're we're, you know, curious about some of the other chips. Like, you know, we made some experiments, like, taking a look at it. But but NVIDIA definitely has the best chips. But, you know, like, looking at AMD, Tranium, TPUs, etcetera, like, starting to be, like, pretty good. So so we're always curious about that. And then then, of course, there's there's a lot of, like, older chips too. And and I I think the the challenge has been, like, for a lot of the LMs, like, you're kinda constantly pushing, like, you know, how many parameters can you fit. So so, typically, latest and greatest LMs, like, they will always require, you know, a a full node, maybe even in the future, even multiple nodes. Who knows? So, like, eight b three hundreds. Like, they will require the latest chip and, like, eight of them because that's how many you can fit on a node. But but there's plenty of models outside of that, like older LMs, smaller ones that have... You know, they're they're optimized for performance might be fewer parameters. And then outside language, there's there's a lot of stuff around, voice and and, you know, audio transcription or speech synthesis or or, like, you know, pharma, like, bio stuff. Like, a lot of that stuff actually... They don't require, like, the latest and greatest GPUs. Like, they actually get more cost efficient serving on on hoppers or or even m pairs or, like, you know, different types of, like, l 40 x or RTX Pro 6,000, stuff like that. So we actually... I mean, we are ramping up, like, very, very aggressively on on Blackwalls, primarily GBs now. But we we see actually, like, plenty of demand also in the older chips, which I think is quite exciting. There's just so much stuff going on. People forget about the hoppers, but, like, the... They... They're our most common type right now still at modal, like, h one hundreds. We run some very, like, large production use cases based on hoppers.
00:18:19 · Jacob Effron
[00:18:19] S0: How efficient, like, is model to chip matching today, like, know, in the industry at large? Do you feel like most people are using the, like, optimal, you know, chip or or, you know, from a price perspective for what they're what they're working, or is there still a lot of room for improvement there?
00:18:30 · Erik Bernhardsson
[00:18:30] S1: There's a little bit of room to improve. I I mean, one of the things that we wanna do over time is, like, kinda automate that for people. Think in practice, it's, like, less of a problem than it might seem like because because often, like, you're kinda memory constrained. So, like, knowing how much memory you need for a model, like, you basically pick the cheapest GPU that fits the model. That's usually, like, the the heuristic that works in almost all cases. So if you have... You know, if your model is 80 gigs, like, you can either fit it on a hopper or or an older a 180 gig. I I guess in that case, it depends a little bit on the flops. But but in many cases, it's like whatever whatever GPU, like, is the smallest that fits your model. I think over time, like, there's a bunch of stuff we've been thinking about that we're, like, basically, you deploy something a model, and then we manage, like, different chips and maybe automatically give you discounts because, like, we can... You know, the more options we can have to like run it on a lot of different chips, like that lets us aggregate capacity in a better way and run things at higher utilization because then we can switch between different GPU types as you know, there's like underutilization on those. So that's something actually we're we're we're quite interested in doing over the next six months. It's, like, giving people an option to to deploy things in a way where, like, we automatically switch types underneath in a way that's still consistent with your dollar per request or or dollar per token or whatever. I I think there's some incredible cool stuff to do there.
00:19:49 · Jacob Effron, Erik Bernhardsson
[00:19:49] S0: Alright. Super interesting. And then one thing I imagine is so interesting in in your role too is, like, you're obviously predicting for, like, what your compute fleet should be. And, obviously, you see what the workloads are today, and you probably have some sense of of, you know, what models run well on what chips. Then you're also trying to predict, like, hey, a year or two from now, what kind of workloads will we have? I think there's lots of different debates about whether we're gonna... Most tokens running will be really large models or actually we saturate in those tasks and these models will get smaller and smaller. As you think about kind of the evolution of your fleet in the next year or two, like, what... Yeah. What's your, you know, perspective on on, like, where you'll see relatively more different kinds of workloads and less? [00:20:26] S1: I I think there's been kind of, like, basically over a trend of, like, like, larger and larger models over the last three, four years. [00:20:33] S0: Bet against that.
00:20:33 · Jacob Effron, Erik Bernhardsson
[00:20:33] S0: Bet against that. [00:20:34] S1: I I kind of... You know, I I I definitely feel like that trend is gonna continue. Like, we're gonna need... Like, if b three hundreds had more memory, I'm sure, you know, a Kiwi k three would be bigger. Like, people are kind of, like, you know, training the models in a way that they fit on these these chips. So I I I think on the... At least on the LM side, like, the... You know, LM side, in a way, like, kind of a crude, like, type of world model. Like, you're trying to, like, encode as much information as you can in these LMs. Like, you need, like, very big, like, almost, like, lookup tables of facts, you know, anyway. It's kind of a conception how to think about it. And and and that sort of pushes, like, the memory requirements a lot. I mean, at some point, like, it might taper off. Like, who knows? But but, like, for now, like, people will make use of every possible memory. Like, the more memory, like, they'll just train more parameters for sure. [00:21:21] S0: You obviously sit in, like, the the... In some ways, unlike the eye of the hurricane of, like, a very exciting, large, and competitive market. And so I'm curious as you think about what ultimately will kind of differentiate players in this space and win out. I think there's so many true believers in the size of the overall inference market, and I think lots of folks are asking what actually, you know, wins out in the end of the day. I know you have a good thesis on this, so I would love it for you to to share with our listeners.
00:21:49 · Erik Bernhardsson
[00:21:49] S1: I I mean, I think a lot of the inference market will be quite rough. Right? Like, think the sort of straight down the middle, just like, you know, serving tokens, you already see sort of open router basically commoditizing the underlying providers. I tend to think, you know, like, the the the real alpha or, like, the real, like, point of differentiation, like, there will be a lot of differentiation just in, like, more efficient serving. And... But but it is a little bit of a race to the bottom, and I think and I think there's gonna be a lot of opportunities to sort of stay ahead of competition, and you're gonna have to, like, put a lot of efforts into that. But but I think a more sort of durable differentiation will definitely be in like, know, providing the entire platform of a lot of different things, which is like, I think something we are quite well positioned to do. We have the sandboxes, we have reinforcement learning, a lot of customers doing also normal fine tuning. A lot of customers, obviously, you know, deploying other types of models. We have batch processing. I think I I think my my my view in general is, like, customers, they don't wanna buy, like, single trick ponies. Like, they wanna buy, like, a multiproduct set of tools. So that to me is, like, ultimate... Like, probably the biggest source of differentiation is, like, you know, how do you provide that that entire set of tools? How do you provide this, like, very general purpose thing that lets engineers, like, iterate very fast, deliver value? And then, like, this is a day to day. Like, still gonna have to, like, go and do the knife fighting of, like, you know, staying ahead of your competitors in terms of performance and use the latest kernels and the much more efficient KV cache. And there's gonna be a lot of that stuff for a long time because there's just so much money in it. But I I... I'm kind of pointing out. I I think that's sort of that's like less durable as an advantage. Think that, know, because over time, like, people... These tricks will sort of spread and people figure out how to how to copy them. So the more durable thing I think is like being that sort of portfolio of a lot of stuff around inference, not just the inference in itself.