Unsupervised Learning with Jacob Effron
00:17:20 · Danila Shtan
[00:17:20] S0: Right? So if you hit this, like, use case where the customer needs 700 gigabytes per second from a, I don't know, three petabytes of shared file storage connected to a thousand of servers or, like, 8,000 GPUs, you kind of know how to serve a massive cohort. So I'd say in terms of like performance or features per se, the customers are not that different. Then you meet another subset of, like, what the customers really want. Because there are def... Definitely different customers in terms of not their scale, not their size, not their workloads, but rather how mature they are. Now, what are their obligations, like, for their clients? Right? Because we can talk about like a new labs. Like new labs routinely have seat rounds of... In the billions nowadays. Right? So most of that budget goes into commutes. So you can have like a contract for a billion dollars with a newly founded Neolab who... Like, they don't care that much about anything but GPU, storage, performance, availability. This is it. They... Like, oftentimes, they don't even care about the location of those GPUs. Yeah. They don't care if that's US, Europe, like France or Iceland. Like, why why bother? Like Yeah. Internet connection is good. Storage is there. GPUs are working as a... And we are talking about a billion dollar contract. Right? At the same time, we now see and like start onboarding enterprise customers. These are companies whose like the contract with us will be smaller, sometimes significantly smaller. But they are more established. They have customers, and they have applications for that customers. They have procedures. They need to make sure they're compliant to a number of regulations in, like, multiple countries.
00:19:24 · Jacob Effron, Danila Shtan
[00:19:24] S1: Yeah. [00:19:24] S0: So now we are getting into that whole angle of being the real cloud, you know, like certifications, compliance Yeah. Security posture checks. Let us look at your source code. How do you solve problem x? How do you, like, protect your customers from attack vector y? Etcetera, etcetera, etcetera. And I'd say if you are talking about differentiated customers, this is the most differentiating thing as opposed to what you would expect from the size or the performance. [00:19:57] S1: You know, obviously, I think a lot of people are thinking through this this trade off of of, like, throughput... Or a trade off of, like, latency and and and kind of cost. And I imagine you have some customers that have workloads that are super latency sensitive and they'll do small batches or whatever is required to get to that latency. Then you have other customers that are much less latency sensitive, maybe background agents or something. Is that all just determined on the software side, I guess, if you have those different customers of, like, basically how, you know, the batch size that you're running these workloads in? Or Yeah. Do you actually construct the clusters in any different ways based on those workloads?
00:20:31 · Danila Shtan, Jacob Effron
[00:20:31] S0: No. No. No. No. This was the premise from the very beginning. We are building the universal cloud platform, and we need to make sure that our baseline is high enough to fit most of the workloads. Yeah. Then there is the SLA play. Right? So customers come, like, example, for the managed inference part of the business. Customers come with the different demands. And then you are in this kind of a benchmark game. Right? They want this particular footprint based on this particular traffic pattern, feeds these SLAs in terms of latency. Latency is time to first token. Or the whole like request latency. And you start optimizing for that. But this is never done on the hardware level because hardware is good enough. This is rarely done on the lower infrastructure level because it's universal. And again, good enough, no compromise for performance, etcetera, etcetera. And you're going further away to the orchestration level, to the model level, way further up the stack, all the way to how exactly does the request from the customer, like the endpoint, gets to the inferencers and what are your performance hits along the way. [00:21:50] S1: Yeah. That makes sense. [00:21:51] S0: So this is, like, 100% software. [00:21:53] S1: I guess a lot of people have talked to me. You it feels like this this disag, you know, splitting prefill and decode is very in the air these days, and a lot of people talking about that. Like, is that something you guys think about now? Or or at least how do you how do you think about that kind of in the future?
00:22:05 · Danila Shtan, Jacob Effron
[00:22:05] S0: I think we are in the stage where everyone's talking about that and no one's doing that properly. So because like, it is a complex problem to solve. And particularly in inference, you have a lot of other problems, like way lower hanging fruits that you can fix address and hit those SLAs. So we are quite cautious about that. We don't have it, for example, as a go to solution for every customer, for every workload. Because like, first, it doesn't make sense. Second, it's really hard to operate and to work with. Like, provides proper SLAs around that is, like, proving quite difficult as of today. We have some promising results in our, like, prototype services, but not as of today. At the same time, we have some, like, disaggregated setups for the most demanding customers in production right now. It's just more, you know, wide glove manual approach. Yeah. But especially for these newer architectures of, like like, you know, those gray black walls, whole node that has really fast interconnect, a lot of challenges that you've had before with disaggregate steps are kind of solved if you're able to fit in the footprint of a rack. Because, like, the throughput is so much faster. Like, interconnect is so much faster. You kinda stop caring, and that this is a way of just balancing the prefilled decode nodes to match the particular pattern that you see from the customer. [00:23:41] S1: What... What's your mental model for, like, the workloads that do fit in a rack?
00:23:41 · Jacob Effron, Danila Shtan
[00:23:41] S1: What... What's your mental model for, like, the workloads that do fit in a rack? [00:23:44] S0: I don't know. It's a very case by case. Well, these are definitely significant workloads. Like, we are talking, like, thousands of requests per second. Right? We are talking, like, billions of tokens that flow through. But at the same time, because we are the cloud, we can, for example, like share the rack between two workloads and not be that bothered about that. Right? [00:24:05] S1: Yeah. [00:24:06] S0: So for example, on the cloud side, we... For for this particular product, we only allow to consume, like, a track level. You cannot order two Grace Blackwell nodes. It doesn't make sense. [00:24:21] S1: Yeah. [00:24:21] S0: Right? Like, you have to allocate the whole rack and then, like, structure workloads within it. But for example, when you're a client of such a solution, as we do with managed inference, for example, you can collocate workloads and you can find a way to do that. So in a sense, you don't need to fit anything in the rack. It's just, is there a workload that will benefit from this disbalance, like, intentional disbalance of prefilling the code nodes? Yeah. And can you fit more than one into a single rack, and how do you solve the packing problem? [00:24:58] S1: Yeah. That makes sense. And you kinda mentioned there's, like, lower hanging fruit on the on the inference side, you know, than than doing that maybe. What what are, like, some of the lower hanging fruits today that you still think we're we're not as as as an industry optimized on?
00:25:11 · Danila Shtan
[00:25:11] S0: Well, first of I think as an industry, we didn't optimize shit. So, like, pardon my French. But in a sense, I strongly believe that running inference, particularly in the like variety of open weight, source ish models that are available today, is not a solved problem. Like, if you take something, just run with it as it's published, like in the initial support in some of the open source inference servers run with the novel models, your results will probably be underwhelming. It's not that fast. It's not that cheap, especially with like current prices on the infrastructure. It's easier to go and use like, I don't know, Fable 5.1, Astra six, etcetera, etcetera, etcetera. But because of the open architecture, because like of everything that we know as of today, you can optimize the way you process those requests. You can optimize the way the inference server is configured. You'll put a lot of effort in kernel optimizations, sometimes kernel over optimizations based, like, on the particular pattern that you see in the traffic. Right? There is a lot of underexplored potential in model weights optimization. And I'm not just talking about like stuff like speculative decoding, for example. I'm talking about like model pruning, selective pruning, compression. Because I believe that... And we see that within our customer base. 95% of our customers do not run vanilla open source models. These are modified, fine tuned, etcetera, etcetera, etcetera. It means that these are the models that start to address a particular use case that the customer has. Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain like frontier intelligence and reasoning for a particular subset of tasks. And you don't care about the rest. And now we are getting somewhere in terms of like, maybe you don't need such a large model anymore. Maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it. Now, of course, that doesn't work with, like, open ended, you know, deep research or broad coding tasks anymore.