Unsupervised Learning with Jacob Effron
00:14:55 · Jacob Effron, S2
[00:14:55] S1: One thing I think it would be so interesting to get your thoughts on is just the the differences in, you know, kind of what end customers end up wanting. Like, obviously, it can be easy to think, okay, everyone just wants to, you know, come to you and and get similar things. But I imagine you segment your customers or think about differences in what is important to different customers. How do you think about the key differences in these customer desires today? [00:15:18] S2: Hi. I'm Anne Maris, and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership, and see you next episode. [00:15:35] S1: How do you think about, like, the the key differences in in in these customer desires today?
00:15:35 · Jacob Effron, Danila Shtan
[00:15:35] S1: How do you think about, like, the the key differences in in in these customer desires today? [00:15:42] S0: It's not that different, to be honest. Because, like, every customer is, like, its own unique flower. Right? But at the same time, workloads in principle are not that different. It's either training, and you kinda know how training looks like, or inference, and you kinda know how inference looks like. Circling back a couple of years ago, for example, we kinda underestimated the storage component. We thought GPUs are the most important thing. And they are. But alongside the whole training angle, you see a massive demand for storage. And it's not for the, like, amount of, AGDs or VMUs that you can provide to the customer. It's about throughput. It's about latency. In my prior life, pre AI workloads and infrastructure build outs, I was talking about gigabytes per second, if we were talking about network attached storage. Right? It was like... It's really... Right now, we have customers routinely kinda like hitting our... Like a common share, like one volume, right, connected to a number of GPU servers with like, I don't know, five, six, 700 gigabytes per second of throughput. This is extreme. Like two years ago, I didn't even think about that. And now this is more or less like something we expect. At the same time, if you are targeting these numbers for the highest demand... Like, the the most demanding customers, your baseline is so high, you kinda can serve anyone. [00:17:20] S1: Yeah.
00:17:20 · Danila Shtan
[00:17:20] S0: Right? So if you hit this, like, use case where the customer needs 700 gigabytes per second from a, I don't know, three petabytes of shared file storage connected to a thousand of servers or, like, 8,000 GPUs, you kind of know how to serve a massive cohort. So I'd say in terms of like performance or features per se, the customers are not that different. Then you meet another subset of, like, what the customers really want. Because there are def... Definitely different customers in terms of not their scale, not their size, not their workloads, but rather how mature they are. Now, what are their obligations, like, for their clients? Right? Because we can talk about like a new labs. Like new labs routinely have seat rounds of... In the billions nowadays. Right? So most of that budget goes into commutes. So you can have like a contract for a billion dollars with a newly founded Neolab who... Like, they don't care that much about anything but GPU, storage, performance, availability. This is it. They... Like, oftentimes, they don't even care about the location of those GPUs. Yeah. They don't care if that's US, Europe, like France or Iceland. Like, why why bother? Like Yeah. Internet connection is good. Storage is there. GPUs are working as a... And we are talking about a billion dollar contract. Right? At the same time, we now see and like start onboarding enterprise customers. These are companies whose like the contract with us will be smaller, sometimes significantly smaller. But they are more established. They have customers, and they have applications for that customers. They have procedures. They need to make sure they're compliant to a number of regulations in, like, multiple countries.
00:19:24 · Jacob Effron, Danila Shtan
[00:19:24] S1: Yeah. [00:19:24] S0: So now we are getting into that whole angle of being the real cloud, you know, like certifications, compliance Yeah. Security posture checks. Let us look at your source code. How do you solve problem x? How do you, like, protect your customers from attack vector y? Etcetera, etcetera, etcetera. And I'd say if you are talking about differentiated customers, this is the most differentiating thing as opposed to what you would expect from the size or the performance. [00:19:57] S1: You know, obviously, I think a lot of people are thinking through this this trade off of of, like, throughput... Or a trade off of, like, latency and and and kind of cost. And I imagine you have some customers that have workloads that are super latency sensitive and they'll do small batches or whatever is required to get to that latency. Then you have other customers that are much less latency sensitive, maybe background agents or something. Is that all just determined on the software side, I guess, if you have those different customers of, like, basically how, you know, the batch size that you're running these workloads in? Or Yeah. Do you actually construct the clusters in any different ways based on those workloads?
00:20:31 · Danila Shtan, Jacob Effron
[00:20:31] S0: No. No. No. No. This was the premise from the very beginning. We are building the universal cloud platform, and we need to make sure that our baseline is high enough to fit most of the workloads. Yeah. Then there is the SLA play. Right? So customers come, like, example, for the managed inference part of the business. Customers come with the different demands. And then you are in this kind of a benchmark game. Right? They want this particular footprint based on this particular traffic pattern, feeds these SLAs in terms of latency. Latency is time to first token. Or the whole like request latency. And you start optimizing for that. But this is never done on the hardware level because hardware is good enough. This is rarely done on the lower infrastructure level because it's universal. And again, good enough, no compromise for performance, etcetera, etcetera. And you're going further away to the orchestration level, to the model level, way further up the stack, all the way to how exactly does the request from the customer, like the endpoint, gets to the inferencers and what are your performance hits along the way. [00:21:50] S1: Yeah. That makes sense. [00:21:51] S0: So this is, like, 100% software. [00:21:53] S1: I guess a lot of people have talked to me. You it feels like this this disag, you know, splitting prefill and decode is very in the air these days, and a lot of people talking about that. Like, is that something you guys think about now? Or or at least how do you how do you think about that kind of in the future?
00:22:05 · Danila Shtan, Jacob Effron
[00:22:05] S0: I think we are in the stage where everyone's talking about that and no one's doing that properly. So because like, it is a complex problem to solve. And particularly in inference, you have a lot of other problems, like way lower hanging fruits that you can fix address and hit those SLAs. So we are quite cautious about that. We don't have it, for example, as a go to solution for every customer, for every workload. Because like, first, it doesn't make sense. Second, it's really hard to operate and to work with. Like, provides proper SLAs around that is, like, proving quite difficult as of today. We have some promising results in our, like, prototype services, but not as of today. At the same time, we have some, like, disaggregated setups for the most demanding customers in production right now. It's just more, you know, wide glove manual approach. Yeah. But especially for these newer architectures of, like like, you know, those gray black walls, whole node that has really fast interconnect, a lot of challenges that you've had before with disaggregate steps are kind of solved if you're able to fit in the footprint of a rack. Because, like, the throughput is so much faster. Like, interconnect is so much faster. You kinda stop caring, and that this is a way of just balancing the prefilled decode nodes to match the particular pattern that you see from the customer. [00:23:41] S1: What... What's your mental model for, like, the workloads that do fit in a rack?