Unsupervised Learning with Jacob Effron
00:12:31 · Danila Shtan
[00:12:31] S0: Like, they're iterating, like, going further and further and further. NVIDIA is a behemoth now. Right? Whatever they do influences the whole market, supply chains, just everything. So they have to, like, live with that. So at some point, you get a sample, and it's more or less like what you'll get in production. Then you start building. Like you're testing everything. You like incorporate support for the new hardware, for the new restrictions or capabilities into the software platform. You figure out a way to isolate workloads from from the chat. We are we are a cloud. We are not providing bare metal hosts. Right? Like, so we need to split. We need to make sure that our virtualization works, that our storage works, that our networking works. Sometimes we are on the bleeding edge of this. So we, like... For some of the generations, we were probably the first team... Like, maybe one of the first teams that actually built, like, a cluster of a certain size. So we've hit some issues with, like, with the software that no one has seen before. So we had to report that back to NVIDIA, like, propose a patch, get there, like fix from it, make sure that it runs, etcetera, etcetera, etcetera. So like, we kind of feel like we are at the forefront of it. But all in all, it's a pretty normalized process. It may sound messy. Right? There are like multiple iterations. You go back and forth. You're in constant contact with like NVIDIA hardware folks, with NVIDIA software folks. You're trying to solve the issues on your end at the same time, incorporating into a more complex platform, thinking how can you expose that to customers. Because, for example, working with, like, your usual black hole chip, like, you know, the b three hundreds, and then transferring that experience to GB three hundreds, which I recall different architecture in terms of isolation. You are talking racks rather than servers. So you need to find a way how to do maintenance on the rack instead of the server, which you just switch off and you move the workloads to another one. Despite all of that, this kind of works all the time. And every day, you like push it a little bit further. And then in the end, yes, there is a product and the customers use it. And that's it.
00:14:55 · Jacob Effron, S2
[00:14:55] S1: One thing I think it would be so interesting to get your thoughts on is just the the differences in, you know, kind of what end customers end up wanting. Like, obviously, it can be easy to think, okay, everyone just wants to, you know, come to you and and get similar things. But I imagine you segment your customers or think about differences in what is important to different customers. How do you think about the key differences in these customer desires today? [00:15:18] S2: Hi. I'm Anne Maris, and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership, and see you next episode. [00:15:35] S1: How do you think about, like, the the key differences in in in these customer desires today?
00:15:35 · Jacob Effron, Danila Shtan
[00:15:35] S1: How do you think about, like, the the key differences in in in these customer desires today? [00:15:42] S0: It's not that different, to be honest. Because, like, every customer is, like, its own unique flower. Right? But at the same time, workloads in principle are not that different. It's either training, and you kinda know how training looks like, or inference, and you kinda know how inference looks like. Circling back a couple of years ago, for example, we kinda underestimated the storage component. We thought GPUs are the most important thing. And they are. But alongside the whole training angle, you see a massive demand for storage. And it's not for the, like, amount of, AGDs or VMUs that you can provide to the customer. It's about throughput. It's about latency. In my prior life, pre AI workloads and infrastructure build outs, I was talking about gigabytes per second, if we were talking about network attached storage. Right? It was like... It's really... Right now, we have customers routinely kinda like hitting our... Like a common share, like one volume, right, connected to a number of GPU servers with like, I don't know, five, six, 700 gigabytes per second of throughput. This is extreme. Like two years ago, I didn't even think about that. And now this is more or less like something we expect. At the same time, if you are targeting these numbers for the highest demand... Like, the the most demanding customers, your baseline is so high, you kinda can serve anyone. [00:17:20] S1: Yeah.
00:17:20 · Danila Shtan
[00:17:20] S0: Right? So if you hit this, like, use case where the customer needs 700 gigabytes per second from a, I don't know, three petabytes of shared file storage connected to a thousand of servers or, like, 8,000 GPUs, you kind of know how to serve a massive cohort. So I'd say in terms of like performance or features per se, the customers are not that different. Then you meet another subset of, like, what the customers really want. Because there are def... Definitely different customers in terms of not their scale, not their size, not their workloads, but rather how mature they are. Now, what are their obligations, like, for their clients? Right? Because we can talk about like a new labs. Like new labs routinely have seat rounds of... In the billions nowadays. Right? So most of that budget goes into commutes. So you can have like a contract for a billion dollars with a newly founded Neolab who... Like, they don't care that much about anything but GPU, storage, performance, availability. This is it. They... Like, oftentimes, they don't even care about the location of those GPUs. Yeah. They don't care if that's US, Europe, like France or Iceland. Like, why why bother? Like Yeah. Internet connection is good. Storage is there. GPUs are working as a... And we are talking about a billion dollar contract. Right? At the same time, we now see and like start onboarding enterprise customers. These are companies whose like the contract with us will be smaller, sometimes significantly smaller. But they are more established. They have customers, and they have applications for that customers. They have procedures. They need to make sure they're compliant to a number of regulations in, like, multiple countries.
00:19:24 · Jacob Effron, Danila Shtan
[00:19:24] S1: Yeah. [00:19:24] S0: So now we are getting into that whole angle of being the real cloud, you know, like certifications, compliance Yeah. Security posture checks. Let us look at your source code. How do you solve problem x? How do you, like, protect your customers from attack vector y? Etcetera, etcetera, etcetera. And I'd say if you are talking about differentiated customers, this is the most differentiating thing as opposed to what you would expect from the size or the performance. [00:19:57] S1: You know, obviously, I think a lot of people are thinking through this this trade off of of, like, throughput... Or a trade off of, like, latency and and and kind of cost. And I imagine you have some customers that have workloads that are super latency sensitive and they'll do small batches or whatever is required to get to that latency. Then you have other customers that are much less latency sensitive, maybe background agents or something. Is that all just determined on the software side, I guess, if you have those different customers of, like, basically how, you know, the batch size that you're running these workloads in? Or Yeah. Do you actually construct the clusters in any different ways based on those workloads?
00:20:31 · Danila Shtan, Jacob Effron
[00:20:31] S0: No. No. No. No. This was the premise from the very beginning. We are building the universal cloud platform, and we need to make sure that our baseline is high enough to fit most of the workloads. Yeah. Then there is the SLA play. Right? So customers come, like, example, for the managed inference part of the business. Customers come with the different demands. And then you are in this kind of a benchmark game. Right? They want this particular footprint based on this particular traffic pattern, feeds these SLAs in terms of latency. Latency is time to first token. Or the whole like request latency. And you start optimizing for that. But this is never done on the hardware level because hardware is good enough. This is rarely done on the lower infrastructure level because it's universal. And again, good enough, no compromise for performance, etcetera, etcetera. And you're going further away to the orchestration level, to the model level, way further up the stack, all the way to how exactly does the request from the customer, like the endpoint, gets to the inferencers and what are your performance hits along the way. [00:21:50] S1: Yeah. That makes sense. [00:21:51] S0: So this is, like, 100% software. [00:21:53] S1: I guess a lot of people have talked to me. You it feels like this this disag, you know, splitting prefill and decode is very in the air these days, and a lot of people talking about that. Like, is that something you guys think about now? Or or at least how do you how do you think about that kind of in the future?