Unsupervised Learning with Jacob Effron
00:22:05 · Danila Shtan, Jacob Effron
[00:22:05] S0: I think we are in the stage where everyone's talking about that and no one's doing that properly. So because like, it is a complex problem to solve. And particularly in inference, you have a lot of other problems, like way lower hanging fruits that you can fix address and hit those SLAs. So we are quite cautious about that. We don't have it, for example, as a go to solution for every customer, for every workload. Because like, first, it doesn't make sense. Second, it's really hard to operate and to work with. Like, provides proper SLAs around that is, like, proving quite difficult as of today. We have some promising results in our, like, prototype services, but not as of today. At the same time, we have some, like, disaggregated setups for the most demanding customers in production right now. It's just more, you know, wide glove manual approach. Yeah. But especially for these newer architectures of, like like, you know, those gray black walls, whole node that has really fast interconnect, a lot of challenges that you've had before with disaggregate steps are kind of solved if you're able to fit in the footprint of a rack. Because, like, the throughput is so much faster. Like, interconnect is so much faster. You kinda stop caring, and that this is a way of just balancing the prefilled decode nodes to match the particular pattern that you see from the customer. [00:23:41] S1: What... What's your mental model for, like, the workloads that do fit in a rack?
00:23:41 · Jacob Effron, Danila Shtan
[00:23:41] S1: What... What's your mental model for, like, the workloads that do fit in a rack? [00:23:44] S0: I don't know. It's a very case by case. Well, these are definitely significant workloads. Like, we are talking, like, thousands of requests per second. Right? We are talking, like, billions of tokens that flow through. But at the same time, because we are the cloud, we can, for example, like share the rack between two workloads and not be that bothered about that. Right? [00:24:05] S1: Yeah. [00:24:06] S0: So for example, on the cloud side, we... For for this particular product, we only allow to consume, like, a track level. You cannot order two Grace Blackwell nodes. It doesn't make sense. [00:24:21] S1: Yeah. [00:24:21] S0: Right? Like, you have to allocate the whole rack and then, like, structure workloads within it. But for example, when you're a client of such a solution, as we do with managed inference, for example, you can collocate workloads and you can find a way to do that. So in a sense, you don't need to fit anything in the rack. It's just, is there a workload that will benefit from this disbalance, like, intentional disbalance of prefilling the code nodes? Yeah. And can you fit more than one into a single rack, and how do you solve the packing problem? [00:24:58] S1: Yeah. That makes sense. And you kinda mentioned there's, like, lower hanging fruit on the on the inference side, you know, than than doing that maybe. What what are, like, some of the lower hanging fruits today that you still think we're we're not as as as an industry optimized on?
00:25:11 · Danila Shtan
[00:25:11] S0: Well, first of I think as an industry, we didn't optimize shit. So, like, pardon my French. But in a sense, I strongly believe that running inference, particularly in the like variety of open weight, source ish models that are available today, is not a solved problem. Like, if you take something, just run with it as it's published, like in the initial support in some of the open source inference servers run with the novel models, your results will probably be underwhelming. It's not that fast. It's not that cheap, especially with like current prices on the infrastructure. It's easier to go and use like, I don't know, Fable 5.1, Astra six, etcetera, etcetera, etcetera. But because of the open architecture, because like of everything that we know as of today, you can optimize the way you process those requests. You can optimize the way the inference server is configured. You'll put a lot of effort in kernel optimizations, sometimes kernel over optimizations based, like, on the particular pattern that you see in the traffic. Right? There is a lot of underexplored potential in model weights optimization. And I'm not just talking about like stuff like speculative decoding, for example. I'm talking about like model pruning, selective pruning, compression. Because I believe that... And we see that within our customer base. 95% of our customers do not run vanilla open source models. These are modified, fine tuned, etcetera, etcetera, etcetera. It means that these are the models that start to address a particular use case that the customer has. Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain like frontier intelligence and reasoning for a particular subset of tasks. And you don't care about the rest. And now we are getting somewhere in terms of like, maybe you don't need such a large model anymore. Maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it. Now, of course, that doesn't work with, like, open ended, you know, deep research or broad coding tasks anymore.
00:27:48 · Danila Shtan, Jacob Effron
But I strongly believe that for for real AI adoption, when we are talking about serious enterprise business, when we are talking about, like, practical AI that actually solves problems, You are getting more and more into these specialized agents, limited domains, where you even stop caring about the cost. You care more about predictability of the outcome, and you care about time, as in, like, real seconds and minutes till you get that outcome. Yeah. And I think there is a lot of potential there, and we still... We'll we'll we'll still see a couple of nice breakthroughs there. [00:28:31] S1: There's a ton of work around inference optimization, and a lot of that gets done in open source, right, or, like, you know, contributions that that share throughout the community. There's obviously a bunch of work on on on, like, smaller purpose built models and and compaction and a bunch of these things. Across all these problems, which do impact, you know, how enterprises consume inference, which ones like Nebius should have a solution or offer a product around and which ones you're like, hey. It's great that that exists, and we'll benefit from that, but we don't need to be at the forefront.
00:28:59 · Danila Shtan, Jacob Effron
[00:28:59] S0: We want to be at the forefront of everything. In a sense, these are additive and even maybe multiplicative. Right? Like, if you have a right subset of things, optimizations that you introduce for a particular use case, it means that you can reach the targets easier, better, faster, in a more predictable fashion. So we invest in all avenues. Like, one of our acquisitions back in, like, the beginning of summer was targeting specifically that. Like, we basically bought the company for the talent in model optimization, model compression, speculative decoding, etcetera, etcetera, etcetera, combined it with existing talent. So, like, we have, like, team that is two x this size. And we... Like, we have serious manpower, like, as in tens and tens of people who are working on these problems. Like, we have specialized groups for, like, kernel optimization, specialized groups for, like, model compression problems. Because this is... Like, I feel if you want to talk about, like, running inference as a service at a serious level, [00:30:09] S1: this is the [00:30:09] S0: only way you can do it nowadays. [00:30:11] S1: And I guess, like, you know, maybe along those lines, you guys have done, you know, a few acquisitions, and and one of them that that caught a few people's eyes was, you know, Tavily in the kind of, like, web search space. Maybe talk a little through a little bit, like, you know, the rationale for that, how you think about that fitting into the broader platform.
00:30:11 · Jacob Effron, Danila Shtan
[00:30:11] S1: And I guess, like, you know, maybe along those lines, you guys have done, you know, a few acquisitions, and and one of them that that caught a few people's eyes was, you know, Tavily in the kind of, like, web search space. Maybe talk a little through a little bit, like, you know, the rationale for that, how you think about that fitting into the broader platform. [00:30:27] S0: So first, we were very impressed by the product and their go to market motion. It was almost a year ago now. Right? But we felt like we knew how to help them run even faster with our expertise and technology and low level stuff. Because actually running a web search engine is not a very simple task, not neither from, like, the software development perspective nor from the It's [00:30:56] S1: a a lot of... You used to run one.