Unsupervised Learning with Jacob Effron
00:23:41 · Jacob Effron, Danila Shtan
[00:23:41] S1: What... What's your mental model for, like, the workloads that do fit in a rack? [00:23:44] S0: I don't know. It's a very case by case. Well, these are definitely significant workloads. Like, we are talking, like, thousands of requests per second. Right? We are talking, like, billions of tokens that flow through. But at the same time, because we are the cloud, we can, for example, like share the rack between two workloads and not be that bothered about that. Right? [00:24:05] S1: Yeah. [00:24:06] S0: So for example, on the cloud side, we... For for this particular product, we only allow to consume, like, a track level. You cannot order two Grace Blackwell nodes. It doesn't make sense. [00:24:21] S1: Yeah. [00:24:21] S0: Right? Like, you have to allocate the whole rack and then, like, structure workloads within it. But for example, when you're a client of such a solution, as we do with managed inference, for example, you can collocate workloads and you can find a way to do that. So in a sense, you don't need to fit anything in the rack. It's just, is there a workload that will benefit from this disbalance, like, intentional disbalance of prefilling the code nodes? Yeah. And can you fit more than one into a single rack, and how do you solve the packing problem? [00:24:58] S1: Yeah. That makes sense. And you kinda mentioned there's, like, lower hanging fruit on the on the inference side, you know, than than doing that maybe. What what are, like, some of the lower hanging fruits today that you still think we're we're not as as as an industry optimized on?
00:25:11 · Danila Shtan
[00:25:11] S0: Well, first of I think as an industry, we didn't optimize shit. So, like, pardon my French. But in a sense, I strongly believe that running inference, particularly in the like variety of open weight, source ish models that are available today, is not a solved problem. Like, if you take something, just run with it as it's published, like in the initial support in some of the open source inference servers run with the novel models, your results will probably be underwhelming. It's not that fast. It's not that cheap, especially with like current prices on the infrastructure. It's easier to go and use like, I don't know, Fable 5.1, Astra six, etcetera, etcetera, etcetera. But because of the open architecture, because like of everything that we know as of today, you can optimize the way you process those requests. You can optimize the way the inference server is configured. You'll put a lot of effort in kernel optimizations, sometimes kernel over optimizations based, like, on the particular pattern that you see in the traffic. Right? There is a lot of underexplored potential in model weights optimization. And I'm not just talking about like stuff like speculative decoding, for example. I'm talking about like model pruning, selective pruning, compression. Because I believe that... And we see that within our customer base. 95% of our customers do not run vanilla open source models. These are modified, fine tuned, etcetera, etcetera, etcetera. It means that these are the models that start to address a particular use case that the customer has. Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain like frontier intelligence and reasoning for a particular subset of tasks. And you don't care about the rest. And now we are getting somewhere in terms of like, maybe you don't need such a large model anymore. Maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it. Now, of course, that doesn't work with, like, open ended, you know, deep research or broad coding tasks anymore.
00:27:48 · Danila Shtan, Jacob Effron
But I strongly believe that for for real AI adoption, when we are talking about serious enterprise business, when we are talking about, like, practical AI that actually solves problems, You are getting more and more into these specialized agents, limited domains, where you even stop caring about the cost. You care more about predictability of the outcome, and you care about time, as in, like, real seconds and minutes till you get that outcome. Yeah. And I think there is a lot of potential there, and we still... We'll we'll we'll still see a couple of nice breakthroughs there. [00:28:31] S1: There's a ton of work around inference optimization, and a lot of that gets done in open source, right, or, like, you know, contributions that that share throughout the community. There's obviously a bunch of work on on on, like, smaller purpose built models and and compaction and a bunch of these things. Across all these problems, which do impact, you know, how enterprises consume inference, which ones like Nebius should have a solution or offer a product around and which ones you're like, hey. It's great that that exists, and we'll benefit from that, but we don't need to be at the forefront.
00:28:59 · Danila Shtan, Jacob Effron
[00:28:59] S0: We want to be at the forefront of everything. In a sense, these are additive and even maybe multiplicative. Right? Like, if you have a right subset of things, optimizations that you introduce for a particular use case, it means that you can reach the targets easier, better, faster, in a more predictable fashion. So we invest in all avenues. Like, one of our acquisitions back in, like, the beginning of summer was targeting specifically that. Like, we basically bought the company for the talent in model optimization, model compression, speculative decoding, etcetera, etcetera, etcetera, combined it with existing talent. So, like, we have, like, team that is two x this size. And we... Like, we have serious manpower, like, as in tens and tens of people who are working on these problems. Like, we have specialized groups for, like, kernel optimization, specialized groups for, like, model compression problems. Because this is... Like, I feel if you want to talk about, like, running inference as a service at a serious level, [00:30:09] S1: this is the [00:30:09] S0: only way you can do it nowadays. [00:30:11] S1: And I guess, like, you know, maybe along those lines, you guys have done, you know, a few acquisitions, and and one of them that that caught a few people's eyes was, you know, Tavily in the kind of, like, web search space. Maybe talk a little through a little bit, like, you know, the rationale for that, how you think about that fitting into the broader platform.
00:30:11 · Jacob Effron, Danila Shtan
[00:30:11] S1: And I guess, like, you know, maybe along those lines, you guys have done, you know, a few acquisitions, and and one of them that that caught a few people's eyes was, you know, Tavily in the kind of, like, web search space. Maybe talk a little through a little bit, like, you know, the rationale for that, how you think about that fitting into the broader platform. [00:30:27] S0: So first, we were very impressed by the product and their go to market motion. It was almost a year ago now. Right? But we felt like we knew how to help them run even faster with our expertise and technology and low level stuff. Because actually running a web search engine is not a very simple task, not neither from, like, the software development perspective nor from the It's [00:30:56] S1: a a lot of... You used to run one.
00:30:56 · Jacob Effron, Danila Shtan
[00:30:56] S1: a a lot of... You used to run one. [00:30:59] S0: And of us dabbled a little bit in our previous lives. Yes. Right? So this was the premise. Like, we have a great team that is running real fast in terms of like market penetration and in general go to market motion. And we know how to make it complementary with our expertise on the low level stuff infrastructure and, for example, large scale web index. It's not exactly, like, embedded into the rest of the offerings that we have now today. Like, we are exploring and we have something cooking for, like, for the managed inference combined with, like, agentic tools. Because, for example, we also develop sandboxes nowadays. Right? So this is another, like, building blocks for p... Building... One of one of the building blocks that people use to build their agents. We have several other products that are in the earlier stages of development in that regard too.