Unsupervised Learning with Jacob Effron
00:10:23 · Erik Bernhardsson
[00:10:23] S1: Yeah. I think a massive source of alpha in this market is just, like, not being afraid to do hard things. So we went all in when we realized, like, in order to develop... To to to... In order to to to deliver on the developer experience we wanted to have in the very early days, we realized, like, kinda have to build our own file system to do container images really fast. So we we started building that in the very early days, and and that become, like, over the next few years, you know, still is, like, a very massive source of of alpha. It... It's been it's been a longer journey for sure. Like, we've had to go a much longer route, like building our own file system, building our own container runtime, our own schedule. Like, we don't use Kubernetes and Docker. Like, we we we use, like, very little off the shelf software. We use gVisor, which is a fantastic Google product for isolation. Recently started using also KVM for for certain tasks. But but like decided pretty pretty early on that like a lot of the stuff we're gonna have to build ourselves. And and I'll let us do some really cool stuff. I I think, know, one of the best examples of where that shows up is we can do cold starts extremely fast. Turned out like the the initial... The developer experience that we wanted, very fast feedback loop when you're launching stuff in the cloud. That is also very helpful when you do building auto scaling applications that need to scale up and down very fast. So so we we... So first of all, the the loading the container is very fast, but over time, we also launched a a really... A couple of really cool features around CPU snapshotting and GPU snapshotting. So we can, like, very fast load data from the disk or from network straight into the GPU and and fire up, like, even these, like, large models in a few seconds, which is really cool. And and over the last six months, we we built this incredibly complex system that basically uses RDMA over peer to peer between the nodes to copy data very fast. So I I I think a lot of this stuff is, like, kind of, you know, accumulating and and really starting to pay off. Like, we we built some crazy stuff, and and now that's, you know, we're we're really kind of, you know, reaping the devided... Dividends of of of that... Those sort of investments that we made for many, many years.
00:12:25 · Erik Bernhardsson, Patrick Chase
It was a longer route, but, you know, now we... I think we have some incredibly unique technology that's very hard [00:12:31] S2: to replicate. And and then kind of moving up to to inference, what do you view as... You guys have made, you know, tons of investments there as well and and optimizations. What do you view as the unsolved problems in in the inference world or in the inference stack?
00:12:45 · Erik Bernhardsson
[00:12:45] S1: I I think we're still pretty early with inference. I I think, you know, a lot of stuff is like... It's it's like one of those, like, very empirical sciences. Like, I think people are trying different things and seeing what works. I mean I mean, over time, like, people have kinda figured out a lot of stuff. You know? Take something like disaggregate it prefilled. Right? Like, it's a very nonobvious idea. Like, you you sort of, you know, do the prefill on on on a set of notes and then you copy the data over RDMA and do the decode and kinda swap. There's some, like, very non obvious things, but, we've invested a lot in those types of things. The KV caching, there's a lot of complexity around, like, KV caching. Of course, like running more efficient kernels, that's something people do a lot. There's a lot of these things that, you know, we we have a team of, I think, 20, 30 people right now who are extremely focused on just like LM performance optimization. And, you know, it takes a lot of work. You have to stay up with with research, you have to read papers, you have to look at all the software coming out all the time and try try and and see what works, what doesn't work. It's still kind of an empirical science. Over time, I think, you know, patterns will emerge. I think also the models themselves will change very quickly. Like, now we have, like, mixture of experts and more complex things over time. Like, who knows? Like, maybe we'll swap out the transformers for something else and, you know, and that would change how, like, kvcaching works. Like, there's all these things, like, that could, you know, change overnight if someone figures out, like, a very different paradigm for all these things. And maybe they will, maybe they won't. I don't know. There's also obviously a lot of stuff around quantization and and and other things. So so, yeah, it's it's it's it's a wild world out there. But but, yeah, we're we're we're we're playing around with all these things.
00:14:20 · Jacob Effron, Anne Maris
[00:14:20] S0: But it feels like there's so many folks that are working on, you know, different parts of of of improving inference. And, obviously, what's, I think, been really interesting in the last two, three years is a lot of those optimizations and improvements have been in open source, and for the broader community, maybe shared via research. As a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, hey, we're gonna benefit from just the entire community pushing this space Hi. [00:14:46] S3: I'm Anne Maris, and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your listenership, and see you next episode. [00:15:02] S0: As like a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, we're gonna benefit from just the entire community pushing this space
00:15:02 · Jacob Effron, Erik Bernhardsson
[00:15:02] S0: As like a CEO, how do you think about the level of... The kinds of problems you want your team to go solve versus it's like, we're gonna benefit from just the entire community pushing this space [00:15:12] S1: forward? We actually love open source because I think a lot of our alpha is is in the proprietary side, in the infrastructure layer, like all this cool stuff around auto scaling for instance. I think unlike other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLang team. We contribute very actively to both those repos and do a lot of optimizations with them. And and so I I think this may be a little bit of a different approach than than maybe other inference providers. Of course, there's gonna be, you know, certain things that, you know, we we keep internal, but but almost everything we we push to to open source. And I and I think it's, you know, it's because, like, we have so much alpha in the infrastructure anyway. Like, we might as well just, you know, work with the amazing open source engines that are out there that maybe were... You know, they weren't state of the art, like, you know, a few years ago, but now they are. And and and that's an incredible thing that, like, we'd love to leverage, frankly.
00:15:12 · Erik Bernhardsson, Patrick Chase
[00:15:12] S1: forward? We actually love open source because I think a lot of our alpha is is in the proprietary side, in the infrastructure layer, like all this cool stuff around auto scaling for instance. I think unlike other inference providers, we've actually leaned very heavily into open source. We work very closely with both the VLM team and the SGLang team. We contribute very actively to both those repos and do a lot of optimizations with them. And and so I I think this may be a little bit of a different approach than than maybe other inference providers. Of course, there's gonna be, you know, certain things that, you know, we we keep internal, but but almost everything we we push to to open source. And I and I think it's, you know, it's because, like, we have so much alpha in the infrastructure anyway. Like, we might as well just, you know, work with the amazing open source engines that are out there that maybe were... You know, they weren't state of the art, like, you know, a few years ago, but now they are. And and and that's an incredible thing that, like, we'd love to leverage, frankly. [00:16:07] S2: You know, you were talking about heterogeneous compute and the the different places where you're getting compute from and also how the, you know, kind of inference stack is changing. How important are, you know, the long tail of chips to you guys, the non NVIDIA chips and and things like that as you think about your supply or kind of what the... What your serving stack looks like over time?