Unsupervised Learning with Jacob Effron
00:19:49 · Jacob Effron, Erik Bernhardsson
[00:19:49] S0: Alright. Super interesting. And then one thing I imagine is so interesting in in your role too is, like, you're obviously predicting for, like, what your compute fleet should be. And, obviously, you see what the workloads are today, and you probably have some sense of of, you know, what models run well on what chips. Then you're also trying to predict, like, hey, a year or two from now, what kind of workloads will we have? I think there's lots of different debates about whether we're gonna... Most tokens running will be really large models or actually we saturate in those tasks and these models will get smaller and smaller. As you think about kind of the evolution of your fleet in the next year or two, like, what... Yeah. What's your, you know, perspective on on, like, where you'll see relatively more different kinds of workloads and less? [00:20:26] S1: I I think there's been kind of, like, basically over a trend of, like, like, larger and larger models over the last three, four years. [00:20:33] S0: Bet against that.
00:20:33 · Jacob Effron, Erik Bernhardsson
[00:20:33] S0: Bet against that. [00:20:34] S1: I I kind of... You know, I I I definitely feel like that trend is gonna continue. Like, we're gonna need... Like, if b three hundreds had more memory, I'm sure, you know, a Kiwi k three would be bigger. Like, people are kind of, like, you know, training the models in a way that they fit on these these chips. So I I I think on the... At least on the LM side, like, the... You know, LM side, in a way, like, kind of a crude, like, type of world model. Like, you're trying to, like, encode as much information as you can in these LMs. Like, you need, like, very big, like, almost, like, lookup tables of facts, you know, anyway. It's kind of a conception how to think about it. And and and that sort of pushes, like, the memory requirements a lot. I mean, at some point, like, it might taper off. Like, who knows? But but, like, for now, like, people will make use of every possible memory. Like, the more memory, like, they'll just train more parameters for sure. [00:21:21] S0: You obviously sit in, like, the the... In some ways, unlike the eye of the hurricane of, like, a very exciting, large, and competitive market. And so I'm curious as you think about what ultimately will kind of differentiate players in this space and win out. I think there's so many true believers in the size of the overall inference market, and I think lots of folks are asking what actually, you know, wins out in the end of the day. I know you have a good thesis on this, so I would love it for you to to share with our listeners.
00:21:49 · Erik Bernhardsson
[00:21:49] S1: I I mean, I think a lot of the inference market will be quite rough. Right? Like, think the sort of straight down the middle, just like, you know, serving tokens, you already see sort of open router basically commoditizing the underlying providers. I tend to think, you know, like, the the the real alpha or, like, the real, like, point of differentiation, like, there will be a lot of differentiation just in, like, more efficient serving. And... But but it is a little bit of a race to the bottom, and I think and I think there's gonna be a lot of opportunities to sort of stay ahead of competition, and you're gonna have to, like, put a lot of efforts into that. But but I think a more sort of durable differentiation will definitely be in like, know, providing the entire platform of a lot of different things, which is like, I think something we are quite well positioned to do. We have the sandboxes, we have reinforcement learning, a lot of customers doing also normal fine tuning. A lot of customers, obviously, you know, deploying other types of models. We have batch processing. I think I I think my my my view in general is, like, customers, they don't wanna buy, like, single trick ponies. Like, they wanna buy, like, a multiproduct set of tools. So that to me is, like, ultimate... Like, probably the biggest source of differentiation is, like, you know, how do you provide that that entire set of tools? How do you provide this, like, very general purpose thing that lets engineers, like, iterate very fast, deliver value? And then, like, this is a day to day. Like, still gonna have to, like, go and do the knife fighting of, like, you know, staying ahead of your competitors in terms of performance and use the latest kernels and the much more efficient KV cache. And there's gonna be a lot of that stuff for a long time because there's just so much money in it. But I I... I'm kind of pointing out. I I think that's sort of that's like less durable as an advantage. Think that, know, because over time, like, people... These tricks will sort of spread and people figure out how to how to copy them. So the more durable thing I think is like being that sort of portfolio of a lot of stuff around inference, not just the inference in itself.
00:23:35 · Erik Bernhardsson, Patrick Chase
You know, having all the models and and and and a lot of, you know, stuff like around reinforcement learning maybe or or other things. [00:23:41] S2: You know, the biggest modal customers are are on multiple products. Maybe can you touch on a few of the more interesting ones? [00:23:49] S1: So, yeah, we we have some incredibly cool customers like Cognition is using us at large scale for both for inference and and sandboxes to do a lot of reinforcement learning on model. These sandboxes for all the rollouts. We have customers like Ramp using this for background agents for the internal coding agent that they actually launched as a as a standalone product today that they're selling. We have customers like Suno and and Runway that uses regenerative video and audio. Decagon is another customer that uses us for a lot of customer support stuff, voice and and text. DoorDash uses us for a bunch of internal applications, mostly LMs, but also all kinds of other stuff. We actually have a lot of customers in in in computational biotech using us for for various types of drug discovery, using, you know, molecular simulations and sequence alignment and all kinds of stuff. We have a bunch of customers in robotics, one x and and a few other ones using us to train robot models, like doing a lot of training, but also batch processing of video and things like that. A lot of customers in the data generation space. So so customers who... They're building environments and and running benchmarks in model often using sandboxes.
00:24:55 · Jacob Effron
[00:24:55] S0: One thing you kind of mentioned is portfolio of products, and I think that's like one of the most interesting parts of business you're building. I imagine, honestly, one of the hardest parts must be just prioritization of all the different things you could build. There's probably so many new things that come up of really exciting tools you can build for developers, and I guess I'm curious, obviously, Sandboxes, you have incredible products around coding agents, reinforcement learning, how do you think about building down that vector? There's also the fine tuning or all as a service stuff that some of your competitors seem like they they prioritize. Like, how do you think about, I guess, you know, where to prioritize within that portfolio?
00:25:34 · Erik Bernhardsson
[00:25:34] S1: I I think it's an incredibly hard problem at, I mean, most startups, but especially multiproduct companies. Right? Like, the sort of balance of, like, you know, first of all, like, when do you go multiproduct, and and then where do you put your bets? Like, we're a 150 engineers or something like that to... Today, and and I struggle with it every time, like, every day. Like, you know, I wish we could, you know, do that and then, you know, and whatever. And and, like... But I kinda know that even if we had a thousand engineers or 10,000 engineers, like, it's all, you know There's lots of build. Dollars. Like, know, I wish we could build a rocket and go to the moon and whatever. Like, it just never goes away. So so, like, you kinda have to be just, like, ruthless with prioritization and and really think about, you know, what's the highest ROI and, like, where do you draw the line? And you have to be sometimes, like, pretty aggressive also saying, like, actually, like, I know we started on this project. I know you're really excited about it, but, like, this other thing kinda came up. So, like, we're gonna have to pause this for a second. We'll get back to it later. We've had to do that many times too. It sucks when it happens, but I I really think, you know, that sort of prioritization and, like, how do you how do you allocate resources? I mean, resource allocation is, like, basically, like, you know, the number one thing that I think a CEO has has to has to really, you know, be ruthless about it, a company, and like so much value is being created, like allocating the the work to the right resources.