Y Combinator Startup Podcast
00:20:41 · Jared Friedman, Jeffrey Morgan, Garry Tan, Aaron Epstein
[00:20:41] S3: And speaking of Apple Silicon, should we talk about local models? Because you're you're in a bit of a unique position and be because you have large businesses both in cloud hosted models and locally hosted models that will run on your laptop. What are you seeing in those two worlds, and what do you [00:20:56] S0: think is gonna happen? I think it's incredibly exciting because it's similar to the closed versus open model question. It'll be a mix in our mind. And that's what we hear from customers as well, where for easier tasks, could run them locally and with lower latency and of course lower costs when it comes to the per token costs. Ultimately you're buying hardware upfront. And you'll use that in conjunction with these cloud models. What's exciting about this next generation of hardware, which we've had for a few years now, is just how good they are at running the 20,000,000,000 parameter to 40,000,000,000 parameter range of models, sometimes up to 120,000,000,000 parameters. [00:21:32] S2: Yeah. Qwen 3.838 b is now as good as OPUS 4.6 for coding. Is that right? [00:21:38] S0: That's what the benchmark showed. [00:21:40] S2: Yeah. That's wild. [00:21:40] S0: It's incredibly exciting because you can run that on not the the lowest memory MacBook, but the second lowest memory MacBook you can find from this store. So it's incredible. [00:21:50] S3: And are you seeing your users do that? [00:21:52] S1: Like, how are you [00:21:52] S3: seeing people use the local models versus the cloud hosted models for in practice?
00:21:52 · Jared Friedman, Jeffrey Morgan
[00:21:52] S3: seeing people use the local models versus the cloud hosted models for in practice? [00:21:56] S0: From the side of which models they're running, we're seeing a really solid mix of US and Chinese trained models being used for local. And we have the incredible models from the original Llama models, of course, but also the Gemma models from DeepMind. These are great choices for local. But when it comes down to use cases, coding agents by and large are most effective with the large cloud models. You're solving really hard problems, you're writing code, tests, it's really difficult versus some of the document processing workflow use cases that run extremely well locally because they don't have as difficult of a task in the end to end problem you're trying to solve. And so that's where we see this hybrid execution model where some of the easier, more straightforward tasks run locally. And then you have a router that can help decide, hey, we need to go to a large cloud model for this. And I think what that means for customers is that you're really dropping the costs even further when you're going from open models. Not just because they're cheaper to run-in the cloud, but because now you can run them effectively for free on the hardware you're buying for your business anyways. And what we're seeing ultimately from the cloud coding agent models is it is predominantly Chinese models being consumed today. And for the local models, it's a really strong blend of US, Europe,
00:23:10 · Jared Friedman, Jeffrey Morgan, Garry Tan
[00:23:10] S3: and Chinese origin models. Yeah. These two graphs are pretty stunning in comparison. Basically, for local models, The US and Chinese models are neck and neck, or like tied. And for cloud hosted models, The US is recoloring the x axis. It's 100% Chinese models. Basically, we need more US labs to make large models. That what this graph is showing? [00:23:32] S0: Effectively. And, you know, with the launch of the Nemotron Ultra model, we're seeing kind of the first wave of that, and it's really exciting. [00:23:38] S2: NVIDIA as a company is so interesting because their moat is not, like, trying to start new software businesses or sell, you know, tokens. Seem to be quite interested in just releasing a lot of open source and helping the ecosystem. And then the fact that they do that then helps them stay ahead of the game on the hardware side. [00:24:00] S0: I think so. Ultimately, NVIDIA, what's so incredible is there's helping power an ecosystem around open models, whether that's the hardware, the models. We've seen the new DGX station computers that they're working on which have I want GB 300 on your desk [00:24:17] S2: How do we get on that desk. [00:24:18] S0: That isn't deafening [00:24:20] S2: Yeah. Loud. Yeah. Do you know the price point on that thing yet? [00:24:23] S0: I don't know it off the bat. [00:24:24] S3: It's gotta be like Garry buying it. [00:24:25] S2: I know. Right? Well, I looked it up. It's... I mean, you can probably run a Frontier model for, like, I mean, very slowly for like $2.02, $300,000. Is that is that right?
00:24:25 · Garry Tan, Jeffrey Morgan, Harj Taggar
[00:24:25] S2: I know. Right? Well, I looked it up. It's... I mean, you can probably run a Frontier model for, like, I mean, very slowly for like $2.02, $300,000. Is that is that right? [00:24:36] S0: I think it's even more competitive than that. Yeah. And you can run more than a Frontier model at high speeds at a a price point that isn't very far off what you can buy from a classic workstation computer. [00:24:48] S2: Oh, no way. [00:24:48] S0: If you think about quite a few of the customers we talked to, some of them are banks for example or industrial businesses. They already have these NVIDIA workstation GPUs in every single engineer's desk. Some of them tens of thousands of them. And so this is No. I want one. [00:25:02] S4: They've been selling them for [00:25:03] S0: How do [00:25:03] S2: we get [00:25:04] S4: all the CAD work [00:25:04] S0: for for this is for the kind of original RTX a a six thousands, but this is the next generation [00:25:10] S2: of that. Yeah. Yeah. I'm gonna have to email Jen some [00:25:12] S0: This was at this was at GTC. So what we see here was at GTC. And we were one of the first people along with Elon and a few others to receive the DGX Spark as which sits on your desk and provides, you know, 128 gigabytes of unified memory to run that kind of 20 to 120B model range. But obviously that's just the beginning of a whole new range of hardware that can run the biggest models. [00:25:35] S2: So did you say you can buy a bunch of these and chain them and actually run a 400B model?
00:25:35 · Garry Tan, Jeffrey Morgan, Harj Taggar
[00:25:35] S2: So did you say you can buy a bunch of these and chain them and actually run a 400B model? [00:25:40] S0: You can, absolutely. They have this really fast network link. And so you can stack them on your desk, almost like a miniature data center rack. [00:25:48] S4: There's thing that people do with the Mac minis. Absolutely. Now this is like the production version of it. [00:25:52] S0: I think that's what's so exciting is you're seeing both from Apple and Nvidia this incredibly incredible leap to next generation hardware that's built for these models and is effective at running [00:26:03] S2: them. So you'd say like this is this is the platform to get. Like you could make Apple Studios work, but like, if you want something that just can work, get DGX Spark. [00:26:13] S0: From our testing, both are very competitive. [00:26:15] S2: Got [00:26:15] S0: it. So I think a lot of Whatever you can what get you can get. Yeah. And then also, the tech stack you're looking for. I think there's an incredibly mature tech stack through the MLX project with Apple. They've done some amazing work to run LMs on the Mac Studio, but also the smaller Macs. And of course, the DGX Spark stacks. Just incredible. We're super excited as partners with NVIDIA for that. [00:26:35] S4: And this is gonna be a cool renaissance for personal desktops.
00:26:35 · Harj Taggar, Jeffrey Morgan, Jared Friedman
[00:26:35] S4: And this is gonna be a cool renaissance for personal desktops. [00:26:40] S0: I think so. And it's it's funny with Ollama's journey. We started local. Clearly, the coding agent demand is in the cloud. But that's gonna come back locally in our minds because the hardware will catch up when you have a GB 300 on your desk and you want the fastest coding loop that's as fast as running your tests or as fast as making code editor change. We all remember the GitHub Copilot experience of having the autocomplete come up in a few milliseconds... A hundred milliseconds. That experience will make its way back to the desk, which has just been a journey of starting local, going to the cloud, and then we think that'll come back local, and you'll end up using the two together. [00:27:15] S3: Speaking of, like, what you can get, in order to run a Ollama Cloud, you need like a shit ton of GPUs. What are you seeing in the GPU market? [00:27:24] S0: I think what we're seeing is ultimately the prices are changing very quickly. And the supply and demand volatility is very high there. And so I think ultimately for if you're a startup, getting access to some of the B200, B300 GPUs you need to run these latest models is very hard. Thankfully, there's a great set of inference providers building on top of that. And so we're seeing this extreme demand. Unlike, you know, and when you Are think you [00:27:50] S3: able to get all the GPUs that you need? Are you constantly like, growth limited by how many GPUs you can get your hands on? What's the current state?