Y Combinator Startup Podcast
00:25:35 · Garry Tan, Jeffrey Morgan, Harj Taggar
[00:25:35] S2: So did you say you can buy a bunch of these and chain them and actually run a 400B model? [00:25:40] S0: You can, absolutely. They have this really fast network link. And so you can stack them on your desk, almost like a miniature data center rack. [00:25:48] S4: There's thing that people do with the Mac minis. Absolutely. Now this is like the production version of it. [00:25:52] S0: I think that's what's so exciting is you're seeing both from Apple and Nvidia this incredibly incredible leap to next generation hardware that's built for these models and is effective at running [00:26:03] S2: them. So you'd say like this is this is the platform to get. Like you could make Apple Studios work, but like, if you want something that just can work, get DGX Spark. [00:26:13] S0: From our testing, both are very competitive. [00:26:15] S2: Got [00:26:15] S0: it. So I think a lot of Whatever you can what get you can get. Yeah. And then also, the tech stack you're looking for. I think there's an incredibly mature tech stack through the MLX project with Apple. They've done some amazing work to run LMs on the Mac Studio, but also the smaller Macs. And of course, the DGX Spark stacks. Just incredible. We're super excited as partners with NVIDIA for that. [00:26:35] S4: And this is gonna be a cool renaissance for personal desktops.
00:26:35 · Harj Taggar, Jeffrey Morgan, Jared Friedman
[00:26:35] S4: And this is gonna be a cool renaissance for personal desktops. [00:26:40] S0: I think so. And it's it's funny with Ollama's journey. We started local. Clearly, the coding agent demand is in the cloud. But that's gonna come back locally in our minds because the hardware will catch up when you have a GB 300 on your desk and you want the fastest coding loop that's as fast as running your tests or as fast as making code editor change. We all remember the GitHub Copilot experience of having the autocomplete come up in a few milliseconds... A hundred milliseconds. That experience will make its way back to the desk, which has just been a journey of starting local, going to the cloud, and then we think that'll come back local, and you'll end up using the two together. [00:27:15] S3: Speaking of, like, what you can get, in order to run a Ollama Cloud, you need like a shit ton of GPUs. What are you seeing in the GPU market? [00:27:24] S0: I think what we're seeing is ultimately the prices are changing very quickly. And the supply and demand volatility is very high there. And so I think ultimately for if you're a startup, getting access to some of the B200, B300 GPUs you need to run these latest models is very hard. Thankfully, there's a great set of inference providers building on top of that. And so we're seeing this extreme demand. Unlike, you know, and when you Are think you [00:27:50] S3: able to get all the GPUs that you need? Are you constantly like, growth limited by how many GPUs you can get your hands on? What's the current state?
00:27:50 · Jared Friedman, Jeffrey Morgan, Garry Tan
[00:27:50] S3: able to get all the GPUs that you need? Are you constantly like, growth limited by how many GPUs you can get your hands on? What's the current state? [00:27:58] S0: We're lucky that we've partnered with quite a few providers to work together to pool a bunch of GPUs together, which allows us to stay on top of our demand. But that's a lot of work. And it's definitely a lot of spending time thinking through which model will get run where, how fast should it be, which region is it in, what will the latency be for the customer. There's a lot of hard problems to solve in that stack. And I think what's really exciting about products like OpenRouter, Ollama, the OpenCode project is for an end user developer, they can sign up and get access to this without having to go negotiate prices on a B200, B300. Think about their twenty four month forecast in order to get access to some of these these, you know, GPUs. [00:28:43] S2: YC's next batch is now taking applications. Got a startup in you? Apply at ycombinator.com/apply. It's never too early, and filling out the app will level up your idea. Okay. Back to the video. [00:28:56] S3: Suppose you were like a a startup founder, and you were just starting out now, and you're building some AI company, and you haven't raised a lot of money, and so you like want to like use as many tokens as possible, like inexpensively. Like what would your advice be to that person about like how they can get like huge mileage with like a limited budget?
00:29:16 · Jeffrey Morgan
[00:29:16] S0: This new class of models, like DeepSeek Flash is a great example, and I think there'll be quite a few more, where it's ultra low cost per token. It's also low cost per task, which is a really important metric. And that class of models, in my mind will be the first ones that come down to this idea of like unlimited tokens. We all remember ChatGPT, you didn't really have to think about how many tokens you were using. You would just use it every day, you had unlimited. Ultimately, I think we return to that, but it's gonna take a lot of work in the model, the architecture to be custom trained for high volume token usage. And if you think about the start of the year, we really want, the open models got to the frontier of intelligence. We bridged the gap. We're maybe like less than three months behind between the frontier closed models and the open models. But the next problem to solve is extreme efficiency. Seeing for example, the GBT Luna model become very, very price effective for customers has been a huge boom. We talked a ton of customers where that kind of pricing enables widespread adoption within a team. I think we're gonna see that with open models. We already are seeing that with open models. I think the DeepSeek flash model is leading that charge. If we go back to the model breakdown on Ollama's cloud, the highest growth area is definitely the DeepSeek model and this is largely powered by the DeepSeek flash adoption. And so this new class of flash models where they're good enough for 80% of the tasks, they're really fast and they're ultra cheap. This new class of model that I think will enable some of those use cases.
00:30:40 · Jared Friedman, Jeffrey Morgan
[00:30:40] S3: Yeah, those are gonna be like the workhorse model to do like all the grunt work. [00:30:43] S0: Exactly, yeah. You won't have to be thinking about how many requests am I making, how many tokens. You'll be much more inclined to consume as much as you can because, you know, it's able to solve the hardest... Not the hardest, but you know, difficult problems. If you think back to the coordination layer we were talking about earlier too, being able to coordinate these flash models together to do different tasks can also yield great results that a bigger model can. So by having these cheaper models, not only are they more accessible, they can run faster and you can access them at higher volume, but you can start to chain them together and build new problems that are solved by orchestration on top. And that's a really exciting area for new startups, for existing inference providers, for some of the larger businesses today that solve workflow problems. Ultimately, being able to chain these models together is gonna be super helpful. And you won't have to think about the underlying costs.
00:31:35 · Garry Tan
[00:31:35] S2: Yeah. I guess when we first started talking about AGI, even on this podcast, there was sort of debate about, you know... And I think a lot of AI researchers would come out and say, like, there's just gonna be a giant god model, and it's gonna do everything. But, you know, I think so far, like, it hasn't quite worked out that way. Like, obviously, you still... You know, if you're if you have to literally hack the NSA, maybe you need mythos or something. But for the majority of use cases, like, you're talking about orchestration and you're talking about, like, smaller models, you you you know, that the task composition actually probably gives you a bunch of ways to make it more repeatable. It's more trustworthy. Like, it actually does work at a cost that is, like, possible. So, you know, if if it was gonna be a god model versus, like, lots of, you know, smaller special purpose or even just simpler models. It's turning out to be the latter so far.