Y Combinator Startup Podcast
00:08:52 · Jeffrey Morgan
[00:08:52] S0: Yeah, absolutely. And like I said before, the models are coming out faster, and faster, and faster. And so we've developed a playbook to what does a successful day zero model launch look like? And there are a lot of things to get right. There's making sure that it's supported and your favorite inference engine, which is generally sometimes a multi week process to make sure it's fast, to make sure it's accurate, to make sure it's up to spec with the reference. There's also finding the right use cases and harnesses for developers and users to make use of this model. Generally these models have new capabilities. This morning, there's announced DeepSeek launched their first multimodal model from a large model LLM standpoint. They had previously had some smaller OCR models, which unlocks a whole bunch of use cases. But what they also changed was the DeepSeek harness which launched recently so that it could support this capability. So step two is then to find the harnesses, make sure they're prepared to actually run this model and to do that effectively. But every model is different, they all have different challenges and architecture changes and tool calling mechanics and getting all this right is really hard. I think the key thing to do at the end of the day is to run benchmarks against the final product ahead of release and make sure that it's running as the research team at the lab specified.
00:10:03 · Harj Taggar
[00:10:03] S4: So I think one very important part that you play in the whole ecosystem is you kinda create a very legible standard to be able to make sure you have the best way to use the hardness for each new model, each new tool call, and all of it is consistent across all the different models, which is pretty hard to do.
00:10:21 · Jeffrey Morgan
[00:10:21] S0: Yeah. I think there are three things really that we try to package together. One is harness, for example. Whether it's an off the shelf harness or an SDK to help use the model, some existing harness that is designed for this. And what's great now, there's so many great open source harnesses. The Codex harness is open source, OpenCode's a gray one and one of the most popular harnesses from Ollama users. Step one's getting that right, then you got to package it with the model, making sure that it's available, it's reliable, if it's in the cloud that there's enough capacity for it, because day zero tends to be the largest growth day obviously. And then importantly, there's the hardware and the providers. And that's actually where there's a lot of collaboration to be had, whether it's like an inference provider optimizing the model or with some of our partners, whether it's Nvidia or for example, working with the Apple Silicon stack, making sure that it actually runs the model really fast. Because if the model's capable but it's really slow, that's not a great experience. So getting those three things packaged into a box and generally you get the model, you're lucky if there's the ability to access the model a few weeks in advance. A lot of this stuff comes together in the last twenty four hours before the model gets released. And so it's generally a fire drill.
00:11:31 · Harj Taggar, Jeffrey Morgan
[00:11:31] S4: So you can almost make an operating system in the old world where you needed to really integrate very tightly with all the drivers, all the hardware, and then at the application level to make sure that all the apps were really tuned up well. And you are the glue for all of it. Right? [00:11:46] S0: I do think the OS, which is generally a cliche analogy to use, is a good one because you've got the drivers for the hardware and the providers and the inference layer. But you also have the application runtime and making sure that the harness works. Gluing that together, it's a very combinatorically large problem to solve. And so doing that well is really difficult. And so, but over time you develop pieces that allow you to quickly develop that, test it, release it, and kind of have this common runtime that can match any harness to any model. And that's kind of the role we're really playing for developers. [00:12:21] S4: You guys are very hardcore engineers, and you fine tune things all the way to... From the origins with Apple Silicon all the way now to DGX. How do you build such a deep technical bench with that?
00:12:21 · Harj Taggar, Jeffrey Morgan
[00:12:21] S4: You guys are very hardcore engineers, and you fine tune things all the way to... From the origins with Apple Silicon all the way now to DGX. How do you build such a deep technical bench with that? [00:12:34] S0: I think a lot of our team, we aren't AI researchers by background. We're from VMware and Docker and from other networking companies. And so the classic compute problems are kind of reinventing themselves in the inference land. So largely, that's where we like to focus our time. But I think at the end of the day, it all comes down to the developer experience. What is it like when the developer makes a call to the API and gets tokens back? What happens? And there's more and more happening in that layer right now. And getting that right needs all the layers of the stack to work well together. And I think one of the challenges with open models has been that hasn't been happening at the rate of what a frontier model lab puts out, where they have the classic five layer cake that Jensen mentioned, which is like the apps, you have the model, you have the infrastructure and inference, you've got the chips, and you've got the energy. And they've got all that ready to go for developers on day zero. And that's really the thing that we're trying to reproduce for open models. Of course, we're not going to do every layer of the stack, but we can help orchestrate that. [00:13:37] S4: You layer the layers.
00:13:37 · Harj Taggar, Jeffrey Morgan, Jared Friedman
[00:13:37] S4: You layer the layers. [00:13:39] S0: Yeah, and maybe in open models there's more than five layers. Like that model layer actually has a lot to it, right? There's the model weights, but there's also a lot of the orchestration components that each model is uniquely good at. There's this developer API layer. There's so many opportunities to build between the model and the application layer that are kind of hidden in today's five layer cake stack. [00:13:58] S3: That's interesting. Do you think any of those hidden layers might get unbundled and become their own companies or providers?