Y Combinator Startup Podcast
00:06:31 · Garry Tan, Jeffrey Morgan
[00:06:31] S2: I mean, we're also entering this moment where AI safety is becoming more and more of an issue at the Frontier labs. So the Frontier may well slow down to figure out its alignment and containment issues. And then meanwhile, the open source models and open weight models are continuing to grow and get better. [00:06:51] S0: Yeah. And we saw the announcement and the release of the GLM 5.3 model and its capabilities from a cybersecurity standpoint being extremely impressive. It creates a big opportunity for whether it's startups or existing businesses in the security and governance space to really jump in and help. Because I think if you look at that AT&T article I was speaking about, largely, the blocker to adopting open models is largely around security and safety. But from our experience talking to customers, whether it's in Europe, whether it's here in The US, if you can solve the safety problems, by and large, adopting the Chinese origin model labs is completely on the table and it's really exciting for these businesses. [00:07:27] S2: There's sort of this interesting moment right now where a Hugging Face had to use open weight models to actually even detect the hack from the frontier. [00:07:36] S0: A common question we get is like, well, where where can I use open models that are leaps and bounds of an advantage over using a frontier closed model? And one of the key use cases is security testing, and making sure that your software's secure.
00:07:36 · Jeffrey Morgan, Jared Friedman, Garry Tan
[00:07:36] S0: A common question we get is like, well, where where can I use open models that are leaps and bounds of an advantage over using a frontier closed model? And one of the key use cases is security testing, and making sure that your software's secure. [00:07:51] S3: Yeah, because like if you try to get Claude to like pen test your product, it will just refuse to do that. [00:07:56] S0: Correct, yeah, by and large. [00:07:57] S2: Whereas there are literally obliterated security researcher models that you can find on Hugging Face that allow you to do it. [00:08:04] S0: That is true. There are ones that are custom trained to be even more liberal to go and attack these problems. But even the out of the box models, they do come with safety training, but they're a little better understanding if you're doing this for a good use case versus one that's more of a negative or malicious use case. [00:08:26] S3: A cool thing about Ollama is that because you guys are such a key distribution channel for these models, my understanding is that typically the model developers are contacting you before the general release to like coordinate launches and stuff like that, and you often get sort of like previews of what's about to about to happen, and you get these incredible growth spikes when a new model drops. Like, I wonder if you could like tell us a bit about like, what it's like to operate this thing at scale.
00:08:52 · Jeffrey Morgan
[00:08:52] S0: Yeah, absolutely. And like I said before, the models are coming out faster, and faster, and faster. And so we've developed a playbook to what does a successful day zero model launch look like? And there are a lot of things to get right. There's making sure that it's supported and your favorite inference engine, which is generally sometimes a multi week process to make sure it's fast, to make sure it's accurate, to make sure it's up to spec with the reference. There's also finding the right use cases and harnesses for developers and users to make use of this model. Generally these models have new capabilities. This morning, there's announced DeepSeek launched their first multimodal model from a large model LLM standpoint. They had previously had some smaller OCR models, which unlocks a whole bunch of use cases. But what they also changed was the DeepSeek harness which launched recently so that it could support this capability. So step two is then to find the harnesses, make sure they're prepared to actually run this model and to do that effectively. But every model is different, they all have different challenges and architecture changes and tool calling mechanics and getting all this right is really hard. I think the key thing to do at the end of the day is to run benchmarks against the final product ahead of release and make sure that it's running as the research team at the lab specified.
00:10:03 · Harj Taggar
[00:10:03] S4: So I think one very important part that you play in the whole ecosystem is you kinda create a very legible standard to be able to make sure you have the best way to use the hardness for each new model, each new tool call, and all of it is consistent across all the different models, which is pretty hard to do.
00:10:21 · Jeffrey Morgan
[00:10:21] S0: Yeah. I think there are three things really that we try to package together. One is harness, for example. Whether it's an off the shelf harness or an SDK to help use the model, some existing harness that is designed for this. And what's great now, there's so many great open source harnesses. The Codex harness is open source, OpenCode's a gray one and one of the most popular harnesses from Ollama users. Step one's getting that right, then you got to package it with the model, making sure that it's available, it's reliable, if it's in the cloud that there's enough capacity for it, because day zero tends to be the largest growth day obviously. And then importantly, there's the hardware and the providers. And that's actually where there's a lot of collaboration to be had, whether it's like an inference provider optimizing the model or with some of our partners, whether it's Nvidia or for example, working with the Apple Silicon stack, making sure that it actually runs the model really fast. Because if the model's capable but it's really slow, that's not a great experience. So getting those three things packaged into a box and generally you get the model, you're lucky if there's the ability to access the model a few weeks in advance. A lot of this stuff comes together in the last twenty four hours before the model gets released. And so it's generally a fire drill.
00:11:31 · Harj Taggar, Jeffrey Morgan
[00:11:31] S4: So you can almost make an operating system in the old world where you needed to really integrate very tightly with all the drivers, all the hardware, and then at the application level to make sure that all the apps were really tuned up well. And you are the glue for all of it. Right? [00:11:46] S0: I do think the OS, which is generally a cliche analogy to use, is a good one because you've got the drivers for the hardware and the providers and the inference layer. But you also have the application runtime and making sure that the harness works. Gluing that together, it's a very combinatorically large problem to solve. And so doing that well is really difficult. And so, but over time you develop pieces that allow you to quickly develop that, test it, release it, and kind of have this common runtime that can match any harness to any model. And that's kind of the role we're really playing for developers. [00:12:21] S4: You guys are very hardcore engineers, and you fine tune things all the way to... From the origins with Apple Silicon all the way now to DGX. How do you build such a deep technical bench with that?