Y Combinator Startup Podcast
00:10:21 · Jeffrey Morgan
[00:10:21] S0: Yeah. I think there are three things really that we try to package together. One is harness, for example. Whether it's an off the shelf harness or an SDK to help use the model, some existing harness that is designed for this. And what's great now, there's so many great open source harnesses. The Codex harness is open source, OpenCode's a gray one and one of the most popular harnesses from Ollama users. Step one's getting that right, then you got to package it with the model, making sure that it's available, it's reliable, if it's in the cloud that there's enough capacity for it, because day zero tends to be the largest growth day obviously. And then importantly, there's the hardware and the providers. And that's actually where there's a lot of collaboration to be had, whether it's like an inference provider optimizing the model or with some of our partners, whether it's Nvidia or for example, working with the Apple Silicon stack, making sure that it actually runs the model really fast. Because if the model's capable but it's really slow, that's not a great experience. So getting those three things packaged into a box and generally you get the model, you're lucky if there's the ability to access the model a few weeks in advance. A lot of this stuff comes together in the last twenty four hours before the model gets released. And so it's generally a fire drill.
00:11:31 · Harj Taggar, Jeffrey Morgan
[00:11:31] S4: So you can almost make an operating system in the old world where you needed to really integrate very tightly with all the drivers, all the hardware, and then at the application level to make sure that all the apps were really tuned up well. And you are the glue for all of it. Right? [00:11:46] S0: I do think the OS, which is generally a cliche analogy to use, is a good one because you've got the drivers for the hardware and the providers and the inference layer. But you also have the application runtime and making sure that the harness works. Gluing that together, it's a very combinatorically large problem to solve. And so doing that well is really difficult. And so, but over time you develop pieces that allow you to quickly develop that, test it, release it, and kind of have this common runtime that can match any harness to any model. And that's kind of the role we're really playing for developers. [00:12:21] S4: You guys are very hardcore engineers, and you fine tune things all the way to... From the origins with Apple Silicon all the way now to DGX. How do you build such a deep technical bench with that?
00:12:21 · Harj Taggar, Jeffrey Morgan
[00:12:21] S4: You guys are very hardcore engineers, and you fine tune things all the way to... From the origins with Apple Silicon all the way now to DGX. How do you build such a deep technical bench with that? [00:12:34] S0: I think a lot of our team, we aren't AI researchers by background. We're from VMware and Docker and from other networking companies. And so the classic compute problems are kind of reinventing themselves in the inference land. So largely, that's where we like to focus our time. But I think at the end of the day, it all comes down to the developer experience. What is it like when the developer makes a call to the API and gets tokens back? What happens? And there's more and more happening in that layer right now. And getting that right needs all the layers of the stack to work well together. And I think one of the challenges with open models has been that hasn't been happening at the rate of what a frontier model lab puts out, where they have the classic five layer cake that Jensen mentioned, which is like the apps, you have the model, you have the infrastructure and inference, you've got the chips, and you've got the energy. And they've got all that ready to go for developers on day zero. And that's really the thing that we're trying to reproduce for open models. Of course, we're not going to do every layer of the stack, but we can help orchestrate that. [00:13:37] S4: You layer the layers.
00:13:37 · Harj Taggar, Jeffrey Morgan, Jared Friedman
[00:13:37] S4: You layer the layers. [00:13:39] S0: Yeah, and maybe in open models there's more than five layers. Like that model layer actually has a lot to it, right? There's the model weights, but there's also a lot of the orchestration components that each model is uniquely good at. There's this developer API layer. There's so many opportunities to build between the model and the application layer that are kind of hidden in today's five layer cake stack. [00:13:58] S3: That's interesting. Do you think any of those hidden layers might get unbundled and become their own companies or providers?
00:13:58 · Jared Friedman, Jeffrey Morgan
[00:13:58] S3: That's interesting. Do you think any of those hidden layers might get unbundled and become their own companies or providers? [00:14:06] S0: Absolutely. There was a really good talk from some of the Anthropic team, the platform team, and they talked about three big things. One was knowledge. How do you connect your company's data and context to the model? One is coordination. So as you know, when you make a request to in your Cloud app or Codex, it goes off and spins out a bunch of sub agents, some of those in the cloud, some locally. There's a coordination problem. And then lastly, there's an execution problem, which we talk a lot about as sandboxes. But these agents, more and more of them are moving to the cloud. There's a huge compute problem to be solved there. And if you look at what's happened classically in the cloud business, open source has meant that there are best of breed companies for each of those problems. Whereas you might have had kind of the, if you look back to the original generation cloud products, you have like the Heroku of the world, you have Google App Engine, where all those things were bundled together. But what developers ended up preferring is best of breed products for each one.
00:15:00 · Garry Tan, Jeffrey Morgan
[00:15:00] S2: It's the nivolism, which is all things are just bundling or unbundling. Exactly. Like the frontier model. Frontier Model Labs want you to be totally in their walled garden of manage agents and their context, their memory layer. And then meanwhile, Little Tech and all the founders out there and all the open source developers don't wanna be caged in. So we're gonna make all this other stuff. And it'll it'll be an interesting moment to figure out like what ends up winning. I mean, it'll probably be some mix of both. [00:15:27] S0: I think so. And you've got this abundance of open model tokens that's being created. There are just dozens of open model providers that are able to serve these tokens. The new scarcity, the problems now are what's above the tokens, right? How do you orchestrate an agent from A to B? These are problems that are tons of new systems and engineering problems that are just really hard to solve for an individual dev. Like there's no way they're going to build all those layers.