The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:44:52 · Harry Stebbings
[00:44:52] S1: the use cases that they have throughout their business. Do you think you will see that commoditization at the model layer, whereby enterprise clients are able to really efficiently package the workflows that they do, so it does commoditize the model layer. Because right now it's not commoditized quite.
00:45:07 · Brendan Foody
[00:45:07] S0: Yeah, so I think the key distinction is that I think the API layer will get commoditized. You can definitely build stickiness and workflows that people people have on top of those APIs. Like for example, I have all of these routines running in Cloud Code, and I feel like it would probably be difficult, or I at least wouldn't put in the time to move those routines over, and I have a bunch of similar things running in ChatGPT. So I think that there's going to be various ways that people can build stickiness, but for pure API based products, where it's like if we are just spending $10,000,000 a year on a specific workflow, obviously we're gonna have an eval for that. And every time a new model comes out, we're gonna benchmark that and understand exactly how we should be hot swapping between models and distilling models. Why does the API layer get commoditized? Because the switching costs are zero. Like when the switching costs are zero, there's a new frontier model every two months. That means that we very quickly are going to swap them out, right? And ultimately the decisions that we make boil down to the score on the eval corresponding to that workflow. And so it's very easy to compare model to model one for one in a perfectly hot swappable way, which is almost the definition of a commodity.
00:46:19 · Harry Stebbings
[00:46:19] S1: I'm still reeling from your token spend with agents more than headcount. Because actually, Mark Benioff said the other day that they spent 300,000,000 on Anthropic, which seemed like a lot of money. But actually, when you baked it down, it worked out to be about 3.8% of developer salaries is being spent on Anthropic, which actually is much less than one would think. Yeah. What do you think that is in twenty four months' time? For Salesforce? Yeah. I
00:46:44 · Brendan Foody
[00:46:44] S0: don't know about twenty four months' time, but I would bet that in five years, the average enterprise spends more on compute than headcount. The reason for that is that the models are just becoming so capable that it seems like there is just enormous ROI to being able to have models do something for 100 ks a year, that is going to continue compounding at an exponential rate in a way that human intelligence is not going to. And so humans will still play an important role at the things models can't do, but I expect that cost of inference, cost of compute will exceed that. The reason that that's so interesting to me is that having an eval for your specific workflow, like say we take the case of Salesforce, having an eval for how good is a specific model at code generation and their use case is often a 10x lever on the price performance of that model. Because they can distill the model, they can have an open source model that is performing as well, if not better, for a dramatically lower cost. And so as we see this enormous shift towards compute and significant inference spend across every workflow in the enterprise, they're going to need to have evals that act as a source of truth for whether those workflows are being done correctly and whether they're using the right models to accomplish that. With the greatest of respect,
00:48:05 · Harry Stebbings, Brendan Foody
[00:48:05] S1: are evals today not relatively unhelpful? It's like, how good are you at driving around the corner for the driving test in a very specific way, but that's not how it works in the real world, and it's actually not very practical. [00:48:16] S0: That's exactly the problem, right? Is that we used to have this paradigm of all of the academic benchmarks that were totally disconnected from the outcomes that enterprises actually care about, where people were building everything ranging from GPQA for PhD level reasoning to IMO for Olympiad math to Humanities last exam for this long tail of academic problems no one really cares about. And now they're focused on how do we get the model to do this end to end workflow coordinating with multiple colleagues for a financial model or a slide deck like we were discussing. How do we get the model to build an entire SaaS application end to end? That's why there's this enormous build out in pushing the frontier of evaluation as a critical research problem for the next frontier of model development.
00:48:16 · Brendan Foody, Harry Stebbings
[00:48:16] S0: That's exactly the problem, right? Is that we used to have this paradigm of all of the academic benchmarks that were totally disconnected from the outcomes that enterprises actually care about, where people were building everything ranging from GPQA for PhD level reasoning to IMO for Olympiad math to Humanities last exam for this long tail of academic problems no one really cares about. And now they're focused on how do we get the model to do this end to end workflow coordinating with multiple colleagues for a financial model or a slide deck like we were discussing. How do we get the model to build an entire SaaS application end to end? That's why there's this enormous build out in pushing the frontier of evaluation as a critical research problem for the next frontier of model development. [00:49:05] S1: Okay, next frontier of model development. If I listen to everything that you just said, I would draw two conclusions. One, shit, we should just invest all of our money into OpenAI and Anthropic. And then the realization dawned on me that the majority of startups, and you can shoot me down, again, shoot me down, is the majority of startups, say, especially on the West Coast, use frontier models to see where they can go and how far they can push them. And then they use open source, often Chinese models, to get as close to that as possible at a much better cost basis, in which case OpenAI and Anthropic are inherently challenged by that much more cost efficient open source model, right or wrong?