The Frontier with Brendan Foody
00:21:11 · Jonathan Frankle
It starts off with one example. A Waymo car starts off with one little driving course, probably on some Google campus somewhere, where there are some street signs and some traffic lights and some fake pedestrians that, you know, some fake people who like carefully walk in front of the car and we see if the car stops or not. What does that look like for you? It could be one example where you have one spreadsheet or one question you just keep asking, and you have a couple of golden answers, and you see if the output matches according to some mediocre LLM judge. But one example is a great place to start. And then two examples, and then three examples, whatever you would vibe test. I think whenever you see anyone use a new model, everybody's got their favorite question, whether it's the unicorn drawing or, you know, how many R's are there in Strawberry or what have you. I know that whenever a new model comes out, I have my favorite questions I ask. Internally at Databricks in our agents when the new model is there. And those become your e-vals. And then you start looking at like where do things go wrong? Where did somebody else complain? It's kind of vibe testing transforms into evals over time. And then eventually you can get systematic once you've deployed something at scale. You know, Waymo now, like, you know, as they run into these weird corner cases, they issue a recall because it's a real product. They add that to their eval suite and they collect a lot more data about that situation. I think that's how you bootstrap it.
00:22:26 · Brendan Foody, Jonathan Frankle
[00:22:26] S1: When these enterprises put a lot of time and effort into creating a robust eval, have you seen cases where they're comfortable sharing that with foundation model labs? Or in most cases, are you finding that they're extremely sensitive to that? [00:22:39] S0: I think they're extremely sensitive to that. And I'm gonna choose my words carefully here. But if you're a company, your data is basically your lifeblood. Like it's some sense, you know, and data is both the literal data you have and kind of the human capital, the data that's invested in that human capital. If you're building an LLM, data is also your lifeblood, but it's your lifeblood in the sense that you would like more of it. And so it's a very scary thing if you're an enterprise where you have your crown jewels, yeah, and you know that there's a company that both really wants to help you, but also really, really needs more data. That's a complicated proposition. So I don't know, being at Databricks, it's been nice to kind of be Switzerland in this conversation, where in some sense we're the place where the data lives, we're the place where the models arrive, and we can kind of intermediate a little bit to make it possible to measure agents and give some feedback without, you know, having this weird relationship where we would like the data for ourselves.
00:23:35 · Brendan Foody
[00:23:35] S1: And we're finding that, of course, this is the source of the real-to-sim gap in so many ways. If every enterprise had robust evals that they could share with the labs, we would solve the real-to-sim gap very quickly. But because they don't, there needs to be this huge workforce of contractors that we scale out to help Frontier Labs approximate that distribution throughout the entire economy. And of course, there is a gap on public benchmarks even today, as much as uh we're trying to make a lot of effort to resolve that. How do you think about public benchmarks today? Which parts of them are useful, which parts of them aren't? And what are what are the largest gaps that uh we as an industry need to solve?
00:24:15 · Jonathan Frankle
[00:24:15] S0: I mean, it's easy to bash public benchmarks. Nobody has ever liked a public benchmark. Like I think I've never, in my, you know, very long decade of having been an AI, which I guess makes me ancient at this point. I've never seen a public benchmark where someone said, Oh, that benchmark is amazing and clearly measures the real world. And people definitely aren't overfitting to that benchmark. I mean, CFAR 10, everybody hated, but that's also 32 by 32 images, where if you look at those images, you can't tell that that's an airplane or not. Um, ImageNet has like, I think it's 10% of the categories in ImageNet are just dog breeds. And then there's one category for cats. Like you can tell the people who created ImageNet and how they feel about that, but it's not a realistic benchmark. But the ImageNet benchmark alone, I think, is one of the most important contributions to the fact that we have all of this, you know, AI today. That benchmark showed that deep learning was valuable and led to a huge amount of investment over five years. And even though we totally beat the crap out of that benchmark to the point of uselessness, it's hugely advanced the field. So I think we can live in this in-between where new benchmarks challenge us and new public benchmarks are important because they give us a shared measuring stick. And we tend to bend them out of shape pretty quickly these days. And it's easy to wonder whether people are, you know, bench maxing or people are, you know, literally training on the benchmark data. And that's a risk. So they're useful to the field, they're useful to advancing science. That doesn't mean they're perfect, and they tend to have bias built into them of first of all, what is possible to build in the academic world. And second, what the academic world thinks is important for AI. For example, there's a lot of emphasis on coding right now, which is understandable. Coding is really popular. I don't think we're putting enough emphasis on kind of knowledge-related tasks or kind of knowledge synthesis tasks. That's very Databricks pilled. So take that, Freddy Will, and very enterprise pilled. You know, I think that's true.
00:25:56 · Brendan Foody, Jonathan Frankle
[00:25:56] S1: I mean, like software engineering is something like 1% of the economy, right? And in Silicon Valley, we feel like it's everything. [00:26:02] S0: Well, come to New York. Come get a bagel with me and you can see the rest of the economy. And I'm sure we can look out this window right now and we can see only tech companies. But I think there's also like even a bias in the RL environment world toward coding-related environments. The idea that a Docker container is sufficient to contain an environment is a very coding-pilled way to look at the world. When I look at the world from a Databricks or enterprise pill perspective, many of the environments I look at are tens of petabytes. Go put that in a Docker container. [00:26:29] S1: Yeah.
00:26:29 · Jonathan Frankle
[00:26:29] S0: And go make that something that you can easily modify and reset. Like that's actually a new systems problem that needs to be solved at scale. How do you allow an AI agent to work in a 10 petabyte environment, make some changes, and immediately revert that so it doesn't break prod? And how do you do that without literally breaking prod because the environment you're working in is also your production environment? That's something very different than I think the academic world is focused on. And it's not a bad thing. These are just different perspectives, different problems. But I think the thing that's always missing in any academic task, you know, going back to time immemorial in computer science is scale. The story goes that when Google came into existence, it basically killed the field of information retrieval in academia for a decade. Because why would you do information retrieval on a nice synthetic task in academia when you literally have the whole internet? It's just a different scale of problem. And we're watching that happen in academia again. Like look at the number of faculty, yours truly included in some sense, who either never arrived in academia, sorry to everybody at Harvard, or who, you know, have left academia recently because I'm sorry, but working on 16 H100s is just not the same as working at cursor training incredible models or working at Databricks training models or what have you. It's the same thing with benchmarks. We're missing that scale. There's no academic benchmark that works on a tens of petabytes data set, nor can there be.