"The Cognitive Revolution"
00:12:41 · Nathan Labenz, Anastasios Angelopoulos
[00:12:41] S0: I asked what the right mix of evaluation techniques is now and how confident we should be in our ability to evaluate these models at all. [00:12:51] S1: Well, it's a very good question in part because we now know that models are able to tell whether they're being evaluated or not. So if you are subjecting models to tasks that are transparently some form of test or evaluation, then they are going to exhibit a line of behavior there because they know that that's what they're supposed to do. And then elsewhere, on other types of real world tasks where it really matters, they might be misaligned and you would never know. It's a big problem. And what that means is that the only actual way to measure alignment is by getting into the flow of people's real work and checking whether or not the model is aligned or not, you know, in situ, like actually in production. So that's the that's the perspective that we take on the problem is that by measuring actual production traces from real people in the flow of real work, we're able to see is the model following the guardrails. Because we can check what the guardrails are with an independent auditor and then go and check whether or not the model took the right actions or not, what it touched, what it didn't touch, and report any violations. We can check whether or not the model... Anything that it says, whether it's reporting to the user that it changed the file in the correct way and or checked every line in the spreadsheet, we can check whether the model actually did that.
00:14:09 · Nathan Labenz
[00:14:09] S0: One big kind of narrative violation that we've heard from other sources, namely Andon Labs, who, as I'm sure you're aware, does these, like, Gonzo, you know, put a AI in charge of a business kind of experiments, is that OpenAI models, they have found to be more aligned, less inclined to do sort of, you know, sneaky, bad behavior in the course of running their businesses. I see basically a a similar signal on the Arena alignment index where Correct. GPT is holding the top five slots.
00:14:46 · Anastasios Angelopoulos
[00:14:46] S1: And what we've seen from this is that if you annotate all the traces that go through Arena, about, you know, single digit percentage of that... Of them, even from OpenAI and Anthropic models exhibit some form of misalignment. So perhaps the high level takeaway would be, OpenAI and Anthropic are the two most aligned labs. Alignment tends to follow capabilities. So more capable models also tend to be more aligned perhaps because these companies have, you know, really made an effort to make their models more aligned. But also the rates of misalignment are perhaps higher than people initially would think. Single digit percentage misalignment is a ton. It's a ton. It means that anybody that's had a thousand chats has experienced 10 at least, you know, 10 to 50, let's say, examples of a misaligned behavior like a lie or deception. And so it's pretty interesting. OpenAI is number one on our leaderboards right now. It is overanthropic, I believe, in all three of our signals. So there's actually not one of our signals in which anthropic is leading. It really speaks to the efforts that the OpenAI team has made to make the models more aligned in production. And, yeah, it is a narrative violation, but the reality of the matter is that these two labs are clearly the front runners in this area. Both are working very hard, to avoid misalignment. And then the third runner-up is, SpaceX AI, you know, just behind Anthropic and then Google. And then there's a big gap, between these and the and the front... And the, open source labs. You know, Tencent is number five. The deceptive completion rate of the Tencent model is 13%. So 13% of the time that it's making these claims, it's it's making them deceptively.
00:16:42 · Nathan Labenz, Anastasios Angelopoulos, Sagar Vaze
[00:16:42] S0: The day before, artificial analysis had published version five of its index with most of its benchmarks now private. The open weight models fell to the back, which some read as a sign of benchmark maxing. Prakash asked whether Arena could detect benchmark maxing on its own platform. [00:16:58] S1: Well, let me just say, first of all, zooming out, I'm actually pretty concerned about the way that the industry is heading with respect to benchmarks. I love the artificial analysis guys. They're great people. I don't think that it's healthy for our industry to be relying on privately held benchmarks that are just like a dataset that's collected by some companies. You know, a lot of them, you know, in cahoots with the labs to sort of like, you know, sell data that's very similar to these benchmarks to have people improve, and then they're not auditable by the community. I don't really think that that's a sustainable way for us to be evaluating models. I have a serious problem with it because it's not actually solving the fundamental issue of quote unquote benchmarking. You can benchmarks on a dataset that's private. All you need to do is find data that's similar to that dataset and then... Or, you know, test on it enough times. And because that set of questions is fixed and and you have the, you know, the issue of multiple comparisons, you're you're gonna be able to, like, you know, improve on it [00:17:54] S4: more and more and more and more until you get to number one.
00:17:54 · Sagar Vaze, Anastasios Angelopoulos
[00:17:54] S4: more and more and more and more until you get to number one. [00:17:56] S1: I don't really get the narrative about the benchmarking of the open models and so on and so forth simply because we don't really know whether the frontier closed source models in The US weren't benchmark maxing against the closed datasets that are now going into this index. You know, these companies have very close relationships with, like, Cognition and, you know, some of the other companies and artificial analysis that are building these closed datasets. What we really need is benchmarks that speak to the utility of AI to real people and real workflows. And there should be a consistent and scalable methodology to do this that doesn't overfit. When you have live fresh data coming in all of the time from, like, real production use cases, it can't be overfit. It cannot be overfit, at least statistically. You can argue that you're doing better on that distribution than you are on other distributions, but at the end of the day, for that customer set, you're doing better as you go more and more because every question is fresh. It's like you're constantly taking new tests. It's not like you have the same test and you're taking it many times. And so I worry if that's the direction that the ecosystem is going, that we're going to set ourselves up for failure in the long run.
00:17:56 · Anastasios Angelopoulos, Prakash Narayanan
[00:17:56] S1: I don't really get the narrative about the benchmarking of the open models and so on and so forth simply because we don't really know whether the frontier closed source models in The US weren't benchmark maxing against the closed datasets that are now going into this index. You know, these companies have very close relationships with, like, Cognition and, you know, some of the other companies and artificial analysis that are building these closed datasets. What we really need is benchmarks that speak to the utility of AI to real people and real workflows. And there should be a consistent and scalable methodology to do this that doesn't overfit. When you have live fresh data coming in all of the time from, like, real production use cases, it can't be overfit. It cannot be overfit, at least statistically. You can argue that you're doing better on that distribution than you are on other distributions, but at the end of the day, for that customer set, you're doing better as you go more and more because every question is fresh. It's like you're constantly taking new tests. It's not like you have the same test and you're taking it many times. And so I worry if that's the direction that the ecosystem is going, that we're going to set ourselves up for failure in the long run. [00:19:10] S6: So let's talk a little bit about model routing. So I believe you guys have just introduced a model router. Can you explain business reasons or the user reasons for introducing a router? Let me tell you what we