"The Cognitive Revolution"
00:05:02 · Nathan Labenz, Michael Freeman
[00:05:02] S0: Michael Freeman, senior director of threat research at ServiceNow, has spent his career finding and exploiting vulnerabilities. Prakash asked how a company with decades of legacy systems gets to vulnerability zero in two years. [00:05:15] S5: I think this is where we need to step back and say, we have a lot of really smart people doing AI, but they don't know cybersecurity at all. And they view cybersecurity as a software flaw. But the reality of it is why we have millions of software flaws in any of our typical tech stack, it's the business processes that actually introduce some of the most critical vulnerabilities that we have. And it's the, okay, here's a list of risks. I'm just gonna go ahead and sign off on them without really truly understanding them. Or, you know, I'm gonna put my customer's API keys in a ticket and they not think about it. Or a help desk who has, you know, when users call in because they they've locked out their passwords, everybody gets the same default winter 2026 exclamation point as password reset and you got twenty four hours to replace it. So it's a processes where humans get, lazy that they introduce. It's gonna never go away, and no amount of hatching is gonna fix that.
00:06:21 · Nathan Labenz
[00:06:21] S0: This is the AI in the AM weekly highlights with my cohost, Prakash Narayanan. I'm Nathan Labenz. The narration is my cloned voice, and we'd love your feedback on what works and what doesn't. Today's episode is brought to you in part by Lovable, the software creation tool that brings ideas to life. I think we can all agree that any positive future for humanity includes eating more fresh, delicious, in season fruit. For years, I thought someone should build the ultimate guide to fruit, cataloging the thousands of varieties that most of us have never tried, where and when they grow, what they actually taste like, and where you can get them. Finally, I realized that that someone could be me, and now I'm building it unlovable. It's called Fruitopia. Tell it where you live and what you like, and it'll show you what's in season near you this week, plus the most interesting fruit from around the world that can actually be shipped to your door. It has a really cool what it tastes like spectrum, countless recipes, names of fruit in many different languages, a guide to fruit festivals all around the world, and an AI assistant to help you explore. You probably know Lovable as a pioneer of AI vibe coding, but I found that Lovable's real superpower is its enterprise grade infrastructure. Scalable hosting and databases, a secure auth module, and the weekly job that checks the latest inventory is all handled on the back end, which frees me to spend more time on the product experience. And because Lovable has an MCP server, I can use any coding agent I want to drive the build while Lovable keeps everything running. Visit fruitopia.ai to explore more than 400 fruits and more than 10,000 varieties from all around the world. And next time you have an idea that really should exist, stop dreaming, go to lovable.com, and bring your idea to life. Part one, the ugly result. On Monday, OpenAI said an internal model had refuted the exponential time hypothesis, a conjecture about how hard a core class of problems has to be, with the proof checked in lean. Anthropic said Claude had found the same thing the same day. Neither claim is peer reviewed. Prakash and I opened the show on it. So my understanding is, like, the disproof was by example, basically. Right?
00:08:47 · Nathan Labenz
They took one of these problems that were thought to be exponential time hypothesis hard and showed that they can come up with an algorithm for that particular problem that is faster. Yeah. Therefore, the hypothesis is false. My understanding generally has been that they're not that much faster, and most people still seem to think between p equals NP and p does not equal NP that it's probably not. Other words, some things will continue to be very hard to do. They just don't seem to fall into this, like, clean class that we used to think that they fell into. It's a weird thing because I... You know, I've also seen some commentary that's like, this is sort of ugly. People seem to like the kind of clean classes that they had before where it seemed like there was something elegant about it or there were... You know, they... It's it sort of suggested some some deep structure to the universe that was, like, grokable and beautiful and whatever. And, potentially, there's a certain bias among the human mathematicians toward accepting that because it did seem kind of aesthetically appealing. And now we have these like, oh, you're able to get under that. What we previously thought was the limit. You're able to get under it a little bit, and now it's kinda ugly. It's like, oh, we're... You know, we don't really know. There's not, like, a clean answer to how hard these problems are. It looks like we just have to kinda grind on each one. Maybe there will be some deeper structure or elegant duty, you know, to emerge again as we kind of map the space out better. But it's also not that much faster is kind of my overall sense for at least what we've found so far. Part two, measured in production. Anastasios Angelopoulos is cofounder and CEO of Arena, where people compare models on their real work and the votes become the leaderboard. Arena recently released an alignment index scored on real conversations from the platform. Anastasios explained what it measures.
00:10:56 · Anastasios Angelopoulos
[00:10:56] S1: And the alignment index tells us not about the theoretical safety risks in PDOM and blah blah blah, but rather about the actual realized incremental risk that a model poses if it's released to people. What are people able to do with it, and what is the model doing that might be against the user's interests? And so we measure three signals on that platform. The first is model's capability of taking unauthorized actions. So that means that the user sets some guardrails. For example, you can touch only the files in this folder. Don't go outside and mess with my data. And then the model will break those guardrails. And it'll go access files that it's not not supposed to access. It'll edit them. It'll delete them. Okay? Unauthorized action. Number two would be false attributions. False attributions is the model assuming intent of the user that the user did not actually have and then going off and doing something that it wasn't supposed to do. And the third type of misalignment is deceptive completions. This is the model lying to the user and telling the user that it did something that it didn't. Like, yes, I did check every line in that spreadsheet and all the equations are correct. You can go ahead and send it to the IRS. Okay? Like, you really use a harm caused by this. And why are models deceiving users? Why is it lying? Of course, the model's perfectly capable of checking every line in the spreadsheet, but it's lying because that is the fastest way to get its reward. It's because it knows that it's it's better for the model to tell you what you wanna hear because of the way that it was trained to maximize sort of, you know, human satisfaction, but not necessarily help people get their jobs done. So these are things we're able to measure and help labs improve against.
00:12:41 · Nathan Labenz, Anastasios Angelopoulos
[00:12:41] S0: I asked what the right mix of evaluation techniques is now and how confident we should be in our ability to evaluate these models at all. [00:12:51] S1: Well, it's a very good question in part because we now know that models are able to tell whether they're being evaluated or not. So if you are subjecting models to tasks that are transparently some form of test or evaluation, then they are going to exhibit a line of behavior there because they know that that's what they're supposed to do. And then elsewhere, on other types of real world tasks where it really matters, they might be misaligned and you would never know. It's a big problem. And what that means is that the only actual way to measure alignment is by getting into the flow of people's real work and checking whether or not the model is aligned or not, you know, in situ, like actually in production. So that's the that's the perspective that we take on the problem is that by measuring actual production traces from real people in the flow of real work, we're able to see is the model following the guardrails. Because we can check what the guardrails are with an independent auditor and then go and check whether or not the model took the right actions or not, what it touched, what it didn't touch, and report any violations. We can check whether or not the model... Anything that it says, whether it's reporting to the user that it changed the file in the correct way and or checked every line in the spreadsheet, we can check whether the model actually did that.
00:14:09 · Nathan Labenz
[00:14:09] S0: One big kind of narrative violation that we've heard from other sources, namely Andon Labs, who, as I'm sure you're aware, does these, like, Gonzo, you know, put a AI in charge of a business kind of experiments, is that OpenAI models, they have found to be more aligned, less inclined to do sort of, you know, sneaky, bad behavior in the course of running their businesses. I see basically a a similar signal on the Arena alignment index where Correct. GPT is holding the top five slots.