"The Cognitive Revolution"
00:17:54 · Sagar Vaze, Anastasios Angelopoulos
[00:17:54] S4: more and more and more and more until you get to number one. [00:17:56] S1: I don't really get the narrative about the benchmarking of the open models and so on and so forth simply because we don't really know whether the frontier closed source models in The US weren't benchmark maxing against the closed datasets that are now going into this index. You know, these companies have very close relationships with, like, Cognition and, you know, some of the other companies and artificial analysis that are building these closed datasets. What we really need is benchmarks that speak to the utility of AI to real people and real workflows. And there should be a consistent and scalable methodology to do this that doesn't overfit. When you have live fresh data coming in all of the time from, like, real production use cases, it can't be overfit. It cannot be overfit, at least statistically. You can argue that you're doing better on that distribution than you are on other distributions, but at the end of the day, for that customer set, you're doing better as you go more and more because every question is fresh. It's like you're constantly taking new tests. It's not like you have the same test and you're taking it many times. And so I worry if that's the direction that the ecosystem is going, that we're going to set ourselves up for failure in the long run.
00:17:56 · Anastasios Angelopoulos, Prakash Narayanan
[00:17:56] S1: I don't really get the narrative about the benchmarking of the open models and so on and so forth simply because we don't really know whether the frontier closed source models in The US weren't benchmark maxing against the closed datasets that are now going into this index. You know, these companies have very close relationships with, like, Cognition and, you know, some of the other companies and artificial analysis that are building these closed datasets. What we really need is benchmarks that speak to the utility of AI to real people and real workflows. And there should be a consistent and scalable methodology to do this that doesn't overfit. When you have live fresh data coming in all of the time from, like, real production use cases, it can't be overfit. It cannot be overfit, at least statistically. You can argue that you're doing better on that distribution than you are on other distributions, but at the end of the day, for that customer set, you're doing better as you go more and more because every question is fresh. It's like you're constantly taking new tests. It's not like you have the same test and you're taking it many times. And so I worry if that's the direction that the ecosystem is going, that we're going to set ourselves up for failure in the long run. [00:19:10] S6: So let's talk a little bit about model routing. So I believe you guys have just introduced a model router. Can you explain business reasons or the user reasons for introducing a router? Let me tell you what we
00:19:28 · Anastasios Angelopoulos
[00:19:28] S1: saw internally at Arena. At some point, we were looking at all our traces that are being done on our coding agents internally. By the way, we're in the top you know, less than 1% of business in terms of AI spend. We're spending thousands of dollars a month per employee on AI just to help us accelerate our workflow. That's why our sort of revenue to employee ratio is crazy off the charts because we're using a lot of agents internally to do our work. Now what happened is that we were looking through our traces, and we saw these traces, tons and tons of traces that were like, thank you, and then that costs you $40. What? Why is this happening? It's because somebody did a long running task, they let their agent go, it created some artifacts, blah blah blah, very long conversation. They left for a day, they came back, the cash was invalidated, and they're happy with the results. So they say thank you and enter, and that... That's a $40 charge. Terrible. Terrible. Terrible. We had... We're burning money. We're burning VC money. Right? We shouldn't do that. What we should do is we should figure out which tokens are best spent on frontier models, you know, which tokens are best spent on a cheaper model that can do the job, where we should be caching intelligently and, you know, avoiding some of these other types of cache hits, more intelligent caching behavior. And that's why we've been building the Arena router, we call MAX, short for argmax, Is it maximizing capabilities subject to cost constraints?
00:21:06 · Nathan Labenz, Anastasios Angelopoulos
[00:21:06] S0: What can you tell us about the new Gemini four Carbon? I think this has literally, come out and appeared on the Arena leaderboard while we've been live. [00:21:17] S1: And, you know, where it really shines, it shines on steerability. It's able to land user corrections, really effectively. It it shines on task completion. It shines on user preference. Users love working with this model. It's great at three d design. It's great at general agentic work with documents. Congrats to Google. This is just like, what a big comeback. I think that, leadership must be really happy.
00:21:45 · Nathan Labenz
[00:21:45] S0: Hey. We'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude. To show off Claude's latest capabilities, I asked Claude to take lead in making this ad. Given a short creative brief, Claude, using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you're watching. My job? Give feedback and decide. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude Code Routines, it keeps working in the cloud even after you close your laptop. For problems worth solving, get started with Claude at claude.aitcr. That's claude.aitcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Claude.aitcr. This episode of the Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems starting from any coding tool in a governed agentic engineering model. OutSystems, you can coordinate and govern your entire agentic workforce within your enterprise ecosystem, accelerate impact with industry proven agentic solutions that are prebuilt, governed, and customizable to your business. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission critical apps. Teams any size can use OutSystems to build, deploy, and manage AI apps and agents quickly and cost effectively without compromising reliability and security. With OutSystems, you can accelerate ideas from concept to completion. It's the leading agentic systems platform that is unified, agile, and enterprise proven, allowing you to build your agentic future with AI solutions deeply integrated into your architecture. OutSystems, all your agentic future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr. Part three, an orthogonal axis.
00:24:28 · Nathan Labenz
Debo Olaosebikan is co founder and CEO of Kepler Computing, which came out of stealth in September with a new kind of memory for AI chips, a ferroelectric capacitor stacked on top of each transistor made in older fabs. He started with where the energy goes.