00:08:51 · Sebastian Johnson, Thore Graepel
[00:08:51] S0: And why do you think, you know, given how impactful that AlphaGo moment was in Move 37, We then had LLM, the LLM architecture come up and be this next big wave of AI. Why do you think, you know, researchers went in that direction and built the sort of all in one knowledge and reasoning as opposed to, you know, to your point, system one and system two? [00:09:13] S1: Yeah. So I think if you now... If you go forward from AlphaGo and say to the JetGPT moment, you will see a similarity. You know, our policy net already predicted the next move, and large language models predict the next token. So that part of of the AlphaGo system was in fact further developed and has taken us where we are. But the other side, you know, the reasoning side was clearly much much less developed. There were some attempts to do some kind of search, of course, in the in the space of tokens. But I think there's this additional idea that's needed here, which is to move from token to proposition. You know, a token, the the basic unit of intelligence, as they say nowadays in large language models, is just some little semantic bit that doesn't say anything about the world. A token is never right or wrong. Right? It's just a token. But when we talk about knowledge and reasoning, we need to speak about propositions. A proposition is a statement about the world that can be true or false or about which we can have some kind of belief. And that is the natural unit in which reasoning happens. And that is where we can get to reasoning that can advance our knowledge about the real world.
00:10:39 · Sebastian Johnson
[00:10:39] S0: And why is that important? You know, why is reasoning really important? And like, what are some of the I guess there's two things I want to talk about. The issues that we have with LLMs being a black box and whether that's, hallucination and kind of lack of auditability. And I think the point that was really interesting was actually the economic point that is then made. So let's just focus on like what are the issues with the current end the architecture and what are the issues that we're seeing from that today? And then let's talk about, well, okay, if we get reasoning right, what are the economic impacts that can be unlocked?
00:11:09 · Thore Graepel
[00:11:09] S1: Yeah. So the problems you mentioned, some of them, there are hallucinations. The systems are intrinsically stochastic, so they usually don't actually max out at 100% in... Even in benchmarks. They, you know, they they don't typically get all the questions right all the time. So that's one of the problems. But possibly the bigger problem is the trust problem because they are black boxes, and we cannot see why they came up with the answer that they did. And, you know, you might think, oh, but we have chain of thought reasoning, and we can inspect the reasoning chain. But first of all, that reasoning chain isn't the actual causal trace that led to the answer. Often... And often, it it looks very confused. Have you ever seen a desperate large language model producing tokens in the chain of thought traces where it says, oh my god. The user wants this. I have to be quick and get the answer. You know, it it just isn't what you would call a reasoning trace of a of a proper reasoning process. And the the problem is, of course, in critical applications. You know, when you have a chatbot, these things don't matter too much. But when this is a system that does medical diagnosis or critical engineering or is an air traffic control system or you know controls robots that are very powerful entities and you know can can do harm if they're not controlled in the right way then we need systems in where we can understand why they come to the conclusions that they do, why they make this the decisions that they do. And in the architecture that we're proposing, this is entirely transparent. You can see how the system reasoned about it. You can see what sources of evidence it used, how it judged those sources of evidence, how it synthesized everything into an answer. And I believe that's what's needed if we want to apply AI in all those critical domains where it actually matters.
00:13:20 · Sebastian Johnson
[00:13:20] S0: Yeah. And we... When when you look at the proposal, think there's three core components you're putting forward, which I think reached kind of what you just covered, a record of what's established and what, you know, degree of certainty. I think you you described it as a persistent epistemic state, the judgment system itself, and this sort of like objective. Is that right? Can you kind of, I guess, explain a bit about the the the core components of the proposal?
00:13:45 · Thore Graepel
[00:13:45] S1: Yes. Exactly. So, first of all, the key thing is that the system actually maintains beliefs about what is the case, you know, what what is its current state of knowledge. And that, again, could be a scientist who has some beliefs about the scientific question or it could be a robot a robot that needs to represent what its current situation is and the environment and task in front of it. So it's a very general state notion. And then what is reasoning? Reasoning is the application of moves to this state to advance that state. So if I have... Because have some set of beliefs at the moment, I can now apply a move and say, oh, why don't I take my problem and decompose it into three subproblems? Now I've advanced my state of knowledge, and now I can tackle these subproblems. I can pull it in a bit of evidence and maybe resolve some uncertainty. And the key thing, of course, is that we need to understand what is the goal of this process, what is the objective that drives it, and the objective that drives it is the reduction in uncertainty. So we need to maintain our beliefs using the language of probability theory and we track if through the actions that we execute the reasoning actions our uncertainty is in fact reduced. And if that's the case then we're reasoning in the right direction and that is our measure of success. And then as the system improves over time that also serves as the reward signal because what is good reasoning? Good reasoning is when you can use very few and clever moves to reduce your uncertainty. You know there's often many different strategies that can lead to a solution and a good reasoner would find the shortest path or the least expensive path. You can even think about it in terms of bits per bug, know, you try to get as reason with in the cheapest possible way to get the maximum, reduction in uncertainty at the minimal price.
00:16:02 · Sebastian Johnson
[00:16:02] S0: Yes. It's... Yeah. It's really interesting. There's a bit where you... I think in the paper, you you talk about where the analogy with AlphaGo sort of ends, I think. It's almost like the the the biggest challenge that you're facing is applying this sort of like very, very scientific approach. And this... I think it's, I guess, very scientific reasoning process, which works perfectly for a game like AlphaGo where you always know where you stand. So you have all the information on the board, you know your position. There are very fixed rules about what you can and can't do. And the value as well is, like, very... Is very clear. Right? They they... You know, you wanna progress, you wanna win. Right? So for for the reasoning in that game, you can kind of see a really clear structure. How do we try and rebuild that for, like, a much more general intelligence, do you think?