00:00:49 · Sebastian Johnson, Thore Graepel
[00:00:49] S0: Absolutely. And it's a really interesting topic. We we... When we talk about LLMs, we kind of talk about that black box nature. But for those who maybe who who don't understand reasoning, can you kind of give a quick explanation of what you really mean by reasoning? [00:01:03] S1: Yeah. Exactly. So, reasoning in our view is really that you have some set of beliefs, and they could be very fuzzy beliefs like a scientist who approaches a question for the first time or a doctor who sees a patient for the first time. And now they need to reason about what's going on and they have the problem in hand and they decompose the problem say, they can put forward hypotheses, they can look for evidence that supports or not those hypotheses. They synthesize results. So it's that process which takes what we call your belief state or epistemic state and reduces uncertainty in it. And if you now compare that with how a large language model works, a large language model really has no notion of truth whatsoever. It predicts the next token in the sequence based on context, but it doesn't carry beliefs about the world that it can then update in light of new evidence. And that's really the missing piece that we're building with metis reasoning.
00:01:03 · Thore Graepel, Sebastian Johnson
[00:01:03] S1: Yeah. Exactly. So, reasoning in our view is really that you have some set of beliefs, and they could be very fuzzy beliefs like a scientist who approaches a question for the first time or a doctor who sees a patient for the first time. And now they need to reason about what's going on and they have the problem in hand and they decompose the problem say, they can put forward hypotheses, they can look for evidence that supports or not those hypotheses. They synthesize results. So it's that process which takes what we call your belief state or epistemic state and reduces uncertainty in it. And if you now compare that with how a large language model works, a large language model really has no notion of truth whatsoever. It predicts the next token in the sequence based on context, but it doesn't carry beliefs about the world that it can then update in light of new evidence. And that's really the missing piece that we're building with metis reasoning. [00:02:14] S0: And, you know, I think when you talk about when you talk LLMs, my understanding is that LLMs kind of have the issue or the problem that makes them a sort of black box is that they have all of the knowledge and they have the reasoning in one sort of architecture, right, inside the LLM, which makes it quite hard to separate or trace. So they've got the knowledge, they've the reasoning, and you can't actually access it. I guess the only way to access it is by asking the LLM itself to sort of explain any sort of reasoning. What does an alternative look like?
00:02:44 · Thore Graepel
[00:02:44] S1: Yeah. That's exactly right. We call this actually the original sin of the transformer architecture, that everything is enmeshed in the weights, you just can't get it out. Even if I gave you a trillion parameters of a model now, the only way to ask it is to prompt it. And so, fundamentally, our architectural decision is to separate these things, the knowledge on which the reasoning operates and the reasoning process. And that unlocks a number of desirable properties for for AI. Because if you explicitly represent your knowledge, then you can inspect that knowledge. You can see what's actually there. You can edit the knowledge. You can remove harmful knowledge or knowledge you're not supposed to have, or you can add knowledge, and you can access control it. You know, you can give access to different people or system depending on on the situation. And these, of course, are the advantages that we knew all along with databases. We had just lost them because they're now... You know, the knowledge is now locked up in the transformer. But on the other side, we have the reasoning process. And now if that process acts on the knowledge, now we can improve that reasoning process separately. In reinforcement learning terms, we can learn a good reasoning policy that becomes better and better at reasoning as it reasons about more problems. And that means we have these two new scaling axes. One, growing the knowledge explicitly and the other one growing the reasoning capability through that learning process.
00:04:21 · Sebastian Johnson, Thore Graepel
[00:04:21] S0: And so at Metis reasoning is is is the approach to build both separately, to build the reasoning capability and the sort of the the knowledge base, the database there, and to try and then kind of get value out of, you know, applying one to the other? Or are you focusing, you know, primarily on the reasoning process itself? [00:04:39] S1: Yeah. No, it's in fact both. And they interact in very rich ways. So you could imagine, suppose the system is confronted with a new question for the first time, and then it reasons through that question and gives the answer together with the reasons for why it gave that answer. That is crucial. And now, of course, we don't throw that away. We can store the answer together with its provenance and all the associated evidence on the knowledge side in in the knowledge graph, if you like. And that means that if that answer is required another time because the user asks another time or it comes up in some other reasoning process as a sub problem, then the system can now just access the cache knowledge in that knowledge store. But that knowledge store isn't just, you know, a bunch of tokens. That knowledge store now is certified knowledge that has already been verified by the system. So through that mechanism, it can become orders of magnitude faster.
00:05:44 · Sebastian Johnson
[00:05:44] S0: And in the paper, one of the things that really... I I thought was really interesting is that you you were talking about AlphaGo. You you were on the AlphaGo team, what, just over ten years ago, you know, with the famous Move 37. Now in the paper, you're kind of arguing that Move 37 was more reasoning, right? It wasn't the similar sort of architecture that we have today in AI. It was a different thing. Can you explain a bit about that? How it's different to modern LLMs? And then I guess like where we went wrong potentially, like why did we move from AlphaGo in this amazing moment at Move 37 to the LLM architecture where we kind of lost reasoning?
00:06:23 · Thore Graepel
[00:06:23] S1: Yes. Yeah. It's a great point. So, going back, ten years in time, so to speak, what what does the AlphaGo architecture actually look like? It has these two sides to it. Yes. There are neural networks. They are very important. There were two of them. The policy network decides what are promising moves in a given position, and the value network is able to judge if a position is good for black or for white. But those were not the only components. They were combined with a tree search, you know, a game tree search. And that is really the the knowledge structure that I've been talking about. The structure that holds the current knowledge of the system about the game or about the game state. And this this tree structure is searched and expanded by the reasoning process and uses the neural networks to interface with the world, you know, to understand, you know, things about about the the go position. And it's really these two systems together that produce the playing strength of AlphaGo. And, you know, in the words of the... Of psychologist Kahneman, these are system one and system two. System one is the quick intuitive pattern matching system, and system two is the more elaborate reasoning systems that thinks forward into the future. And move 37, coming back to that question, was this very surprising move that no human master would have made. You know, the one where Michael Redmond, the commentator said, oh, this must be a mistake. Right? It's how bizarre it looked to good Go players. And we know... Although people think that was the... Like, the gut feeling of the of the computer, It really wasn't. We know that the policy net assigned a very low probability of one in 10,000 to that move so that the policy net wouldn't have played it. It was the reasoning process, the process that looked at the consequences of playing that move, which eventually dared to play that unusual move. And I think there's a really interesting lesson here that if we want to break out of the current patterns of knowledge to generate truly new knowledge, then we need, those reasoning processes, those explicit reasoning processes.