00:16:48 · Thore Graepel
[00:16:48] S1: Yeah. Yeah. You're quite right. So, of course, a game like Go or chess is a very constrained space. It's not stochastic. It's not an open world. And, clearly, the the general reasoning problem is harder. But we do have tools to tackle it. For example, in the game of Go, there is no uncertainty about the board state. Or if I make a move, I know exactly what's gonna happen as a consequence of making that move in in the next step. And these things are not the case in the real world, but we can use the calculus of probability to quantify the uncertainty that arises. That's number one. And number two is that in fact we can use large language models for various tasks that arise within the reasoning. So we're not at all saying that you know current AI is useless or something like that. No, no, it is super useful. We just want to direct it at more specific tasks which are usually related to some form of semantic interpretation. So how how would we use a large language model in this context? I'll give you an example. Our system... If you ask a large language model, what is the capital of France? It'll just tell you Paris, and it'll usually be right, but there will be no trace and, you know, where does the answer come from? From the weights? Where do the weights come from? From the training corpus. So there's not much more you get than the answer. Metis reasoning, in contrast, would go about it as follows. It would first ask the question, oh, does France even have a capital? Is France the kind of thing that can have a capital? Right? It's kind of questioning the assumptions. And then it doesn't jump to a conclusion, but it puts forward hypotheses. And we would... Might very well ask a large language model to help us formulate the hypotheses because nothing can go wrong here. Even if it proposes a nonsensical thing like Mickey Mouse as the answer, we can always root it out later by looking at evidence. And so now we have a number of hypotheses what the capital of France might be, and now we pull in external evidence from Wikipedia or from some other source and test these hypotheses against the evidence. And then, of course, our system would probably get get you Paris if that's the framing that... Where where that is the correct question. But it wouldn't have relied at all on on that information being within an LLM.
00:19:26 · Thore Graepel, Sebastian Johnson
It's just using the LLM to judge the hypotheses against the evidence, and that makes it so powerful. But then, of course, next time, it will save time because it will have stored that fact together with the evidence, and it will give you an instant answer the next time you ask it. That's why it doesn't hallucinate, you see, because it doesn't rely on the LLM fantasizing about the answer. It just uses the LLM to interpret the evidence. [00:19:57] S0: And I'm cautious. We're out of time. Wanna ask one more question, which is that you've you've kind of come out of stealth. You've released this white paper. At the moment, it is... You know, you're making an argument and you have a strong theory. What's next? You know, are building some sort of reasoning model itself? Are you going to start trying to benchmark or prove that it has no hallucinations? Like, what's next in terms of, you know, what comes after the paper?
00:20:20 · Thore Graepel
[00:20:20] S1: Yeah. I mean, first of all, I have to mention that I have a fantastic team already, including, you know, physicists, mathematicians, roboticists, and, you know, part... Some of them are in Singapore. Some of them are on the West Coast, but, you know, London will be will be our headquarter. And so we've already been able to build this system, and we have prototypes. We run it on PhD level questions through my business partner's biotech company. We had the opportunity to already have scientists work with the system and give feedback. So we're really making some some great progress here. Also, in addition to building the core system, we're currently experimenting with robotics. We have a robotics lab in Singapore where we're playing with little unit three g ones and and try out this reasoning in in that domain because we believe that robotics is the ultimate application domain. One day robotics will have to be a general reasoner that can solve problems for us. But even in the smaller movements and actions that robots do, we believe that this is ultimately a reasoning problem, maybe except for the finest dexterity questions. And so that is a domain that we're currently looking at as well.
00:21:52 · Sebastian Johnson, Thore Graepel
[00:21:52] S0: Very interesting. Okay, well, look, I'm looking forward to seeing, you know, the progress, journey that you've gone. It's really exciting. And I think there's an increasing noise around the issues with NLM's, especially on this hallucination and lack of accountability. So I think it's really exciting. You're kind of working industry with the tailwinds behind you. So thank you so much for taking the time and let's catch up soon. [00:22:12] S1: Great. Thank you very much.