No Priors: Artificial Intelligence | Technology | Startups
00:16:59 · Maxim Bar Kogan, Sarah Guo
This is the action where I think someone should take a closer look. And that's why we want to experience models for this purpose, and it's you know, most of the hard things that we're doing in this space. [00:17:14] S1: Yeah. You and I actually both love to play blitz chess, and I look at Guardian as a system that's a little bit analogous. Like, it's not clear either of us is going to be competitive with Magnus in a real game. But if the if you play if you play enough times with the right data and all you have to do is make intuitive decisions under time pressure very, very quickly, it's actually a different game. Right? And do you think do you think that makes sense, or am I reaching here?
00:17:14 · Sarah Guo, Maxim Bar Kogan
[00:17:14] S1: Yeah. You and I actually both love to play blitz chess, and I look at Guardian as a system that's a little bit analogous. Like, it's not clear either of us is going to be competitive with Magnus in a real game. But if the if you play if you play enough times with the right data and all you have to do is make intuitive decisions under time pressure very, very quickly, it's actually a different game. Right? And do you think do you think that makes sense, or am I reaching here? [00:17:43] S0: Yeah. I actually I didn't think about it, but, yeah, there's a lot of analogies because I think if you look at top chess players in the world, like, most of the moves that they make are intuitive. They don't calculate forward. They've seen so much games and they've played so much games that they already have a good sense of what is their right move and that they're not taking too much risk here by taking this move without calculating. And then if you look at those games, every once in a while, they do stop for suddenly a really long period of time to just calculate forward a lot of options because they know this is a critical move in the game. There's risk, You need to think through what you're doing and you need to decide correctly. I think that's very similar. It's the efficient way to run computation, right? You don't want to spend too much intelligence where you don't have to, and you want to spend a lot of intelligence, overwhelmingly a lot, in situations where there's high risk.
00:17:43 · Maxim Bar Kogan, Sarah Guo
[00:17:43] S0: Yeah. I actually I didn't think about it, but, yeah, there's a lot of analogies because I think if you look at top chess players in the world, like, most of the moves that they make are intuitive. They don't calculate forward. They've seen so much games and they've played so much games that they already have a good sense of what is their right move and that they're not taking too much risk here by taking this move without calculating. And then if you look at those games, every once in a while, they do stop for suddenly a really long period of time to just calculate forward a lot of options because they know this is a critical move in the game. There's risk, You need to think through what you're doing and you need to decide correctly. I think that's very similar. It's the efficient way to run computation, right? You don't want to spend too much intelligence where you don't have to, and you want to spend a lot of intelligence, overwhelmingly a lot, in situations where there's high risk. [00:18:37] S1: You guys are a team mostly based in Israel today. I think the world has accepted that there is a cohort of amazing Israeli security talent that comes out of, you know, the military and offensive security and then, you know, repeat entrepreneurs like you guys. I think the DNA at Onyx is a little bit different here. Your co founder Gil came out of building synthetic data and working at NVIDIA. Like, What would you characterize the talent that Onyx is particularly good at? And then, are people actually training interesting frontier models in Israel now?
00:19:14 · Maxim Bar Kogan
[00:19:14] S0: So, of I think Israel started maybe a bit late in the game, but it's catching up quickly. So, I think there's now amazing companies in Israel building world models, building AI infrastructure that's top of its glass, building chips. So, I think Israel, in general, is becoming very strong in AI, and we're proud to be a part of that movement. And I think you're right, our company has a very mixed DNA between cyber and AI, which kind of reflects mine's and Gil's backgrounds. Most of the people in our company, most of our research engineering come from a unit in the Israeli intelligence where we actually deal with math and cyber and the intersection thereof. And so, I think it is also reflected in kind of the type of talent that we bring in. I think it's important for a few reasons. The first and foremost is that we want to be more than just a security company long term. We think that to solve this problem well, it's going to require deep AI expertise, but then the problem is not just cybersecurity. The problem is how do we control advanced AI long term? And that problem, even if you just forget about enterprise security and the different gaps in various controls that they currently have, first principle is that problem just sounds very important to me. So, I think it will be crucially important. If you have AI companies that are $10,000,000,000,000 companies, we think you want a company that is not the vendor of the AI itself to oversee and help you control what AI is doing. And we think that's an opening that's a $100,000,000,000 plus opening for a really important company. And then if you think about what it's gonna take to control advanced AI long term, then we're just scratching the surface because long term, you're going to have to also understand much better what models are thinking, what's happening on the internals of these models as they're operating. That's also a lot of where our research is focused.
00:21:25 · Sarah Guo, Maxim Bar Kogan
[00:21:25] S1: So, the industry is quite divided on this issue, I mean, amongst the people who think about whether or not mechanistic interpretability or research into better understanding models is possible? Like, that's a question. So, it's something you believe in. [00:21:43] S0: We believe that there's been a lot of strong progress in that direction. We believe that understanding the internal weights and activations, what is the internal structure, the mathematical structure of these systems, is gonna be at least part of the solution. And in many ways, we think that, and this is maybe, we'll only know when we get there, but we think that for our level of intelligence, it's kind of difficult to understand very quickly what is the internal structure of a large language model. What is the internal structure of way Our it's an [00:22:27] S1: level, like, human intelligence or our level of your model's camera? Okay. Human intelligence.
00:22:27 · Sarah Guo, Maxim Bar Kogan
[00:22:27] S1: level, like, human intelligence or our level of your model's camera? Okay. Human intelligence. [00:22:30] S0: Oh, yes. Yeah. I think like yeah, exactly. I think as humans, it might still be very difficult to understand what weights and activations mean, and maybe mechanistic interpretability. It seems like, oh, maybe that's too hard or shouldn't be possible. But as we're starting to have models that are much smarter than us, at least in some important ways, we think that we'll be able to start tracking mechanistic capability much more effectively. And I think it's gonna be extremely rewarding, by the way, long term for understanding intelligence in general, like not just overseeing, but just understanding what intelligence is, how it works. What's the difference between the smarter model and the less smart model? [00:23:13] S1: I completely agree that the opportunity to understand and trust and secure and govern these super intelligent AIs is a very large opportunity. If we just scroll back today, the security person in me says, Well, then I have to give you all the permissions and understanding that I have to give these companies too. Like, how do you get customers or, you know, the Fortune 100s you're working with already or, you know, tech natives, everybody cares about their own security and business, to trust you now as a You're like less than 100 people, right?