No Priors: Artificial Intelligence | Technology | Startups
00:12:45 · Sarah Guo, Maxim Bar Kogan
[00:12:45] S1: As somebody who has worked in security for a long time, my first very traditional instinct on a problem like this is like, that sounds like a problem for a proxy with a policy engine. We make some rules and make the rules smarter. Like, why doesn't that work, or did you try it? [00:13:03] S0: There are a few things that, I mean, proxy's integration method, I would say. So, there are some AI systems where like you would want to integrate with a proxy if that's the easiest way to do it. But number one, there's a lot of systems where that's just not viable technically, because AI today runs on the cloud, on someone else's infrastructure, on your endpoint, and just proxy is not always an option. And the second thing is the question, okay, great, you're proxying, so you're seeing the data. You're seeing it, but that's not the hard problem. The hard problem is understanding what I should do now. Turns out that in the case of AI systems, that is the hard question. Like, what is the engine that needs to underwrite these different actions and say if they're okay or not? Because we need to be able to understand what another AI system is thinking, what is it planning to do, and then have our own opinion on that. And consider we're trying to understand some of the smartest models in the world are doing the right thing. So, who are we to do it? How are we going to do it correctly? Right? And so, that turns out to be a really difficult technical question. [00:14:11] S1: Part of the solution for Onyx has been training its own models. Like, what can you say about that?
00:14:18 · Maxim Bar Kogan
[00:14:18] S0: If you try today, let's say we're trying to build a solution to oversee and kind of control how other agents are operating. Maybe the first thing a lot of our listeners might think is say, well, I'll just ask Cloud Code to do it. In a sense, they would be right because Cloud Code is great. And maybe we can ask it to spawn a version of itself for every agent that we have and kind of keep monitoring everything that agent is starting to do. And if you think that there's a problem, intervene. So, that approach, obviously it's pretty naive, and there are some ways in which it totally fails, we could talk about, but it has some merit to it. So, it does seem intuitive that it's a good idea to have capable agents reviewing what other agents are doing, same as we have capable humans reviewing what other humans are doing. But then the problems that you're gonna run into is how do I make this work from cost, latency, reliability perspective? Because if I need to run an agent for every agent you're running as your security vendor, you're gonna be paying for me more than you're paying for your AI, right? So, it's pretty much a dealbreaker. And also, it's going be so slow, so you're not going be happy with whatever latency you're going to get. And so, the challenge then becomes how do I know what are the times where I need to interject with these smart agents to look at what's happening? And that's when actually what you want to do is you want to train very smart models that are, actually let me correct with that, very not smart models, but models that are just good at one thing. They're very small. They almost can't do anything else other than be able to say, should I have a smarter agent look at this? And if you manage to bake in that intuition into those small models well, in the sense that they don't miss a lot of stuff and they don't call that other agent too much, then you can get to a really good balance of we're very performant, we have smart agents overseeing things when needed, our costs are low and our latency is low. And then that becomes the challenge because you need to make sure that as the frontier models get smarter and the hardnesses become more evolved, you need to be able to have models that are on your side that are small and effective and continuously being able to say, now is the time.
00:16:59 · Maxim Bar Kogan, Sarah Guo
This is the action where I think someone should take a closer look. And that's why we want to experience models for this purpose, and it's you know, most of the hard things that we're doing in this space. [00:17:14] S1: Yeah. You and I actually both love to play blitz chess, and I look at Guardian as a system that's a little bit analogous. Like, it's not clear either of us is going to be competitive with Magnus in a real game. But if the if you play if you play enough times with the right data and all you have to do is make intuitive decisions under time pressure very, very quickly, it's actually a different game. Right? And do you think do you think that makes sense, or am I reaching here?
00:17:14 · Sarah Guo, Maxim Bar Kogan
[00:17:14] S1: Yeah. You and I actually both love to play blitz chess, and I look at Guardian as a system that's a little bit analogous. Like, it's not clear either of us is going to be competitive with Magnus in a real game. But if the if you play if you play enough times with the right data and all you have to do is make intuitive decisions under time pressure very, very quickly, it's actually a different game. Right? And do you think do you think that makes sense, or am I reaching here? [00:17:43] S0: Yeah. I actually I didn't think about it, but, yeah, there's a lot of analogies because I think if you look at top chess players in the world, like, most of the moves that they make are intuitive. They don't calculate forward. They've seen so much games and they've played so much games that they already have a good sense of what is their right move and that they're not taking too much risk here by taking this move without calculating. And then if you look at those games, every once in a while, they do stop for suddenly a really long period of time to just calculate forward a lot of options because they know this is a critical move in the game. There's risk, You need to think through what you're doing and you need to decide correctly. I think that's very similar. It's the efficient way to run computation, right? You don't want to spend too much intelligence where you don't have to, and you want to spend a lot of intelligence, overwhelmingly a lot, in situations where there's high risk.
00:17:43 · Maxim Bar Kogan, Sarah Guo
[00:17:43] S0: Yeah. I actually I didn't think about it, but, yeah, there's a lot of analogies because I think if you look at top chess players in the world, like, most of the moves that they make are intuitive. They don't calculate forward. They've seen so much games and they've played so much games that they already have a good sense of what is their right move and that they're not taking too much risk here by taking this move without calculating. And then if you look at those games, every once in a while, they do stop for suddenly a really long period of time to just calculate forward a lot of options because they know this is a critical move in the game. There's risk, You need to think through what you're doing and you need to decide correctly. I think that's very similar. It's the efficient way to run computation, right? You don't want to spend too much intelligence where you don't have to, and you want to spend a lot of intelligence, overwhelmingly a lot, in situations where there's high risk. [00:18:37] S1: You guys are a team mostly based in Israel today. I think the world has accepted that there is a cohort of amazing Israeli security talent that comes out of, you know, the military and offensive security and then, you know, repeat entrepreneurs like you guys. I think the DNA at Onyx is a little bit different here. Your co founder Gil came out of building synthetic data and working at NVIDIA. Like, What would you characterize the talent that Onyx is particularly good at? And then, are people actually training interesting frontier models in Israel now?
00:19:14 · Maxim Bar Kogan
[00:19:14] S0: So, of I think Israel started maybe a bit late in the game, but it's catching up quickly. So, I think there's now amazing companies in Israel building world models, building AI infrastructure that's top of its glass, building chips. So, I think Israel, in general, is becoming very strong in AI, and we're proud to be a part of that movement. And I think you're right, our company has a very mixed DNA between cyber and AI, which kind of reflects mine's and Gil's backgrounds. Most of the people in our company, most of our research engineering come from a unit in the Israeli intelligence where we actually deal with math and cyber and the intersection thereof. And so, I think it is also reflected in kind of the type of talent that we bring in. I think it's important for a few reasons. The first and foremost is that we want to be more than just a security company long term. We think that to solve this problem well, it's going to require deep AI expertise, but then the problem is not just cybersecurity. The problem is how do we control advanced AI long term? And that problem, even if you just forget about enterprise security and the different gaps in various controls that they currently have, first principle is that problem just sounds very important to me. So, I think it will be crucially important. If you have AI companies that are $10,000,000,000,000 companies, we think you want a company that is not the vendor of the AI itself to oversee and help you control what AI is doing. And we think that's an opening that's a $100,000,000,000 plus opening for a really important company. And then if you think about what it's gonna take to control advanced AI long term, then we're just scratching the surface because long term, you're going to have to also understand much better what models are thinking, what's happening on the internals of these models as they're operating. That's also a lot of where our research is focused.