No Priors: Artificial Intelligence | Technology | Startups
00:10:24 · Maxim Bar Kogan
[00:10:24] S0: Security is always a space where you have some overlap between different tooling. In this case, and you have the concept of defensive debt as well. So, you want to have defenses at different levels of your technology stack to solve the problem. And that said, I think in this space, a lot of enterprises are kind of helpless because I'll take an example, the identity approach. Traditionally, if we have a software system that's running in our company, our first and most important control will be to limit what permission it has. And then no matter what, even if it goes wrong, even if it's compromised, it can't typically do stuff that it was originally allowed to do. But with these autonomous AIs, with these assistants, with these coding agents, we kind of want them to have our permissions because we want to tell Cloud Code to do something or Cloud Code work to do something, and we want to then go have lunch, we want to come back and see that it's done. And we want to give it so many diverse tasks as well that we kind of can't find the right set of permissions to do. So, suddenly our identity security software is not very useful. Then if you think about endpoint security or API security, if we tell our Cloud Code that we want to recreate a database and it should delete it and recreate it, that's great. That's gonna save our DevOps team and our platform teams a lot of time. It's a great benefit of Cloud Code. But if Cloud Code is working on an unrelated task and suddenly thinks that maybe the right thing to do is to delete our database and recreate it. Maybe we don't want that to happen. And unfortunately, our endpoint providers or API security tools, they don't know what cloud was thinking, why is it doing what it's doing, right? So, a lot of these existing tools, they don't have the context to understand what these very flexible, unpredictable systems are doing. If you're not building some kind of controls that are built for these systems, then you're either going to end up limiting them a lot, making them almost much less useful to the enterprise, or you're going to miss a lot of pretty dangerous things that they might be doing.
00:12:45 · Sarah Guo, Maxim Bar Kogan
[00:12:45] S1: As somebody who has worked in security for a long time, my first very traditional instinct on a problem like this is like, that sounds like a problem for a proxy with a policy engine. We make some rules and make the rules smarter. Like, why doesn't that work, or did you try it? [00:13:03] S0: There are a few things that, I mean, proxy's integration method, I would say. So, there are some AI systems where like you would want to integrate with a proxy if that's the easiest way to do it. But number one, there's a lot of systems where that's just not viable technically, because AI today runs on the cloud, on someone else's infrastructure, on your endpoint, and just proxy is not always an option. And the second thing is the question, okay, great, you're proxying, so you're seeing the data. You're seeing it, but that's not the hard problem. The hard problem is understanding what I should do now. Turns out that in the case of AI systems, that is the hard question. Like, what is the engine that needs to underwrite these different actions and say if they're okay or not? Because we need to be able to understand what another AI system is thinking, what is it planning to do, and then have our own opinion on that. And consider we're trying to understand some of the smartest models in the world are doing the right thing. So, who are we to do it? How are we going to do it correctly? Right? And so, that turns out to be a really difficult technical question. [00:14:11] S1: Part of the solution for Onyx has been training its own models. Like, what can you say about that?
00:14:18 · Maxim Bar Kogan
[00:14:18] S0: If you try today, let's say we're trying to build a solution to oversee and kind of control how other agents are operating. Maybe the first thing a lot of our listeners might think is say, well, I'll just ask Cloud Code to do it. In a sense, they would be right because Cloud Code is great. And maybe we can ask it to spawn a version of itself for every agent that we have and kind of keep monitoring everything that agent is starting to do. And if you think that there's a problem, intervene. So, that approach, obviously it's pretty naive, and there are some ways in which it totally fails, we could talk about, but it has some merit to it. So, it does seem intuitive that it's a good idea to have capable agents reviewing what other agents are doing, same as we have capable humans reviewing what other humans are doing. But then the problems that you're gonna run into is how do I make this work from cost, latency, reliability perspective? Because if I need to run an agent for every agent you're running as your security vendor, you're gonna be paying for me more than you're paying for your AI, right? So, it's pretty much a dealbreaker. And also, it's going be so slow, so you're not going be happy with whatever latency you're going to get. And so, the challenge then becomes how do I know what are the times where I need to interject with these smart agents to look at what's happening? And that's when actually what you want to do is you want to train very smart models that are, actually let me correct with that, very not smart models, but models that are just good at one thing. They're very small. They almost can't do anything else other than be able to say, should I have a smarter agent look at this? And if you manage to bake in that intuition into those small models well, in the sense that they don't miss a lot of stuff and they don't call that other agent too much, then you can get to a really good balance of we're very performant, we have smart agents overseeing things when needed, our costs are low and our latency is low. And then that becomes the challenge because you need to make sure that as the frontier models get smarter and the hardnesses become more evolved, you need to be able to have models that are on your side that are small and effective and continuously being able to say, now is the time.
00:16:59 · Maxim Bar Kogan, Sarah Guo
This is the action where I think someone should take a closer look. And that's why we want to experience models for this purpose, and it's you know, most of the hard things that we're doing in this space. [00:17:14] S1: Yeah. You and I actually both love to play blitz chess, and I look at Guardian as a system that's a little bit analogous. Like, it's not clear either of us is going to be competitive with Magnus in a real game. But if the if you play if you play enough times with the right data and all you have to do is make intuitive decisions under time pressure very, very quickly, it's actually a different game. Right? And do you think do you think that makes sense, or am I reaching here?
00:17:14 · Sarah Guo, Maxim Bar Kogan
[00:17:14] S1: Yeah. You and I actually both love to play blitz chess, and I look at Guardian as a system that's a little bit analogous. Like, it's not clear either of us is going to be competitive with Magnus in a real game. But if the if you play if you play enough times with the right data and all you have to do is make intuitive decisions under time pressure very, very quickly, it's actually a different game. Right? And do you think do you think that makes sense, or am I reaching here? [00:17:43] S0: Yeah. I actually I didn't think about it, but, yeah, there's a lot of analogies because I think if you look at top chess players in the world, like, most of the moves that they make are intuitive. They don't calculate forward. They've seen so much games and they've played so much games that they already have a good sense of what is their right move and that they're not taking too much risk here by taking this move without calculating. And then if you look at those games, every once in a while, they do stop for suddenly a really long period of time to just calculate forward a lot of options because they know this is a critical move in the game. There's risk, You need to think through what you're doing and you need to decide correctly. I think that's very similar. It's the efficient way to run computation, right? You don't want to spend too much intelligence where you don't have to, and you want to spend a lot of intelligence, overwhelmingly a lot, in situations where there's high risk.
00:17:43 · Maxim Bar Kogan, Sarah Guo
[00:17:43] S0: Yeah. I actually I didn't think about it, but, yeah, there's a lot of analogies because I think if you look at top chess players in the world, like, most of the moves that they make are intuitive. They don't calculate forward. They've seen so much games and they've played so much games that they already have a good sense of what is their right move and that they're not taking too much risk here by taking this move without calculating. And then if you look at those games, every once in a while, they do stop for suddenly a really long period of time to just calculate forward a lot of options because they know this is a critical move in the game. There's risk, You need to think through what you're doing and you need to decide correctly. I think that's very similar. It's the efficient way to run computation, right? You don't want to spend too much intelligence where you don't have to, and you want to spend a lot of intelligence, overwhelmingly a lot, in situations where there's high risk. [00:18:37] S1: You guys are a team mostly based in Israel today. I think the world has accepted that there is a cohort of amazing Israeli security talent that comes out of, you know, the military and offensive security and then, you know, repeat entrepreneurs like you guys. I think the DNA at Onyx is a little bit different here. Your co founder Gil came out of building synthetic data and working at NVIDIA. Like, What would you characterize the talent that Onyx is particularly good at? And then, are people actually training interesting frontier models in Israel now?