00:07:32 · John Collison, Sam Altman
[00:07:32] S0: The exploits get a lot of headlines. The prevented exploits do not. How quickly are people making progress with Cyberblue? I don't want to [00:07:42] S1: declare victory at all on the PHP thing, because I still don't think we know. And I assume some bad things are going to happen. Relative to my worst fears, if I can, like, think back ten days ago, the first day of that went much better than I expected. And we heard a bunch of stories from people that were using our models to, I mean, not only build patches and build cyber defense, but like individual companies who were just like, I need to just do a quick scan appointment. Like, I'm take gonna my thing down for an hour. I'm gonna use this thing, I wanna put it back up because my application's offline. Now I feel like I really jinxed it, I shouldn't have said that. But the first day did go a lot better than I thought, and I think people have been able to use that. The thing that I've heard a lot of cyber experts talk about is they just no longer are confident that we have the right assumptions baked into our threat model. And they're like, look, if everything is what we think and hardware exploits are as hard as we think they are, then yes, we can see how this is going to work. But if AI is going to discover really deep, bugs in hardware that require physics we didn't understand, but all of a sudden means some assumption doesn't hold, then I think people aren't sure what happens.
00:08:53 · John Collison, Sam Altman
[00:08:53] S0: One thing that always strikes me is that when I talk to very knowledgeable people inside OpenAI and similar companies, there is a pessimism that sandboxing is possible, basically. And that would not have been my mental model going in. It would have been this, yeah, you, have to build a really tight sandbox, and we haven't kind of hardened them historically enough compared to the current capabilities. But ultimately, you can have a maximum security prison for the model to to work in and keep it inside there. It seems like you guys have kind of a pessimism that sandboxing models is possible. [00:09:35] S1: My instinct... Well, first of all, think we knew much better than we're doing now, and we should. I think we we clearly... That is the the main thing to focus on right now is a better security model. If you imagine a theoretical AI that is, like, a million times smarter than the smartest security researcher, would you, like, really be willing to trust that sandbox, especially if the model can, like, discover new science that people don't already... [00:10:01] S0: You you can use the models to construct the sandboxes. [00:10:03] S1: I think you can. Well, so this gets... Let me come back to that. [00:10:07] S0: Yeah.
00:10:07 · John Collison, Sam Altman
[00:10:07] S0: Yeah. [00:10:07] S1: I think sandboxing works to a point. Then I think monitoring works to a further point Mhmm. Where we can look at what the models are doing, not only in their chain of thought, we can do activation monitoring and try to really understand, like, are the models... Are we happy that the models are not deceiving us? And then, eventually, I think you need to believe with huge confidence that the models are truly aligned and are going to value and try faithfully to do... To follow their actual... The intent of the people operating them, and that they're going to value sort of people being in control of the future forever. And if you had a model that you could fully trust the alignment, that you were absolutely certain of the alignment of, then yes, I think you can have that to help you build better and better sandboxes. But, like, if you really let yourself project forward into the, like, what does it mean to have something a million times smarter than we are? I I I don't know. Like, I think that just gets [00:11:09] S0: strange. Because the alignment becomes hard? [00:11:12] S1: Because you need to really, really trust the alignment to trust that these sandboxes stay in ahead of capabilities. [00:11:20] S0: Can you not give models competing incentives and give them incentives to defect from each other? [00:11:24] S1: I think this is a promising direction. I suspect that this idea... I hate the word agent swarms. Let's call them agent teams. [00:11:35] S0: But it seems to
00:11:35 · John Collison, Sam Altman
[00:11:35] S0: But it seems to [00:11:36] S1: be I taking suspect that teams of agents have very interesting scaling law properties, but also very interesting alignment properties. And I think that's, like, a direction we should push in more, just from capabilities, from a safety perspective too. [00:11:53] S0: I know it when you talk about agent teams, not swarms. A swarm is such a... I know. It's a locust. Yeah. Yeah. Not good. Not good branding, in keeping with many historical AI industry traditions. [00:12:06] S1: We are still good at this. [00:12:08] S0: We'll get back to branding. But I know that there seems to be a big gap in what the leading AI users are doing and kinda standard corporate use in the use of Swarm is for, like, all the advanced math... Sorry, teams. All the advanced math uses agent teams. Yeah. All the advanced hacking uses agent teams. And then, like, common or garden corporate stuff is not using big swarms. Why has that not done? Because we've had this for many months. [00:12:39] S1: Not to the same degree. Like like, the one way to look at this problem, or this question, is how much compute can we productively throw at a single problem? And, you know, a few years ago... We forget how fast this is all gone. But a few years ago, it was, like, impossible to throw more than a dollar of compute at a problem and have it help you at all. [00:13:00] S0: Why?
00:13:00 · John Collison, Sam Altman
[00:13:00] S0: Why? [00:13:02] S1: You could have GPT-four, like, grind out for you, you know, like a 100,000 tokens or something, and it was going to answer your question. And it certainly was not gonna be able to go right productively, like write you a piece of software. We just... We had not figured out how to use a lot of test time compute to get you a I better see. And then we figured out a chain of thought, and the models could think hard about a problem. And now, And then, like, people really started using a lot of compute. And now you can throw a whole team at something, and you can do that, you know, 10,000 times in parallel. You need chain [00:13:37] S0: of thought and test time compute before you can get the swarms. Because you can't have a useful swarm with the one two two two three. Yes. [00:13:42] S1: But so the effect of this, and I think this is not being well followed, paid attention to in the world, but the incredible exponential of how much compute you can throw at a single problem and get better results is one thing. The delta between, like, what most people are using AI for and what the people are using AI the most productively for has just continued to spread and spread and spread. How do you allocate
00:13:42 · Sam Altman, John Collison
[00:13:42] S1: But so the effect of this, and I think this is not being well followed, paid attention to in the world, but the incredible exponential of how much compute you can throw at a single problem and get better results is one thing. The delta between, like, what most people are using AI for and what the people are using AI the most productively for has just continued to spread and spread and spread. How do you allocate [00:14:11] S0: compute budgets now? Because presumably, even though researchers are paid a lot, it's eclipsed by compute. And so capital allocation at OpenAI is sort of compute allocation, and it's hard to a priori know how much it will cost to go do something. Yeah. It's actually also hard to just, like, control the... Like, you can't quite prompt an agent where it's like, go do a really good job of this and don't spend more than $10,000. [00:14:40] S1: You can. [00:14:41] S0: Oh, okay. Will it do a job Yeah.