00:13:00 · John Collison, Sam Altman
[00:13:00] S0: Why? [00:13:02] S1: You could have GPT-four, like, grind out for you, you know, like a 100,000 tokens or something, and it was going to answer your question. And it certainly was not gonna be able to go right productively, like write you a piece of software. We just... We had not figured out how to use a lot of test time compute to get you a I better see. And then we figured out a chain of thought, and the models could think hard about a problem. And now, And then, like, people really started using a lot of compute. And now you can throw a whole team at something, and you can do that, you know, 10,000 times in parallel. You need chain [00:13:37] S0: of thought and test time compute before you can get the swarms. Because you can't have a useful swarm with the one two two two three. Yes. [00:13:42] S1: But so the effect of this, and I think this is not being well followed, paid attention to in the world, but the incredible exponential of how much compute you can throw at a single problem and get better results is one thing. The delta between, like, what most people are using AI for and what the people are using AI the most productively for has just continued to spread and spread and spread. How do you allocate
00:13:42 · Sam Altman, John Collison
[00:13:42] S1: But so the effect of this, and I think this is not being well followed, paid attention to in the world, but the incredible exponential of how much compute you can throw at a single problem and get better results is one thing. The delta between, like, what most people are using AI for and what the people are using AI the most productively for has just continued to spread and spread and spread. How do you allocate [00:14:11] S0: compute budgets now? Because presumably, even though researchers are paid a lot, it's eclipsed by compute. And so capital allocation at OpenAI is sort of compute allocation, and it's hard to a priori know how much it will cost to go do something. Yeah. It's actually also hard to just, like, control the... Like, you can't quite prompt an agent where it's like, go do a really good job of this and don't spend more than $10,000. [00:14:40] S1: You can. [00:14:41] S0: Oh, okay. Will it do a job Yeah.
00:14:41 · John Collison, Sam Altman
[00:14:41] S0: Oh, okay. Will it do a job Yeah. [00:14:43] S1: If you say do the best job you can with $10,000 of compute, like, yes, it will understand that. I mean, there's like a there's another kind of, like, hard question for society about what does it mean when you have two companies competing, and one says you can spend 10,000, and one says you can spend a 100,000 on the same problem. Like, it's clear what's gonna happen there, but there's, like, a question about how society thinks about computer allocation and wealth and things like that. But we have our own version of this inside of OpenAI. We allocate, you know, a lot of compute to internal productivity, and then people get very wide latitude to make their own decisions unless they're really being abusive. [00:15:23] S0: But do just encourage people to, like, do useful stuff, and if they spend $10,000,000 in compute, it's like, oh, interesting. [00:15:29] S1: Correct. We try never to reward people for using a lot of compute. Sure. Like, no token leaderboards. It's actually slightly bad to be high on the token leaderboard, certainly not a promotion. But but it's like... It's a skeptical default, if you can show that you're doing something important, you will have nothing but appreciation.
00:15:29 · Sam Altman, John Collison
[00:15:29] S1: Correct. We try never to reward people for using a lot of compute. Sure. Like, no token leaderboards. It's actually slightly bad to be high on the token leaderboard, certainly not a promotion. But but it's like... It's a skeptical default, if you can show that you're doing something important, you will have nothing but appreciation. [00:15:46] S0: But at a certain point, this will need to change as... Because you're describing the increasing orders of magnitude, and right now you have a researcher who does some very interesting math for $10,000,000, and you're like, oh, a little pricey, but cool math, and then a $100,000,000. You know, when they're doing the billion dollar math, you're like, look, I'm really happy that you're making so much math progress, but I really need you to cool the jets with the math. [00:16:09] S1: Look, we did let it get out of control for, like, one week on the math, intentionally so, because you have [00:16:13] S0: to be like... And it worked? [00:16:14] S1: Well, you don't know how... You don't know how far it's gonna go until you try. [00:16:18] S0: Yeah yeah. [00:16:19] S1: And I think it's important to have... When there's a new paradigm, I think it's important to let people be fairly unconstrained. So when we did the Navier-Stokes proof, we understood absolutely nothing about the scaling laws of teams, of agents. You know, are you supposed to, like... [00:16:35] S0: The very recent Navier Stokes Yeah.
00:16:35 · John Collison, Sam Altman
[00:16:35] S0: The very recent Navier Stokes Yeah. [00:16:37] S1: Are you supposed to give them, like, the smartest model? Are you supposed to give them, like, many copies, fewer copies, longer rollouts, shorter rollouts? We did not understand that at all. And now, we understand it a little bit. I think we can still understand it, like, much better. But like has happened in every other new paradigm we've discovered, I am confident that we will find our, you know, 10x improvements per year at how we allocate dollars or computer flops, whatever you want measure, into useful things. So I think this is a thing that our company is particularly good at, but also, like, fits an important property of the field, which is let people explore at the limit of what is possible, and then trust that the incredible optimization pressure will figure out how to do this in an efficient way. I... And, you know, directionally speaking, I would rather us be more willing to let people use crazy amounts of compute, and let users have extremely high end plans to figure out the edge of the possible and then figure out how to make that efficient at mass scale. [00:17:43] S0: But when does the optimization function come in? [00:17:45] S1: Well, think we're already doing it. I mean, like, I bet we could redo Navier Stokes at one one hundredth of the cost a year from when it was done. So we did it in, like... [00:17:57] S0: Not just knowing how the proof works, but, like, just with better approaches. [00:18:00] S1: Yes. Smarter models and better approaches. [00:18:02] S0: How much shall the math cost?
00:18:02 · John Collison, Sam Altman
[00:18:02] S0: How much shall the math cost? [00:18:04] S1: Order 10,000,000 per, like... For for, like, Navier Stokes, I think. [00:18:08] S0: Okay. Maybe a little more. 10,000,000 ish. That that order yeah. Yeah. But but for one proof. And so... 10,000,000? [00:18:16] S1: Well, no, no. Because then we kept getting better so fast. Like, we understood more. It. And we spent way less than that on the average one of those. [00:18:22] S0: But, like, hundreds and millions of dollars of math. [00:18:24] S1: Yeah. Yeah. Yeah. That kind [00:18:25] S0: of work. Okay. Interesting. [00:18:26] S1: But that include, like, that includes, like, much more other discovery than we... What we have published. Yes. So it was, like, maybe we spend We that [00:18:33] S0: more than the 700? We do. Of course. When did get those? Second album?