01:05:55 · John Coogan, Dylan Patel, Max Kan
[01:05:55] S1: think [01:05:55] S3: AI has to improve drastically in order to deliver a trillion dollars of revenue. And now we at SemiAnalysis are very bullish on this happening, on AI delivering a trillion dollars of revenue, you know, just open eye and anthropic and the leading labs within the next couple years. You know, the two... You know, Max and Andrew will call me bearish for saying in two years, but, generally, you know, if if those areas don't deliver, then, yeah, we're we're gonna, like, we're gonna saturate all the existing tasks pretty quickly and spendable. And and diffusion of, like, white collar work replacement will be too slow, and these companies will actually not even maybe be able to make their compute commitments. Right? It's like that's that's like sort of... And I think the AI labs, I think us at SemiAnalysis, I think the people who are really in the know see the tremendous progress that's happening. I mean, Max or Andrew, why don't you talk about the material science eval a little bit that we did and and some of the things that our PhD collaborator sort of found. Yeah. [01:06:53] S2: Sure. You wanna take this one, Andrew?
01:06:53 · Max Kan, Andrew Megalaa, Tyler
[01:06:53] S2: Sure. You wanna take this one, Andrew? [01:06:54] S5: Yeah. Basically, like, our our collaborators, like, found that, like, the models are much, much better than they previously expected. And so they have, like, a moment where they have to, like, sit back and be like, wait. Like, this is actually better than a lot of, like, the stuff people are putting out there in papers. And, like, in fact, in, like, one of the problems, like Astra, for example, solved, like, a subtask of a task better than, like, a state of art state of the art reference published reference. And so that was, like, a big moment for them. So it's very clear that the models are very spiky in these categories and are increasingly better and reaching, like, above, you know, superhuman baselines. But, you know, still, like, even on our benchmark, the best model, Astra material size, only reached, like, 28%. I know you've submitted it. [01:07:42] S7: Can you talk about the the value of, like, using human baselines in these? I noticed there was no human baseline in these. Is that because this bedfrog is just like so general and like it wouldn't make sense to... For someone to be so good at all these different things? Or why is there no kind of human baseline compared to a person here?
01:07:42 · Tyler, Andrew Megalaa, Dylan Patel
[01:07:42] S7: Can you talk about the the value of, like, using human baselines in these? I noticed there was no human baseline in these. Is that because this bedfrog is just like so general and like it wouldn't make sense to... For someone to be so good at all these different things? Or why is there no kind of human baseline compared to a person here? [01:08:01] S5: Yes. Someone actually tweeted about this and I replied to him. It was... It's funny because it's very hard to put a human baseline on these. Right? Like, if Astra just gets better than, like, state of the art, like, teams of researchers, like, what what is a human baseline there? I mean, maybe for software engineering, could say, like, this could take x days of software engineering. But, like, some of the stuff is just, like, teams of hardware engineers would have had to do to get done. Like, in our hardware design task or our chip design task, like, that's that's not one person. That's, like, many, many people working together, and it's very difficult to even put a number on, like, the amount of hours or days that it would take for those people to accomplish... Like, finish even those tasks that we have. [01:08:47] S3: That makes sense. Well past the point where we're trying to measure, you know, our models similar to humans. It's like we're trying to measure when will models be superhuman because we're not gonna pay for models that are similar to, like, regular humans. People are only gonna pay for models that are superhuman in certain domains and then utilize them to advance humanity.
01:08:47 · Dylan Patel, Jordi Hays, Max Kan
[01:08:47] S3: That makes sense. Well past the point where we're trying to measure, you know, our models similar to humans. It's like we're trying to measure when will models be superhuman because we're not gonna pay for models that are similar to, like, regular humans. People are only gonna pay for models that are superhuman in certain domains and then utilize them to advance humanity. [01:09:07] S0: I'm so curious. We were talking about this earlier on the show. How how should people read into Elon's post? I believe it was Saturday that that, like, Grok bot is just too expensive to run. Sorry. Tangent. But I'm curious what the what the house take is on that. [01:09:25] S2: Like, it might be too expensive to run today, especially when consider sort of, like, the CPU and, you know, sandboxing cost needed per user. But I think one thing to understand about these personal agents is that it's very likely you will not need sort of the smartest frontier model a year from now in order to, like, serve the majority of people. And sort of the the cost to serve will actually undergo this, you know, potential 100 x plus. I follow the same trend as essentially a decreased cost of intelligence that Dylan was talking about. So even if, like, personal agents are really sensitive to serve today, I think it's very likely that they are totally economical to serve a year
01:09:25 · Max Kan, Jordi Hays, Andrew Megalaa
[01:09:25] S2: Like, it might be too expensive to run today, especially when consider sort of, like, the CPU and, you know, sandboxing cost needed per user. But I think one thing to understand about these personal agents is that it's very likely you will not need sort of the smartest frontier model a year from now in order to, like, serve the majority of people. And sort of the the cost to serve will actually undergo this, you know, potential 100 x plus. I follow the same trend as essentially a decreased cost of intelligence that Dylan was talking about. So even if, like, personal agents are really sensitive to serve today, I think it's very likely that they are totally economical to serve a year [01:10:02] S0: from now. I know. So what's the logic? You have a product that people love that's like, you know, gaining traction and then you basically, like, have everyone hit their usage limits, you know, suddenly and then say, oh, it's too expensive. [01:10:17] S5: Like has Elon's [01:10:18] S0: been on the record saying that he thinks that Grok bot will one day, and again, I don't know how much to read into this, but he's like, he thinks that Grok bot will will consume the majority of their compute over a longer time horizon. So like, what what what... You know, assume the cost is gonna drop a 100 x, like, why do you why do you stop, like, subsidizing that when it's seemingly working?
01:10:42 · Dylan Patel, Jordi Hays
[01:10:42] S3: Well, Elon, you have to remember what Elon's trying to do right now. Right? He's taking SpaceX and turning it into a Neo Cloud. Right? In in in some sense. Right? He's building ridiculous amounts of compute. And, you know, he's only got... You know, if you talk take his balance sheet, you know, he's got, what, like, $80,000,000,000 of cash on it. $20,000,000,000 is gonna go to, you know, rockets and stuff like that. But then the other 60,000,000,000 he can spend on compute, he's raising another 40,000,000,000 right now. At the same time, he claims he wants to build 13 gigawatts. 13 gigawatts, if you build all of that out, is is, you know, you know, 400 plus billion dollars. Right? You know, 500 plus billion dollars. So so where does he get the cash? Right? He has to monetize his compute as effectively as possible to able to pay for all the CapEx that he wants to do. And given what's going on in the spot compute market, right, you know, being able to sell GPUs for, you know, $15 an hour, when the TCO of them is only, like, $4 an hour, you you have this massive delta. And so he's exploiting that, and he needs to focus on how do I build the most compute, period. Right? And that's that's step one. Step two is, okay. How do I use that compute most effectively to dominate the universe? Right? It's sort of like, I don't know if you're, like... You take a evil look at it. I'm sure it's just to... It it is just to benefit humanity. But the only way [01:11:57] S0: you can build 13 gigawatts a little bit.