01:02:34 · Dylan Patel, Jordi Hays
The domains that people operate in. Right? The amount of siloing is just so massive because the complexity here is far beyond that of any software engineering. And so the models need to get really, really good at these things before they can assist these people and and, you know, help close this loop in a full circle, right, and then optimize the heck out of these things, make these experiments that you're running, you know, have have that data actually influence the design and go towards the right process, improve the yields, help you shrink faster, help you do whatever it is, and and then actually generate hundreds of billions, if not trillions of dollars of economic value to the world. [01:03:11] S0: Talk about the incentive for people building in these domains using the models to not wanna talk about what they're doing publicly. Right? Because I feel like part of the current sort of disillusionment with the state of AI is that you have a bunch of new software being created, software engineers are obviously happy. Look, I shipped this new feature. Or or you have, like, really cool... [01:03:35] S3: I don't know, bro. I think I think software engineers are depressed, by the way.
01:03:39 · Jordi Hays
[01:03:39] S0: Yeah. But at the same time, you have, like, the excitement of, like, Karpathy making, like, Hacker Tube. Right? And everyone's like, oh, that's really cool that, like, you can do a two, three, sentence prompt and create this sort of novel product that maybe wouldn't have been worth putting the effort into historically. But I feel like when you talk about these domains like material science, I know from having built businesses in the past, like if I figure out like one simple trick, I assume that like it's only gonna be like, you can only arbitrage it for like a certain amount of time before it becomes sort of common knowledge. And so there's an incentive or or even another example, like if I find an amazing freelancer, like I'm not gonna try to introduce them to like 20 people because I don't want my competitors or other companies in the category to have access to them. I assume over time, they'll probably... That that person will meet a bunch of people. But, like, right now when you talk about, you know, these domains where you can have like a a 20% efficiency gain or things like that, there's so much in... Like, I assume that you guys are calling all these people and like trying to understand the kind of gains that they're getting, but they're not exactly raising their hand and going on LinkedIn or going on X and saying, oh, we're getting this massive speed up or efficiency gain here and we want to tell everyone. Because they want to get as much out of it as they can before it becomes more common knowledge.
01:05:00 · Dylan Patel, John Coogan
[01:05:00] S3: There's there's definitely a big difference in hustler culture, right, in in sort of software and founder land and SF and sort of these things versus, you know, these industries that have... That are manufacturing and making the world go, right, and work and operate. I agree that they're not gonna stick their hand up and scream, what they're doing. They'll they'll do some vague stuff. Hey. We improved our our manufacturing efficiency. Hey, customer. Here's what we did. It'll get out there. Right? You make your product better. Everyone will understand that you made your product better. Your financials get better. Everyone will understand that your financials got better. These are just, like, obvious facts that are public. And so, you know, we're in the very early innings of these things, to be clear, you know, whether it be material science or semiconductor engineering or whatever whatever very hard science thing it is. But our our sort of, like, one of our main, you know, focuses from GDPX looping it back is to focus on these areas where we [01:05:55] S1: think
01:05:55 · John Coogan, Dylan Patel, Max Kan
[01:05:55] S1: think [01:05:55] S3: AI has to improve drastically in order to deliver a trillion dollars of revenue. And now we at SemiAnalysis are very bullish on this happening, on AI delivering a trillion dollars of revenue, you know, just open eye and anthropic and the leading labs within the next couple years. You know, the two... You know, Max and Andrew will call me bearish for saying in two years, but, generally, you know, if if those areas don't deliver, then, yeah, we're we're gonna, like, we're gonna saturate all the existing tasks pretty quickly and spendable. And and diffusion of, like, white collar work replacement will be too slow, and these companies will actually not even maybe be able to make their compute commitments. Right? It's like that's that's like sort of... And I think the AI labs, I think us at SemiAnalysis, I think the people who are really in the know see the tremendous progress that's happening. I mean, Max or Andrew, why don't you talk about the material science eval a little bit that we did and and some of the things that our PhD collaborator sort of found. Yeah. [01:06:53] S2: Sure. You wanna take this one, Andrew?
01:06:53 · Max Kan, Andrew Megalaa, Tyler
[01:06:53] S2: Sure. You wanna take this one, Andrew? [01:06:54] S5: Yeah. Basically, like, our our collaborators, like, found that, like, the models are much, much better than they previously expected. And so they have, like, a moment where they have to, like, sit back and be like, wait. Like, this is actually better than a lot of, like, the stuff people are putting out there in papers. And, like, in fact, in, like, one of the problems, like Astra, for example, solved, like, a subtask of a task better than, like, a state of art state of the art reference published reference. And so that was, like, a big moment for them. So it's very clear that the models are very spiky in these categories and are increasingly better and reaching, like, above, you know, superhuman baselines. But, you know, still, like, even on our benchmark, the best model, Astra material size, only reached, like, 28%. I know you've submitted it. [01:07:42] S7: Can you talk about the the value of, like, using human baselines in these? I noticed there was no human baseline in these. Is that because this bedfrog is just like so general and like it wouldn't make sense to... For someone to be so good at all these different things? Or why is there no kind of human baseline compared to a person here?
01:07:42 · Tyler, Andrew Megalaa, Dylan Patel
[01:07:42] S7: Can you talk about the the value of, like, using human baselines in these? I noticed there was no human baseline in these. Is that because this bedfrog is just like so general and like it wouldn't make sense to... For someone to be so good at all these different things? Or why is there no kind of human baseline compared to a person here? [01:08:01] S5: Yes. Someone actually tweeted about this and I replied to him. It was... It's funny because it's very hard to put a human baseline on these. Right? Like, if Astra just gets better than, like, state of the art, like, teams of researchers, like, what what is a human baseline there? I mean, maybe for software engineering, could say, like, this could take x days of software engineering. But, like, some of the stuff is just, like, teams of hardware engineers would have had to do to get done. Like, in our hardware design task or our chip design task, like, that's that's not one person. That's, like, many, many people working together, and it's very difficult to even put a number on, like, the amount of hours or days that it would take for those people to accomplish... Like, finish even those tasks that we have. [01:08:47] S3: That makes sense. Well past the point where we're trying to measure, you know, our models similar to humans. It's like we're trying to measure when will models be superhuman because we're not gonna pay for models that are similar to, like, regular humans. People are only gonna pay for models that are superhuman in certain domains and then utilize them to advance humanity.