00:08:51 · Ruby Justice Thelot
[00:08:51] S1: So, yeah, this this reminds me of, another post from nostalgebraist on Less Wrong, who I consider another one of these, like very good AGI centrists. I think you're kind of one of them. I think Seb Krier is one of them. Is it? So nostalgebraist has posted this thing on LessWrong. I'm trying to find the exact text, but it's something like LLMs are adversarially optimized for humans to think that they're really good when they still have, like, substantial weaknesses. None of this is to say, of course, that they are useless. There it is, yeah. Today's LMs are so LLM agents are so smart in so many ways, and yet they constantly make fairly basicdumb mistakes in my experience. This was seven days ago, so with basically the newest LLMs that are available, and says, Yeah, working with these things is often frustrating. This is a very long comment I'm trying to pull through it. It's becoming increasingly unclear what co working process I should expect to work well. They just not have the judgment to do nontrivial things in autonomous settings. They're like smart but naive first year grad students: book smart, hard working, lacking in tastediscernment and ability to prioritize, liable to go off on silly low value tangents, which they treat with the utmost serious if not closely watched and steered. But unlike real grad students, they can't accumulate knowledge or practical know how over time.
00:10:26 · Herbie Bradley
[00:10:26] S0: Yeah, mean, this reminds me of a lot of the things in software engineering where the actual quality of code or quality of decision making particularly and architectural decision making is below what an experienced senior engineer might feel happy with. And yet, because it's a highly verifiable domain, many people at the frontier of using LLMs for coding are just kind of like yellowing, the agent's form on your task, don't worry too much about code quality, just keep iterating, and you'll get to some output that is plausible or reasonable. So far, I think it has, in low stakes domains, been that you can in fact do this, but nonetheless, there have been many examples of some Vibe created failures and people's Vibe created apps for security holes as a result of this, and I think this is actually going to make it harder for the broader economy to adopt AI at scale, especially at the huge agents form level, if they are required to sort of go all in on this kind of ignore quality to some extent or sacrifice some level of some dimension of decision making that senior engineers or others would be happy with in order to just token max because it would improve overall productivity.
00:11:37 · Ruby Justice Thelot
[00:11:37] S1: Right. I think a counterargument to this whole AGI Centrism thing is straight lines on graphs, the model will just continue to get better, trust the exponential, and also the maximum possible capabilities maxes would have accurately predicted the last five or so years of AI progress and AI development. So I guess I need some epicentic humility with that in mind. But if they're not correct, like if AI capability does continue to improve in a jagged way rather than improving in a fully general way and leading to a real for real superintelligence within a few years, what do you think would be the most likely reasons that that doesn't happen?
00:12:28 · Herbie Bradley, Ruby Justice Thelot
[00:12:28] S0: I think probably it's that the at some point, the straight line on the graph diverges with, you know, actual what we would consider desirable qualities of a superintelligence, like the ability to make human like decisions across almost every domain, but better even to be able to do persuasion or other big tasks, which are very hard to quantify or define, as well as humans. You can define arbitrary benchmarks and evals to hill climb on, but in some cases that may diverge from real world qualitative utility. I think that is part of the reason why we're saying, for example, doctors' subjective experience or how good models are at diagnosing medical cases to be that the models have not improved a huge amount often in the last nine months. It's because, you know, you can construct many different benchmarks and evals for this task, but, ultimately, they don't quite capture everything. And so the question really for the diverges, you know, capabilities, Maxis, and what you might consider AI centrists is something like, you know, what are the problems of generalizing good hearting, and what happens if you sort of extrapolate the train on every quantifiable thing to the limit, and then what remains? [00:13:51] S1: Well, what remains? [00:13:52] S0: Well, that's the question. My take is still that there are a number of things in human activity, especially economically valuable things, that are very hard to quantify. And one reason why, for example, I don't expect creative writing capability to surpass the best humans or other things like this.
00:13:52 · Herbie Bradley, Ruby Justice Thelot
[00:13:52] S0: Well, that's the question. My take is still that there are a number of things in human activity, especially economically valuable things, that are very hard to quantify. And one reason why, for example, I don't expect creative writing capability to surpass the best humans or other things like this. [00:14:11] S1: If the AGI centrists are right, then does that imply that we will see some sort of market crash or correction, or have the markets not fully priced in the capabilities and axi perspective yet? [00:14:24] S0: Oh, yeah. I think these two things are mostly divorced, and the markets themselves are mostly dependent on the rate of diffusion. And right now, there's a huge overhang in terms of the amount of capabilities that exist and what people are actually using them for, what companies are using them for in the real economy. And so that diffusion cycle will continue even if we froze capabilities progress, And then the markets would depend on like how big is the level of demand from that. [00:14:49] S1: How true is that? Like, what would you consider current capabilities that would be able to have economically beneficial, useful applications that haven't been diffused? Like one clear example I can think of is the school system, I think could be massively improved with current AI. And this hasn't happened. But this to me is more political constraints because public schools are not subject to economic incentives in the way private businesses are. [00:15:17] S0: So could
00:15:17 · Herbie Bradley, Ruby Justice Thelot
[00:15:17] S0: So could [00:15:18] S1: we imagine that, you know, there are just like insurance companies run by boomers who don't understand the latest tech who aren't using it where it would be helpful? [00:15:27] S0: I mean, more or less, yes. I think it is just across all of the big, let's say, white collar domains, where there are large companies. AI adoption has progressed to the point where many of these companies have bought Claude Code seats or GPT seats for all their employees, but the intensity of use is not there. And there is still the fact that you ultimately need some degree of creativity or, you know, open thinking in order to be able to figure out what are the best applications of AI for your company, for your team, for your job, and a lot of people are just sort of not really doing that. And as a result, you see situations like, you know, a Fortune 500 company has ChatGPT seats for everyone, but only 1% of the company is using it by... Is, like, maxing out the subscriptions, and everyone else is, like, sort of not really using it very much, or they're not really using the most useful applications. It's just like we're still very, very early in this diffusion cycle, I think, and so demand could easily, you know, 10 x, a 100 x from here just with current capabilities.