The MAD Podcast: How AI Gets Built — with Matt Turck
00:00:00 · Nathan Benaich, Matt Turck
[00:00:00] S0: The cyber risk is, like, much more real than, like, AI exterminating humanity. You either, like, die trying to get to the frontier or you just live long enough to eventually serve inference. Individuals would prefer to live next to a nuclear power plant than live next to a data center. It's bananas what's happened in the last year alone, so let's see AGI next year. [00:00:18] S1: That is Nathan Benaich, venture capitalist and author of the State of AI Report. Every year, Nathan and his team compile all the key facts, figures, trends of the AI industry into a 250 slide report that is one of my favorite reads and reference documents. Nathan is back for a third time on this show and as usual, we covered many of the stories of the year as well as some of his predictions. Why AGI could happen in 2027, AI that is now building AI, what really happened in the Hugging Face incident, the data center backlash, and how he runs his own fund on agents. I'm Matt Turck, and this is the MAD Podcast. So every year, you do this big report that we're going to extensively talk about, and you end the report with predictions. And, like, historically, your predictions have been somewhat measured, but this year, you went all out. And the last prediction is AGI 2027. So, why is that? Unpack that for us.
00:01:16 · Nathan Benaich
[00:01:16] S0: Yeah. I mean, it's been like ten years roughly speaking of writing this report. And, you know, originally it was like a year is like long enough time that enough stuff happens that you have, you know, enough content to write the report. And honestly, like this year was really, really hectic. Every other day towards the end of the timeline for publishing, which was like October 8, I had to make updates. And there was a point where I was like, please, just like no more news over like one day that I can at least record my like launch video, etcetera. So by the time I was like making the predictions and, you know, I went through a few like scary cybersecurity ones, which I guess only got worse in the last couple of weeks. Just the experience of just seeing like everybody else who's using these products, just like rapid capability improvement, you know, week after week. And I think there's many, many use cases I have now where like, I think the model is like brilliant. I'm particularly like an astro boy. I really love that system. And I could see like in the next twelve months that whether it's, you know, legal finance, like my investment job, like thinking intelligently about topics I don't know that the model could potentially be like better than what I could do in any, you know, compressed amount of time that I have to do the work. And so for me, that would be like approximately AGI.
00:02:28 · Matt Turck, Nathan Benaich
[00:02:28] S1: Yeah. And a big theme of the year that you highlight in the report is RSI, Recursive Self Improvement, which is this idea of AI building AI, which we've discussed extensively on this podcast throughout the episodes. Is that part of the story that would accelerate us towards AGI for you? [00:02:46] S0: Yeah. I think so. Right now, the challenge is, like, how do you frame problems that the model is currently not good at in a way that is amenable to RSI, which would be, you know, potentially having like a software environment that approximates the task on the computer or some kind of verifiable reward, which I'm sure you've discussed a lot about. And I think if we can get to a point where in like any arbitrary domain, we're just limited by the amount of compute and synthetic data we can generate, then yeah, for like most knowledge work, which I I I do think to some degree that's that's possible, particularly as like models get better as as their own verifiers. That would be one of the main reasons why AGI would come would come faster. [00:03:26] S1: Yep. You had some stats that you mentioned in the report from Anthropic where they say Claude leads 26% of its measured model R and D in August.
00:03:36 · Nathan Benaich
[00:03:36] S0: Yeah. Yeah. Yeah. Exactly. And that's like I think up from 1% a couple of months ago and that was August, which sounds really far away. So God knows what it is in October. And so there's that I think that was that was really interesting. But then there was there was another... A stat that I really liked from a blog post that they had in June this year about recursive self improvement, which explained the concept. Then they presented this kind of dual arm experiment where they were testing out different model generations versus a human in this setup where first the human would undertake a research agenda and, you know, go a couple of steps down this path deciding what to work on doing the work. And then at some point, that experiment would fork and the human would continue or a specific model generation would continue. And it would be, you know, the old sonnet versions to Opus and then Fable. And then and the experiment would end and human graders would look at the human path versus the model path. And it was something like the old generation of models humans preferred like 10% of the time or something like that. But for the Mythos class models, humans preferred that research direction 64% of the time, which suggests that models are getting better at predicting the right steps or the right interesting kind of, you know, sort of leafs in the in the nodes to pursue. And then the second kind of piece of data I liked was some research from OpenAI with their Codex usage amongst their AI staff where they tried to classify the sort of token use in the overall process of their of their research. And that would be using tokens for deciding what research question to test, how to test it, what experiment you should set up, like writing the training code, running the, you know, the data engineering pipelines, writing the infrastructure, etcetera, and then marketing the paper, writing the paper, etcetera. And the vast majority of the token usage was on kind of explaining the research writing, which kind of makes sense, you know, sort of knowledge work followed by all the infrastructure for running the experiment. Very, very few tokens were used for what question should I explore and like, why should I explore it and what's the right path?
00:05:42 · Nathan Benaich, Matt Turck
And so I like this question a lot because like I was trained in science and biology and I just remember, you know, this is like ten years ago, fifteen years ago, not that long. But you know, at the start of the PhD, you would sit there with, you know, a ton of papers and you know, you go on the internet and read, read, read, read, read, try to understand what's the status quo, how do people test certain ideas, where are the gaps, what experiment should I run, and then in concert with your supervisor, like decide on the path. And the best supervisors would be the ones who could guide you down the most fruitful path to get hopefully your nature paper and then off you were. And the worst supervisors were the ones that like didn't really know or didn't have great ideas. So that idea of like research taste, which is what kind of gets bandied around on Twitter a lot, is I think one of the hallmarks of intelligence and is one of the critical components towards RSI working well and is one that I just like really enjoy because it speaks to my interest in science. [00:06:41] S1: Yeah. Indeed taste is sort of become a cliched Yeah. Word over the last couple of years. But in the specific context of RSI, just to play back what you just said, it's effectively choosing the right problem. Yeah. That's the right Yeah
00:06:41 · Matt Turck, Nathan Benaich
[00:06:41] S1: Yeah. Indeed taste is sort of become a cliched Yeah. Word over the last couple of years. But in the specific context of RSI, just to play back what you just said, it's effectively choosing the right problem. Yeah. That's the right Yeah [00:06:57] S0: exactly. And you know a lot of famous inventors will say, you know, it's like invention is a result out of good taste and a lot of the work is choosing what direction to work on, not necessarily just boiling the ocean. And so if we wanna be using tokens correctly, using our compute budgets correctly, but also showing that our systems are smart then they shouldn't just be And on boring [00:07:20] S1: conquering taste is something that you see happening in your various conversations as something know something happening in the next few months? [00:07:30] S0: Maybe longer, at least a year. I think the problem is like how do you objectively measure taste? I mean in art I guess you can because it's like perfectly observable and it's... And maybe in like wine you can sort of express like why this taste is better than that taste. But in like science and engineering where there's different paths to getting to the right results, it's not immediately clear. Like would it be the model that consumes the less tokens to get to an answer that humans believe is the best? Is it like the journey that matters more? And I think maybe the whole like uproar in the maths community of, you know, Astra solving or whatever model it was solving various like [00:08:11] S1: There's dope. Open questions.