00:10:22 · Qasim Wani
[00:10:22] S0: Yeah. So so this is actually like a really interesting question of like, why why do RSI outside or why work on this problem outside the Frontier lab and not directly inside the lab? I think that's sort of like a bet that we are taking, which I think is somewhat interesting. There's this really good book that I highly recommend called Why Greatness Cannot Be Planned. It's by Kenneth Stanley. It's a really fun read for the technical folks. All that the reader talks about in that book is to be able to solve the most ambitious ideas, you have to have some degree of nonlinear thinking. Yeah. If your ideas are the exact same as everyone else's, you're going have the same outcomes as everyone else. And part of what our hypothesis is that the labs are focused on superintelligence. Like is on, like our side is on the critical path towards the labs achieving superintelligence. The problem is that, and this is sort of what we discovered empirically at xAI is, RSI is fundamentally a data problem. It is not a model problem. It is a data problem. That's all it is. And this is sort of like my biggest qualm with the data industry is, as someone who personally evaluated dozens of vendors back when I was at xAI, the quality of samples is not that great. Majority of the vendors that you work with don't really know what it takes to build a frontier grade model. They have some ideas based on their interaction working with the labs. The labs usually won't tell them this was good because of so and so. They kind of just YOLO things and hope that it's good. And so our hypothesis is that if you take set of guys from the labs and instead of them working in the labs, but now creating high quality data and selling that to the labs, you actually get to do two things. One, you get to democratize superintelligence by leveling the playing field. Like I don't want to be in a world where Opus is the only model lab that has access to AGI or ASI. So that's like just healthy for the ecosystem. And second is that there's just like a lot more entropy. There's more interesting things that might happen by working on these kinds of problems and not being limited to a certain type of a model architecture or model recipe. And this is why, you know, that's sort of like how we think about why doing this outside and inside.
00:12:54 · Ruby Justice Thelot, Qasim Wani
[00:12:54] S1: And so why is data, like why is data the bottleneck for RSI, not model capabilities? [00:13:01] S0: I think having high quality data is what enables better model capabilities. The way that I think about this is that we have entered a new economy where the frontier labs have become the literal market exchange for goods and services. And from that perspective, there's only like two real ways of contributing to this economy, which is either you sell compute via energy or you sell data via like human intellect. Yeah. There's like nothing else that matters besides these two things. So both compute and data are important. The problem with the data industry is that I think it's extremely undervalued. We have multiple model companies that are worth over $1,000,000,000,000. We have multiple chip companies worth over $1,000,000,000,000. But the sum total of the data industry is not more than $150,000,000,000. And that to me feels like a very gross underestimate of how valuable access to high quality data is. Internally at the labs, we would spend, as we all know, ungodly amounts of money on data curation and coming up with high quality data. So it's literally all how good's your data is, how good's your model. [00:14:13] S1: It's kind of funny hearing you say that when a lot of other people in AI are like, oh, data doesn't even matter that much because of RSI, right? The model will just be able to RSI its way to overcome whatever data bottlenecks it has, which is like a thing that I've always been skeptical of. It seems like it's kind of magical thinking.
00:14:32 · Qasim Wani
[00:14:32] S0: There's a benchmark that Patronus, a company that I respect, released earlier today, where it sort of asked agents to create RL environments. And I think the best performing model had a pass rate of 49 to 50%, which means that for that specific capability of how good is a model in creating an environment, not even like the full recipe, it's still only like halfway there. I mean there's like all kinds of like nuance in like how high quality are these environments and like the sophistication around it. There was this paper called Code Meet Us that Xiaomi released a couple weeks ago where they open sourced about 21,000 environments that were purely generated synthetically. And this was open source. It sort of went viral on x. But a lot of the reviewers, including Vowels AI, they sort of discovered that when the agents were creating these tasks synthetically, and they're using frontier models, I would assume Fable or Astra, when the agents were creating these tasks synthetically, a lot of the tasks were focused around can an agent create the missing patch from some repo or some code base and like identify how to like bug fix stuff. During the construction of these tasks, the agents accidentally leaked the code that was deleted in the commit, Which is actually like very concerning because that basically tells the agent how to reward hack or cheat and get to the right answer. I would love to be in a world where agents can like create the data themselves. If an agent can create perfect environment that has no reward hacking behavior, that is actually like on the Pareto frontier of model capabilities, that is to me the terminal state. Like it's the definition for superintelligence for me is can an agent create a high quality environment that has no reward hacking behavior? And as someone who's been doing this for a long time now, we are nowhere close to that. Now that doesn't mean that we won't get there. Like, that's really the goal for for our our company is how do we go from where we are today to that end state.
00:16:44 · Ruby Justice Thelot, Ruby Justice Thelot, Qasim Wani
[00:16:44] S1: Mhmm. [00:16:45] S2: How do you think about, like, any risks associated with RSI? Because, like, a lot of what we've seen today on the timeline has just been, like, we need to pause before we... But Yeah. It seems like you're, like, thinking [00:16:57] S1: about it. It's been specifically, like, a [00:16:59] S0: lot of, a lot [00:16:59] S1: of freaking out, a lot of major updating based on this [00:17:02] S2: meter thing. Yeah. Yeah. Yeah. But you seem to be approaching it from a more, like, I don't know, problem solving way.
00:17:08 · Qasim Wani
[00:17:08] S0: I think I think my my, like, impression of this is a little bit different than my friends at Anthropic who are more like doomsdays here, like pdoom is super high. I'm, you know, an engineer and an entrepreneur, I really think about how can I be helpful in solving these problems? Like we obviously need like better AI policy and like better governance around this. But one way of being useful here is how can you engineer your way out of destruction? And I think this is a very solvable problem if you fix the data. Hence why, if the data is, let's say, if the data has word hacking behavior, which a lot of the vendors that we used to work with back when I was at xAI, we would have models that would do all kinds of catastrophic stuff. And that is really, really bad because as a researcher, you have so many priorities, you're not going to spend a lot of time looking at every single sample and you kind of just YOLO things every now and then regardless of you know what kind of lab you are. But it's really important as a data vendor or a data lab to ensure that the quality of your samples are super high because you run the risk of models having all kinds of weird behaviors, and those weird behaviors may be like in 1% of the samples. And that 1% of sample like economically could be like worth hundreds of billions of dollars, which is why what we're doing is we're sort of taking guys who've worked at Frontier Labs, who understand the model recipes really well, and making safe, aligned, but data sets focused around RSI capabilities to ensure models are safe and not destructive.
00:18:55 · Ruby Justice Thelot, Ruby Justice Thelot
[00:18:55] S1: Yeah. Mhmm. [00:18:56] S2: And I guess before, like, we even get to RSI, like, within labs, there are already teams that are doing a lot of handoff of, like, particular... Like, they're just pressing yes. Right? Like, they're just using Claude to press yes. [00:19:09] S1: So how do you also Wait. What teams? [00:19:12] S2: Like Like, researchers? Yeah. [00:19:14] S1: You can [00:19:15] S2: yeah. You can assume that they're, like, using Claude to make to hand off decisions. [00:19:19] S1: Yeah. So, [00:19:21] S2: like, what do you make of that sort of thing where it's like, yeah. Maybe we don't have recursive self improvement in the sense that you and I conceptualize it, but we do have the models making a huge amount of the decisions, like micro decisions about where certain directions go or micro engineering decisions, know, things like that.