00:17:08 · Qasim Wani
[00:17:08] S0: I think I think my my, like, impression of this is a little bit different than my friends at Anthropic who are more like doomsdays here, like pdoom is super high. I'm, you know, an engineer and an entrepreneur, I really think about how can I be helpful in solving these problems? Like we obviously need like better AI policy and like better governance around this. But one way of being useful here is how can you engineer your way out of destruction? And I think this is a very solvable problem if you fix the data. Hence why, if the data is, let's say, if the data has word hacking behavior, which a lot of the vendors that we used to work with back when I was at xAI, we would have models that would do all kinds of catastrophic stuff. And that is really, really bad because as a researcher, you have so many priorities, you're not going to spend a lot of time looking at every single sample and you kind of just YOLO things every now and then regardless of you know what kind of lab you are. But it's really important as a data vendor or a data lab to ensure that the quality of your samples are super high because you run the risk of models having all kinds of weird behaviors, and those weird behaviors may be like in 1% of the samples. And that 1% of sample like economically could be like worth hundreds of billions of dollars, which is why what we're doing is we're sort of taking guys who've worked at Frontier Labs, who understand the model recipes really well, and making safe, aligned, but data sets focused around RSI capabilities to ensure models are safe and not destructive.
00:18:55 · Ruby Justice Thelot, Ruby Justice Thelot
[00:18:55] S1: Yeah. Mhmm. [00:18:56] S2: And I guess before, like, we even get to RSI, like, within labs, there are already teams that are doing a lot of handoff of, like, particular... Like, they're just pressing yes. Right? Like, they're just using Claude to press yes. [00:19:09] S1: So how do you also Wait. What teams? [00:19:12] S2: Like Like, researchers? Yeah. [00:19:14] S1: You can [00:19:15] S2: yeah. You can assume that they're, like, using Claude to make to hand off decisions. [00:19:19] S1: Yeah. So, [00:19:21] S2: like, what do you make of that sort of thing where it's like, yeah. Maybe we don't have recursive self improvement in the sense that you and I conceptualize it, but we do have the models making a huge amount of the decisions, like micro decisions about where certain directions go or micro engineering decisions, know, things like that.
00:19:40 · Qasim Wani
[00:19:40] S0: I think that's great. I think we need that. There's still a lot of human involvement in a lot of these processes, But at the same time, certain functions have been almost completely automated, like models kicking off a training run and monitoring for a big run and these types of pre training runs. A lot of that has been automated. But things like auditing for data quality, that is not a solved problem. In fact, on GDPX, the public task that we have shown is how good is an agent in detecting for potential reward hacking behavior on about 1,000 rollouts. And here, just for context, the task was we had some model post training RL run that we have done, and we give an agent, and during this run the reward actually regressed. It went from 0.14 to something super, super low, and this is a fairly normal thing that happens at a lab, and you're like, I want to know what happened and why it happened. We give the agent this task, Astra, Fable, like all the models, and we give them 20,000,000 tokens worth of context, which is about 1,000 rollouts, and we ask it to spend five hours to figure out what's wrong. And almost every single time, all the models gave the complete different answer and an incorrect answer, and they only were able to get the highest score was like 24%. And we made the task fairly simple, where competent engineer could just look through the logs and figure out, oh, this is a weird thing that happened with the harness. But the point that I'm bringing up is certain parts have been automated, but certain other parts that are super critical, like just looking at the data, we're nowhere close to fully automating it.
00:21:27 · Ruby Justice Thelot, Qasim Wani, Ruby Justice Thelot, S3
[00:21:27] S2: Yeah. Yeah. Well, I think we're coming up on time, but this has been super interesting. Thanks so much for joining us. [00:21:32] S0: Thanks for having me. [00:21:34] S2: A huge thanks to MTS sponsors. [00:21:36] S1: ElevenLabs, AI that communicates at human level across every channel and modality. 11labs.io/mts. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their back end. Start on the free plan or get up to a $100,000 in credits for your startup at neon.com/mts. Special thanks to our sponsor, Kong, the AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. Konghq.com. [00:22:07] S3: Finally, prove what your AI investment is worth with Larridin. Head to lairdon.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune one hundred, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go to market opportunities. Mongodb.com Chargebee, billing and monetization infrastructure for the AI era. Chargebee.com. [00:22:40] S2: Blitzy, autonomous software development for enterprise code bases. Ship 5x faster. Blitzy.com. Adquick. Make your brand a billboard out of home advertising as easy to scale as digital. Adquick.com. Arena. Measuring AI performance in the real world. Arena.ai.
00:22:40 · Ruby Justice Thelot, S3
[00:22:40] S2: Blitzy, autonomous software development for enterprise code bases. Ship 5x faster. Blitzy.com. Adquick. Make your brand a billboard out of home advertising as easy to scale as digital. Adquick.com. Arena. Measuring AI performance in the real world. Arena.ai. [00:22:58] S3: Support for the show comes from VCX, the public ticker for private tech. The U. S. Stock market started history's greatest wave of wealth creation. From factory workers in Detroit to farmers in Omaha, anyone could own a piece of the great American companies. But today, our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit getvcx.com for more info. That's getvcx.com. Carefully consider the investment material before investing, including objectives, risks, charges, and expenses. This and other information can be found in the innovations fund prospectus at getvcx.com. This is a paid sponsorship.