00:23:19 · Jonathan Siddarth
For example, in coding, humans may tend to prefer code that's commented really well, that's structured in a way that is easy to maintain, and is more modular. But those preferences are learned in RLHF. And once you've done RLHF, would then do another round of evals, where you're evaluating the model's performance to see if the model improved. As the model improves, you'll figure out where the model breaks next and then generate data to improve the model there again. It's a loop between you first do pretraining. Pretraining is done relatively infrequently. And then post training is a highly iterative process. You start first with SFT, then you would do reinforcement learning with human feedback, you would evaluate the model again, figure out where it's breaking, where it's doing well, and then generate data in the areas where the model is weakened. Now what I just described is how things used to largely work until last year. Now for these verifiable domains like coding and math, instead of doing reinforcement learning with human feedback, you can do reinforcement learning because you can automatically check when you got the correct answer or not in these verifiable domains. And with reinforcement learning, the way you teach the model is not through imitation learning, where you're learning from humans, but it's through experiential learning where you create this environment with prompts and verifiers and you create basically a mini world model where as long as the agent gets the right result, that particular trajectory that the agent took to come to the right answer gets reinforced. And when the agent got to a result that didn't get the right answer, that doesn't get reinforced. So the model kind of learns from its own experience. It's a form of self play which I think is really, really cool. If you remember, from the AlphaZero days, when DeepMind was teaching their Go Playing algorithm, the first version of Google's AlphaGo system learned from expert humans, which was quite good. The next generation, which was even better, just played against itself and figured out which paths tended to it winning and which paths tended to it losing and learned from that.
00:25:51 · Jonathan Siddarth, Molly O'Shea
And there is this Move 37 in one of the games where the model came up with a move that felt alien to the best Go players in the world because the model had figured out a trajectory to win that beat the best human experts. And I'm super excited for AI in the enterprise when an agent takes a path that is so novel and so interesting and so different that really exceeds superintelligence. And for me that's a big part of artificial superintelligence. Why should human intelligence be the bound for what we do? These models could exceed us in many interesting ways. [00:26:30] S2: In today's high speed business world, staying ahead means using the smartest tools possible, including the powerful capabilities of artificial intelligence. Meet Turing Intelligence. Turing builds customizable AI systems designed to solve your mission critical challenges no matter your industry. From expert guidance to tailored projects, Turing helps top companies realize AI that's more capable, more adaptable, and more effective. With Turing, discover how AI can accelerate your business growth. To learn more, visit turing.com/sorcery. Spelt sourcery. That's turing.com/sorcery.
00:27:06 · Molly O'Shea
[00:27:06] S0: I think that's a really interesting point and it's also a point that scares people a lot. It's created a culture of doomers because they're afraid of AI and technology rapidly advancing so fast or so quickly out of our control that it puts safety at a risk. I'm gonna put that to the side, like, quite... Like, a little bit, but it brings it brings into light this tweet that David Sacks had not too long ago, like, right after the GPT-five release. And there's a lot of pushback on this release. It didn't go as expected. Many, were actually, like, quite disappointed. They were expecting, like, something crazy to happen, and turns out it was just a little bit of an incremental, seemingly incremental. I don't wanna offend anyone. Seemingly incremental improvement to the model. But I'm curious from your standpoint, what did you make of that moment in that tweet that David Sacks had put out that was, you know, it was addressing the doomers of, okay, actually, like AI is not moving that fast. We actually might have more time before we have real crazy moves happening.
00:28:23 · Jonathan Siddarth
[00:28:23] S1: Yeah, let me unpack this. I think there are two parts to your question. There's one part which is, are we slowing down? Like, are things going to be more incremental going on? That's one part to your question. And then there's another part which is, how should we think about safety in these AI systems. So let me take that separately. First, I think GPT-five is fucking awesome. I think it's awesome. I think we've just gotten used to magic that maybe we are not fully seeing the impact of what's just out there. And I'm not just saying that because we work with the company that built it. I think GPT-five is awesome. I think one thing is real though, which is the AI doomers talk about this concept of rapid takeoff. It's sort of the I mean, you've seen The Matrix or the Terminator movies, like somehow we'll hit a singularity and there's just this rapid takeoff where humanity gets left behind. I don't think rapid takeoff is how things will unfold. I think it's going to be steady continuous progress. Every step of the way we're going to keep moving forward and it's going to be great. And it's great for a few reasons. The reason I think rapid takeoff will not happen and I know why the doomers believed in rapid takeoff is because GPT-three felt like magic. It was like fucking magic. It was magic because there were a lot of one time jumps in there. Jump number one on the data front. We ate the internet. We ate the internet. There is only one internet and we ate it in GPT-three, right? And GPT-four, again, in all of these and all the models like GPT-four, like Gemini 2.5 Pro, Grok2, Grok3, Claude, it's the same formula. We all ate the internet. And that was a huge one time jump. And we put in all of humanity's knowledge. All the books that you owned the licenses of were fed into the models. So humanity's collective intelligence was eaten in the GPT-three, GPT-four era. And we scaled up compute. Some of the early gains in compute were we also got that, right? And computers takes time to build. You've got to solve energy and you've got to build data centers. And you see what Elon is doing with projects, you see what Sam Altman and OpenAI are doing, you see what Meta is doing. Compute also takes time to build, right? So we got a huge jump by eating the Internet and using all the easily accessible compute.
00:30:58 · Jonathan Siddarth, Molly O'Shea
But the scaling laws, which is an empirical law, is continuing to hold, meaning you take a big model with lots of parameters, you dial up compute, you dial up data, and the models smoothly keep improving. Right? I'm a huge optimist in humans and human potential. I think we humans will solve bigger, more interesting problems instead of the more mundane stuff that we do today. I believe I will be 100x more productive. If I'm 100x more productive, maybe I'll run 100 companies in parallel. Elon might run 600 companies in parallel. I think the solution is simple. Keep hiring the best researchers on the planet. Keep scaling up compute, keep scaling up data, we will get to ASI and it's honestly the most important problem humanity could be working on. Because if we solve intelligence, we can solve diseases, we can solve aging, we can solve perhaps interstellar travel. Travel. Right? It's gonna be it's gonna be awesome. [00:32:01] S0: On the safety side, I think the narrative has changed a little bit. Kalshi actually has a market out on this of will models pause research for safety reasons? And this goes until 2027. And so right now, it's at about 9% chance, but it was trending upwards of, like, 40 not too long ago. It's really shifted and it's taken a nosedive in the last week or so. So why do you think that is? Why do you think the narrative on safety is changing?
00:32:35 · Jonathan Siddarth
[00:32:35] S1: People have realized that the best way to deploy these systems is with humans in the loop. And with humans in the loop, a lot of the safety issues go away. You have to design these systems for partial autonomy. People's mental models when they think about AI is self driving cars. I feel like the self driving car industry did a lot of damage to AI in one particular way. With self driving cars 99% accuracy is not enough. That last 1% really matters. But with a lot of AI systems, if you're automating the job of an investment analyst or if you're automating the job of a CEO or a CFO, you don't have to be 100% accurate as long as there is a human overseeing the results of the model. So I think we know how to build good safety harnesses around these systems with a human verifying the results of the model. So I think a lot of the risks, many of the risks were somewhat exaggerated. And we have good techniques to address safety in the post training phase. You can teach the models to decline to answer questions that could be dangerous in the CBRN side the space, a chemical, biological, radioactive nuclear. So we have some good techniques to contain the negative behaviors of these models. Safety is important but I think it was some of the risks were over exaggerated and they could be solved with really good engineering. I think what's the bigger risk is we don't deploy these systems quickly enough. I mean there was that MIT study that showed how ninety five percent of Gen AI pilots fail. I'm not surprised by that result at all. They fail because oftentimes firstly there is a big data gap. To solve real world enterprise workflows you need real world enterprise data and that data doesn't exist on the internet. To have that data, you have to hire people doing those workflows and train the models the right way, which is what we're doing. So there's an enterprise data gap. And there's also an enterprise expertise gap. There's a lot of first mile schlep and last mile schlep to make these systems actually work well. And there is work to do. And the bigger risk is an insurance company or a company in financial services doesn't adopt or make the best use of these models quickly enough. And if you don't do it quickly, their competitors will and whoever does it really well will win.