00:17:58 · Jonathan Siddarth
[00:17:58] S1: Yeah. Let me talk the general recipe of training a frontier model. And a frontier model is a trillion parameter model, like let's say a GPT-four class model, like the general recipe. This is roughly what all the labs follow. So there are two main steps. There's pretraining and posttraining. Pretraining is a form of unsupervised learning, meaning humans are not involved. It's like lots and lots of compute. In pretraining you build what's called a base model. And all the base model knows is how to auto complete stuff. Pretraining is done completely inside the Frontier Labs. And during pretraining, you feed the models a subset of the Internet. It's a filtered subset of the Internet. There are these data sets like Common Crawl, C4, GitHub, Archive. You also feed a lot of books. You might feed videos. You feed all these, let's call it humanity's knowledge corpus. You feed it to these models. And the model is basically doing the task of language modeling, which is learning how to predict the next token. Right? This is a model that just knows how to auto complete stuff. An example of a pre trained model is something like GPT-three. We haven't yet built ChatGPT, this is just GPT-three, right, as the result of pre training. To this model, if you say something like Molly runs Sourcery, it'll then respond by saying Molly is interviewing Jonathan for the next episode. That's how the model will respond. It's just autocompleting. It's not an assistant yet. But the magical thing about pretraining is in the quest to autocomplete stuff, the model has learned a lot of internal representations about the world. It's like raw brain mass. It's like an artificial brain that hasn't been molded yet. But the intelligence is there. Next step is post training. And OpenAI came up with this paper called Instruct GPT, which lays out this process in more detail. I encourage your readers to listen to it, to read the paper. In post training, this is when GPT-three becomes ChatGPT. You teach this base model to become a helpful, honest, harmless assistant. And post training has two main steps: supervised fine tuning and reinforcement learning. In supervised fine tuning, you hire lots and lots of human experts to create prompts and completions that teach the model how to answer questions. The task is still language modeling. You're still predicting the next token.
00:20:42 · Jonathan Siddarth
But you feed it question answer pairs that could either be single turn, meaning question and answer, or multi turn, where you're having a dialogue with the model. At this step, if you ask the model let's say a human contractor has the question, Molly runs Sourcery, the next completion for it might be, what would you like to know about Molly might be an appropriate response that you teach the model to do. You teach the model how respond. In the case of coding, a contractor might create a question like, hey, give me some Python code to train a text classifier to detect x posts into positive sentiment, neutral sentiment, negative sentiment. And then the contractor actually writes code. And let's imagine the code had some type of exception or some type of bug. The contractor would give the model a demonstration of what the right answer is. It's humans showing the model question answer pairs so the model learns. Again, this is still language modeling. The model is still learning how to predict, but now you've aligned the model. If you ask the model a dangerous question and there are these areas in the community it's called CBRN, Chemical, Biological, Radioactive, Nuclear, there are categories of things that for safety you might want to train the model to decline to answer. That gets done in this step two, this supervised fine tuning step. The step after that is reinforcement learning with human feedback, where when the model has gotten slightly better at responses, you'd have the model output two responses, two completions, and you'd have human contractors pick what is a completion that a human tends to prefer. For example, if the question is, tell me more about Molly O'Shea and Sourcery, and there's, one set of responses that is maybe super verbose and is not easy to consume, whereas there's another response that's in crisp bullet points and maybe that links to the conversation we are having. Maybe a human contractor maybe picks that, oh that response is better. This RLHF STEP is used to train a reward model. And the reward model basically scores responses based on what's a response that a human tends to prefer. And this is used to update the model that has been trained so far to output completions that humans tend to prefer.
00:23:19 · Jonathan Siddarth
For example, in coding, humans may tend to prefer code that's commented really well, that's structured in a way that is easy to maintain, and is more modular. But those preferences are learned in RLHF. And once you've done RLHF, would then do another round of evals, where you're evaluating the model's performance to see if the model improved. As the model improves, you'll figure out where the model breaks next and then generate data to improve the model there again. It's a loop between you first do pretraining. Pretraining is done relatively infrequently. And then post training is a highly iterative process. You start first with SFT, then you would do reinforcement learning with human feedback, you would evaluate the model again, figure out where it's breaking, where it's doing well, and then generate data in the areas where the model is weakened. Now what I just described is how things used to largely work until last year. Now for these verifiable domains like coding and math, instead of doing reinforcement learning with human feedback, you can do reinforcement learning because you can automatically check when you got the correct answer or not in these verifiable domains. And with reinforcement learning, the way you teach the model is not through imitation learning, where you're learning from humans, but it's through experiential learning where you create this environment with prompts and verifiers and you create basically a mini world model where as long as the agent gets the right result, that particular trajectory that the agent took to come to the right answer gets reinforced. And when the agent got to a result that didn't get the right answer, that doesn't get reinforced. So the model kind of learns from its own experience. It's a form of self play which I think is really, really cool. If you remember, from the AlphaZero days, when DeepMind was teaching their Go Playing algorithm, the first version of Google's AlphaGo system learned from expert humans, which was quite good. The next generation, which was even better, just played against itself and figured out which paths tended to it winning and which paths tended to it losing and learned from that.
00:25:51 · Jonathan Siddarth, Molly O'Shea
And there is this Move 37 in one of the games where the model came up with a move that felt alien to the best Go players in the world because the model had figured out a trajectory to win that beat the best human experts. And I'm super excited for AI in the enterprise when an agent takes a path that is so novel and so interesting and so different that really exceeds superintelligence. And for me that's a big part of artificial superintelligence. Why should human intelligence be the bound for what we do? These models could exceed us in many interesting ways. [00:26:30] S2: In today's high speed business world, staying ahead means using the smartest tools possible, including the powerful capabilities of artificial intelligence. Meet Turing Intelligence. Turing builds customizable AI systems designed to solve your mission critical challenges no matter your industry. From expert guidance to tailored projects, Turing helps top companies realize AI that's more capable, more adaptable, and more effective. With Turing, discover how AI can accelerate your business growth. To learn more, visit turing.com/sorcery. Spelt sourcery. That's turing.com/sorcery.
00:27:06 · Molly O'Shea
[00:27:06] S0: I think that's a really interesting point and it's also a point that scares people a lot. It's created a culture of doomers because they're afraid of AI and technology rapidly advancing so fast or so quickly out of our control that it puts safety at a risk. I'm gonna put that to the side, like, quite... Like, a little bit, but it brings it brings into light this tweet that David Sacks had not too long ago, like, right after the GPT-five release. And there's a lot of pushback on this release. It didn't go as expected. Many, were actually, like, quite disappointed. They were expecting, like, something crazy to happen, and turns out it was just a little bit of an incremental, seemingly incremental. I don't wanna offend anyone. Seemingly incremental improvement to the model. But I'm curious from your standpoint, what did you make of that moment in that tweet that David Sacks had put out that was, you know, it was addressing the doomers of, okay, actually, like AI is not moving that fast. We actually might have more time before we have real crazy moves happening.
00:28:23 · Jonathan Siddarth
[00:28:23] S1: Yeah, let me unpack this. I think there are two parts to your question. There's one part which is, are we slowing down? Like, are things going to be more incremental going on? That's one part to your question. And then there's another part which is, how should we think about safety in these AI systems. So let me take that separately. First, I think GPT-five is fucking awesome. I think it's awesome. I think we've just gotten used to magic that maybe we are not fully seeing the impact of what's just out there. And I'm not just saying that because we work with the company that built it. I think GPT-five is awesome. I think one thing is real though, which is the AI doomers talk about this concept of rapid takeoff. It's sort of the I mean, you've seen The Matrix or the Terminator movies, like somehow we'll hit a singularity and there's just this rapid takeoff where humanity gets left behind. I don't think rapid takeoff is how things will unfold. I think it's going to be steady continuous progress. Every step of the way we're going to keep moving forward and it's going to be great. And it's great for a few reasons. The reason I think rapid takeoff will not happen and I know why the doomers believed in rapid takeoff is because GPT-three felt like magic. It was like fucking magic. It was magic because there were a lot of one time jumps in there. Jump number one on the data front. We ate the internet. We ate the internet. There is only one internet and we ate it in GPT-three, right? And GPT-four, again, in all of these and all the models like GPT-four, like Gemini 2.5 Pro, Grok2, Grok3, Claude, it's the same formula. We all ate the internet. And that was a huge one time jump. And we put in all of humanity's knowledge. All the books that you owned the licenses of were fed into the models. So humanity's collective intelligence was eaten in the GPT-three, GPT-four era. And we scaled up compute. Some of the early gains in compute were we also got that, right? And computers takes time to build. You've got to solve energy and you've got to build data centers. And you see what Elon is doing with projects, you see what Sam Altman and OpenAI are doing, you see what Meta is doing. Compute also takes time to build, right? So we got a huge jump by eating the Internet and using all the easily accessible compute.