00:13:41 · Molly O'Shea
[00:13:41] S0: Because research has become such an important part of this next era of AI, it would be would be wrong not to bring up the fact that this is your background. So you were in research and in AI development before this. So how did that lead to founding Turing?
00:14:01 · Jonathan Siddarth
[00:14:01] S1: Yeah. Turing's founding DNA is AI research. My co founder and I met at Stanford. I worked at the Stanford AI Lab and the Stanford Info Lab. My co founder worked at the Stanford NLP Lab. And in an alternate universe, both of us would have gotten our PhD in computer science and AI. And at Stanford, there's this radioactive spider that tends to bite people and makes them start companies instead. And that's what happened to me and my co founder. And so we saw the power of AI early. And when Turing was started, it was all about using AI to find the world's smartest software engineers at scale, use AI to vet them, match them to companies, use AI to manage them. So when the AGI wave hit, I remember that fateful meeting with OpenAI when they were training GPT-three. And OpenAI wanted to teach GPT-three to code and to do function calling and tool use, which are building blocks for training agents. When we saw it, it was obvious what this was going to become. This was the future. And we were perfectly positioned. We had the world's largest platform of software engineers where we could build the ability to generate high quality data from. And it was clear that the models needed not just coding data, but data in every advanced domain in STEM, in healthcare, legal, finance, etc. And that really helped us the fact that our DNA was AI research. And now we are publishing papers on what type of data is most likely to help these models advance in coding, in chip design, in multi modality and coming up with really hard evals for these models.
00:15:52 · Molly O'Shea, Jonathan Siddarth
[00:15:52] S0: On the research side, how are you evaluating researchers and how are you recruiting them? [00:15:58] S1: On the research side, our focus is on two things: number one, ensuring that our data is the highest quality in advancing the models along the four pillars of superintelligence: multimodality, reasoning, tool use, and coding And number two, we do research on what type of data the models might likely need in the future. That's why we were on RL gyms very, very early. We built thousands of these reinforcement learning gyms to train agents. We were able to do that again because we have an internal R and D team that's constantly going to AI conferences, understanding where the frontier is, publishing papers, that's been really key. So the focus is on ensuring that the data that we generate is high quality. I think of that as horizon one. That's the here and now. And horizon two is what type of data will these models need three months in the future. That's horizon two. And today a lot of that work is on embodied AI and robotics and coming up with benchmarks that the models would struggle at today. Like we are creating data for coding now, for agentic coding, that would stump all of today's models and agents built on top of those models. It gives us a little bit of a sadistic pleasure in flunking all of the models today and a little bit of demonstrating where the human intelligence frontier is. But I would give that maybe six months before the models climb that hill, and then we'll generate data that's even harder for the models.
00:17:33 · Molly O'Shea
[00:17:33] S0: Could you just explain, like, maybe step by step for people that don't understand how models are created, the partners in which they work with and then also the teams they have internally that make all these processes run? Because I don't think many people know, like, what what does the inside of OpenAI look like? What does it take to get to GPT-five? What is that process?
00:17:58 · Jonathan Siddarth
[00:17:58] S1: Yeah. Let me talk the general recipe of training a frontier model. And a frontier model is a trillion parameter model, like let's say a GPT-four class model, like the general recipe. This is roughly what all the labs follow. So there are two main steps. There's pretraining and posttraining. Pretraining is a form of unsupervised learning, meaning humans are not involved. It's like lots and lots of compute. In pretraining you build what's called a base model. And all the base model knows is how to auto complete stuff. Pretraining is done completely inside the Frontier Labs. And during pretraining, you feed the models a subset of the Internet. It's a filtered subset of the Internet. There are these data sets like Common Crawl, C4, GitHub, Archive. You also feed a lot of books. You might feed videos. You feed all these, let's call it humanity's knowledge corpus. You feed it to these models. And the model is basically doing the task of language modeling, which is learning how to predict the next token. Right? This is a model that just knows how to auto complete stuff. An example of a pre trained model is something like GPT-three. We haven't yet built ChatGPT, this is just GPT-three, right, as the result of pre training. To this model, if you say something like Molly runs Sourcery, it'll then respond by saying Molly is interviewing Jonathan for the next episode. That's how the model will respond. It's just autocompleting. It's not an assistant yet. But the magical thing about pretraining is in the quest to autocomplete stuff, the model has learned a lot of internal representations about the world. It's like raw brain mass. It's like an artificial brain that hasn't been molded yet. But the intelligence is there. Next step is post training. And OpenAI came up with this paper called Instruct GPT, which lays out this process in more detail. I encourage your readers to listen to it, to read the paper. In post training, this is when GPT-three becomes ChatGPT. You teach this base model to become a helpful, honest, harmless assistant. And post training has two main steps: supervised fine tuning and reinforcement learning. In supervised fine tuning, you hire lots and lots of human experts to create prompts and completions that teach the model how to answer questions. The task is still language modeling. You're still predicting the next token.
00:20:42 · Jonathan Siddarth
But you feed it question answer pairs that could either be single turn, meaning question and answer, or multi turn, where you're having a dialogue with the model. At this step, if you ask the model let's say a human contractor has the question, Molly runs Sourcery, the next completion for it might be, what would you like to know about Molly might be an appropriate response that you teach the model to do. You teach the model how respond. In the case of coding, a contractor might create a question like, hey, give me some Python code to train a text classifier to detect x posts into positive sentiment, neutral sentiment, negative sentiment. And then the contractor actually writes code. And let's imagine the code had some type of exception or some type of bug. The contractor would give the model a demonstration of what the right answer is. It's humans showing the model question answer pairs so the model learns. Again, this is still language modeling. The model is still learning how to predict, but now you've aligned the model. If you ask the model a dangerous question and there are these areas in the community it's called CBRN, Chemical, Biological, Radioactive, Nuclear, there are categories of things that for safety you might want to train the model to decline to answer. That gets done in this step two, this supervised fine tuning step. The step after that is reinforcement learning with human feedback, where when the model has gotten slightly better at responses, you'd have the model output two responses, two completions, and you'd have human contractors pick what is a completion that a human tends to prefer. For example, if the question is, tell me more about Molly O'Shea and Sourcery, and there's, one set of responses that is maybe super verbose and is not easy to consume, whereas there's another response that's in crisp bullet points and maybe that links to the conversation we are having. Maybe a human contractor maybe picks that, oh that response is better. This RLHF STEP is used to train a reward model. And the reward model basically scores responses based on what's a response that a human tends to prefer. And this is used to update the model that has been trained so far to output completions that humans tend to prefer.