00:21:18 · Ronak Malde
[00:21:18] S0: And the important part is, I think with humans, like, as kids, we are constantly exploring the world. We see a soccer ball, we see a football, and we kind of just try out things to build this mental model, and that makes us incredibly intelligent, whereas with agents that doesn't exist right now. And so we've done a lot of thinking in how do you actually quantify curiosity? And I think one interesting way to think about this is, so there's this idea in RL called a Markov decision process, And it's basically, can characterize the world as a bunch of state action pairs. So a state is like, I'm holding the ball in my hand, and then an action is I throw it, and then the following state is where does the ball end up? What LLMs right now are really good at is basically going straight from state zero to state one, their next token predictors, right? But what's missing is that intermediate part of what is that transition function? When you actually like go from or S0 to S1, what is that transition function that happens? And LLMs don't really have a way of encoding that right now. So I think the interesting thing here is with humans, we are very aware of when we don't know what a transition function is in the world, and we say, I'm basically going to try to figure out what this transition function is, and I think this is where curiosity comes in. And so if there's a way to basically quantify, here's all of the world that I don't know about right now. And my goal as an agent is to figure out if I do this task, what happens? I build this mental heuristic. Then I think we are starting to solve this curiosity problem and get to a world where you can autonomously kind of learn and continually learn from the world.
00:23:01 · Pat Grady, Ronak Malde
[00:23:01] S2: How do you make sure that an agent's curiosity is aligned with a user's objectives? [00:23:07] S0: So I think that's actually one of the reasons why I'm very passionate about working on continual learning. So there's two kind of directions that it seems like research is going right now. There's RSI, Recursive Self Improvement, and there's continual learning. And both of these are very tightly tied together, right, or very similar concepts. They're essentially just how does an agent actually learn from experience. The thing is RSI is all about taking a model, putting it in a black box, and basically making it the best AI researcher itself so that it can improve itself, right? But this is kind of a scary world, right? Because it exactly begs the question of, are these models and agents actually going to be aligned and have the best interests of humans in mind? Where I think continual learning takes a very different approach, is it says, I want to learn from experience, but learn from the experience of actually interacting with humans, and then getting better at the tasks that they set out and that they want to do. And I think that's where the key difference is, is you need to allow these models and build the algorithm so that it keeps humans in the loop. I don't want to end up in a world where these agents are going rogue, I think the way to do that is to keep on this alignment path.
00:23:07 · Ronak Malde, Lauren Reeder
[00:23:07] S0: So I think that's actually one of the reasons why I'm very passionate about working on continual learning. So there's two kind of directions that it seems like research is going right now. There's RSI, Recursive Self Improvement, and there's continual learning. And both of these are very tightly tied together, right, or very similar concepts. They're essentially just how does an agent actually learn from experience. The thing is RSI is all about taking a model, putting it in a black box, and basically making it the best AI researcher itself so that it can improve itself, right? But this is kind of a scary world, right? Because it exactly begs the question of, are these models and agents actually going to be aligned and have the best interests of humans in mind? Where I think continual learning takes a very different approach, is it says, I want to learn from experience, but learn from the experience of actually interacting with humans, and then getting better at the tasks that they set out and that they want to do. And I think that's where the key difference is, is you need to allow these models and build the algorithm so that it keeps humans in the loop. I don't want to end up in a world where these agents are going rogue, I think the way to do that is to keep on this alignment path. [00:24:19] S1: Guys have done a bunch of unique things across product and research at Trajectory to bring these two halves of the whole together. Tell us more about what makes building at Trajectory unique, and who are some of the people you bring together to make it happen?
00:24:31 · Ronak Malde
[00:24:31] S0: Yeah, so the cool part that we are sitting in is the intersection of frontier research and product, where we have to modify the core RL algorithms in order to get to a place where you can actually learn from the world, but also package that into a really beautiful and clean interface for people to train these models. And so some of the research team that we've built that had contributed to, like for example, the co founders to Gemini three and Gemini Robotics, other folks to GPT four point zero at OpenAI, and building Muse infrastructure at MSL. But then on the product side as well, having people that have built scalable infrastructure at Stripe for payment SDKs. Also, have the youngest design engineer at Netflix. These sort of people, and I think the intersection between them is where the magic starts to happen. So the way we've actually structured the team, normally you have like a research stack that's this spaghetti code base in one corner of the company, and then you have the product that's kind of well built out. What we've done is actually productionized the code base for training as well. So this means microservices, Kubernetes scaling, auto scaling of every single part of the trainer, the rollout engine. And then on the design side as well, our design engineers are kicking off training runs all of the time and understanding deeply what their research experience is. I think this is a really interesting and cool place to be in. A lot of this is also only possible because of Frontier coding agents. Actually, most of the time now, we are spending our time whiteboarding, and we'll spend a lot of time just arguing about system design. And then the actual implementation is just a software factory of agents that will go in and actually implement it.
00:26:08 · Lauren Reeder, Ronak Malde
[00:26:08] S1: So building a product here is not easy. What is the role of a product builder in a world of continual learning? [00:26:13] S0: I think as a product builder, you should be thinking about how you actually take these traces and experience in the real world and start to shape and improve the product autonomously. So we're starting with the model, right, for Trajectory, and I think there's some interesting work to do there. Eventually that will be the model and the harness and basically self improving both of those. And I think once you have the harness and products are kind of becoming more AI native, and so the harness is kind of becoming most of the product for a lot of these companies, right? Well, and if you have the right to kind of self improve the harness and then the product, then you have the right to kind of improve and self, kind of self improve the entire company. And I think there then products and product builders start to become almost like a gardener, like you have your plant and it's going to grow autonomously on its own, And your goal as a product builder is to basically just set these trellises in the ground of here's where I want to shape my product. And I think this is going to be a very exciting world where you can just put a product out to a couple 100 users, it's going to grow on its own. And now we're gonna have basically you just imbue your taste, into where the company grows. So I think that's gonna be a very exciting world and companies should kind of start building towards that.
00:27:26 · Pat Grady
[00:27:26] S2: Yeah. For somebody listening who's thinking, boy, Trajectory sounds like a cool place to work, what's the pitch? Like, why why might somebody want to work at Trajectory?