00:07:39 · Ronak Malde, Lauren Reeder
[00:07:39] S0: So, basically, they were up until now riding on top of the Frontier closed source models, models, right? Right? And they've been able to build incredible interfaces and products on top of that, that help tons of law firms in the world. But what they realized is they have an immense amount of knowledge of data, of all of this nuance built up that they are actually able to see what the law firms actually care about, and the built up experience of just thirty years of working in this domain, and with using frontier models, they're unable to actually take that data and use that to their advantage. And so what they realized is, hey, okay, they're able to construct, for example, a legal agent benchmark and what they released, and using all of the expert information that they have, and then if they get the power to basically use that data to train a model, then they get to shape their own destiny. So with Trajectory, what they were able to do is train a model that is six times cheaper, and it goes from around 4% accuracy for an open source model to around 18%. And this is, there's still so much room to grow, obviously, what they're able to do, but now suddenly they get the power of a research lab. And on top of that, the nice part is these models end up being six times cheaper than the Frontier, and so you're able to expend so many more tokens to achieve these tasks. [00:08:55] S1: So you picked the name Trajectory, as it's related to this Continual Learning concept. How did you pick the name and tell us how it fits into what you guys are doing?
00:08:55 · Lauren Reeder, Ronak Malde
[00:08:55] S1: So you picked the name Trajectory, as it's related to this Continual Learning concept. How did you pick the name and tell us how it fits into what you guys are doing? [00:09:02] S0: Yeah, so it's funny because the world right now is very focused on tokens, and I think a lot of people view tokens as a unit of work, but if you think about the way that we interact with other humans, I'm not counting the amount of words that you're actually talking about, right? And with coworkers, we, a lot of the time, are thinking about a task or a trajectory, and that's the unit of work that I really want to get to it. In RL, it basically is synonymous with a rollout or what you would actually do in a training stack, and with continual learning, you have an RL training stack that's like a rollout engine and a trainer. A rollout engine is just like in a simulation, basically like doing a bunch of rollouts and doing these trajectories. In the real world for continual learning, your rollout engine becomes just the inference and the tokens in the real world, and now suddenly your trajectories are the tasks of economic value that are happening. And so I want this basically paradigm to propagate, where the unit of work that we think about is a trajectory. Beyond that, I think for also people that don't understand RL, it is also super nice that we are basically letting companies just own that path and really grow on their own. I think it has this nice double entendre.
00:10:13 · Pat Grady, Ronak Malde
[00:10:13] S2: And, Kashy, I like this idea of a trajectory as like a unit of economically valuable work. You've also a couple times talked about what does good look like, which kind of sounds like an eval or a benchmark. What are the core primitives for continual learning, or what are the key ingredients that kind of go into this? [00:10:30] S0: Yeah. So our goal at Trajectory is to actually limit these ingredients so that it is literally just, you know what good looks like. However, right now there are a lot of components that companies need, and then we, as our product as well, try to bridge that gap and make that as easy as possible. So right now, as a company, you need a couple of things. One, you need to be able to trace all of your trajectories that are happening in production, And there's lots of AI native companies that have gone through this transformation where they're logging not just the main agent call, but also all the sub agents, what's happening in the product as well, any feedback that the user is giving. The ideal world is you get this kind of perfect timeline of all of those things so that you can track them. So that's the first step. [00:11:15] S2: And just on that, what are people doing that in today? Like, where do those traces live?
00:11:15 · Pat Grady, Ronak Malde
[00:11:15] S2: And just on that, what are people doing that in today? Like, where do those traces live? [00:11:20] S0: Yeah. In my view, I think the entire ecosystem of continual learning needs to be a lot of players that basically build up all this infrastructure. So some companies are doing this in house, but there's also a lot of great products out there like LangChain, like Braintrust, Raindrop, where you can basically plug into your infrastructure or your production agents, and then basically trace all of these things. So what we're finding is a lot of AI native companies will use a lot of these products in order to get that first step out of the way. [00:11:49] S2: Okay, so we got traces and then Yeah,
00:11:49 · Pat Grady, Ronak Malde
[00:11:49] S2: Okay, so we got traces and then Yeah, [00:11:51] S0: so that's traces. The second part is then making these replayable. You need some sort of infrastructure to say, Hey, okay, given a trace that I've seen in production, how can I actually then replay that for any eval to see what good looks like, or right now for training as well? And so that's where companies also need to invest some sort of time. What we make really easy in our product as well is to bridge that gap. And so if you bring your traces and you say, okay, here's just the concept of an environment that I might want, in more technical terms, it's a Docker file or some sort of system requirements, then you can basically replay that entire trace. So that's the second part. And then the third part is then the actual modification or the algorithm. And I think that's where it's really difficult to do this in house as a company, is basically the entire training stack, the entire loss function, the reward function, of all these sort of components that need to be built up to translate these tasks into actual improvements for the model. And so that's the main part that we want to abstract away, off the bat. [00:12:56] S2: To your point on abstracting away some of that, what has been abstracted thus far? What's next? You know, what's kind of further out? Like, how does this product come together over time?
00:13:08 · Ronak Malde
[00:13:08] S0: Right now, it's mainly you bring traces and reward and some notion of environment. The two parts here that require a bunch of manual work is the environment, actually constructing this thing that matches the real production. And a lot of the time it's actually impossible to create a perfect environment, right? And then the second part is reward. A lot of the times a legal workflow or investment banking workflow doesn't have some perfect verifiable reward that you can train off of, right? And so these are the two areas that we're really pushing on the research side. So for the environment, there's an algorithm called GRPO. That's the main kind of post training algorithm that exists right now. What you need to do there is you basically, like, imagine you have a player that's in a video game and it's basically going around and hopping around a bunch of platforms, and then at the end of the day, you get some reward. What's needed here in GRPO is basically comparing like the same task, the same level of the video game, and you need to play that out eight times or 16 times, and we call that a group. Now the thing is, with humans, a lot of the time we don't need to replay the exact same environment over and over again. We're just in the world experiencing and doing one rollout. And so the goal is, could we modify GRPO as an algorithm to get to a world where it is what we call group size one instead of needing a group of 16? And so we've made a lot of strides here where now if you can take 10,000 trajectories and basically they're all from different tasks, we can actually learn from just that one behavior or that one rollout. And so that's one algorithm that we actually released a blog post about, is going into this group size one paradigm. And so that actually eliminates the need for an environment and all of this infrastructure that needs to be built out. The second part is for reward. And so this is another factor where, let's say I have, again, 10,000 traces. The default is that I need to go and label every single one of those traces to say, here's verifiable reward, or here's what good looks like. And this is tractable, it's just a lot of work. And so now our goal is how can we actually reduce that amount of time needed to take this notion of what good looks like and do that in much less amount of work.