00:11:15 · Pat Grady, Ronak Malde
[00:11:15] S2: And just on that, what are people doing that in today? Like, where do those traces live? [00:11:20] S0: Yeah. In my view, I think the entire ecosystem of continual learning needs to be a lot of players that basically build up all this infrastructure. So some companies are doing this in house, but there's also a lot of great products out there like LangChain, like Braintrust, Raindrop, where you can basically plug into your infrastructure or your production agents, and then basically trace all of these things. So what we're finding is a lot of AI native companies will use a lot of these products in order to get that first step out of the way. [00:11:49] S2: Okay, so we got traces and then Yeah,
00:11:49 · Pat Grady, Ronak Malde
[00:11:49] S2: Okay, so we got traces and then Yeah, [00:11:51] S0: so that's traces. The second part is then making these replayable. You need some sort of infrastructure to say, Hey, okay, given a trace that I've seen in production, how can I actually then replay that for any eval to see what good looks like, or right now for training as well? And so that's where companies also need to invest some sort of time. What we make really easy in our product as well is to bridge that gap. And so if you bring your traces and you say, okay, here's just the concept of an environment that I might want, in more technical terms, it's a Docker file or some sort of system requirements, then you can basically replay that entire trace. So that's the second part. And then the third part is then the actual modification or the algorithm. And I think that's where it's really difficult to do this in house as a company, is basically the entire training stack, the entire loss function, the reward function, of all these sort of components that need to be built up to translate these tasks into actual improvements for the model. And so that's the main part that we want to abstract away, off the bat. [00:12:56] S2: To your point on abstracting away some of that, what has been abstracted thus far? What's next? You know, what's kind of further out? Like, how does this product come together over time?
00:13:08 · Ronak Malde
[00:13:08] S0: Right now, it's mainly you bring traces and reward and some notion of environment. The two parts here that require a bunch of manual work is the environment, actually constructing this thing that matches the real production. And a lot of the time it's actually impossible to create a perfect environment, right? And then the second part is reward. A lot of the times a legal workflow or investment banking workflow doesn't have some perfect verifiable reward that you can train off of, right? And so these are the two areas that we're really pushing on the research side. So for the environment, there's an algorithm called GRPO. That's the main kind of post training algorithm that exists right now. What you need to do there is you basically, like, imagine you have a player that's in a video game and it's basically going around and hopping around a bunch of platforms, and then at the end of the day, you get some reward. What's needed here in GRPO is basically comparing like the same task, the same level of the video game, and you need to play that out eight times or 16 times, and we call that a group. Now the thing is, with humans, a lot of the time we don't need to replay the exact same environment over and over again. We're just in the world experiencing and doing one rollout. And so the goal is, could we modify GRPO as an algorithm to get to a world where it is what we call group size one instead of needing a group of 16? And so we've made a lot of strides here where now if you can take 10,000 trajectories and basically they're all from different tasks, we can actually learn from just that one behavior or that one rollout. And so that's one algorithm that we actually released a blog post about, is going into this group size one paradigm. And so that actually eliminates the need for an environment and all of this infrastructure that needs to be built out. The second part is for reward. And so this is another factor where, let's say I have, again, 10,000 traces. The default is that I need to go and label every single one of those traces to say, here's verifiable reward, or here's what good looks like. And this is tractable, it's just a lot of work. And so now our goal is how can we actually reduce that amount of time needed to take this notion of what good looks like and do that in much less amount of work.
00:15:21 · Ronak Malde, Pat Grady, Lauren Reeder
So what we've realized is, let's say I have 10,000 traces. If I can smartly select a 100 of them that are basically representative of all of these broader 10,000, and then have the customer or the user basically densely say, okay, here's what good looks like, here's a rubric or verifiable criteria for just those a 100 traces. And then what we can do is basically train a meta grader that learns from those behaviors of the 100 to the broader 10,000. And with that, you get a really clean interface that allows you to just propagate this reward across the board. So I think obviously it's not the perfect solution in that direction. The ideal goal actually is that you have some sort of observability interface where you're like, here's the five things my agent's good at, here's the five things it's bad at, and I just circle those five things and hit train. That's the world I want to get to, and that's where we're pushing that research direction in. [00:16:16] S2: Very cool. [00:16:16] S1: So you have a product that you just released to do some of this. Tell us about it. What have you, what are you really seeing in the product and who is it for?
00:16:16 · Lauren Reeder, Ronak Malde
[00:16:16] S1: So you have a product that you just released to do some of this. Tell us about it. What have you, what are you really seeing in the product and who is it for? [00:16:22] S0: Yeah, up until now, it's taken a massive research team, a ton of time and a ton of compute in order to train these models. What we wanted to do is just make the easiest and most intuitive interface to train models and customize them for what you care about. And so Trajectory, the platform that we released, is basically a very simple system where you can take your agent traces and what good looks like. It's a one click train interface that just takes a couple agent prompts, and then you can eval the model and then deploy it all in one place. And I think the really magical part about this, a lot of our philosophy in product actually comes from Apple, and when, in like the early 2000s, you had a computer that had access to the internet, and you had a phone that you could make phone calls, and I think no one really thought to put the two together until Apple said, okay, what if you actually put the internet on a handheld device and combine the two? And I think by doing this, of basically putting training, eval, and deployment in one place, making that just dead simple for every single person to do, and then you can close the loop and basically now upload that model back to production, gather new traces, it suddenly is this magical paradigm that I think every single company is going to adopt very soon.
00:16:22 · Ronak Malde, Pat Grady
[00:16:22] S0: Yeah, up until now, it's taken a massive research team, a ton of time and a ton of compute in order to train these models. What we wanted to do is just make the easiest and most intuitive interface to train models and customize them for what you care about. And so Trajectory, the platform that we released, is basically a very simple system where you can take your agent traces and what good looks like. It's a one click train interface that just takes a couple agent prompts, and then you can eval the model and then deploy it all in one place. And I think the really magical part about this, a lot of our philosophy in product actually comes from Apple, and when, in like the early 2000s, you had a computer that had access to the internet, and you had a phone that you could make phone calls, and I think no one really thought to put the two together until Apple said, okay, what if you actually put the internet on a handheld device and combine the two? And I think by doing this, of basically putting training, eval, and deployment in one place, making that just dead simple for every single person to do, and then you can close the loop and basically now upload that model back to production, gather new traces, it suddenly is this magical paradigm that I think every single company is going to adopt very soon. [00:17:37] S2: Very cool. Who is the product targeted at? Like, who is the ideal customer for you guys today?