00:00:00 · Ronak Malde, Lauren Reeder
[00:00:00] S0: A lot of our philosophy in product actually comes from Apple. In, like, the early two thousands, you had a computer that had access to the Internet, and you had a phone that you could make phone calls. And I think no one really thought to put the two together until Apple said, okay. What if you actually put the Internet on a handheld device and combine the two? And I think by doing this of of basically putting training, eval, and deployment in one place, making that just dead simple for every single person to do, And then you can close the loop and basically now upload that model back to production, gather new traces. It suddenly is this magical paradigm that I think every single company is going to adopt very soon. [00:00:53] S1: Ronak, thank you for joining us. It's a pleasure to have you. We would love to dig into a bunch of what you're building, how you see the world of research, and where you think the future is going. Maybe to start, you guys spend a lot of time thinking about continual learning. Can you define it for us? I think you've also said this field is in a state of pseudo continual learning. What is real, and how will we know when we get to the real thing?
00:01:14 · Ronak Malde
[00:01:14] S0: That's a great question. So when I think about continual learning, I actually think the human brain is a really good model for where we should aspire to get to. Like, in some sense, we are true continual learners. We learn online, we are constantly updating, iterating, and I think the world we're in right now with agents and AI is kind of, like you said, pseudo continual learning. When I was growing up, my family used to do a bunch of road trips, right? And I don't know how you guys did things, but we would go to AAA back in the day, and like, they would give us a bunch of these maps, they would like highlight everything, and then we'd go on a road trip, and like, if there's any point where we wanted to change our route, we'd have to either call AAA or like go to a physical location, and I feel like that's the world we're in right now, where we have these amazing agents, they're very intelligent, we have these mapping algorithms, but you can only consult offline a couple of places, and anytime you wanna update your course, have to just do an entire mass batch update. What the first goal, and what we're building at Trajectory, is first the ability to just democratize that access, to even just training these models and customizing them for what people want. I think of that as kind of like MapQuest, where suddenly it's still offline, but we at least got the ability for everyone to not need to go to AAA, but get the ability to map these algorithms. But the final stage of continual learning is something more like Google Maps, where every single time you're driving, every route that you make, it is updating. It's learning aggregate what the fastest route is and where the traffic is. That's the world I wanna get to.
00:02:47 · Pat Grady
[00:02:47] S2: That's a really good analogy. So, we had... Recently, we had Rich Sutton and David Silver on the show, and both of them kinda made the comment that continual learning isn't really possible with today's architectures. I'm curious in your point of view, what's doable, what's not doable, what's the state of play, do we need major architectural leaps to kind of bring this to fruition, where are we today?
00:03:11 · Ronak Malde
[00:03:11] S0: Yeah, so it's funny because when you think about a transformer in LLMs, I feel like a lot of people have seen maybe Karpathy's TinyGPT video or something where you basically have this stack of attention blocks and transformers that are built on top of each other. And a lot of LLM scaling has just been stacking these transformers on top of each other. But when you look at like any paper, like DeepSeek v4.1 flash that just came out recently, their architecture is actually kind of lobotomized or changed a lot of that base architecture, where you have things like fast and slow memory within the weights. You have skip connections that are going between layer two and layer 50. And so I think these architectural changes are actually already happening within the base models, and it will be a key component, I do agree, for continual learning. Where I want to sit is basically leverage all of that technology and research that's happening at the architectural level, but then actually connect the dots to the real world, because at the end of the day, the real value of any AI progress, but certainly continual learning, is how are people actually using models in the real world? And I think companies have this idea of basically what good looks like in their mind, and the hardest part of this entire loop is then translating that to actual improvements for the model. I think that's where that missing layer is.
00:04:33 · Pat Grady, Ronak Malde
[00:04:33] S2: Yeah. So how do you connect the dots to the real world and how much of that is product or productizable and how much of that is just handholding and helping people through the process? [00:04:44] S0: Yeah, so up until now, it's been really hard to just go from this fact of like data to intelligence. Up until now, that's only been in the labs and you need a massive research team. When, so in the last couple of years, we started to have vertical companies going into this space as well and building a massive research team. So I was previously at Windsurf where we embarked on this journey and trained a model, so we won. So did Cursor as well with Cursor Composer, but the problem was you needed hundreds of millions of dollars of R and D in compute. You needed a really massive research team to invest in, and then you also needed a lot of time. So it took us six to nine months to actually get to that point where we could train a frontier model, and that is with a kind of handholding sort of consulting model. That's still sort of the world that we live in. I think any good product exists when you have every single company in the world trying to do the same thing over and over and over again, and if you can package that into a really beautiful interface that just says, Here are the knobs and levers that you care about to translate what good looks like, and then everything else is packaged away, I think that's a beautiful interface to do that.
00:04:44 · Ronak Malde, Lauren Reeder
[00:04:44] S0: Yeah, so up until now, it's been really hard to just go from this fact of like data to intelligence. Up until now, that's only been in the labs and you need a massive research team. When, so in the last couple of years, we started to have vertical companies going into this space as well and building a massive research team. So I was previously at Windsurf where we embarked on this journey and trained a model, so we won. So did Cursor as well with Cursor Composer, but the problem was you needed hundreds of millions of dollars of R and D in compute. You needed a really massive research team to invest in, and then you also needed a lot of time. So it took us six to nine months to actually get to that point where we could train a frontier model, and that is with a kind of handholding sort of consulting model. That's still sort of the world that we live in. I think any good product exists when you have every single company in the world trying to do the same thing over and over and over again, and if you can package that into a really beautiful interface that just says, Here are the knobs and levers that you care about to translate what good looks like, and then everything else is packaged away, I think that's a beautiful interface to do that. [00:05:56] S1: What did you and your co founders see when you were at Windsurf and when they were working on models as well to help you realize, hey, this is repeatable, that there is a project build. When did you have that moment and that realization?