The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:24:31 · Harry Stebbings, Harrison Chase
[00:24:31] S1: Can I ask, how does continual learning change agents and how they work? [00:24:36] S0: Continual learning, I don't think so much changes how agents work, but I think it changes the systems that you wanna have running around the agent. So what I mean by that is is is I think the core loop for a continual learning agent, the core inner loop maybe to say, will will still largely be the same. It will be an LLM running in a loop, calling tools, interacting with the model in in an environment. I I think what continual learning adds is an outer loop, an outer loop that looks at the exhaust of agents and then and and then does things with it. And, again, what what it does could be different depending on what continual learning means for you. So I think continual learning is also one of those words that doesn't have a super precise definition or maybe [00:25:18] S1: How would you define it? [00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file.
00:25:19 · Harrison Chase, Harry Stebbings
[00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file. [00:25:44] S1: If we look at the the context layer, we mentioned data being a part of that. We're seeing data companies scale to hundreds of millions of revenue in the fastest time ever. I mean, like, ever, ever, ever, and everyone is selling data. How do you expect this market to play out given your neutrality on not being involved in it, but being adjacent to it? Will this be a market of consolidation in years to come? How do you see it play out?
00:26:09 · Harrison Chase
[00:26:09] S0: I actually think the the data market is still, like, pretty early in the grand scheme of things in so much as we're talking about kind of like a context layer for agents. So what I mean by that is, like, there are a bunch of databases that are... Clickhouse is maybe, like, a really good example of one that's doing incredibly well recently. I think they're still largely focused on kind of, like, their database. They're good old fashioned database. They're good at storing data and querying data, but it's still in pretty kind of traditional ways. Clickhouse existed pre pre AI. When we talk about kind of, like, a context layer for agents, and again, like there will be a bunch of data produced from, like, software systems and great. Like, I think, like, existing databases, Clickhouse, Mongo, Elastic, whatever, are great for that. When we talk about a context layer for agents, I think a key part of what exists there is, yes, there's a database, but I actually think there's a system around the database. I I think, like, a context layer for agents will look more similar to, like, an agent around a database than just a database. And the reason for this is that taking all of these learnings and ingesting them into a database, you need some sort of, like, intelligence to do that. And I think that intelligence is actually really, really hard. The hardest part about memory is actually not the database at all. When we talk to a number of companies, the way that they do memory is they just store it in a single file. Very, very kind of simple. The the hard part is taking all of the observations that happen, whether it's from the user interacting with, like, the... Let's say, it's a traditional SaaS interacting with the the software or the user interacting with the agent and distilling that into information that's valuable enough to go into this context layer. That's the hard part in memory right now.
00:27:35 · Harry Stebbings, Harrison Chase
[00:27:35] S1: Why is that so hard? [00:27:36] S0: Because it's tricky to get LLMs to do what you want them to do. And this type of, like, meta reflection, I think, is something that LLMs actually still struggle at a lot. Like, reflecting on kind of, like, prompts and improving prompts, reflecting on agent architecture, improving agent architecture. We should build a benchmark for this internally. We act... We do have some benchmarks for this in terms of engine, which is our agent that's running on things. But I think this type of meta reflection is still really difficult for agents. I think it's also generally, like, a hard thing for humans to do as well. Like, I don't know if we actually understand... Some people probably do, so I shouldn't say this. But I don't know if we understand, like, like, what like, what what goes on inside our brain? Like, I feel like our brains are still kind of, like, complex things that we don't fully understand. And so if we're trying to now replicate this kind of like learning process in a system, I think there's a bunch of different pieces. And so I think models aren't great at kind of like reflection, and I don't think we know what this system should look like. What should the inputs be? What should the outputs be? What should the harness for this agent be, what should the different workflows be. [00:28:30] S1: Will the models be meaningfully better at reflection in twelve months? [00:28:33] S0: Probably. I mean, seems like they'll get better at everything.
00:28:33 · Harrison Chase, Harry Stebbings
[00:28:33] S0: Probably. I mean, seems like they'll get better at everything. [00:28:35] S1: Do you think the speed of model improvement will continue in the same manner of acceleration that it has done, or do you think we are approaching some form of asymptotic plateauing? [00:28:45] S0: I think the speed of model improvement is something that's probably really tough to say without being inside of a lab. I would say I think there are a lot of tasks for which the model improvements will flatline in the sense that, like, it's already good enough. Like, I think there are a bunch of tasks where the models are just good enough. [00:28:59] S1: Which would be like coding? [00:29:00] S0: Yeah. Coding's getting there, like summarization, routing tasks. There are probably more difficult tasks or tasks that we don't even know to think about, know to throw a model at. I would probably say, like, this type of, like, meta learning is one of those tasks that the models aren't good at now. I would imagine is very important to the labs in terms of kind of like cracking RSI or or things like that. And so I do think the models will get better at that over time, and it's possible that that progress actually looks much faster than than previous. Think I it's really tough to have an educated opinion without being inside of a lab. There are some tasks where it's saturated. There are other tasks where I think it will probably just increase faster than before. [00:29:40] S1: What about agents today? Sounds crazy that you think will be incredibly common in two to three years.
00:29:40 · Harry Stebbings, Harrison Chase
[00:29:40] S1: What about agents today? Sounds crazy that you think will be incredibly common in two to three years. [00:29:47] S0: I mean, I think this memory layer for agents is still massively unsolved. I think one thing about agents that is rare today but will be incredibly common in the next few years is is this memory layer. So what I mean by that is if you look at, like, Instinct or Muse, I think one of the things that they've nailed is this kind of, like, personalization and learning and seeming like it's something for you, whether it's because it's speaking in the tone or because it's just following up, like that proactiveness. And so I think memory is super, super early. We've been talking with folks about memory for like two years and trying to do things in the open source. We we launched an alpha and then... And it killed a commercial solution for memory. It's just so early on in memory, and it's so domain specific. What memory looks like for Instinct or Muse is actually pretty different than what it probably looks like for a company or org harness, and it's pretty different than what it looks like for legal or finance or things like that. So I think memory is one thing that really makes a difference. You see this make a different... I think Instinct and Muse have done a great job at this, and so you see how it makes a difference there. But I think it's still very, very early on, and I would expect all products in a few years to have some aspect of memory [00:30:46] S1: in With respect, is that not quite obvious?