The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:23:23 · Harry Stebbings, Harrison Chase
[00:23:23] S1: And that would [00:23:23] S0: be data. That would be data. That would be memory. That would be specific APIs, specific tools that you can use. Controlling the harness is, like, a little bit harder, but still... You can still put a lot of your domain knowledge around what should do, both routing and things in there. And then the model is the hardest control, but potentially the highest ceiling. The thing that lets you iterate on each of those is evals and traces. So there is a ton of value in these, but there's only value in that you, like, capture insights that you bring back to the context or bring back to the harness or bring back to the model. And that exact process can look different. So post training, great for bringing these insights from traces back to the model. For bringing things back to the context or memory, you're probably gonna wanna run like another agent over those traces and then have that agent output kind of like learnings from memory. And then for bringing the... These learnings in the traces back back to the harness, you're probably gonna wanna run an agent over those traces and update the code of the harness itself. And we actually have something that does this for people. It's called engine. It's an agent in Langsmith that runs over traces and tries to update code. And then when you update these things, you're gonna need to know that it's actually better, and this is where Evalues come in handy. [00:24:31] S1: Can I ask, how does continual learning change agents and how they work?
00:24:31 · Harry Stebbings, Harrison Chase
[00:24:31] S1: Can I ask, how does continual learning change agents and how they work? [00:24:36] S0: Continual learning, I don't think so much changes how agents work, but I think it changes the systems that you wanna have running around the agent. So what I mean by that is is is I think the core loop for a continual learning agent, the core inner loop maybe to say, will will still largely be the same. It will be an LLM running in a loop, calling tools, interacting with the model in in an environment. I I think what continual learning adds is an outer loop, an outer loop that looks at the exhaust of agents and then and and then does things with it. And, again, what what it does could be different depending on what continual learning means for you. So I think continual learning is also one of those words that doesn't have a super precise definition or maybe [00:25:18] S1: How would you define it? [00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file.
00:25:19 · Harrison Chase, Harry Stebbings
[00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file. [00:25:44] S1: If we look at the the context layer, we mentioned data being a part of that. We're seeing data companies scale to hundreds of millions of revenue in the fastest time ever. I mean, like, ever, ever, ever, and everyone is selling data. How do you expect this market to play out given your neutrality on not being involved in it, but being adjacent to it? Will this be a market of consolidation in years to come? How do you see it play out?
00:26:09 · Harrison Chase
[00:26:09] S0: I actually think the the data market is still, like, pretty early in the grand scheme of things in so much as we're talking about kind of like a context layer for agents. So what I mean by that is, like, there are a bunch of databases that are... Clickhouse is maybe, like, a really good example of one that's doing incredibly well recently. I think they're still largely focused on kind of, like, their database. They're good old fashioned database. They're good at storing data and querying data, but it's still in pretty kind of traditional ways. Clickhouse existed pre pre AI. When we talk about kind of, like, a context layer for agents, and again, like there will be a bunch of data produced from, like, software systems and great. Like, I think, like, existing databases, Clickhouse, Mongo, Elastic, whatever, are great for that. When we talk about a context layer for agents, I think a key part of what exists there is, yes, there's a database, but I actually think there's a system around the database. I I think, like, a context layer for agents will look more similar to, like, an agent around a database than just a database. And the reason for this is that taking all of these learnings and ingesting them into a database, you need some sort of, like, intelligence to do that. And I think that intelligence is actually really, really hard. The hardest part about memory is actually not the database at all. When we talk to a number of companies, the way that they do memory is they just store it in a single file. Very, very kind of simple. The the hard part is taking all of the observations that happen, whether it's from the user interacting with, like, the... Let's say, it's a traditional SaaS interacting with the the software or the user interacting with the agent and distilling that into information that's valuable enough to go into this context layer. That's the hard part in memory right now.
00:27:35 · Harry Stebbings, Harrison Chase
[00:27:35] S1: Why is that so hard? [00:27:36] S0: Because it's tricky to get LLMs to do what you want them to do. And this type of, like, meta reflection, I think, is something that LLMs actually still struggle at a lot. Like, reflecting on kind of, like, prompts and improving prompts, reflecting on agent architecture, improving agent architecture. We should build a benchmark for this internally. We act... We do have some benchmarks for this in terms of engine, which is our agent that's running on things. But I think this type of meta reflection is still really difficult for agents. I think it's also generally, like, a hard thing for humans to do as well. Like, I don't know if we actually understand... Some people probably do, so I shouldn't say this. But I don't know if we understand, like, like, what like, what what goes on inside our brain? Like, I feel like our brains are still kind of, like, complex things that we don't fully understand. And so if we're trying to now replicate this kind of like learning process in a system, I think there's a bunch of different pieces. And so I think models aren't great at kind of like reflection, and I don't think we know what this system should look like. What should the inputs be? What should the outputs be? What should the harness for this agent be, what should the different workflows be. [00:28:30] S1: Will the models be meaningfully better at reflection in twelve months? [00:28:33] S0: Probably. I mean, seems like they'll get better at everything.
00:28:33 · Harrison Chase, Harry Stebbings
[00:28:33] S0: Probably. I mean, seems like they'll get better at everything. [00:28:35] S1: Do you think the speed of model improvement will continue in the same manner of acceleration that it has done, or do you think we are approaching some form of asymptotic plateauing? [00:28:45] S0: I think the speed of model improvement is something that's probably really tough to say without being inside of a lab. I would say I think there are a lot of tasks for which the model improvements will flatline in the sense that, like, it's already good enough. Like, I think there are a bunch of tasks where the models are just good enough. [00:28:59] S1: Which would be like coding? [00:29:00] S0: Yeah. Coding's getting there, like summarization, routing tasks. There are probably more difficult tasks or tasks that we don't even know to think about, know to throw a model at. I would probably say, like, this type of, like, meta learning is one of those tasks that the models aren't good at now. I would imagine is very important to the labs in terms of kind of like cracking RSI or or things like that. And so I do think the models will get better at that over time, and it's possible that that progress actually looks much faster than than previous. Think I it's really tough to have an educated opinion without being inside of a lab. There are some tasks where it's saturated. There are other tasks where I think it will probably just increase faster than before. [00:29:40] S1: What about agents today? Sounds crazy that you think will be incredibly common in two to three years.