The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:22:46 · Harry Stebbings, Harrison Chase
[00:22:46] S1: Can I ask you? You mentioned traces there. If, like, traces are the, like, exhaustive agent work, where is the value then? Is it the app, the observability layer, the model layer? Where is it? [00:22:58] S0: If you're building an agent, I think the value is at what makes that agent good at the domain or vertical that you're you're working in. So what I what I mean by that is, again, I think there's different ways to control what an agent does. There's the context, there's the harness, and there's the model. Having specific proprietary things for for any of those can be an advantage. I think controlling the the context is the easiest and and the most obvious thing to do when starting [00:23:23] S1: And that would
00:23:23 · Harry Stebbings, Harrison Chase
[00:23:23] S1: And that would [00:23:23] S0: be data. That would be data. That would be memory. That would be specific APIs, specific tools that you can use. Controlling the harness is, like, a little bit harder, but still... You can still put a lot of your domain knowledge around what should do, both routing and things in there. And then the model is the hardest control, but potentially the highest ceiling. The thing that lets you iterate on each of those is evals and traces. So there is a ton of value in these, but there's only value in that you, like, capture insights that you bring back to the context or bring back to the harness or bring back to the model. And that exact process can look different. So post training, great for bringing these insights from traces back to the model. For bringing things back to the context or memory, you're probably gonna wanna run like another agent over those traces and then have that agent output kind of like learnings from memory. And then for bringing the... These learnings in the traces back back to the harness, you're probably gonna wanna run an agent over those traces and update the code of the harness itself. And we actually have something that does this for people. It's called engine. It's an agent in Langsmith that runs over traces and tries to update code. And then when you update these things, you're gonna need to know that it's actually better, and this is where Evalues come in handy. [00:24:31] S1: Can I ask, how does continual learning change agents and how they work?
00:24:31 · Harry Stebbings, Harrison Chase
[00:24:31] S1: Can I ask, how does continual learning change agents and how they work? [00:24:36] S0: Continual learning, I don't think so much changes how agents work, but I think it changes the systems that you wanna have running around the agent. So what I mean by that is is is I think the core loop for a continual learning agent, the core inner loop maybe to say, will will still largely be the same. It will be an LLM running in a loop, calling tools, interacting with the model in in an environment. I I think what continual learning adds is an outer loop, an outer loop that looks at the exhaust of agents and then and and then does things with it. And, again, what what it does could be different depending on what continual learning means for you. So I think continual learning is also one of those words that doesn't have a super precise definition or maybe [00:25:18] S1: How would you define it? [00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file.
00:25:19 · Harrison Chase, Harry Stebbings
[00:25:19] S0: I I would define it as a system that gets better over time as it runs. And again, there's different ways it could get better over time. Could update the context. It could update the code of the harness, or it could update the weights of the model. And the outer loop will look roughly the same for all of those. It will look like take traces, get learnings, fix something, measure it to make sure it improved, and then repeat. But that exact kind of like changing, again, updating weights of model, very different from updating a text file that is like a memory file. [00:25:44] S1: If we look at the the context layer, we mentioned data being a part of that. We're seeing data companies scale to hundreds of millions of revenue in the fastest time ever. I mean, like, ever, ever, ever, and everyone is selling data. How do you expect this market to play out given your neutrality on not being involved in it, but being adjacent to it? Will this be a market of consolidation in years to come? How do you see it play out?
00:26:09 · Harrison Chase
[00:26:09] S0: I actually think the the data market is still, like, pretty early in the grand scheme of things in so much as we're talking about kind of like a context layer for agents. So what I mean by that is, like, there are a bunch of databases that are... Clickhouse is maybe, like, a really good example of one that's doing incredibly well recently. I think they're still largely focused on kind of, like, their database. They're good old fashioned database. They're good at storing data and querying data, but it's still in pretty kind of traditional ways. Clickhouse existed pre pre AI. When we talk about kind of, like, a context layer for agents, and again, like there will be a bunch of data produced from, like, software systems and great. Like, I think, like, existing databases, Clickhouse, Mongo, Elastic, whatever, are great for that. When we talk about a context layer for agents, I think a key part of what exists there is, yes, there's a database, but I actually think there's a system around the database. I I think, like, a context layer for agents will look more similar to, like, an agent around a database than just a database. And the reason for this is that taking all of these learnings and ingesting them into a database, you need some sort of, like, intelligence to do that. And I think that intelligence is actually really, really hard. The hardest part about memory is actually not the database at all. When we talk to a number of companies, the way that they do memory is they just store it in a single file. Very, very kind of simple. The the hard part is taking all of the observations that happen, whether it's from the user interacting with, like, the... Let's say, it's a traditional SaaS interacting with the the software or the user interacting with the agent and distilling that into information that's valuable enough to go into this context layer. That's the hard part in memory right now.
00:27:35 · Harry Stebbings, Harrison Chase
[00:27:35] S1: Why is that so hard? [00:27:36] S0: Because it's tricky to get LLMs to do what you want them to do. And this type of, like, meta reflection, I think, is something that LLMs actually still struggle at a lot. Like, reflecting on kind of, like, prompts and improving prompts, reflecting on agent architecture, improving agent architecture. We should build a benchmark for this internally. We act... We do have some benchmarks for this in terms of engine, which is our agent that's running on things. But I think this type of meta reflection is still really difficult for agents. I think it's also generally, like, a hard thing for humans to do as well. Like, I don't know if we actually understand... Some people probably do, so I shouldn't say this. But I don't know if we understand, like, like, what like, what what goes on inside our brain? Like, I feel like our brains are still kind of, like, complex things that we don't fully understand. And so if we're trying to now replicate this kind of like learning process in a system, I think there's a bunch of different pieces. And so I think models aren't great at kind of like reflection, and I don't think we know what this system should look like. What should the inputs be? What should the outputs be? What should the harness for this agent be, what should the different workflows be. [00:28:30] S1: Will the models be meaningfully better at reflection in twelve months? [00:28:33] S0: Probably. I mean, seems like they'll get better at everything.