The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:04:31 · Harrison Chase
[00:04:31] S0: We build tools to make it easy to build agents. So we build developer tools primarily. We have an open source framework, which basically connects the model to everything else that you need to build a useful agent. So tools, data, computation, things like that. And then we also have Langsmith. Langsmith is observability in Evals. Agents go wrong all the time. How do you know how they go wrong, and how do you measure them so that you can increase their performance over time. That's what Langsmith does. Why is it difficult to build agents? The hard part about building agents is they don't behave how you want them to behave. So the models are these, like, nondeterministic black boxes. Right? And they take natural language as input, and then they output natural language. And inside the black box, they're also nondeterministic. And so what that means is that the input you get is, like, infinite in dimension. It can be anything. You put a chatbot in front of someone, they can type anything that they want. Right? And so that makes it really hard to know what they're gonna type. And then once it goes into this model... And by the way, agents are not just one model call. They're a series of model calls, they're interacting with other tools. And so you have these small block... Black boxes that are then put inside this large system, and you get basically these cascading types of errors. So you have this this very big kind of like input space. You have this very big output space because two similar inputs can go in and they can arrive at completely different answers. And the thing that's making all this hard is the LLM itself. Yes. They're very, very smart, but you exactly what they're going to do, and they change answers based on small changes to the input. And so you build the system, but it's really non robust. And so getting the system to behave reliably, like, that is the hardest part. Like, when we talk about agents three years ago, four years ago to agents today, they've gotten a lot better. And the main thing that's happened is, one, the models have gotten better. And two, we've figured out how to build systems that basically harness these models better. And and that second part, the harnessing of these models, is what we focus on.
00:06:24 · Harry Stebbings, Harrison Chase
[00:06:24] S1: How does your business change in a way that people don't see as the models get better? [00:06:29] S0: The main way our business changes as the model gets better is we basically have to erase big parts of our product. So what I mean by that... And and Cursor said this really well as well. I think this was one of their actual operating principles as a company is like erase the product or delete the product, I think is what they called it, where basically, like, these models get better and things that you had to do six months ago, you no longer have to do. And so this is very relevant in our open source. So our open source is all about basically the the harness and the framework for taking these models and helping them do things. And the right way to take these models and help them do things changes every month, three months, six months, choose your time period, but it changes pretty rapidly. [00:07:07] S1: How do you build for such an uncertain future? Do you just have to get in bed with the labs to know the roadmap so you don't waste six months building something that then narrates?
00:07:15 · Harrison Chase
[00:07:15] S0: There's no easy answer. On one hand, what are you going to do? Not build. Like, that's what I like to say to founders who ask this. Like, what do I do? Like, stuff is changing. You know, the models are changing. If I build something now, it might not be relevant in six months. What's the alternative? You just don't build anything. And so I think you have to make kind of like educated guesses as to what parts are going to be around and what parts aren't going to be around in the future. So for example, early on, the way that we would do tool calling in Linkchain was by having the model generate JSON, DSL, and then parsing it. And that was something that I think we all thought or knew was like a little bit of a hack at at at the time, but we had to do it because that was the only way to get it to to actually steer things because you have to get it to do this type of tool calling. And then a few months later, OpenAI came out with tool calling. And I think we recognized pretty early... And this is like a more native version of tool calling that's directly in in in in the model and in the APIs. And I think we recognized pretty early on that that was the future, and so we invested a lot of efforts and just completely switched from JSON parsing to this tool calling. And so I think you have to basically recognize what things are short term hacks and what things are long term kind of like will always be there. The models will never know your proprietary data. They will never be able to kind of like actually get the output of APIs. You have to take their, like, API call, pass it to the API, get the output, and then continue in the loop. And so I think the system around the model connecting it to other places will always be there. But a lot the hacks we do around chain of thought or planning or tool calling, things that can just be much more closely integrated into the model or into the model API will go away, and I think you just have to be quick to recognize that and go on.
00:08:48 · Harry Stebbings
[00:08:48] S1: Why do we need neutral harnesses? We see, you know, bluntly, very brilliant agent builders from the models themselves. Why does your layer not just get eaten by them?
00:08:58 · Harrison Chase
[00:08:58] S0: I think the reason we need neutral harnesses is that there is a lot of state that you build up either before you build the agent or as part of the agent running. And the state can include things like memory, whether that's short term memory or long term memory. Memory also comes in different form factors. It can be from interacting with the agent versus from the agent learning from the environment. And the reason that this state gets built up is the agent interacts. It goes out. It learns. It chats, and and it learns things to make it better. I think I think one of the most interesting things right now is how to make these agents do better over time and learn. And some of that learning and some of the doing better over time will be in the model weights, but a lot of it is outside the model, and a lot of it is in this in this context. When you build up this context, you really don't want it to be linked to a single kind of, like, model provider. The reason for that is that then it becomes linked there. You can't go to other models. Being able to change models is very offensive and defensive. So it's offensive in the fact that you wanna be able to take advantage of the best model when it comes out no matter who it's coming from. It's defensive in the fact that you don't wanna get locked in, so can't raise rates, things like that. So you wanna be able to switch models. The harness can be very tied to the model. It can also be neutral. The context right now is really intricately linked to the harness. So the harness's job is to basically take the model and interact with all the other things. All the other things includes data and APIs, but also includes memory. And so if you build this memory in a way that is linked to the harness and the harness is linked to the model, then you're kind of stuck.
00:10:23 · Harry Stebbings
[00:10:23] S1: So the true value of the harness structure is the portability of data that it allows the user?