The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:02:41 · Harry Stebbings
Thousands of AI tools and millions of dollars of AI spend are measured. You bet on AI. Larridin proves that it's working. Visit larridin.com to learn more. That's larridin.com. While Larridin tracks AI impact, Superhuman helps you move faster. Can I be honest with you for a second? Here's what my week actually looks like. Back to back meetings all day, and the real pressure builds up in the gaps between them. I'll come out of six founder meetings with a whole stack of follow ups waiting, and my inbox, it's hundreds of messages. And somewhere in there is the one thing that really matters, which I always tell my mother is her. I've tried AI tools for this, but they always lived in another tab. I'd have to stop, go find it, paste in the context, write the perfect prompt. It was just enough friction that it never really stuck. Superhuman Go from the makers of Grammarly, totally different. It's right there where you need it, when you need it, in the doc, when I'm reading, the email I'm writing, when a thread gets dense, I highlighted and asked Go to pull out what matters without losing my place. And after a run of back to backs, I asked what was actually decided, and Go turns it into clear action items or drafts, the reply I need to send. It's already up to speed, and I'm always the one in control. Try Superhuman Go from the makers of Grammarly and find out more at superhuman.com. You have now arrived at your destination. Harrison, dude, I'm so looking forward to this. I've been looking forward to it for a while because I remember Miles telling me about you when he was at Benchmark and just singing your praises. So I wanted to make it happen for a while. So thank you for joining me.
00:04:23 · Harrison Chase, Harry Stebbings
[00:04:23] S0: I'm excited to be here. [00:04:24] S1: For those that don't know, sixty seconds, what is Langchain and where do you like sit in the stack?
00:04:31 · Harrison Chase
[00:04:31] S0: We build tools to make it easy to build agents. So we build developer tools primarily. We have an open source framework, which basically connects the model to everything else that you need to build a useful agent. So tools, data, computation, things like that. And then we also have Langsmith. Langsmith is observability in Evals. Agents go wrong all the time. How do you know how they go wrong, and how do you measure them so that you can increase their performance over time. That's what Langsmith does. Why is it difficult to build agents? The hard part about building agents is they don't behave how you want them to behave. So the models are these, like, nondeterministic black boxes. Right? And they take natural language as input, and then they output natural language. And inside the black box, they're also nondeterministic. And so what that means is that the input you get is, like, infinite in dimension. It can be anything. You put a chatbot in front of someone, they can type anything that they want. Right? And so that makes it really hard to know what they're gonna type. And then once it goes into this model... And by the way, agents are not just one model call. They're a series of model calls, they're interacting with other tools. And so you have these small block... Black boxes that are then put inside this large system, and you get basically these cascading types of errors. So you have this this very big kind of like input space. You have this very big output space because two similar inputs can go in and they can arrive at completely different answers. And the thing that's making all this hard is the LLM itself. Yes. They're very, very smart, but you exactly what they're going to do, and they change answers based on small changes to the input. And so you build the system, but it's really non robust. And so getting the system to behave reliably, like, that is the hardest part. Like, when we talk about agents three years ago, four years ago to agents today, they've gotten a lot better. And the main thing that's happened is, one, the models have gotten better. And two, we've figured out how to build systems that basically harness these models better. And and that second part, the harnessing of these models, is what we focus on.
00:06:24 · Harry Stebbings, Harrison Chase
[00:06:24] S1: How does your business change in a way that people don't see as the models get better? [00:06:29] S0: The main way our business changes as the model gets better is we basically have to erase big parts of our product. So what I mean by that... And and Cursor said this really well as well. I think this was one of their actual operating principles as a company is like erase the product or delete the product, I think is what they called it, where basically, like, these models get better and things that you had to do six months ago, you no longer have to do. And so this is very relevant in our open source. So our open source is all about basically the the harness and the framework for taking these models and helping them do things. And the right way to take these models and help them do things changes every month, three months, six months, choose your time period, but it changes pretty rapidly. [00:07:07] S1: How do you build for such an uncertain future? Do you just have to get in bed with the labs to know the roadmap so you don't waste six months building something that then narrates?
00:07:15 · Harrison Chase
[00:07:15] S0: There's no easy answer. On one hand, what are you going to do? Not build. Like, that's what I like to say to founders who ask this. Like, what do I do? Like, stuff is changing. You know, the models are changing. If I build something now, it might not be relevant in six months. What's the alternative? You just don't build anything. And so I think you have to make kind of like educated guesses as to what parts are going to be around and what parts aren't going to be around in the future. So for example, early on, the way that we would do tool calling in Linkchain was by having the model generate JSON, DSL, and then parsing it. And that was something that I think we all thought or knew was like a little bit of a hack at at at the time, but we had to do it because that was the only way to get it to to actually steer things because you have to get it to do this type of tool calling. And then a few months later, OpenAI came out with tool calling. And I think we recognized pretty early... And this is like a more native version of tool calling that's directly in in in in the model and in the APIs. And I think we recognized pretty early on that that was the future, and so we invested a lot of efforts and just completely switched from JSON parsing to this tool calling. And so I think you have to basically recognize what things are short term hacks and what things are long term kind of like will always be there. The models will never know your proprietary data. They will never be able to kind of like actually get the output of APIs. You have to take their, like, API call, pass it to the API, get the output, and then continue in the loop. And so I think the system around the model connecting it to other places will always be there. But a lot the hacks we do around chain of thought or planning or tool calling, things that can just be much more closely integrated into the model or into the model API will go away, and I think you just have to be quick to recognize that and go on.
00:08:48 · Harry Stebbings
[00:08:48] S1: Why do we need neutral harnesses? We see, you know, bluntly, very brilliant agent builders from the models themselves. Why does your layer not just get eaten by them?