The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:05:03 · Parag Agrawal
[00:05:03] S0: Yeah. Exactly. The second thing is humans operate in a very narrow zone. So if you think about how we use web search, we type keyword queries, which are short and underspecified. We wait for about half to one second. If web search takes more than that, we're impatient. And then we get 10 blue links, and then we random walk across them and a collection of searches to get what we're doing. Agents are not like that. Agents are gonna perhaps tell you exactly what they're looking for, not like three keywords, but like a full sentence. Like, this is what I'm trying to do. Agents will either be super impatient. Like imagine a voice agent, the agent will be like, I need an answer like now, a hundred milliseconds. I can't wait five hundred milliseconds because the human is waiting on me and they're going to wait five hundred milliseconds. So I need web search to do it in hundred. Are there going to be a background agent who's like, I don't care. Just give me the best answer possible. And so the variance of what you can do within web search changes completely. In fact, the one thing that is the least interesting is what we've built for humans. Right? You either have much time or too little time. Almost never the same amount. The output is not the same. So the input's different. The time you have is different. The output is different. The output is in blue links. The output is tokens or files on a file system depending on the type of agent. And so now all of a sudden, you say, okay. Now the problem's inputs are different, outputs are different, and constraints are different. And you get to spend very different amounts of compute on it. So imagine someone running an agent built with a LUNA model. And then imagine someone running an agent built with a Fable model. They're very different models. How you wanna optimize signal to noise and tokens for each of them is so different in terms of what you do in the web search track.
00:06:39 · Harry Stebbings
[00:06:39] S1: So what do mean optimize signal for noise? So think of it this way.
00:06:43 · Parag Agrawal
[00:06:43] S0: Let's say you took web search, which was cheap and fast and low compute. One way of conceptualizing the web search problem is you start with, like, a trillion documents that are on the web, some few trillion. Given any search, I now need to narrow it down to a thousand tokens that your model's context window should see. So the problem is going from a trillion URLs with, let's call it, a few thousand tokens each down to a thousand total tokens. So how do we do it in website? We first say, okay, we're gonna do retrieval. So for most documents, I'm gonna spend zero compute. For small number of documents, I'm gonna spend minuscule amounts of compute to figure out which 10,000 to look at. Once I get these 10,000, I'm gonna spend a little bit more compute for each of these 10,000 documents to narrow it down to 1,000 documents. Mhmm. And I'm gonna keep doing this with bigger and bigger models and rankers with more and more features features until I can narrow it down to a thousand tokens for your model. So you're essentially allocating compute to web search in order to save compute on the model. That's roughly what's going on here. So if you wanna save compute for the model, the question is how much compute should you allocate? So if your LUNA model is really cheap, you don't want to do too much compute in web search because it's okay to leak a little bit more information into LUNA's context because it's cheap. Into Fable, you wanna do the work before you waste Fable's time because that's gonna be expensive in time and money for you if web search gives you worse answers because you cheaper out on web search.
00:08:10 · Harry Stebbings
[00:08:10] S1: Can I ask, how do you deal with the ambiguity of what what agents want? And what I mean by that is for different things, an agent might want different things. And for different people, an agent might want different things. I may really care about accuracy and not at all about latency or cost. I may really care about latency, but not a tool by accuracy, really.
00:08:30 · Parag Agrawal
[00:08:30] S0: So you allow the agent to specify that in the API signature. So our product is an API, which either the programmer can configure based on their application or can leave it to the agent. Our API even has a parameter, which is like, what's the model calling me? If you know the model, we can do things differently. Now you don't have to tell us, but if you tell us, you might get better results. We ship just like you can use, like, a model. You can use a small model or a big model, and you can use it with, like, low thinking or medium thinking or high thinking. We have productized our search system into a few different modes, each optimized for a certain class of use case. For example, we have a really, really fast API, the fastest in the market. It's fast and cheap because with low latency budget, there's only so much compute you can do. And it's built for voice agents. So your voice agent must be all knowing without telling you, guys, I'm searching the web. Hold on while I come back with the answer. Like, that's a silly experience for a voice agent. Right? But you should just, like, magically immediately respond. And you now need to do web search, which is like a rapid. On the other hand, you can take a Fable model, and the thing is gonna think for twenty seconds and then generate for it. You can spend five seconds on web search to make sure it does into five web searches, only two. So you end to end in the agent, you save time and cost. And so that's a different processor on our system. It's called advanced. So you use Turbo if you're building a voice agent, you use advanced if you're like really expensive background agent.
00:09:59 · Harry Stebbings, Parag Agrawal
[00:09:59] S1: And so the primary use case today in terms of customer base is engineering and coding. [00:10:04] S0: It's pretty broad based. I would say the primary use cases, the common theme is knowledge work. Coding is a category of knowledge work. So are AI lawyers. So are productivity applications. So are AI insurance underwriters. So are [00:10:18] S1: AI scientists. I'm trying to understand how much of the mother load is engineering. Is it 80%? [00:10:24] S0: No. The the thing about engineering is coding is a large chunk of inference in the market right now. Coding invokes, I would say, web search in 5% of prompts. Woah. Right? So it's not every prompt invoking web search when you're writing code because most of them rely on your internal context and your internal data and your code base. So the model is spending time reading your internal code and not on the web. Law is not gonna [00:10:52] S1: be much more, is it? [00:10:53] S0: Law is very web search oriented. [00:10:55] S1: Really? I thought it'd be internal data driven. [00:10:57] S0: There is internal data, but there is case law. There is facts. There's facts about companies, facts about people, and you have to go exclude information. So you have to be very comprehensive in law to say, I want to be confident that despite a lot of effort, you can't find this. Insurance underwriting has the same flavor. So there's... Sales is very web search heavy. AI science is very web search heavy. So on a relative basis, inference goes more in compute. But when you think of web search, all of these others start popping too.