The Twenty Minute VC (20VC): Venture Capital | Startup Funding | The Pitch
00:06:39 · Harry Stebbings
[00:06:39] S1: So what do mean optimize signal for noise? So think of it this way.
00:06:43 · Parag Agrawal
[00:06:43] S0: Let's say you took web search, which was cheap and fast and low compute. One way of conceptualizing the web search problem is you start with, like, a trillion documents that are on the web, some few trillion. Given any search, I now need to narrow it down to a thousand tokens that your model's context window should see. So the problem is going from a trillion URLs with, let's call it, a few thousand tokens each down to a thousand total tokens. So how do we do it in website? We first say, okay, we're gonna do retrieval. So for most documents, I'm gonna spend zero compute. For small number of documents, I'm gonna spend minuscule amounts of compute to figure out which 10,000 to look at. Once I get these 10,000, I'm gonna spend a little bit more compute for each of these 10,000 documents to narrow it down to 1,000 documents. Mhmm. And I'm gonna keep doing this with bigger and bigger models and rankers with more and more features features until I can narrow it down to a thousand tokens for your model. So you're essentially allocating compute to web search in order to save compute on the model. That's roughly what's going on here. So if you wanna save compute for the model, the question is how much compute should you allocate? So if your LUNA model is really cheap, you don't want to do too much compute in web search because it's okay to leak a little bit more information into LUNA's context because it's cheap. Into Fable, you wanna do the work before you waste Fable's time because that's gonna be expensive in time and money for you if web search gives you worse answers because you cheaper out on web search.
00:08:10 · Harry Stebbings
[00:08:10] S1: Can I ask, how do you deal with the ambiguity of what what agents want? And what I mean by that is for different things, an agent might want different things. And for different people, an agent might want different things. I may really care about accuracy and not at all about latency or cost. I may really care about latency, but not a tool by accuracy, really.
00:08:30 · Parag Agrawal
[00:08:30] S0: So you allow the agent to specify that in the API signature. So our product is an API, which either the programmer can configure based on their application or can leave it to the agent. Our API even has a parameter, which is like, what's the model calling me? If you know the model, we can do things differently. Now you don't have to tell us, but if you tell us, you might get better results. We ship just like you can use, like, a model. You can use a small model or a big model, and you can use it with, like, low thinking or medium thinking or high thinking. We have productized our search system into a few different modes, each optimized for a certain class of use case. For example, we have a really, really fast API, the fastest in the market. It's fast and cheap because with low latency budget, there's only so much compute you can do. And it's built for voice agents. So your voice agent must be all knowing without telling you, guys, I'm searching the web. Hold on while I come back with the answer. Like, that's a silly experience for a voice agent. Right? But you should just, like, magically immediately respond. And you now need to do web search, which is like a rapid. On the other hand, you can take a Fable model, and the thing is gonna think for twenty seconds and then generate for it. You can spend five seconds on web search to make sure it does into five web searches, only two. So you end to end in the agent, you save time and cost. And so that's a different processor on our system. It's called advanced. So you use Turbo if you're building a voice agent, you use advanced if you're like really expensive background agent.
00:09:59 · Harry Stebbings, Parag Agrawal
[00:09:59] S1: And so the primary use case today in terms of customer base is engineering and coding. [00:10:04] S0: It's pretty broad based. I would say the primary use cases, the common theme is knowledge work. Coding is a category of knowledge work. So are AI lawyers. So are productivity applications. So are AI insurance underwriters. So are [00:10:18] S1: AI scientists. I'm trying to understand how much of the mother load is engineering. Is it 80%? [00:10:24] S0: No. The the thing about engineering is coding is a large chunk of inference in the market right now. Coding invokes, I would say, web search in 5% of prompts. Woah. Right? So it's not every prompt invoking web search when you're writing code because most of them rely on your internal context and your internal data and your code base. So the model is spending time reading your internal code and not on the web. Law is not gonna [00:10:52] S1: be much more, is it? [00:10:53] S0: Law is very web search oriented. [00:10:55] S1: Really? I thought it'd be internal data driven. [00:10:57] S0: There is internal data, but there is case law. There is facts. There's facts about companies, facts about people, and you have to go exclude information. So you have to be very comprehensive in law to say, I want to be confident that despite a lot of effort, you can't find this. Insurance underwriting has the same flavor. So there's... Sales is very web search heavy. AI science is very web search heavy. So on a relative basis, inference goes more in compute. But when you think of web search, all of these others start popping too.
00:10:57 · Parag Agrawal, Harry Stebbings
[00:10:57] S0: There is internal data, but there is case law. There is facts. There's facts about companies, facts about people, and you have to go exclude information. So you have to be very comprehensive in law to say, I want to be confident that despite a lot of effort, you can't find this. Insurance underwriting has the same flavor. So there's... Sales is very web search heavy. AI science is very web search heavy. So on a relative basis, inference goes more in compute. But when you think of web search, all of these others start popping too. [00:11:26] S1: We were talking before about muse and instinct. If there's anything that needs web search, it's personal assistance. It's great. How does that... Yeah. It's great. This is like, woo hoo. This is the greatest thing for my business. How does that change your business?