The MAD Podcast: How AI Gets Built — with Matt Turck
00:45:38 · Matt Turck, Nathan Benaich
[00:45:38] S1: There seems to be a good number of, labs trying to figure out, ways to be more efficient with compute. So from flipping airplanes, which I I don't know much about, to what seems to be the reemergence of, like, diffusion model focused companies [00:45:54] S0: Yeah. [00:45:55] S1: Yeah. To just reduce the cost of of inference, which a fascinating thought by the way, which, you know, people tend to talk about a bunch for a minute and then forget about. But like this massive investment in compute that we would do in theory seems to be at the mercy of like somebody coming up with a much more efficient way to train models. [00:46:14] S0: Yeah. Yeah. Yeah. I mean, I would contend that even if that were to happen, we would just see, like, more consumption. And I... You know, I'm just informed by a fact that I'm on my laptop using Astra High on 1.5 x, and I would love that 1.5 x to be, like, 10.5 x. [00:46:29] S1: A different paradox [00:46:30] S0: of Yeah. The I'm like doing more things than I I could have done before with the same amount of time. So I'm not necessarily sure that, like, you know, one super efficient system will sort of like pop all the investments that we're making. [00:46:43] S1: In your report, you talk about the important intersection between models and harnesses, which seems to be one of those topics that like keeps shifting. People say, well, no, the intelligence is all in the model. And then other people say, it's all in the harness and that seems to be going back and forth. What's your take?
00:47:02 · Nathan Benaich
[00:47:02] S0: I think my broad take is like... It's a bit like when you're injured and you have crutches, you need to have assistance. And at some point you heal and you don't need the assistance anymore. In absence of having like a frontier model, like you need to give it crutches. And those crutches can be like access to tools, like how it handles memory, what data sources it has access to, and then maybe some recipes in order to accomplish a specific task. And I I think for the moment, like, the non frontier, that'll that'll hold. But I'll take the word for... I'll take the sort of frontier lab folks' word for it that, you know, over time, one should assume that all these crutches kinda gets consumed into the model through training or through the harness that the model vendor provides. I mean, there's there's a data point around how much of the Claude code system prompt was rewritten over time as the model itself got better, and it's something like 80% of the system problem was just removed. The harness is also important because the one that's shipped by OpenAI Anthropic is kind of like not incentivized to make you use a less powerful model. Like just in the UI and the and the product, like if I'm doing a legal task and I have the $200 plan and I have the tokens, like I want the quality to be super good. So why would I risk the quality being like not super good by sliding it from high to low or by trying it on a on a lower qualities... Lower quality model. And so I think unless there's some some sort of, like, product UI behavior that is altered to disincentivize the user from only using the most expensive tool, Frontier Labs will continue doing that. And so that's maybe where, like, getting getting good knowledge around, like, okay, can use a less good system, but compensate for it by having a better harness can come into play. But I think that's more of, a product a product decision also for companies that that are not in the frontier labs.
00:48:56 · Matt Turck, Nathan Benaich
[00:48:56] S1: Is that interesting for you as an investor, all the kind of vertical harness companies or horizontal harness companies? Is that an investable theme? [00:49:05] S0: Yeah. I think in isolation, like no, at the moment. I can change my mind, but at the moment, no. But I do think for my companies that are, you know, heavily like AI based or heavily agent based, when we see that, you know, every month like token expenditure because... Is going up through the roof because employees are increasingly getting leverage from these tools and and product usage is going up. Like, that's where I think the trade ends up being like, hey, we should really, like, spend time on seeing can we use, a less frontier model and, like, compensate for it with a better harness so we could just, not burn a lot of money. [00:49:40] S1: Yeah. By the way, you're a solo GP and I think I read somewhere that you use a lot of agents in your investing activity to do everything you do. Example or anything that you thought was pretty cool in the way you use agents throughout this year?
00:49:59 · Nathan Benaich
[00:49:59] S0: The coolest use case I had recently is on like expert research where, you know, I think you can, well, if you're sort of into cheap man's version, it's like you ask your friends. But if you wanna spend a bit of budget and ask beyond your network, then you're gonna ask like GLG or AlphaSense, etcetera. And then they will go find humans for you. They will introduce you. You take the call. It's a ping ponging, like really annoying and limited by language, time zone, etcetera. But there's like a new product that I started using called Lato, which is basically like replacing the fact that I have to do the work with a model doing the work. And so I'm already like heavily using, in this case Codex, I kind of switched depending on what model is better at the time to sort of like just process all my ideas and the context I have on other companies that I looked at over history, how they performed, what questions I asked, what responses I got to just like figure out what is the highest value question I can ask of a team to inform like whether I should spend time or make the investment or not. And so I can take all that information and then with Lapto like construct an expert interview and then decide on cohorts I want and then just hit launch study. And then the system goes out and finds people, consents them to see if they're the right fit for the study. And then they go speak to an AI. And I was like shocked because, you know, I hit launch study, I went to sleep. I had like three days to basically give the company an answer. It was like in competitive markets, so I think the nuances of customer research is important. You'll wake up the first day, have three interviews from like pretty senior people. I have the raw transcript. The transcript can come back into my model, answer the open questions we had. We, me and my model. And then and then figure out, okay, we have enough data points for this, but like it's kinda opened up this question stream that we should ask. Maybe we should actually retune some of the interview questions to really coax the person to answer those questions. So it like updates the study questions and relaunches the survey. Next day, five five people. Again, this repeats. And over like two and a half days, had 12 in...
00:51:59 · Nathan Benaich, Matt Turck
12 interviews over like fourteen hours across like US, India, Europe, Germany, France, is like, it would be physically impossible for me to have spent like fourteen hours across 12 people. Like, there were several people who spoke to this model like for an hour and twenty minutes. [00:52:14] S1: Which just in itself is fascinating. Right? Like the fact that humans feel so comfortable so quickly speaking to an AI. [00:52:22] S0: Yeah. Yeah. It's probably a bit recursive in the sense that, like, we're all in front of a computer all the time and, like, these are generally, like, tech questions. And so I think people prefer to... Unfortunately, prefer to speak to their phone than speak to a human. [00:52:36] S1: That's a fascinating thought. That actually speaking to an AI is what people prefer. Yeah. Which is actually a theme that we heard over the years is like, you know, we we maybe not the years, but the months that people are starting to prefer to talk to an AI customer service agent over a person. [00:52:55] S0: Yeah, because I think in that case it's like a human is pretending to be a robot and doing a really bad job at it because you're a human, you're like, can you please just not gonna talk like a robot? And then now you get a robot that's actually like really good and then the [00:53:07] S1: human can Like a human.
00:53:07 · Matt Turck, Nathan Benaich
[00:53:07] S1: human can Like a human. [00:53:09] S0: Yeah. Okay. Amazing. Yeah. So that I think that that kind of idea of like automated customer research has been very powerful for me and I can just see how, you know, even if you had a portfolio, you would run like this kind of agent interviews on your portfolio companies and how customers are using it. So then at least you get some first party data when you go into the boardroom, you know. Or if you're an LP and like your investors are saying like, oh, this company is really good. Like, is it really good? [00:53:35] S1: A big thing that happened this year in the last few months in particular is to this point, like the rise of personal agents, the instincts and the muse and the rockbots. What do you make of that? [00:53:49] S0: I don't give them my data. I don't really trust trust them at the moment, but I think a lot of people do. [00:53:58] S1: A lot of people do. [00:53:59] S0: Yeah. [00:54:01] S1: So you... What you're saying is that you're you don't give your credit card information to a company that did not exist two months ago and doesn't require you to authenticate?