The MAD Podcast: How AI Gets Built — with Matt Turck
00:43:26 · Matt Turck
[00:43:26] S1: And in world model labs, are you seeing anything exciting? I mean, obviously, he's a Lee's lab was acquired for a lot of money and that's unclear what happens from there, given that the buyer is AMD. Because Yann LeCun started his company AMI in Paris since the
00:43:44 · Nathan Benaich
[00:43:44] S0: World Labs. Yeah. I mean, I've I've, you know, visibility into Odyssey where I was an early investor, like, doing world models. And I think the the interesting thing in that business is like their most latest... Their latest systems like Odyssey three. And there you can see that an autoregressive model trained to basically take all this video data and then have it conditioned on actions such that you know that, okay, I'm taking one action and then I generate the next frame as a result of the action. That paradigm applied to robotic manipulation, to self driving, to like video games, to like entertainment or something, can see generalization across all those different domains and already seeing that model being used for robotic manipulation training. So I I think at a at a broad level, I'm like kinda interested how image, audio, video ends up all mushing together into basically a world model. And then if that actually happens, then that could end up subsuming simulation and game engines. And then that's very important for moving robotics into the real world, but also designing physical systems and like hardware engineering. And to get back to like the AMD World Labs point, it's like if that happens, then the compute substrate that serves those kinds of environments and allows you to train and run inference is super, super important. And therefore having knowledge about how the world model research ecosystem is moving and what kind of hardware improvements you can make to be the number one choice is a big, big revenue pie. That's I think what motivated part... Well, part of the reason that AMD, bought WorldLabs. Sort of in the same way that NVIDIA over many, many years was like very close to research, labs that are doing shaders or three d vision or other things and, you know, giving compute out and constantly getting feedback from researchers such that they always knew like my hardware is the best for where the where the kind of frontier is going.
00:45:38 · Matt Turck, Nathan Benaich
[00:45:38] S1: There seems to be a good number of, labs trying to figure out, ways to be more efficient with compute. So from flipping airplanes, which I I don't know much about, to what seems to be the reemergence of, like, diffusion model focused companies [00:45:54] S0: Yeah. [00:45:55] S1: Yeah. To just reduce the cost of of inference, which a fascinating thought by the way, which, you know, people tend to talk about a bunch for a minute and then forget about. But like this massive investment in compute that we would do in theory seems to be at the mercy of like somebody coming up with a much more efficient way to train models. [00:46:14] S0: Yeah. Yeah. Yeah. I mean, I would contend that even if that were to happen, we would just see, like, more consumption. And I... You know, I'm just informed by a fact that I'm on my laptop using Astra High on 1.5 x, and I would love that 1.5 x to be, like, 10.5 x. [00:46:29] S1: A different paradox [00:46:30] S0: of Yeah. The I'm like doing more things than I I could have done before with the same amount of time. So I'm not necessarily sure that, like, you know, one super efficient system will sort of like pop all the investments that we're making. [00:46:43] S1: In your report, you talk about the important intersection between models and harnesses, which seems to be one of those topics that like keeps shifting. People say, well, no, the intelligence is all in the model. And then other people say, it's all in the harness and that seems to be going back and forth. What's your take?
00:47:02 · Nathan Benaich
[00:47:02] S0: I think my broad take is like... It's a bit like when you're injured and you have crutches, you need to have assistance. And at some point you heal and you don't need the assistance anymore. In absence of having like a frontier model, like you need to give it crutches. And those crutches can be like access to tools, like how it handles memory, what data sources it has access to, and then maybe some recipes in order to accomplish a specific task. And I I think for the moment, like, the non frontier, that'll that'll hold. But I'll take the word for... I'll take the sort of frontier lab folks' word for it that, you know, over time, one should assume that all these crutches kinda gets consumed into the model through training or through the harness that the model vendor provides. I mean, there's there's a data point around how much of the Claude code system prompt was rewritten over time as the model itself got better, and it's something like 80% of the system problem was just removed. The harness is also important because the one that's shipped by OpenAI Anthropic is kind of like not incentivized to make you use a less powerful model. Like just in the UI and the and the product, like if I'm doing a legal task and I have the $200 plan and I have the tokens, like I want the quality to be super good. So why would I risk the quality being like not super good by sliding it from high to low or by trying it on a on a lower qualities... Lower quality model. And so I think unless there's some some sort of, like, product UI behavior that is altered to disincentivize the user from only using the most expensive tool, Frontier Labs will continue doing that. And so that's maybe where, like, getting getting good knowledge around, like, okay, can use a less good system, but compensate for it by having a better harness can come into play. But I think that's more of, a product a product decision also for companies that that are not in the frontier labs.
00:48:56 · Matt Turck, Nathan Benaich
[00:48:56] S1: Is that interesting for you as an investor, all the kind of vertical harness companies or horizontal harness companies? Is that an investable theme? [00:49:05] S0: Yeah. I think in isolation, like no, at the moment. I can change my mind, but at the moment, no. But I do think for my companies that are, you know, heavily like AI based or heavily agent based, when we see that, you know, every month like token expenditure because... Is going up through the roof because employees are increasingly getting leverage from these tools and and product usage is going up. Like, that's where I think the trade ends up being like, hey, we should really, like, spend time on seeing can we use, a less frontier model and, like, compensate for it with a better harness so we could just, not burn a lot of money. [00:49:40] S1: Yeah. By the way, you're a solo GP and I think I read somewhere that you use a lot of agents in your investing activity to do everything you do. Example or anything that you thought was pretty cool in the way you use agents throughout this year?
00:49:59 · Nathan Benaich
[00:49:59] S0: The coolest use case I had recently is on like expert research where, you know, I think you can, well, if you're sort of into cheap man's version, it's like you ask your friends. But if you wanna spend a bit of budget and ask beyond your network, then you're gonna ask like GLG or AlphaSense, etcetera. And then they will go find humans for you. They will introduce you. You take the call. It's a ping ponging, like really annoying and limited by language, time zone, etcetera. But there's like a new product that I started using called Lato, which is basically like replacing the fact that I have to do the work with a model doing the work. And so I'm already like heavily using, in this case Codex, I kind of switched depending on what model is better at the time to sort of like just process all my ideas and the context I have on other companies that I looked at over history, how they performed, what questions I asked, what responses I got to just like figure out what is the highest value question I can ask of a team to inform like whether I should spend time or make the investment or not. And so I can take all that information and then with Lapto like construct an expert interview and then decide on cohorts I want and then just hit launch study. And then the system goes out and finds people, consents them to see if they're the right fit for the study. And then they go speak to an AI. And I was like shocked because, you know, I hit launch study, I went to sleep. I had like three days to basically give the company an answer. It was like in competitive markets, so I think the nuances of customer research is important. You'll wake up the first day, have three interviews from like pretty senior people. I have the raw transcript. The transcript can come back into my model, answer the open questions we had. We, me and my model. And then and then figure out, okay, we have enough data points for this, but like it's kinda opened up this question stream that we should ask. Maybe we should actually retune some of the interview questions to really coax the person to answer those questions. So it like updates the study questions and relaunches the survey. Next day, five five people. Again, this repeats. And over like two and a half days, had 12 in...