The MAD Podcast: How AI Gets Built — with Matt Turck
00:41:36 · Nathan Benaich
[00:41:36] S0: Well, I think it depends on the... Their kind of focus and modality. Like for example, on robotics and embodied AI, I think you can... I can see how an independent lab would still have a chance of competing and winning like because of the immense focus on, you know, enterprise AI, basically, that OpenAI and Anthropic has. Like, I have a slide in the report where I basically just, like, show all the different product lines that OpenAI and Anthropic had, or admittedly, OpenAI had lots and how they progressively just cut all of them. And there was a quote like, you know, this this moment is like too important to be distracted by side quests or some version of that, I'm paraphrasing, which meant that they broadly don't do dedicated robotics. Although maybe there's some of that kind of data in the in the Astra training recipe. Science is a little bit unclear. Anthropic is clearly leaning a lot into biology and drug discovery. OpenAI, maybe more on the model side, less on like we wanna own more of the stack. There's like kind of related to embodied AI, this whole like category of world models, whether that's, you know, a video model with some action in it or like some other definition of world model. But but in general, like Anthropic doesn't really do any of that or at least like from what we can see publicly. I mean, they don't even have an image generation model, for example. I think audio, pretty clearly, labs do have some of those capabilities, but, you know, if 11 labs can, like, keep accelerating the way it does, then clearly there's there's a need for a dedicated capability. I think those are probably the areas where I can see an independent lab, like, working out, but but ones that broadly say we're gonna do frontier intelligence for knowledge work, but just, make it better, faster, cheaper. I think that's tough. I think for me, like, I'll I'll believe it when I see it. But yeah, this idea of like you open your computer or the app looks exactly the same, but it's just better. I don't know if that Yep.
00:43:26 · Matt Turck
[00:43:26] S1: And in world model labs, are you seeing anything exciting? I mean, obviously, he's a Lee's lab was acquired for a lot of money and that's unclear what happens from there, given that the buyer is AMD. Because Yann LeCun started his company AMI in Paris since the
00:43:44 · Nathan Benaich
[00:43:44] S0: World Labs. Yeah. I mean, I've I've, you know, visibility into Odyssey where I was an early investor, like, doing world models. And I think the the interesting thing in that business is like their most latest... Their latest systems like Odyssey three. And there you can see that an autoregressive model trained to basically take all this video data and then have it conditioned on actions such that you know that, okay, I'm taking one action and then I generate the next frame as a result of the action. That paradigm applied to robotic manipulation, to self driving, to like video games, to like entertainment or something, can see generalization across all those different domains and already seeing that model being used for robotic manipulation training. So I I think at a at a broad level, I'm like kinda interested how image, audio, video ends up all mushing together into basically a world model. And then if that actually happens, then that could end up subsuming simulation and game engines. And then that's very important for moving robotics into the real world, but also designing physical systems and like hardware engineering. And to get back to like the AMD World Labs point, it's like if that happens, then the compute substrate that serves those kinds of environments and allows you to train and run inference is super, super important. And therefore having knowledge about how the world model research ecosystem is moving and what kind of hardware improvements you can make to be the number one choice is a big, big revenue pie. That's I think what motivated part... Well, part of the reason that AMD, bought WorldLabs. Sort of in the same way that NVIDIA over many, many years was like very close to research, labs that are doing shaders or three d vision or other things and, you know, giving compute out and constantly getting feedback from researchers such that they always knew like my hardware is the best for where the where the kind of frontier is going.
00:45:38 · Matt Turck, Nathan Benaich
[00:45:38] S1: There seems to be a good number of, labs trying to figure out, ways to be more efficient with compute. So from flipping airplanes, which I I don't know much about, to what seems to be the reemergence of, like, diffusion model focused companies [00:45:54] S0: Yeah. [00:45:55] S1: Yeah. To just reduce the cost of of inference, which a fascinating thought by the way, which, you know, people tend to talk about a bunch for a minute and then forget about. But like this massive investment in compute that we would do in theory seems to be at the mercy of like somebody coming up with a much more efficient way to train models. [00:46:14] S0: Yeah. Yeah. Yeah. I mean, I would contend that even if that were to happen, we would just see, like, more consumption. And I... You know, I'm just informed by a fact that I'm on my laptop using Astra High on 1.5 x, and I would love that 1.5 x to be, like, 10.5 x. [00:46:29] S1: A different paradox [00:46:30] S0: of Yeah. The I'm like doing more things than I I could have done before with the same amount of time. So I'm not necessarily sure that, like, you know, one super efficient system will sort of like pop all the investments that we're making. [00:46:43] S1: In your report, you talk about the important intersection between models and harnesses, which seems to be one of those topics that like keeps shifting. People say, well, no, the intelligence is all in the model. And then other people say, it's all in the harness and that seems to be going back and forth. What's your take?
00:47:02 · Nathan Benaich
[00:47:02] S0: I think my broad take is like... It's a bit like when you're injured and you have crutches, you need to have assistance. And at some point you heal and you don't need the assistance anymore. In absence of having like a frontier model, like you need to give it crutches. And those crutches can be like access to tools, like how it handles memory, what data sources it has access to, and then maybe some recipes in order to accomplish a specific task. And I I think for the moment, like, the non frontier, that'll that'll hold. But I'll take the word for... I'll take the sort of frontier lab folks' word for it that, you know, over time, one should assume that all these crutches kinda gets consumed into the model through training or through the harness that the model vendor provides. I mean, there's there's a data point around how much of the Claude code system prompt was rewritten over time as the model itself got better, and it's something like 80% of the system problem was just removed. The harness is also important because the one that's shipped by OpenAI Anthropic is kind of like not incentivized to make you use a less powerful model. Like just in the UI and the and the product, like if I'm doing a legal task and I have the $200 plan and I have the tokens, like I want the quality to be super good. So why would I risk the quality being like not super good by sliding it from high to low or by trying it on a on a lower qualities... Lower quality model. And so I think unless there's some some sort of, like, product UI behavior that is altered to disincentivize the user from only using the most expensive tool, Frontier Labs will continue doing that. And so that's maybe where, like, getting getting good knowledge around, like, okay, can use a less good system, but compensate for it by having a better harness can come into play. But I think that's more of, a product a product decision also for companies that that are not in the frontier labs.
00:48:56 · Matt Turck, Nathan Benaich
[00:48:56] S1: Is that interesting for you as an investor, all the kind of vertical harness companies or horizontal harness companies? Is that an investable theme? [00:49:05] S0: Yeah. I think in isolation, like no, at the moment. I can change my mind, but at the moment, no. But I do think for my companies that are, you know, heavily like AI based or heavily agent based, when we see that, you know, every month like token expenditure because... Is going up through the roof because employees are increasingly getting leverage from these tools and and product usage is going up. Like, that's where I think the trade ends up being like, hey, we should really, like, spend time on seeing can we use, a less frontier model and, like, compensate for it with a better harness so we could just, not burn a lot of money. [00:49:40] S1: Yeah. By the way, you're a solo GP and I think I read somewhere that you use a lot of agents in your investing activity to do everything you do. Example or anything that you thought was pretty cool in the way you use agents throughout this year?