The Information's TITV
00:17:40 · Akash Pasricha, Umesh Padval, Gopal Srinivasan
[00:17:40] S0: So, Umesh, I may [00:17:42] S2: Let me add to this. Yeah. Look. The... People talk about TPU, but TPU has two solutions. One is for inferencing, and one is for training. So people only mistake TPUs for just training going after GPUs. It's both. They have two solutions, one for inferencing and one for training. Now what is the most important trend, I think, which is I'm seeing in the last six months since GTC, Jensen announced disaggregated solution. What does that mean? Initially, it was TPU or g p... XPU, a CPU, and inferencing engine all provided by one supplier, NVIDIA. It's disaggregated now, and what is happening is the best of breed. So you use the best of breed XPU for training, use the best of breed CPU for the orchestration, and use the lowest cost, highest performance inferencing engine from different suppliers. So it's disaggregating. This is where SambaNova plays tremendously well. Because what they have done is... You saw the deal where they have high performance but at extreme low power. And why is power important? You know, these data centers, when you have massive power of these chips, you need complex liquid cooling technology. SambaNova, because of the power envelope being so low, it does not need liquid cooling. It can do air cooling. Right. What does that do? [00:19:23] S4: Me
00:19:23 · Gopal Srinivasan, Akash Pasricha
[00:19:23] S4: Me [00:19:23] S0: jump in here for one second here. You know, I'm thinking about the projections that some analysts have put out here. So TITV Cowen is saying that TPU revenue by 2028 could be a third of Google Cloud's revenue, which is pretty staggering. Ben, I'll come to you for this one. Mean, how do you think we can put aside whether or not it hits that goal, that margin or that percentage or not? That's a projection. That's not the company saying it. But if it gets there, how does it affect the margins for the Google Cloud business?
00:20:00 · Ben Pouladian
[00:20:00] S3: It just depends where they sell it at. I think, obviously, the TPUs are a crown jewel, and they do a great job building it and running it. But I think stepping back and looking at the bigger picture, it's not all about tokens and what the token factory produces, obviously, per megawatt. You know, you're drawing a parallel to to Google. We were in this world of, how many website visitors you get and then cost per thousand impressions and then ads, then tokens are that same metric, and we're moving towards cost per completed task. So I think what people really need to focus and look on is what is Google's true Agentsic offering on a completed task and compare it to other solutions. And I agree with Umesh that there's this new world of disaggregated inference where you can separate the prefill and decode and have two different solutions. One of them is, for example, like another startup called Dematrix, where they connect with NVLink Fusion. You you can basically take older NVIDIA h 100 GPUs and connect to their system, always... Also air cooled, not liquid cooled, and get the performance of a NVIDIA b 300. So there's ways to mix and match and make things work. The the biggest theme at the end of the day is everyone is super compute constrained, and everyone's looking for a solution right away to get tokens producing because the the first time the token, you know, creates money. But at the same time, it's it's great to see Google really coming out and trying to build this merchant solution. I mean, I never saw it coming. I'm surprised that it's actually happening. It's it's exciting to see the software company morphing into an actual chip company, and they're doing that. And they really wanna showcase Ironwood and the the CPU that they just designed. So all the things are heading in that direction.
00:22:07 · Akash Pasricha
[00:22:07] S0: Umesh, as you think about how this story will play out, so we're compute constrained right now, we're chip constrained. I've been thinking about pricing. And right now, NVIDIA, Google, mean, they can charge whatever the heck they right? Because people need it. Pricing, I imagine, will come down over time here. How do you think that changes the structure of these businesses in the next five years?
00:22:36 · Umesh Padval
[00:22:36] S2: Great question. I've been long enough in the semiconductor and hardware business. I've seen three cycles of up and down. One thing which is common in the semiconductor, is demand and supply always cross each other. Few years, there's feast. Few years, there's famine. And when famine happens, structural problems change. So my prediction is this cycle has lasted for a lot longer than I thought because of the incredible demand from AI infrastructure. But at some point, enough fab capacity is coming online. Enough utilization of GPU is happening because right now, it's at 30 to 40%, and it's gonna go up to 80% just like the CPU utilization in the past went up. And the third thing is applications have really not taken advantage of all the infrastructure. As all these three things happen, there's gonna be a time in the next twelve to eighteen months or somewhere around there where supply is gonna outpace demand. As soon as that happens, believe me, a lot of these neo clouds, which are hundreds of them which have come out, will face a difficult moment in terms of consolidation of that as well as the inferencing companies you've seen raising money every year. I feel both those are going to be challenged when the supply and demand supply outpaces demand, and then pricing pressure will come on all the semiconductor companies. At that point, this is the way history to... You know, if you learn from history, that's what happens.
00:24:13 · Akash Pasricha, Umesh Padval, Ben Pouladian
[00:24:13] S0: But, Umesh, isn't there an argument to be made here that we've seen the Neo Cloud boom with the NVIDIA focused companies. Now we have FluidStack, of course. I mean, don't you think that we... Before the supply demand dynamics equalize here a bit, I mean, I'm sort of seeing maybe Neo Clouds moving more to TPUs as a way to differentiate themselves. Do you see that happening? [00:24:38] S2: Yeah. Meaning, I think people will have GPUs, TPUs. They could have AMD's processors available. They're new clouds, basically. But the business model is GPUs for rent with the software. Right? But if the demand stops Yeah. That falls under supply, what do you do? Customers are gonna say, what contracts? I don't remember what contract is. You can't ship to me. I'm gonna negotiate the price down, and that's gonna put the margin pressure as well as these new clouds are heavily leveraged with debt. And how that works out when the demand outpaces supply is a big question in front of us in the next few years. [00:25:19] S0: Right. Ben, Ben, I'll I'll let you take us home here. You you were gonna say something? [00:25:24] S3: I mean, I agree with the consolidation, but at the same time, one of the differences between a TPU and a GPU, the GPU obviously has CUDA, I don't know, a 100,000 developers around the world that are constantly improving it. And then the NVIDIA libraries behind it, like, cue DSS and cue bio, like, all that stuff runs on GPUs and is optimized for that. [00:25:47] S2: The