The Information's TITV
00:19:23 · Gopal Srinivasan, Akash Pasricha
[00:19:23] S4: Me [00:19:23] S0: jump in here for one second here. You know, I'm thinking about the projections that some analysts have put out here. So TITV Cowen is saying that TPU revenue by 2028 could be a third of Google Cloud's revenue, which is pretty staggering. Ben, I'll come to you for this one. Mean, how do you think we can put aside whether or not it hits that goal, that margin or that percentage or not? That's a projection. That's not the company saying it. But if it gets there, how does it affect the margins for the Google Cloud business?
00:20:00 · Ben Pouladian
[00:20:00] S3: It just depends where they sell it at. I think, obviously, the TPUs are a crown jewel, and they do a great job building it and running it. But I think stepping back and looking at the bigger picture, it's not all about tokens and what the token factory produces, obviously, per megawatt. You know, you're drawing a parallel to to Google. We were in this world of, how many website visitors you get and then cost per thousand impressions and then ads, then tokens are that same metric, and we're moving towards cost per completed task. So I think what people really need to focus and look on is what is Google's true Agentsic offering on a completed task and compare it to other solutions. And I agree with Umesh that there's this new world of disaggregated inference where you can separate the prefill and decode and have two different solutions. One of them is, for example, like another startup called Dematrix, where they connect with NVLink Fusion. You you can basically take older NVIDIA h 100 GPUs and connect to their system, always... Also air cooled, not liquid cooled, and get the performance of a NVIDIA b 300. So there's ways to mix and match and make things work. The the biggest theme at the end of the day is everyone is super compute constrained, and everyone's looking for a solution right away to get tokens producing because the the first time the token, you know, creates money. But at the same time, it's it's great to see Google really coming out and trying to build this merchant solution. I mean, I never saw it coming. I'm surprised that it's actually happening. It's it's exciting to see the software company morphing into an actual chip company, and they're doing that. And they really wanna showcase Ironwood and the the CPU that they just designed. So all the things are heading in that direction.
00:22:07 · Akash Pasricha
[00:22:07] S0: Umesh, as you think about how this story will play out, so we're compute constrained right now, we're chip constrained. I've been thinking about pricing. And right now, NVIDIA, Google, mean, they can charge whatever the heck they right? Because people need it. Pricing, I imagine, will come down over time here. How do you think that changes the structure of these businesses in the next five years?
00:22:36 · Umesh Padval
[00:22:36] S2: Great question. I've been long enough in the semiconductor and hardware business. I've seen three cycles of up and down. One thing which is common in the semiconductor, is demand and supply always cross each other. Few years, there's feast. Few years, there's famine. And when famine happens, structural problems change. So my prediction is this cycle has lasted for a lot longer than I thought because of the incredible demand from AI infrastructure. But at some point, enough fab capacity is coming online. Enough utilization of GPU is happening because right now, it's at 30 to 40%, and it's gonna go up to 80% just like the CPU utilization in the past went up. And the third thing is applications have really not taken advantage of all the infrastructure. As all these three things happen, there's gonna be a time in the next twelve to eighteen months or somewhere around there where supply is gonna outpace demand. As soon as that happens, believe me, a lot of these neo clouds, which are hundreds of them which have come out, will face a difficult moment in terms of consolidation of that as well as the inferencing companies you've seen raising money every year. I feel both those are going to be challenged when the supply and demand supply outpaces demand, and then pricing pressure will come on all the semiconductor companies. At that point, this is the way history to... You know, if you learn from history, that's what happens.
00:24:13 · Akash Pasricha, Umesh Padval, Ben Pouladian
[00:24:13] S0: But, Umesh, isn't there an argument to be made here that we've seen the Neo Cloud boom with the NVIDIA focused companies. Now we have FluidStack, of course. I mean, don't you think that we... Before the supply demand dynamics equalize here a bit, I mean, I'm sort of seeing maybe Neo Clouds moving more to TPUs as a way to differentiate themselves. Do you see that happening? [00:24:38] S2: Yeah. Meaning, I think people will have GPUs, TPUs. They could have AMD's processors available. They're new clouds, basically. But the business model is GPUs for rent with the software. Right? But if the demand stops Yeah. That falls under supply, what do you do? Customers are gonna say, what contracts? I don't remember what contract is. You can't ship to me. I'm gonna negotiate the price down, and that's gonna put the margin pressure as well as these new clouds are heavily leveraged with debt. And how that works out when the demand outpaces supply is a big question in front of us in the next few years. [00:25:19] S0: Right. Ben, Ben, I'll I'll let you take us home here. You you were gonna say something? [00:25:24] S3: I mean, I agree with the consolidation, but at the same time, one of the differences between a TPU and a GPU, the GPU obviously has CUDA, I don't know, a 100,000 developers around the world that are constantly improving it. And then the NVIDIA libraries behind it, like, cue DSS and cue bio, like, all that stuff runs on GPUs and is optimized for that. [00:25:47] S2: The
00:25:48 · Ben Pouladian
[00:25:48] S3: challenge is with, like, the TPU, because it was closed source for Google's own consumption, they need to basically build a developer community from scratch from the outside and open it up to everyone and kind of open source it. So there's a mismatch in timing and history of these two different types of technologies, which will take time to ramp. And at the same time, obviously, with NVIDIA and CUDA, you're getting... The whole team there is basically working to squeeze out more improvements from the stack all the way from the walls of the building, from the electrons coming in to the tokens coming out of Dynamo and all these other different projects that they have, which is pretty it's interesting because they basically designed the whole data center from soup to nuts, optimized on their stack with the DSX program or anything. So Google has some catching up to do if they really want other people to embrace their technology and their chips. It's a good step in the first direction, but you really need to open up the kimono and embrace developer communities to show what you have. And I agree with Umesh on the Neo Clouds. I've... There's a clip where I referenced Bane from Batman. You know, one of one of them is gonna die. It it has to be found in the wreckage. I mean, that's part of it. There'll be consolidation. It comes down to what is your your cost of electricity and your operating costs and your financing costs and what are you producing tokens at. And some people will be backwards. And you have other neo clouds like SpaceX, which now have a full stack and a use case to basically use that token demand for inference on their prod... Products like Grok or X or even the Optimus robots that are coming out. We really need to have a