The Information's TITV
00:15:59 · Akash Pasricha, Ben Pouladian
[00:15:59] S0: Okay. So I hear you. Let's talk about what the TPUs are good at compared to GPUs. Mean, Umesh is pointing to inference here. What are some of the advantages that the TPU has over NVIDIA right now in insofar as the work that it's trying to focus on? [00:16:19] S3: Obviously, the number one reason people turn to it is cost. And then from a business standpoint, the risk is being locked into one vendor, NVIDIA. So the TPU, obviously, has the same type of, you know, matrix multiplication and action that a GPU has, but at the same time, it's not as general purpose. So it's you have to basically design it for specific workloads. So Right. They... If if they use it for inference, that's great. I I think even Meta has, you know, entertained using some TPUs. So if you can find a specific workload or a case that you wanna use it for, it works well. And obviously, has proved it working well horizontally in its stack with its search, its ad business, YouTube overviews, and what it's building with Gemini. So it's a formidable competitor to what the NVIDIA GPU is. And I agree with Umesh that... I mean I mean, Vahdat has definitely taken the front seat and wants to show the world how great the TPU is, and they've ramped up. And they're making deals and working with Blackstone to finance TPUs for other customers. So it's definitely a a multichip world out there for customers. [00:17:40] S0: So, Umesh, I may
00:17:40 · Akash Pasricha, Umesh Padval, Gopal Srinivasan
[00:17:40] S0: So, Umesh, I may [00:17:42] S2: Let me add to this. Yeah. Look. The... People talk about TPU, but TPU has two solutions. One is for inferencing, and one is for training. So people only mistake TPUs for just training going after GPUs. It's both. They have two solutions, one for inferencing and one for training. Now what is the most important trend, I think, which is I'm seeing in the last six months since GTC, Jensen announced disaggregated solution. What does that mean? Initially, it was TPU or g p... XPU, a CPU, and inferencing engine all provided by one supplier, NVIDIA. It's disaggregated now, and what is happening is the best of breed. So you use the best of breed XPU for training, use the best of breed CPU for the orchestration, and use the lowest cost, highest performance inferencing engine from different suppliers. So it's disaggregating. This is where SambaNova plays tremendously well. Because what they have done is... You saw the deal where they have high performance but at extreme low power. And why is power important? You know, these data centers, when you have massive power of these chips, you need complex liquid cooling technology. SambaNova, because of the power envelope being so low, it does not need liquid cooling. It can do air cooling. Right. What does that do? [00:19:23] S4: Me
00:19:23 · Gopal Srinivasan, Akash Pasricha
[00:19:23] S4: Me [00:19:23] S0: jump in here for one second here. You know, I'm thinking about the projections that some analysts have put out here. So TITV Cowen is saying that TPU revenue by 2028 could be a third of Google Cloud's revenue, which is pretty staggering. Ben, I'll come to you for this one. Mean, how do you think we can put aside whether or not it hits that goal, that margin or that percentage or not? That's a projection. That's not the company saying it. But if it gets there, how does it affect the margins for the Google Cloud business?
00:20:00 · Ben Pouladian
[00:20:00] S3: It just depends where they sell it at. I think, obviously, the TPUs are a crown jewel, and they do a great job building it and running it. But I think stepping back and looking at the bigger picture, it's not all about tokens and what the token factory produces, obviously, per megawatt. You know, you're drawing a parallel to to Google. We were in this world of, how many website visitors you get and then cost per thousand impressions and then ads, then tokens are that same metric, and we're moving towards cost per completed task. So I think what people really need to focus and look on is what is Google's true Agentsic offering on a completed task and compare it to other solutions. And I agree with Umesh that there's this new world of disaggregated inference where you can separate the prefill and decode and have two different solutions. One of them is, for example, like another startup called Dematrix, where they connect with NVLink Fusion. You you can basically take older NVIDIA h 100 GPUs and connect to their system, always... Also air cooled, not liquid cooled, and get the performance of a NVIDIA b 300. So there's ways to mix and match and make things work. The the biggest theme at the end of the day is everyone is super compute constrained, and everyone's looking for a solution right away to get tokens producing because the the first time the token, you know, creates money. But at the same time, it's it's great to see Google really coming out and trying to build this merchant solution. I mean, I never saw it coming. I'm surprised that it's actually happening. It's it's exciting to see the software company morphing into an actual chip company, and they're doing that. And they really wanna showcase Ironwood and the the CPU that they just designed. So all the things are heading in that direction.
00:22:07 · Akash Pasricha
[00:22:07] S0: Umesh, as you think about how this story will play out, so we're compute constrained right now, we're chip constrained. I've been thinking about pricing. And right now, NVIDIA, Google, mean, they can charge whatever the heck they right? Because people need it. Pricing, I imagine, will come down over time here. How do you think that changes the structure of these businesses in the next five years?
00:22:36 · Umesh Padval
[00:22:36] S2: Great question. I've been long enough in the semiconductor and hardware business. I've seen three cycles of up and down. One thing which is common in the semiconductor, is demand and supply always cross each other. Few years, there's feast. Few years, there's famine. And when famine happens, structural problems change. So my prediction is this cycle has lasted for a lot longer than I thought because of the incredible demand from AI infrastructure. But at some point, enough fab capacity is coming online. Enough utilization of GPU is happening because right now, it's at 30 to 40%, and it's gonna go up to 80% just like the CPU utilization in the past went up. And the third thing is applications have really not taken advantage of all the infrastructure. As all these three things happen, there's gonna be a time in the next twelve to eighteen months or somewhere around there where supply is gonna outpace demand. As soon as that happens, believe me, a lot of these neo clouds, which are hundreds of them which have come out, will face a difficult moment in terms of consolidation of that as well as the inferencing companies you've seen raising money every year. I feel both those are going to be challenged when the supply and demand supply outpaces demand, and then pricing pressure will come on all the semiconductor companies. At that point, this is the way history to... You know, if you learn from history, that's what happens.