Tech Talks Daily
00:03:47 · Neil C. Hughes, Paul Speciale
[00:03:47] S0: And one of the things I always try and do on here is demystify a lot of the complex technologies we're hearing about. And as AI moves from pilots into production inference there, as you mentioned, what is the data layer underneath? What is it actually needing to do? [00:04:02] S1: Well, I think it needs to do a few things. It needs to be, obviously, to make sure that the the GPUs are fed, as we talked about, but also to keep the costs predictable to make sure that you maintain control of your data. And that means, for example, the jurisdiction of your data that governs your data. That's really key. Right? If you end up looking at the spend that's happening in AI right now, it's absolutely dominated by the compute costs. I'm sure you've heard how expensive memory and GPUs are. Storage is a small part of that spend, but it has so much of a bearing on how efficient the compute is. Right? If we can make sure that those GPUs stay fed and make sure that the data is trusted and make sure that it's sovereign and governed in the right places, that's really what's doing the job. Okay. Those are the three key levers. [00:04:51] S0: We're hearing that a lot on our news feed. It almost feels like we've come full circle from looking back at the cloud and what happened there. Everyone going to cloud then coming back. Why are enterprises bringing AI inference in house? And where does storage fit into that economics too?
00:04:51 · Neil C. Hughes, Paul Speciale
[00:04:51] S0: We're hearing that a lot on our news feed. It almost feels like we've come full circle from looking back at the cloud and what happened there. Everyone going to cloud then coming back. Why are enterprises bringing AI inference in house? And where does storage fit into that economics too? [00:05:06] S1: Well, I think it's... Once again, it's predictable cost. Think about what happens when you have an AI project that succeeds. Users start using more tokens. Right? The more tokens you use, the more expensive things get, and you have end up having this problem that we all had in the cloud where you have these surprise charges, right? These hidden fees. Well, here it's not even hidden. You're just going to spend more on tokens, right? So what you want to do is make sure that that gets minimized. One of the ways you can do that is to control the infrastructure on prem on the servers that the customer owns already. Now they're not renting from some AI provider. But moreover, you can make things cheaper by making sure that you're using data efficiently. One thing you'll notice in the world of these public AI providers, they make it cheaper to use a token that's been cached, about one tenth of the price. So even they realize that compute is expensive. Well, on prem, you can make sure that that happens because you paid for the infrastructure one time. Now you can amortize it the more and more you use it.
00:05:06 · Paul Speciale, Neil C. Hughes
[00:05:06] S1: Well, I think it's... Once again, it's predictable cost. Think about what happens when you have an AI project that succeeds. Users start using more tokens. Right? The more tokens you use, the more expensive things get, and you have end up having this problem that we all had in the cloud where you have these surprise charges, right? These hidden fees. Well, here it's not even hidden. You're just going to spend more on tokens, right? So what you want to do is make sure that that gets minimized. One of the ways you can do that is to control the infrastructure on prem on the servers that the customer owns already. Now they're not renting from some AI provider. But moreover, you can make things cheaper by making sure that you're using data efficiently. One thing you'll notice in the world of these public AI providers, they make it cheaper to use a token that's been cached, about one tenth of the price. So even they realize that compute is expensive. Well, on prem, you can make sure that that happens because you paid for the infrastructure one time. Now you can amortize it the more and more you use it. [00:06:06] S0: And I'm curious. What is it that makes data placement so difficult when performance, cost, sovereignty, which is proving to be a big topic over here in Europe and beyond, and cyber resilient, when they're all pointing in in different directions?
00:06:06 · Neil C. Hughes, Paul Speciale
[00:06:06] S0: And I'm curious. What is it that makes data placement so difficult when performance, cost, sovereignty, which is proving to be a big topic over here in Europe and beyond, and cyber resilient, when they're all pointing in in different directions? [00:06:20] S1: Yeah. That's really a hard one. Right? And and it's one of the things we thought about when we came out with the latest version of our software is that you can't just put everything on flash anymore. Right? It's too simple of an answer, and we all know what's happening with memory prices like NAND flash. It's gone through the roof. It's hard to get. So there's really no single rate... Right place for the data. There's only a right place for each workload, and performance obviously wants things next to the GPUs. It wants it fast, but if you want things inexpensive, you want a denser, colder tier. Right? The thing that you can't trade off is sovereignty. Right? If you need to make sure that that something's located in a certain geography, that's something we can control. But the practical answer is this. If... As long as you have one namespace and that names case can namespace can move data between hot flash, warm HDD, cold tape, or cold cloud, it's a policy decision where you put things. It's no longer a big migration project, which used to be a huge headache.
00:06:20 · Paul Speciale, Neil C. Hughes
[00:06:20] S1: Yeah. That's really a hard one. Right? And and it's one of the things we thought about when we came out with the latest version of our software is that you can't just put everything on flash anymore. Right? It's too simple of an answer, and we all know what's happening with memory prices like NAND flash. It's gone through the roof. It's hard to get. So there's really no single rate... Right place for the data. There's only a right place for each workload, and performance obviously wants things next to the GPUs. It wants it fast, but if you want things inexpensive, you want a denser, colder tier. Right? The thing that you can't trade off is sovereignty. Right? If you need to make sure that that something's located in a certain geography, that's something we can control. But the practical answer is this. If... As long as you have one namespace and that names case can namespace can move data between hot flash, warm HDD, cold tape, or cold cloud, it's a policy decision where you put things. It's no longer a big migration project, which used to be a huge headache. [00:07:21] S0: Yeah. 100% with you there. And for business leaders listening, how should their organization desire... Decide which infrastructure decisions can be automated and which still need human approval? Because it's a big topic right now, isn't it?
00:07:21 · Neil C. Hughes, Paul Speciale
[00:07:21] S0: Yeah. 100% with you there. And for business leaders listening, how should their organization desire... Decide which infrastructure decisions can be automated and which still need human approval? Because it's a big topic right now, isn't it? [00:07:36] S1: It's a huge topic and we face it ourselves. Right? Because we agree that as the data grows, as these things get more complex, you're not gonna add much more human head count to manage all this extra stuff. Right? So our sense is you can automate the recommendations, but keep a human on anything that you cannot undo. Okay? So for example, very simple to automate health checks, forecasting, recommendations, I would automate that. But deleting data, changing policies, moving data, you need a person to approve that. Okay? And that's the way we've implemented in the product. [00:08:11] S0: Love it. And vendor neutrality also sounds attractive, maybe even idealist... Idealistic to a certain extent. But where does it become difficult when storage platforms, cloud services, and AI tools all expose different capabilities?