Tech Talks Daily
00:02:13 · Paul Speciale, Neil C. Hughes
[00:02:13] S1: Yeah. Absolutely. I'm Paul Speciale. I'm the chief marketing officer at Scality. Scality is a vendor of data infrastructure software. We've actually been around a while. We the company started in 2009. And what we specialize in is building data infrastructures for AI, for cloud, and for cybersecurity for data protection for backup. [00:02:35] S0: Awesome. Well, it's a real pleasure to have you on here with me today. There's so much going on in this space. There's a certain amount of hype. You may see a few things on your news feed that might frustrate you as well. But when people talk about an AI data layer becoming an infrastructure control plane, what decisions should that layer actually be able to make?
00:02:35 · Neil C. Hughes, Paul Speciale
[00:02:35] S0: Awesome. Well, it's a real pleasure to have you on here with me today. There's so much going on in this space. There's a certain amount of hype. You may see a few things on your news feed that might frustrate you as well. But when people talk about an AI data layer becoming an infrastructure control plane, what decisions should that layer actually be able to make? [00:02:55] S1: Yeah. I think it's pretty simple. In a production inferencing system, a data layer like ourselves has one job, and that job is to make sure the GPU never computes the same thing twice. Right? So think about how a session works. You initiate a session, you send a query, a prompt, the session either has to recompute everything from the information you sent or if it's seen it before, it can read it back from storage. Okay? Reading it back from storage is so much faster. It's like 10 to 80 times faster than recompute. People think that this inferencing is a compute problem. It actually turns out to be a memory problem. So one of the things we can do is stage all of this existing recomputations in memory. You bring it back. You don't have to do all that recompute overhead. So that's our job, make sure that the GPU never has to do the same thing twice. [00:03:47] S0: And one of the things I always try and do on here is demystify a lot of the complex technologies we're hearing about. And as AI moves from pilots into production inference there, as you mentioned, what is the data layer underneath? What is it actually needing to do?
00:03:47 · Neil C. Hughes, Paul Speciale
[00:03:47] S0: And one of the things I always try and do on here is demystify a lot of the complex technologies we're hearing about. And as AI moves from pilots into production inference there, as you mentioned, what is the data layer underneath? What is it actually needing to do? [00:04:02] S1: Well, I think it needs to do a few things. It needs to be, obviously, to make sure that the the GPUs are fed, as we talked about, but also to keep the costs predictable to make sure that you maintain control of your data. And that means, for example, the jurisdiction of your data that governs your data. That's really key. Right? If you end up looking at the spend that's happening in AI right now, it's absolutely dominated by the compute costs. I'm sure you've heard how expensive memory and GPUs are. Storage is a small part of that spend, but it has so much of a bearing on how efficient the compute is. Right? If we can make sure that those GPUs stay fed and make sure that the data is trusted and make sure that it's sovereign and governed in the right places, that's really what's doing the job. Okay. Those are the three key levers. [00:04:51] S0: We're hearing that a lot on our news feed. It almost feels like we've come full circle from looking back at the cloud and what happened there. Everyone going to cloud then coming back. Why are enterprises bringing AI inference in house? And where does storage fit into that economics too?
00:04:51 · Neil C. Hughes, Paul Speciale
[00:04:51] S0: We're hearing that a lot on our news feed. It almost feels like we've come full circle from looking back at the cloud and what happened there. Everyone going to cloud then coming back. Why are enterprises bringing AI inference in house? And where does storage fit into that economics too? [00:05:06] S1: Well, I think it's... Once again, it's predictable cost. Think about what happens when you have an AI project that succeeds. Users start using more tokens. Right? The more tokens you use, the more expensive things get, and you have end up having this problem that we all had in the cloud where you have these surprise charges, right? These hidden fees. Well, here it's not even hidden. You're just going to spend more on tokens, right? So what you want to do is make sure that that gets minimized. One of the ways you can do that is to control the infrastructure on prem on the servers that the customer owns already. Now they're not renting from some AI provider. But moreover, you can make things cheaper by making sure that you're using data efficiently. One thing you'll notice in the world of these public AI providers, they make it cheaper to use a token that's been cached, about one tenth of the price. So even they realize that compute is expensive. Well, on prem, you can make sure that that happens because you paid for the infrastructure one time. Now you can amortize it the more and more you use it.
00:05:06 · Paul Speciale, Neil C. Hughes
[00:05:06] S1: Well, I think it's... Once again, it's predictable cost. Think about what happens when you have an AI project that succeeds. Users start using more tokens. Right? The more tokens you use, the more expensive things get, and you have end up having this problem that we all had in the cloud where you have these surprise charges, right? These hidden fees. Well, here it's not even hidden. You're just going to spend more on tokens, right? So what you want to do is make sure that that gets minimized. One of the ways you can do that is to control the infrastructure on prem on the servers that the customer owns already. Now they're not renting from some AI provider. But moreover, you can make things cheaper by making sure that you're using data efficiently. One thing you'll notice in the world of these public AI providers, they make it cheaper to use a token that's been cached, about one tenth of the price. So even they realize that compute is expensive. Well, on prem, you can make sure that that happens because you paid for the infrastructure one time. Now you can amortize it the more and more you use it. [00:06:06] S0: And I'm curious. What is it that makes data placement so difficult when performance, cost, sovereignty, which is proving to be a big topic over here in Europe and beyond, and cyber resilient, when they're all pointing in in different directions?
00:06:06 · Neil C. Hughes, Paul Speciale
[00:06:06] S0: And I'm curious. What is it that makes data placement so difficult when performance, cost, sovereignty, which is proving to be a big topic over here in Europe and beyond, and cyber resilient, when they're all pointing in in different directions? [00:06:20] S1: Yeah. That's really a hard one. Right? And and it's one of the things we thought about when we came out with the latest version of our software is that you can't just put everything on flash anymore. Right? It's too simple of an answer, and we all know what's happening with memory prices like NAND flash. It's gone through the roof. It's hard to get. So there's really no single rate... Right place for the data. There's only a right place for each workload, and performance obviously wants things next to the GPUs. It wants it fast, but if you want things inexpensive, you want a denser, colder tier. Right? The thing that you can't trade off is sovereignty. Right? If you need to make sure that that something's located in a certain geography, that's something we can control. But the practical answer is this. If... As long as you have one namespace and that names case can namespace can move data between hot flash, warm HDD, cold tape, or cold cloud, it's a policy decision where you put things. It's no longer a big migration project, which used to be a huge headache.