Tech Talks Daily
00:12:38 · Paul Speciale, Neil C. Hughes
[00:12:38] S1: Yeah. I think the parallel here is to the cloud. Right? We had public cloud starting in the 2006, 2010 era with Amazon and all the like. Public cloud sounded great. And then everybody went into it and had this shock bill of all these surprise charges. I really think the same thing can happen here in AI. So the big changes, infrastructure teams need to become the owner of the AI bill. Okay? They're the ones who can manage it and make it smaller. If they do that, they will start tracking metrics like how much time to the first token, how much reuse is there, right? How much of the context becomes data that they need to actually manage? They can then make decisions more consciously about when does a model need to change, right? They'll find out things like the network matters more than people expect it. It's really the limiting factor a lot of times in the performance to keep the GPUs fed. So I think the lesson is the infrastructure team needs to become the owner of the AI bill, then they'll optimize it. [00:13:40] S0: And I must admit, when you started talking today, especially around the topic of storage, I immediately thought you'd be a great fit for the IT press tool, which is something I've been on multiple times over the last, what, four, five years. And I noticed before you joined me, you were actually on that tour in, what, way back in 2010, 2011, 2012. How much has got. Yeah. How much has changed around storage in in that time? What excites you?
00:14:07 · Paul Speciale
[00:14:07] S1: It's been such an inflection point. I'll tell you back in that era, object storage was kind of emerging. Right? And I was actually with another one of the object storage vendors that was coming out at the same time. You know what we all all were pained by at that time? There was no standard API, no standard interface, no protocol. In fact, I remember being the product manager for such a product and everybody said, the number one feature we need is file, file based access. Well, that kind of defeats the purpose of an object store. So one big inflection point, everybody embraced Amazon's s three API. It's a big deal. Okay? Number two, a big inflection point was cloud native, containerization and Kubernetes. Right? Because it meant the way that storage was provisioned became more automated. That was a really big deal. We had to react differently. We had to have different APIs. And then third, this idea that storage would be instrumental in cybersecurity. Backups, as I said a few minutes ago, are critical. Right? They're kind of your last line of defense. That was a big deal. Before that, we were just thinking, oh, backups need to be faster than tape, so let's put them on disks. Right? That was the whole deduplication era. Now we realize it's not about the backup, it's about the restore. Right? Yeah. So you need to be able to have that copy, it needs to be safe, it needs to be come back really quick. And look, I can't ignore it, but I got to say this last era, this current era, AI, is the other inflection point. Right? We never really thought that we'd be used for AI inferencing. Right? That's a big change. And now making sure that storage is fast enough for that is the real key.
00:15:39 · Neil C. Hughes, Paul Speciale
[00:15:39] S0: It really is. And of course, we're talking today just a couple of weeks after you launched your AI Inference Factory to bring enterprise AI on premises. Where do you go from here? What does 2027 look like for you? What excites you about that and where you... [00:15:53] S1: Well, think now it's a lot of the things we've talked about here. Right? Making sure that people understand that controlling their own fate in terms of their own infrastructure, we think sovereignty is a huge deal, especially in Europe, but also broadly speaking around the world. People are gonna know what the lineage of their data is, what the lineage of their code is. This whole open code model is something we're pushing. AI Inference Factory uses open weight models. We think that's a big deal also in terms of making sure you can lock down and trust the models that you're using. You've seen this too if you use one of the frontier models. They change it on you almost weekly and it's very hard to control. So we think controlling, maintaining the cost, maintaining the sovereignty of AI is what's next for enterprises. If you read a lot of the industry literature, you'll see the prediction is that more and more of inferencing becomes on prem and really to an end state of hybrid. So people are using it partly in the cloud and partly on prem. I think that's the future.
00:15:53 · Paul Speciale, Neil C. Hughes
[00:15:53] S1: Well, think now it's a lot of the things we've talked about here. Right? Making sure that people understand that controlling their own fate in terms of their own infrastructure, we think sovereignty is a huge deal, especially in Europe, but also broadly speaking around the world. People are gonna know what the lineage of their data is, what the lineage of their code is. This whole open code model is something we're pushing. AI Inference Factory uses open weight models. We think that's a big deal also in terms of making sure you can lock down and trust the models that you're using. You've seen this too if you use one of the frontier models. They change it on you almost weekly and it's very hard to control. So we think controlling, maintaining the cost, maintaining the sovereignty of AI is what's next for enterprises. If you read a lot of the industry literature, you'll see the prediction is that more and more of inferencing becomes on prem and really to an end state of hybrid. So people are using it partly in the cloud and partly on prem. I think that's the future. [00:16:52] S0: Awesome. And for anybody listening interested in exploring that AI Inference Factory and and learning more about how they come up to bring their enterprise AI on premises, where should they go? Where can they find out more information and and maybe connect with you or your team?
00:16:52 · Neil C. Hughes, Paul Speciale
[00:16:52] S0: Awesome. And for anybody listening interested in exploring that AI Inference Factory and and learning more about how they come up to bring their enterprise AI on premises, where should they go? Where can they find out more information and and maybe connect with you or your team? [00:17:06] S1: Yeah. Well, do you mind if I give a couple of plugs? Yeah. Course. The website, scalety.com. We have an ancillary blog on another property called solved.scalety.com. You can read all about the thought leadership and these ideas that I've talked about. And then selfishly, me and my CTO host a podcast on YouTube, not anywhere to compete with yours, but it's called the Scale Out Podcast, and we talk about topics that are relevant to us, right, things like k v cache and inferencing and all the tech underneath it. So those are three places people can find out more. [00:17:39] S0: Awesome. I will add links to everything. For people listening, if you go to techtalksnetwork.com, there'll be a blog post associated with this episode. Not only will I include the links that you've mentioned there, I'll also, embed one of your YouTube episodes. So please go check that out. But more than anything, Paul, just thank you for coming on here, talking about all this very complex technology in a language everyone can understand and the value behind it too. Really appreciate your time. [00:18:03] S1: Thank you. I value the opportunity, Neil. Thank you.
00:18:03 · Paul Speciale, Neil C. Hughes
[00:18:03] S1: Thank you. I value the opportunity, Neil. Thank you. [00:18:05] S0: I think Paul's explanation of reusing context gives infrastructure teams a way to to think about costs. The question is how much the work the system repeats, where the information is held, and whether it can be retrieved efficiently. Owning the infrastructure, yes, it brings responsibilities because teams will need to understand utilization, operating cost, model changes, and the rules that are governing their data. So I think ultimately, the business case will depend on the workload and the organization's ability to operate it reliably. I also appreciated Paul's advice on testing restrictions because trying an action that should be refused can reveal whether a policy actually holds in practice. And recovery exercises, they show whether protected data can be restored when needed. So a big thank you to my guest. You'll find all the product information, technical blog, and scale out podcast over in the episode notes at techtalksnetwork.com. But over to you, which of your AI workloads would you consider running on your own infrastructure? And what evidence would you need before deciding to do that? After listening to today's episode, show me your thoughts over at techtalksnetwork.com, and I'll get straight back to you. But that is it for today, so thanks for listening as always. Bye for now.