Paul Speciale says AI inference is a memory problem because reusing stored context is 10 to 80 times faster than recomputing it.
On-premises inference on servers enterprises already own amortizes costs as usage grows, since compute dominates AI spending.
Scality's AI Inference Factory spans flash, HDD, tape and cloud under one namespace, making data placement a policy decision while humans approve irreversible actions.
AI inference turns out to be a memory problem
Enterprise AI inference is a data problem that looks like a compute problem. Reading previously computed context back from storage is 10 to 80 times faster than recomputing it, so the data layer's job is to keep GPUs fed and stop them repeating work. That reframes the economics, because compute dominates AI budgets while storage, a small line item, decides how efficient the expensive part is. On-premises inference on hardware customers already own becomes attractive, and the same system must satisfy sovereignty and cyber resilience rules.
Takeaways
Inference is a memory problem. Paul Speciale says the data layer in a production inference system has one job, to stop the GPU computing the same thing twice. Reading stored context is 10 to 80 times faster than recomputing it, and Scality stages those recomputations in memory.
Compute costs dominate AI spending. Special says spending on AI is dominated by compute costs, while storage is a small share that has a large bearing on compute efficiency. Users of successful projects consume more tokens, so costs rise and produce the surprise charges organizations met in the cloud.
On-premises inference amortizes hardware already owned. Running inference on servers the customer already owns spreads the cost as usage grows. Public AI providers charge about one tenth the price for a cached token, which sets the comparison for teams deciding where inference should run.
Data placement is a policy decision. NAND flash prices are through the roof and supply is hard to get, so nothing can sit on flash. One namespace spanning hot flash, warm HDD, cold tape and cold cloud turns placement into a policy decision, and sovereignty cannot be traded off.
Assume attackers are already inside. Studies such as Veeam's show attackers go after backups first so victims cannot restore and must pay. Organizations need one immutable copy attackers cannot touch and a proven fast restore, and retention must outlast an attacker's undetected dwell time. Scality's cyber guarantee pays $100,000.
Infrastructure teams should own the AI bill. Whoever owns the bill can make conscious decisions about when a model needs to change, tracking metrics such as time to first token and context reuse. The network is often the limiting factor in keeping GPUs fed.
S3 hides real differences between platforms. Data retention and object locking behave differently across platforms, so Speciale warns customers not to let one vendor's differences become lock-in. Scality commits every change upstream and runs on standard servers from HPE, Lenovo, Super Micro and Dell.
Storage's inflection points
Object storage arrived before the S3 API. Early users said the feature they most needed was file-based access, which defeats the purpose of an object store.
Amazon's S3 API became the common language of the category and a major inflection point for the industry.
Containerization and Kubernetes changed how storage is deployed and consumed.
Storage became instrumental in cybersecurity, with backups as the last line of defence.
AI inferencing is the latest turn, and Scality never expected storage to serve it.
Tap a name to go there · tap a line to see why they are linked