20VC: "Anti-Data Centres is a Chinese Psyop" | How Many Planned Data Centers Will Actually Get Built? | Is Energy AI's Biggest Bottleneck? With Thomas Sohmers, Co-Founder @ Positron

Inference Is Memory-Bound, and Economics Limit AI More Than Energy

Inference is becoming the central infrastructure problem in AI, and memory rather than raw compute or energy sets its limits. Positron is building chips for that problem, while Sohmers argues data center backlash is overstated and the real constraints are economics and debt. He opposes the current pace of frontier development but worries that a pause would feed Luddite sentiment.

Takeaways

• Inference is memory-bound, not compute-bound. Training is compute-bound, but inference must read the model weights for every generated token. GPU FLOPs improved about 120 times from 2014 to 2024 while memory bandwidth improved only 17 times, which creates the memory wall.
• KV caching now drives token economics. Providers earn most of their money on cached input and output tokens. SemiAnalysis found about 96 percent of tokens in agentic coding sessions are cached, and long sessions can reach around 100 gigabytes, sometimes exceeding the model weights.
• Economics and debt limit AI more than energy. Positron says it delivers the compute of 500 megawatts of NVIDIA equipment in 100 megawatts, yet builders will still build maximum compute. Sohmers calls permitting restrictions absurd and is not worried about sovereign debt crowding out AI spending.
• Data center opposition is largely a Chinese psyop. Sohmers says the backlash spanning left and right is almost entirely a Chinese psyop, and that water claims are exaggerated. One In-N-Out uses more water than the largest US data centers, and golf courses use far more.
• Pacing AI risks Luddism or concentrated control. Sohmers opposes the current pace but fears a major pause would feed Luddite sentiment and that concentrated control would create a road to serfdom. He says a CCP-controlled superintelligence would deny most people its benefits, and export controls are no long-term fix.
• Scaling continues and per-token pricing likely persists. Sohmers expects scaling to continue, though no mathematical proof exists. He argues value per token has risen 100 to 1,000 fold, predicts per-token pricing stays with possible flat annual fees for unlimited agents, and doubts labs will buy data labeling firms.
• GPT-6 Astra compresses weeks of chip work. Sohmers calls Astra AGI. It took an encryption block through an RTL-to-GDS flow on TSMC N3 in about two days, work he estimates at two to three weeks, and it finds a hidden needle in long context over 95 percent of the time.

How inference memory is being stretched

• Tiered memory moves data from accelerator memory to host memory to flash. Host memory typically holds four to ten times more than accelerator memory.