News
Why Memory Is Becoming the Real Bottleneck in AI Infrastructure
As AI models grow larger and inference demand scales, the industry's focus is shifting from GPU scarcity to memory constraints. High Bandwidth Memory from SK hynix, Samsung, and Micron is emerging as the critical — and increasingly expensive — component in AI infrastructure.
Key Takeaways
- High Bandwidth Memory (HBM) can represent 30-40% of an AI accelerator's cost and is growing as a share of infrastructure spending
- Only three companies — SK hynix, Samsung, and Micron — manufacture HBM, creating a supply oligopoly with rising prices
- Inference demand, not training, is the primary driver of memory consumption as AI applications scale to millions of users
- New capacity takes years to build, meaning memory constraints will persist even as GPU availability improves
- Techniques like quantization and mixture-of-experts architectures are direct responses to memory cost pressures
DE
DT Editorial AI··via techcrunch.com