Ep. 030 - Long Live the Short King: Why 4-hi HBM Wins (Memory)

The AI hardware industry is shifting from high-density HBM stacks toward 4-high and 8-high configurations to address critical memory supply shortages. While previous generations prioritized increasing capacity to accommodate larger model parameters, modern inference and post-training workloads prioritize bandwidth over raw capacity. Because model parameter scaling has slowed due to techniques like loop transformers and efficient quantization, the marginal utility of excessive HBM capacity has diminished. Transitioning to lower-height stacks significantly improves manufacturing yields and allows for a higher volume of accelerator shipments by optimizing the use of scarce DRAM wafers. This strategic pivot reflects a broader industry move toward balancing performance with supply chain realities, as memory constraints are expected to persist throughout the decade.